Understanding the Difference Between the Surface Web, Deep Web, and Darknet

November 29, 2022

The internet, social media, and mobile devices are the fundamental requirements for conducting business and engaging in society. Whether checking email, catching up on industry news or accessing customer information, most of us use the internet (and the deep web) throughout the day, every day, in a variety of capacities. But, do we understand how it works – technically – even at a basic level? Do we understand the differences between the internet and the deep web or what it means to go even darker into the decentralized anonymous networks like the darknet?

Below is a breakdown of the various layers of the internet, from “regular” search engine-compatible websites to complex hidden networks.

The Internet

The term internet is short for internetwork, which is a system created by connecting any number of computer networks together. An internet allows for communication between devices that are a part of that internetwork.

The internet is the most well-known example of an internetwork. This is the internet that we find indispensable to our daily lives, and it links billions of devices across the world through a network of networks using standardized procedures or protocol. The traditional server client architecture and HTTP protocol is the backbone of the internet and used extensively in websites and mobile applications.

Browsing websites on the web is not the only way in which information is shared via the internet. Email, instant messaging, and file transfer protocol (FTP) are other ways to share information like emails, messages, and files.

To clarify, the web is not synonymous with the internet and should not be confused with it. The “world wide web” is simply a way of accessing websites over the medium of the internet.

The Surface Web

The websites we browse each day make up only a small percentage of the internet.

These sites, collectively known as the surface web (or “clearnet”), are visible and accessible to common search engines such as Google and Yahoo. Youtube videos, blogs, Instagram are all examples of surface web content most interact with every single day. While estimates vary, many experts agree that the surface web comprises roughly 4% of all online content. For more reading on how search engines crawl and index web content, there are several articles that describe systems like Google in detail.

High Risk Surface Web

High risk surface web consists of areas of the surface web that have a high degree of hosting criminal or illicit content. Many of the users of the high risk surface web also maintain access to other, darker networks and communities. This includes some “chan”-type imageboards, transient paste sites, and other select non-authenticated forums that mirror dark web sites with surface web top level domains (TLDs).

While .com domains are the most common website domain, DarkOwl regularly tracks various TLDs that are popular with criminals. Our analysts have observed an increase in .top, .ru, and .cc TLDs. Many high risk surface websites popular with Chinese threat actors end in the TLD .cn.

Below the Surface

Beyond the surface web, an estimated 96% of online publicly accessible content is hosted in the deep web and the darknet.

The Deep Web

The deep web consists of website content that cannot be found or directly accessed via surface web search engines such as Google and DuckDuckGo. Examples of deep web sites include websites that require any authentication credentials, such as registered email address and password, unlinked sites that require the direct URL to access, sites that are purposefully designed to keep search crawlers out, and databases. The majority of content resides in the deep web.

Deep web databases commonly have their own search functionality which allows users to access the data contained within them. Government databases, patient medical records, and library catalogs are just a few examples of deep web databases. While these databases do not always require login credentials, many of them do.

Banking website portals for accessing account holder data and credit card statements are technically in the deep web because most banking websites will not allow access to their sensitive servers without authorization. Most social media is technically deep web content.

A specific example is the Denver Property Taxation and Assessment System website which allows users to search property assessment and tax data by entering a Denver-based address into the system. However, if you enter this same Denver-based address into a Google search (and even include terms such as ‘property assessment’ or ‘tax data’), you will not find any documents or URL results from the Denver Property Taxation and Assessment System website. This database and its search functionality are one example of a deep web database that is hidden from surface web search engines and technically resides in the deep web.

The Darknet and The Dark Web

Beyond the deep web is the darknet.

The darknet is any anonymous network, built on top of the internet, that is purposefully hidden, meaning it has been designed specifically for anonymity. Unlike the deep web, the darknet is only accessible with specialized tools and software – browsers and other protocols beyond direct links or credentials. You cannot technically directly access the darknet by simply typing a darknet address into your web browser, even though browsers like Brave offer private tabs with Tor for enhanced privacy.

Most people associate the darknet with Tor, but Tor is one of many darknets available. Let’s explore some of these darknets in more detail:

Tor, or The Onion Router, is an overlay network comprised of volunteer-operated servers that help route internet client-server traffic to provide obfuscation to users accessing the network. Theoretically, Tor is like a virtual private network (VPN) but with a VPN the servers are centralized to the VPN provider. With Tor, users connect through a series of virtual tunnels rather than making a direct connection between a client and server. The identity of the client IP address is unknown to the requesting website server, and the IP address of the server is only known to the exit node but not the originating client.

I2P, or the Invisible Internet Project, is an anonymous overlay network – using the distributed peer-to-peer (p2p) model – intended to protect communication from surveillance and monitoring. It was designed as a self-contained internet and behaves much like an ‘internet’ inside the internet. The I2P router relies heavily on Java to I2P browser, but i2pd is built on C++ for those adverse to Java. I2P routing is described as “garlic routing” where Tor is “onion routing.” It originated in 2003 as a ‘fork’ of Freenet.

ZeroNet is another example of a decentralized peer to peer network built on the Blockchain that functions as a darknet. Zeronet relies on BitTorrent network trackers to resolve network handshaking between peers. Instead of IP addresses in the network, nodes are assigned their own public key, more specifically a Namecoin (Bitcoin) cryptography address. The entry IP address of the network user is not technically private and Zeronet developers offer bundling Zeronet with Tor for additional anonymity.

Freenet is distributed, peer to peer anonymous network which allows users to anonymously share files, browse and publish “freesites” (web sites accessible only through Freenet) and chat on forums. It is a distributed ‘data store’ allowing the content to be available on the network even though the originator or publisher is no longer on the network. Communications by Freenet nodes are encrypted and are routed through other nodes to make it extremely difficult to determine who is requesting the information and what its content is. The distributed data store nature of the Freenet environment is ideal for microblogging and media sharing, but also puts the users of the network at risk of unknowingly hosting illicit or CSAM content as encrypted fragments of media are stored on the hard drive of every user in the network.

Lokinet is another example of an overlay decentralized network built on the Blockchain that serves as a darknet providing enhanced anonymity and privacy. Lokinet relies on a multi-hop low-latency onion routing protocol (LLARP) for routing traffic but is not limited to TCP traffic for serving HTTP requests. Since Lokinet sits in the network layer, it can handle and IP-based protocol like UDP and ICMP making it a highly secure option for web-based video and voice conferencing applications. The Loki network is developed atop the Oxen Blockchain which also hosts the end-to-end encrypted chat application, Session.

Yggdrasil is an IPv6 overlay fully encrypted mesh network where each node in the network is assigned a cryptographic public key (like Zeronet) and routing is highly adaptable, using spanning trees for synchronization. Keys are assigned in alignment with the most optimized routes between all network participants. Yggdrasil nodes serve as routers and paths are built automatically without any data store or shared address book of the network. Yggdrasil is still considered ‘proof of concept’ and is designed as a more scalable version of CJDNS.

Navigating these networks can be frustrating and challenging for any OSINT/Darknet investigator and the public often incorrectly uses the terminology associated with these different layers of the internet. Any website that hosts or serves illicit content whether it is in the surface web, deep web, or darknet is technically a segment of the “dark web.” Dark web and darknet are often used interchangeably by us and other information security researchers.

Join us next time when we explore more darknets and darknet adjacent chat platforms like Telegram and Discord. Get on the list so you don’t miss it!