We had a Deep Web specialist on the show today. Bill Wardell talked about the “Deep Web” and how it offers a virtually untapped bounty of information and services most people don’t know exist.
What is the Deep Web?
It’s basically the 90% of the web the the big 2 search engines haven’t spidered or accessed.
Google only has about 10% of the web spidered. Given that they reported indexing around 26 billion pages on the web, the other 90% of what experts call the Deep Web is a pretty big chunk of data.
How is the Deep Web different from the Surface Web?
“To put it very simply, the web is defined as a collection of hyperlinks that are indexed by search engines. In other words, the pages/content that appears when we do a Google search, is the Internet as we know it – called the surface web.
The Dark Web, also known as the deep web, invisible web, and dark net, consists of web pages and data that are beyond the reach of search engines. Some of what makes up the Deep Web consists of abandoned, inactive web pages, but the majority of data that lies within have been crafted to deliberately avoid detection in order to remain anonymous.
Michael K. Bergman – who first coined the phrase “deep web” describes how searching on the Internet today can be compared to dragging a net across the surface of the ocean; a great deal may be caught in the net, but there is a wealth of information that is deep and therefore missed.” – Wikipedia
Learn how the Deep Web can help researchers, bloggers, marketers, and other folks looking to tap little known information and resources. Lots of cool Deep Web research links to share with you under the show player today as well.
Deep Web Research Tools and Resources
- Deep Web Research Starting Point
- Deep Web Resources (PDF) – This is the crucial source of resources and tools for making the most of the Deep Web.
- Marcus Zillman’s Blog
- Resource Shelf
- Research Buzz
- Top 100 Lists – from Alt Search Engines
- Exalead – example of specialty search engine
- Specialty Video Search – Blinkx.com
- Check out Bill Wardell at The Cyberhood Watch
{ 9 comments… read them below or add one }
Any page that doesn’t want to get indexed shouldn’t really be considered deep web, it’s not useful anyway. Though I agree that there are LOT of pages that search engines haven’t been able to index because the site owners didn’t use the correct navigation schemes on their sites.
We shouldn’t really be bothered about abandoned sites, parked pages etc. should we?
I never even heard about the ‘Deep Web’. Very interesting piece. Good work.
Joseph Condron@Yellow Magpie´s last blog ..Quotes On Challenging Conventions
Very Interesting article… Never knew there is so much of information in the deep web. http://www.turbo.com is a great deep web search engine.
The “deep web” is a pretty interesting concept to think about. The thing that sticks out to me the most, though, is the fact that Google only has spidered 10% of the internet. I really didn’t realize before just how much content there is out there that hasn’t been discovered by Google. It makes you realize just how huge the internet is.
Alec@Trash Bag Holder´s last blog ..Keeping Things Green With Trash Bags
Web services, APIs and mashups seem to be the way this content is being pulled and rendered. This will deepen considerably after the Internet of Things gets going. Good post, thanks.
dave tribbett´s last blog ..Casting an Information Shadow
Great post. Here is a good article that adds some additional detail to the topic and a good set of links to the deep web search engines and other helpful sites.
dave tribbett´s last blog ..The Deep, Dark Invisible Web
This is great stuff Jack, and something most people don’t even start to think about…I didn’t until reading this.
Thanks for the resources too.
Joseph Ratliff´s last blog ..How Do You Expect To Do Any One Thing Well, When You’re Trying To Do Many Things At The Same Time?
ARTICLE: Exploring the Deep Web, the 90% of the web the big search engines haven’t spidered or accessed .. http://digg.com/u1Mp63 .. RT THIS
This comment was originally posted on Twitter
Exploring the “Deep Web”…. interesting…hmmm…. http://bit.ly/cMLxNH http://url4.eu/1P8ks
This comment was originally posted on Twitter