There has been discussion for years about the fact that there’s a lot more information on the Web than is accessed when we do a search using one of the common search engines such as Google. Experts consider that there may be up to 500 times the information available in the ‘invisible web’ than there is in the ‘searchable web’.
One way of addressing this is to consider the federated search approach – the powerpoint above is my humble attempt to illustrate how this works from a couple of years ago.
Another way is to use specialised search engines that are designed to search the deep or invisible web – usually within a particular field or discipline. About a week ago I received a note from Amy Quinn from an oganisation called College Degree, alerting me to a wonderful list of 99 Resources to Research & Mine the Invisible Web. I’ve had a play with several of these now, and can imagine they’d be pretty useful in particular contexts. A number of them are examples of the repository-specific search engines that I refer to in my slide-show, while others are more subject specific and search multiple repositories.
In addition to the links to search engines, the list also contains examples of specific directories, catalogues and databases etc, and has a useful section on social media also. It’s worth scrolling right to the bottom for some articles and guides about the ‘deep’ or ‘invisible’ web.
Great slideshow. Here is a good article that adds some additional detail to the topic and a good set of links to the deep web search engines and other helpful sites.
Thanks for this link Dave – I’ve now got it bookmarked 🙂 Like the way you’ve teased out the ideas, and thanks for the list of search engines at the end!