Grid Computing & Web Crawlers
Home | Blog | Contact Us | Policies | SiteMap
Topic Thread:      Computer & Internet   »   Distributed Computing   »   Grid Computing   »   Grid Computing & Web Crawlers

Recommended:   Web Crawlers & Bots,    Grid Computing & Protein Folding,    Spyware Removers & Antispyware


Grid Computing & Web Crawlers




What are Distributed Computing and Grid Computing?

Distributed computing is a means by which computer resources at more than one physical location can be tied together to perform a task. The Worldwide Web is perhaps the best example of distributed computing. Distributed computing systems range in size from global computing to local intranets.

Grid computing is a method by which the computing power of individual PCs or other computer resources throughout the world can be harnessed to perform large-scale projects. Grid computing joins these resources together through special software to simulate a single large computer or virtual computer. Previously, such massive computing power was available only through use of supercomputers. You can participate in these projects; become a partner with us! The projects are largely non-profit; you donate your idle computing time to help forward progress in scientific research that might otherwise not be possible.


Web Crawlers

Web crawling is a popular terminology used to describe the procedure by which a search engine locates and interrogates websites in order to add (index) and update information about these sites in the search engine’s database. The programs utilized to perform this activity are referred to as web crawlers, web spiders or bots.


The Grub Distributed Web Crawling Project

The Grub distributed computing project, which is no longer operational, was conducted by LookSmart, Ltd. Grub’s goal was to track down every website in the world to provide a real-time map of the Worldwide Web. Leveraging the power of distributed computing, Grub allowed anyone with an internet connection to participate in this bold endeavor. By downloading the unique screensaver, you could donate your computer’s unused bandwidth to discover diverse websites throughout the world in a manner similar to what is routinely done by the internet’s major search engines.


The Boitho Distributed Crawler

Boitho is a Norwegian search engine that actively employs the distributed web crawler concept. The goal of Boitho is to create a thumbnail image of each web page it crawls. At the time of this writing, Boitho had indexed 171,169,451 web pages.

If you wish to participate in an active scientific grid computing project, see Grid Computing & Protein Folding.



WARNING:  Never install or run any distributed computing client at your place of employment without first receiving written permission from your employer to do so, and do not install it on any computer or workstation other than those for which you have been authorized. Because they transfer data to and from an outside server, some employers consider distributed computing clients to be invasive software and could levy punitive actions for unauthorized use which could include termination of employment.



Authored by Kenneth L. Anderson.  Original article published 5 January 2004, updated 17 January 2006.


Our Computer & Internet section includes hardware, software and learning products for business and home computing environments. We feature reviews, news, educational articles, products and services relating to business computing, personal computing and the internet. For additional resources, check Related Links to the left or use the Computer & Internet Sitemap for a complete listing of computer and internet resources available on our metasite.


Receive updates to this and other pages on Twitter!


Ten Spider™ and tenspider™ are trademarks of Ten Spider Enterprises, LLC, and are protected by United States and international trademark laws.
Valid XHTML 1.0!