In English

Boitho distributed crawler (BETA).

This is a distributed crawler, where everybody can donate there superfluous computer resources and spare bandwidth, to help us create a bigger and better search engine.

Our goal is to make a general internet search engine with a thumbnail picture of all the pages. The problem is that the bandwidth and computer resources needed to make a thumbnail of a internet page is several times the resources needed just to download a HTML page. This means that Boitho has to spend more resources on crawling than other search engines.

To make the most of our available resources, and to allow volunteers to donate their superfluous bandwidth and idle CPU time, we have developed a distributed crawler for Boitho, like seti@home and Grub. That way people can install a program on their computers and help us with the crawling.

If you have any questions you can contact Runar Buvik, e-mail: runarb (at) boitho (dot) com, phone: +47.93030627

Download

BoithoCrawler.exe

How to install

If you are running an old version of the crawler, you will have to close it, and then wait until the taskbar icon disappears.
Download and run the file BoithoCrawler.exe
Follow the instructions on the screen
Start the crawler by clicking on the start-menu -> All programs -> BoithoCrawler -> Start Boitho Crawler

If you want to you can create an account. That way you get bether statistics over how many URLs an robots.txt you have crawled. Go to Tools -> Options -> User Info -> Create User

Statistics

http://dcsetup.boitho.com/cgi-bin/dc/topCrawlers.cgi

Minimum system requirements

Operating system: Windows XP, Windows 2000 or Windows Millennium.
Hard disk: 1 MB for the program, and 30 Mb for cachefiles.

Blocking us from accessing you site

You may block access by create/modify you robots.txt file. Please refer to the crawlers homepage for blocking instructions.

Known bugs

The bandwidth meter does not always work properly if you have two or more network cards.