Grub - New crawling technology

[SensoVision]

Recently LookSmart present it's new technology which should revolutionize search engine technology...with Internet growing so rapidly even for most powerful search engine it takes a while to renew it's database. Now LookSmart offer solution to this problem presenting it's Grab technology which is distributed project free to host by anoyone on the web. You can host it on your home machine or on web site and save your bandwaith by this because when you host Grub only determine pages is changed and only when it find some changes Grab automatically submit them not recrawling whole site.  Probably many of you spot this crawler in your logs

Crawl your own stuff with www.inactivedomain.com

as I did, it's one of the thousands crawlers of Grub and soon they promise that this number grow to millions. Read here to learn more about this subject. Also visit you can visit http://www.inactivedomain.com/ directly and download [erspmal copy of client program if you're interested of course.

[SensoVision]

I didn't see any official news on this issue yet but it seems that www.wisenut.com start to use data collected by Grub clients. When I log on into my Grub account I see new button called Wisenut and there is such statement:

Wisenut Search Engine

The Wisenut search engine now uses crawl results from Grub to update its index. If you run the client, your crawling results will help improve the performace of Wisenut!

Also I've try to search for sites which I have in "local crawling" and not listed in Zeal and this sites really appear in Wisenut listing.

[Queen Bee]

Interesting thread... Will look into it.

Grub? What a weird name!

[12noon]

I installed Grub last week, but took it back out because i wasn't sure how to use it.

[SensoVision]

I installed Grub last week, but took it back out because i wasn't sure how to use it.

what kind of problems you have with it? maybe we could help with it if you wish? since I think Grub is a good way to get into Wisenut without paying huge cash to LookSmart...

[12noon]

I'll put it back in and then ask once i've done that.
It's bedtime now, but will add it again in the morning.

[12noon]

bump

I've got grub installed and crawling and i've even added the grub file to my server, but i don't know how to get it to crawl my website. It says :-

URLs crawled : 27.9
Crawled this month : 21.7
Local URLs crawled : N/A
Crawled today : 26.4
Overall ranking : 5.226th
Monthly ranking : 1.041st
Daily ranking : 218th
Local crawl ranking : 20.497th
Group ranking : N/A

Is my site a local URL ?
 

[SensoVision]

yup your site is local URL, if you using Windows Grub client you'll see which URLs you're crawling. Also you may check if your site isn't in Wisenut already(it's depend how long ago you start to crawl since I'm not sure how frequenly they refresh database there).

[12noon]

I only added it back in a few hours ago, but i'm on cable and the thing is flying through the websites.
I'll do a search later and see what the results are.
BTW i'm using windows grub version 1.4.3

[SensoVision]

don't forget to enable local crawling and ensure that you list your site(you can do this when you loggin in into your account on Grub.org)

[12noon]

I've just looked at the grub message board and others have the same problem as me. Local crawling doesn't work for everyone and i'm unlucky.
 

[justjeni]

   I don't get this - what does that do when it crawls the web from you computer or web site?

[Queen Bee]

"Grub's goal is revolutionary - to track down every site in the world and provide a real-time map of the Web.... By downloading the unique screensaver, you can donate your computer's unused bandwidth to probing the hidden depths of the Web."

"People who choose to download and run the client will assist in building the Web's largest, most accurate database of URLs. This database will be used to improve existing search engines' results by increasing the frequency at which sites are crawled and indexed. "

If that helps you any. Guess we should just download it and find out for ourselves, eh?

[justjeni]

  Hmmmm...

If that helps you any. Guess we should just download it and find out for ourselves, eh?

I guess!!!

[SensoVision]

ups sorry for delay, this small programs which you can put on your desktop or on your site, it will crawl links around the web and keep a track whenver information is updated or not, later all the data is transferring to central servers and processed. It's distributing project which as hope their inventors will became search engine of the future as for even major search engines is hard to update all the sites daily and renew their databases accordingly so Grub's main target was to keep info always up-to-date by using computers of voulounteers who run their client program and check various web sites.
Also by participating in this project you'll be able to add your sites to local crawling and defne which pages you want to be crawled(unfortunately as wrote 12noon Grub isn't yet implement techonology which will pick all pages from your site and you need to list them manually for indexing so I don't sure that for your site if it has 1000 pages would be wise to enable local crawling, at least no before Grub do somehting about this).