What a search engine spider sees...

My minor research project on internet search continues.

A search engine comprises three major components:

(1) A spider (also called a "crawler" or a "bot") that goes to sites and reads them. The crawler copies everything (text, pictures, documents, everything) back to the second part of the system…

(2) The database and index - the database stores all data sent back by the spider and indexes of all the data are constructed to make finding information easier using...

(3) The front end - a program that accepts your search request, looks in the index and returns results to you.

When you search at a search engine your are searching the engine’s copy of the internet, not the internet itself.

If you ever wondered what a spider actually sees when it visits your site have a look at this tool which will scan a page and show you the results, as well as provide useful page data such as word density, meta keywords, links, and other bits of page information.

I found out about the spider tool via A Welsh View.

Tags: Search, Google, Yahoo