robots.txt

[Taylor]

What is robots.txt? And why do some people look for it on my server while others don't. Is it a browser thing?

Thanks


------------------
-Taylor

[posterboy]

The crawlers have an agreement to honor your robots.txt. This because some web sites prefer that certain files, or groups of files, should not be indexed by the search engines. Note that excluding the crawlers can only be done on a directory basis, not by individual files. here's mine.

# mail webmaster@gordo.penguinpowered.com
User-agent: *
Disallow: gb

The User-agent allows excluding named crawlers, to certain places, in this case we are talking to all of them.
The disallow gb keeps them from indexing my guestbook pages, in the directory gb
There's further information on this on the web, if you need it. MOST of the engines are very good about honoring this.
Ray


------------------
ray@raymondjones.net
HTTP://www.raymondjones.net

[Taylor]

Thanks Ray, that was very helpful. I was curious since I see it often in my logs.

I will have to read some more and come up with one for something fun to do when I have time.



------------------
-Taylor