GO Away Robots!
I got this info from Hothint today about this robot code that makes search engines to not get access to certain pages for websites... Even Google or Altavista has this .txt page but you can also use this .txt page as a map to the pages of the website...
Heres A Standard for Robot Exclusion to help you more about this.