User-agent: * Disallow: /js/ Disallow: /aspnet_client/ Disallow: /cgi-bin/ Disallow: /icons/ Disallow: /loewen/ Disallow: /small/ Disallow: /tiger/ Disallow: /zebra/ Disallow: /woelfe/ Disallow: /super/ User-Agent: Googlebot-Image Disallow: / # don't let search engines see the RDF feed, it's just confusing. User-agent: Googlebot/2.X (+http://www.googlebot.com/bot.html) User-agent: Googlebot/2.1 User-agent: InfoNaviRobot(F107) User-agent: TV33_Mercator_1-1.0 User-agent: AVSearch-3.0 User-agent: Scooter/2.0 User-agent: Slurp/2.0 User-agent: SearchengineLicenceSheep_v1.0 User-agent: shadow/2.0 User-agent: MultiText/0.1 User-agent: FAST-WebCrawler/2.2.5 User-agent: Atomz/1.0 User-agent: htdig/ (searchit@netmind.com) User-agent: spider00.logika.net. User-agent: Gulliver/1.2 User-agent: Lycos_Spider_(T-Rex) User-agent: ArchitextSpider User-agent: Arachnoidea User-agent: BackRub/2.1 User-agent: InfoSeek Sidewinder/0.9 User-agent: KIT-Fireball/2.0 Disallow: /searchtools-rss.xml # updated 2003-02-04 (disallow rtestprob links)