# robots.txt User-agent: Slurp Crawl-delay: 8 Disallow: /cgi-bin/ Disallow: /mro/ User-agent: Yeti/1.0 (+http://help.naver.com/robots/) User-agent: Mozilla/5.0+(compatible;+jobs.de-Robot++http://www.jobs.de) Disallow: / User-agent: * Disallow: /cgi-bin/ Disallow: /mro/