User-agent: heritrix Crawl-delay: 2 User-agent: * Disallow: /webadmin/ Disallow: /common/ Crawl-delay: 30