# All robots will spider the domain User-agent: * Disallow: # Disallow directory admin User-agent: * Disallow: admin # Disallow directory templates User-agent: * Disallow: templates # Disallow directory files User-agent: * Disallow: files