# Robots Exclusion De-Facto Standard File Robots.txt # # User-agent: * # directed to all spiders Disallow: # Allow everything