#directions for robots, spiders, webcrawlers, etc to index entire site User-agent: * Disallow: