Robots.txt Basics for Small Websites
Learn what robots.txt can do, what it cannot protect and why sitemap lines are useful.
Related tool
Open tool
Robots.txt Generator helps you apply this guide directly in the browser.
Robots.txt is guidance, not security
A robots.txt file tells well-behaved crawlers which paths they should avoid. It does not hide private files from users or malicious bots.
If a folder contains sensitive data, protect it with server rules, authentication or by keeping it outside the public web root.
Point crawlers to your sitemap
Adding a Sitemap line helps search engines discover the official sitemap location.
For most small sites, a simple robots.txt file with Allow, a few Disallow paths and a Sitemap line is enough.