Robots.txt Basics for Small Websites
Learn what robots.txt can do, what it cannot protect and why sitemap lines are useful.
Related tool
Open tool
Robots.txt Generator & Sitemap Helper helps you apply this guide directly in the browser.
Robots.txt is guidance, not security
A robots.txt file tells well-behaved crawlers which paths they should avoid. It does not hide private files from users or malicious bots.
If a folder contains sensitive data, protect it with server rules, authentication or by keeping it outside the public web root.
Point crawlers to your sitemap
Adding a Sitemap line helps search engines discover the official sitemap location.
For most small sites, a simple robots.txt file with Allow, a few Disallow paths and a Sitemap line is enough.