Robots.txt Explained in Plain English
How the robots.txt file controls crawler access, the common mistakes that block your own site, and a correct template to copy.
robots.txt is the first file crawlers read. It tells bots which parts of your site they may or may not visit — one wrong line can hide your entire site from Google.
How It Works
Crawlers request /robots.txt first. Rules apply per user-agent: allow or disallow paths. Because it only governs crawling, not indexing, blocked-but-linked pages can still appear in results without content.
Common Mistakes
Blocking your whole site during development and forgetting to remove it. Disallowing CSS or JS files, which prevents proper rendering. Using robots.txt for noindex — that requires a meta tag instead. And editing the wrong file so a static one overrides your app’s output.
A Safe Template
Allow all public content, disallow only private areas like /admin and /member, and always include a Sitemap line at the end. Test by requesting the file in a browser: what you see is exactly what bots see.
Frequently Asked Questions
Does robots.txt speed up my site?
It saves crawler bandwidth, not visitor bandwidth. Still worthwhile on large sites.
Can I block only Googlebot?
Yes — rules are per user-agent, though spoofed bots can ignore them.
Where should robots.txt live?
At the site root: yourdomain.com/robots.txt — nowhere else.
Check your live robots.txt today in the browser: it should allow all public pages and point to your sitemap.