Open Mailroom
Website owners · Crawlers
robots.txt is a sign on the door. It is not a lock.
A robots.txt file asks cooperating crawlers to respect your rules. It does not make a public page private. The standard explicitly says these rules are not access authorization. RFC 9309.
Three controls, three different jobs
| Control | Its job | Its limit |
|---|---|---|
| robots.txt | Tell cooperating crawlers which paths they may fetch. | A crawler can ignore it. A blocked URL can still be discovered. |
| noindex | Ask supporting search engines to exclude a page from results. | The crawler needs to fetch the page to see the instruction. The page is still publicly accessible. |
| Authentication and authorization | Restrict access to approved users. | You still need correct access checks and secure operations. |
Google documents noindex and its crawl requirement. For confidential material, its guidance recommends password protection. Don’t publish private contact lists and expect robots.txt to protect them.
Training bots are not every bot
OpenAI distinguishes GPTBot, used for potential training content, from OAI-SearchBot and user-initiated ChatGPT-User requests. Its documentation says robots.txt may not apply to the latter. Anthropic distinguishes ClaudeBot, Claude-SearchBot and Claude-User, and says its bots honor robots.txt. Blocking a training crawler therefore is not the same as blocking search or every person using an agent. OpenAI crawler documentation; Anthropic crawler documentation.
An example, not an anti-scraping force field
These rules request that the two named training crawlers stay out. They do not block all agents, undo earlier collection or prevent third-party retrieval. Review and merge them with your existing rules; don’t replace a working file blindly.
# Requests to two specific training crawlers. # Not access control. Not a block on every AI tool. User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: /
Place the relevant file at the origin’s /robots.txt and check each subdomain you use. Keep it reachable. Check current vendor documentation when you make changes: the names, purposes and behavior of crawlers can change.
Reduce exposure at the source
Our practical recommendation: remove unnecessary personal contact details from public pages, use a dedicated public contact address, and keep private documents behind real access controls. Check older PDFs and downloadable files too. If you add rate limits or bot controls, test the effect on legitimate visitors and services.
For a page you want excluded from Google rather than kept secret, use a supported noindex method and let Google fetch it. Blocking that same page in robots.txt can stop Google seeing the noindex instruction. Neither instruction sends a deletion request to companies that already copied your details.
Read it. Use it. Close the tab.