What should my robots.txt contain for AI agents?

What it is

robots.txt is a plain text file at the root of your domain that states which clients may fetch which paths. It is grouped: a `User-agent:` line names a client, and the `Allow:` and `Disallow:` lines under it apply to that client.

We ask for two things only. The file must answer 200, and it must contain at least one User-agent group, because a file with no group states nothing.

We also read any `Sitemap:` lines from it, which is how the sitemap Check finds a sitemap that does not live at the default address.

Why an agent needs it

Every well-behaved crawler reads robots.txt before it reads anything else. A 404 there is not a refusal, but it does leave your position unstated.

A robots.txt that returns your HTML homepage or a soft 404 is worse than an absent one. The client gets 200 and parses a page as policy.

The Fix

Serve a text file at /robots.txt as text/plain. Open with a `User-agent: *` group and the paths you are happy for any client to fetch.

Add a `Sitemap: https://yourcompany.com/sitemap.xml` line so a crawler does not have to guess the address.

Confirm it is a real file and not your framework's catch-all route answering 200 with HTML. Fetch it in a terminal and read the first line.

This one is a change in your own markup or edge configuration, so the Fix Pack cannot write it for you. A Report names every page it affects, with the Evidence.