The AIEO-Readiness crawler

AIEO-Readiness checks whether AI crawlers and agents can reach and read a website. It runs only when a signed-in AIEO.ee user starts an AI readiness check of their own project's domain. Nothing runs on a schedule.

How to recognize it

Every request carries this User-Agent. In robots.txt, match the product token AIEO-Readiness.

AIEO-Readiness/1.0 (+https://aieo.ee/bot)

What one check requests

  1. robots.txt first. If it cannot be reached (server error or timeout), nothing else is requested.
  2. The homepage, once as HTML and once asking for Markdown (Accept: text/markdown).
  3. The sitemap declared in robots.txt (same host only), else /sitemap.xml.
  4. /llms.txt and /llms-full.txt.
  5. Agent discovery files: /.well-known/ai-plugin.json, /openapi.json, /.well-known/api-catalog, /.well-known/oauth-protected-resource, /.well-known/oauth-authorization-server, /.well-known/ai-catalog.json, /.well-known/ard.json, /.well-known/ucp and /auth.md.
  6. Only when the site declares them: its Markdown alternate, a SKILL.md and an MCP server card on the same host.
  7. Two DNS TXT lookups (_agent.<host> and _index._agents.<host>).
  8. At most 3 links listed in the site's llms.txt, each requested once to see whether it opens. These can be on other hosts; their robots.txt is checked first.

How it behaves

  • Obeys robots.txt: the AIEO-Readiness group if there is one, else the * group. A path that is disallowed is never requested.
  • GET requests only, ports 80 and 443 only. No cookies, no logins, no forms, and no JavaScript is executed.
  • At most 80 requests per check, including redirects, and 90 seconds in total.
  • At most 60 requests to the checked site and 30 to any other host per 10 minutes, across all users. A check of the same site within 10 minutes reuses the previous result.
  • Private and internal network addresses are refused.

How to block it

Add this to your robots.txt. The user who asked for the check is told that your site disallows the crawler.

User-agent: AIEO-Readiness
Disallow: /

Questions or a problem with this crawler? Contact the operator of this deployment.