This specification defines how to structure web content for autonomous AI agents — crawlers, chatbots, research assistants, browser-based agents like ChatGPT Atlas and Perplexity Comet, coding agents like Cursor and Claude Code, and any other AI system that discovers, parses, and synthesizes information from the web on behalf of a user.

TL;DR

Make every page agent-ready by ensuring it is discoverable (present in llms.txt and sitemap.xml, allowed for major bots in robots.txt, optionally exposed via agents.json), parseable (full ~30-field frontmatter, semantic headings, JSON-LD, and where relevant a paired agent.md for tool-use surfaces), and attributable (canonical URL plus an ai.txt policy declaring source name and citation format). The compliance checklist at the bottom of this page is the single source of truth.

For the broader strategy, see the AI Agents pillar.

Specification overview

Layer 1: Discovery

llms.txt

Every site should provide a /llms.txt file — a Markdown index that tells AI agents what your site contains and how it is organized. The format was proposed by Jeremy Howard and is documented at llmstxt.org.

Required elements:

• Site name (H1 heading)
• Site description (blockquote)
• Content index (links with descriptions)
• Section organization (H2 headings)

Minimal example:

markdown

Acme is a payments platform. This index lists the canonical references AI agents should consult.

Core concepts

• Payments overview: How card and bank transfers move through Acme.
• Webhooks reference: Event types, retry policy, and signature verification.

API

• Authentication: API key formats and header conventions.
• Errors: Error envelope and code taxonomy.

Full specification: How to Create llms.txt.

agents.json (proposed)

agents.json is an emerging convention for declaring agent-actionable surfaces — APIs, tools, and structured tasks — at a well-known location (/.well-known/agents.json). It complements llms.txt by exposing capabilities, not just content.

Minimal example:

{
  "schema_version": "v1",
  "name_for_model": "acme_payments",
  "description_for_model": "Read and act on Acme payment data.",
  "auth": { "type": "oauth", "authorization_url": "https://acme.com/oauth" },
  "tools": [
    {
      "name": "create_invoice",
      "description": "Create an invoice for a customer.",
      "endpoint": "https://api.acme.com/v1/invoices",
      "method": "POST"
    }
  ],
  "contact_email": "[email protected]"
}

Adoption is partial; treat it as forward-compatible metadata, not a hard requirement.

Sitemap for AI

Standard XML sitemaps help AI crawlers discover content. Enhance with:

• for freshness signals
• to indicate update patterns
• to highlight key pages

A separate sitemap-ai.xml can list only canonical, citation-ready pages — useful when your site mixes marketing and reference content.

robots.txt for AI crawlers

Explicitly allow major agent crawlers; disallow only when you have a specific reason:

User-agent: GPTBot

Allow: /

User-agent: OAI-SearchBot

Allow: /

User-agent: ChatGPT-User

Allow: /

User-agent: ClaudeBot

Allow: /

User-agent: Claude-Web

Allow: /

User-agent: PerplexityBot

Allow: /

User-agent: Perplexity-User

Allow: /

User-agent: Applebot-Extended

Allow: /

User-agent: Google-Extended

Allow: /

Crawler user-agents change. Verify at OpenAI bot docs, Anthropic crawler docs, and Google crawler docs.

Layer 2: Parsing

Frontmatter metadata schema

Every content page should include structured frontmatter. The schema below is the canonical 30-field shape; deviate only where a field truly does not apply.

---
# Identity
title: "Page Title"
slug: "url-slug"
section: "section-name"
canonical_url: "https://example.com/section/url-slug"
status: "published"

Knowledge

canonical_concept_id: "unique-concept-identifier"

knowledge_domain: "domain-name"

concept_type: "core-concept|sub-concept|technique|tool|standard|metric"

entities: ["Primary Entity"]

aliases: ["alt name"]

related_concepts: ["related-id"]

Taxonomy

content_type: "guide|reference|comparison|definition|specification|checklist|tutorial|framework"

primary_audience: "developer|seo-specialist|content-strategist|founder|marketer"

secondary_audiences: ["..."]

reader_modes: ["human", "ai-agent"]

difficulty: "beginner|intermediate|advanced"

ai_platforms: ["chatgpt", "perplexity", "claude", "gemini"]

SEO

description: "120-160 char description."

focus_keyword: "primary keyword"

secondary_keywords: ["k1", "k2"]

AI readiness

canonical_question: "What is X?"

llm_summary: "2-sentence factual summary."

citation_readiness: "reviewed|draft"

Lifecycle

published_at: "YYYY-MM-DD"

updated_at: "YYYY-MM-DD"

last_reviewed_at: "YYYY-MM-DD"

review_cycle_days: 90

version: "1.0"

Relations

series: "series-id"

series_order: 1

related_articles: ["section/slug"]

I18n + authorship

lang: "en"

translations: []

author: "Author Name"

reviewed_by: null

agent.md (tool-use surface)

For pages that document an API, CLI, or other tool, pair the human-readable page with a sibling agent.md file at the same path. agent.md strips marketing prose and gives an agent the deterministic signature it needs.

Minimal example:

# acme.payments.create_invoice

Signature

POST /v1/invoices

Content-Type: application/json

Authorization: Bearer

Input

• customer_id (string, required): Acme customer ID, format cus_.
• amount_cents (integer, required): Positive integer.
• currency (string, required): ISO 4217 code.

Output

• invoice_id (string): Created invoice ID, format inv_.
• status (string): One of open, paid, void.

Errors

• 400 invalid_currency — currency not in ISO 4217.
• 402 insufficient_funds — customer balance below amount_cents.

Idempotency

Pass Idempotency-Key header. Same key returns the original response.

Cursor, Claude Code, and similar coding agents are the primary consumers of agent.md-style surfaces today. Repository-root files (AGENTS.md, CLAUDE.md) follow the same pattern for code-context use.

Content body structure

Answer-first pattern

The first section directly answers the page's core question:

# [Title as Question or Topic]

[Direct answer in 1-2 sentences. Complete and self-contained.]

TL;DR

[Snippet-ready 2-3 sentence summary.]

Heading hierarchy

• H1: page title (exactly one)
• H2: major sections
• H3: sub-sections within H2
• H4: rare; avoid deeper nesting

Required structural elements

Every agent-ready page must include:

1. One H1 matching the frontmatter title.
2. One AI summary blockquote immediately after the H1.
3. A TL;DR section (## TL;DR) with a 2-3 sentence snippet.
4. At least one extractable definition, table, or step list in the body.
5. A FAQ section with 3-8 question-answer pairs phrased as natural questions.
6. A canonical URL in frontmatter and in .
7. At least one JSON-LD block describing the primary entity.

Pages missing any of (1)-(7) are non-compliant and should not be marked citation_readiness: reviewed.

Extractable patterns

Definition:

[Term] is [complete definition in one sentence].
[Optional second sentence on significance.]

Comparison table:

| Dimension | Option A | Option B |
|---|---|---|
| Aspect 1 | Value | Value |

Step-by-step:

1. **Step name** — Description.
2. **Step name** — Description.

FAQ pair:

### Question in natural language?
[Direct answer. No preamble.]

JSON-LD structured data

Every page should include at least one JSON-LD block.

Article:

{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "Page Title",
  "description": "Page description",
  "author": { "@type": "Organization", "name": "Site Name" },
  "datePublished": "2025-01-01",
  "dateModified": "2026-05-01"
}

FAQ:

{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "Question text",
      "acceptedAnswer": { "@type": "Answer", "text": "Answer text" }
    }
  ]
}

Full reference: Structured Data for AI Search.

Layer 3: Attribution

ai.txt

Define how AI agents should attribute your content:

AI Agent Access Policy

User-agent: *

Allow: /

Attribution-required: yes

Source-name: Your Site Name

Source-url: https://yoursite.com

Citation-format: "[Title] — [Source-name] (Source-url/path)"

Full specification: ai.txt Starter Template.

Canonical URLs

Every page must have exactly one canonical URL:

<link rel="canonical" href="https://yoursite.com/section/slug" />

Agents should use this URL for attribution regardless of how they discovered the page. Mirror sites, syndicated copies, and AMP variants must point back to the canonical.

Source metadata

Mark organization-level information once with Organization schema (logo, sameAs, contact) so agents can resolve "who owns this content" without parsing every page individually.

Implementation patterns by platform

Different agents weight signals differently. The patterns below summarize observed behavior; verify against each platform's current docs.

ChatGPT (OpenAI) — Search and Atlas browser

• Honors robots.txt for GPTBot, OAI-SearchBot, and ChatGPT-User.
• Strongly favors pages with clear