Mobile-First Indexing and AI Crawlers: Parity Requirements for Citations

TL;DR

Serve the same primary content on desktop and mobile. Google indexes from Googlebot Smartphone (Google Search Central, 2025); GPTBot, ClaudeBot, OAI-SearchBot, and PerplexityBot fetch with desktop UA strings (Cloudflare, 2025; Cite.sh, 2026). Use a single responsive site, expose the same HTML, and avoid hiding sections on either viewport.

Per-crawler user-agent and viewport reference

Why parity matters in both directions

1. Mobile-only gap (Google side). If a section appears only on the desktop layout (rare but real on legacy sites), Googlebot Smartphone will not index it (Lumar, 2019).
2. Desktop-only gap (AI side). If a section is rendered only into the mobile layout via responsive CSS — or worse, fetched only when a mobile UA is detected — GPTBot, ClaudeBot, and PerplexityBot will not see it because they identify as desktop.
3. Adaptive serving risk. Sites that detect UA server-side and serve different HTML for mobile vs desktop double the parity surface. AI crawlers receive desktop HTML; humans on mobile receive different HTML; either path can drop content.

Patterns by site architecture

Responsive design (single URL, single HTML)

The safest pattern. The same HTML is served to all UAs; CSS adapts the layout. Both Googlebot Smartphone and AI crawlers see identical content. Recommended for new sites.

Dynamic serving (single URL, server-side UA detection)

Different HTML by user agent. Risk: AI crawlers (desktop UA) get one variant; mobile users get another. Audit with both UA strings to confirm parity. Vary responses with Vary: User-Agent so caches do not mix variants.

Separate URLs (m-dot)

example.com for desktop, m.example.com for mobile. Use bidirectional rel="alternate" and rel="canonical" per Google's guidance. AI crawlers will hit the canonical (desktop) version; mobile-first index uses the smartphone version. Parity is mandatory and harder to maintain.

Hidden content and tabs

Google treats content in collapsed accordions, tabs, and display: none blocks as fully indexable as long as it is in the initial HTML. AI crawlers do likewise — they parse the HTML and ignore CSS visibility. Risk lives in JS-driven reveals where content is fetched only on click; AI crawlers will not see it without SSR.

Largest Contentful Paint and mobile retrieval

LCP affects only the rendering pipeline (Googlebot Smartphone, Bingbot partial, Gemini via Google infra). Non-rendering AI crawlers do not measure LCP and instead look at HTML payload size and TTFB. Optimize both: keep below-the-fold images lazy, reserve space to avoid CLS, and keep TTFB under 600ms for the AI fetcher path.

Parity audit checklist

• [ ] One responsive HTML or strict server-side parity between mobile and desktop variants.
• [ ] Every article body section is in the initial HTML response for both UAs.
• [ ] Structured data (JSON-LD) is identical across variants.
• [ ] Internal navigation (header/footer links) matches across variants.
• [ ] Images, captions, and alt text identical or equivalent.
• [ ] FAQ markup not stripped from the mobile variant.
• [ ] Vary: User-Agent set if dynamic serving is used.
• [ ] Test with curl using Googlebot/2.1, GPTBot/1.0, and a real mobile UA.

curl -A "Mozilla/5.0 (Linux; Android 10) Googlebot/2.1" https://example.com/article -o mobile.html
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" https://example.com/article -o ai.html
diff <(grep -oE '<h[1-6][^>]*>[^<]+' mobile.html) <(grep -oE '<h[1-6][^>]*>[^<]+' ai.html)

Common mistakes

• Removing FAQ or comparison tables from the mobile layout to "simplify" it.
• Lazy-loading body text on mobile via IntersectionObserver while keeping it static on desktop.
• Different schema between m-dot and main domain.
• Forgetting Vary: User-Agent with dynamic serving — caches mix variants and serve the wrong HTML to AI crawlers.
• Trusting that Googlebot Smartphone parity is enough; AI crawlers are still desktop-side.

FAQ

Q: Does GPTBot use a mobile or desktop user agent?

Desktop. GPTBot's official user agent string is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot) — desktop-class (Cite.sh, 2026).

Q: Will my mobile-only content be missed by AI search?

Yes if the content only renders when a mobile UA is detected. AI crawlers identify as desktop and will receive your desktop HTML; mobile-only sections will be invisible to them.

Q: Does Google still crawl with Googlebot Desktop?

Rarely. Google completed the mobile-first indexing transition in July 2024 and now indexes from Googlebot Smartphone for nearly all sites (Google, 2025).

Q: Is responsive design enough?

Yes for the parity question. Responsive design serves the same HTML to all UAs and is the simplest way to keep desktop and mobile in sync for AI crawlers and Googlebot Smartphone.

Q: How do I audit content parity?

Fetch with both a mobile UA (Googlebot/2.1 smartphone variant) and an AI bot UA (GPTBot/1.0), then diff the headings, structured data, and main body. Significant differences are parity risks.