Accept-Language Handling for AI Crawlers

AI crawlers including Googlebot crawl from a single region (typically the United States) and most send no Accept-Language header by default; serving locale-adaptive content based on the request header or GeoIP causes only one language to be indexed, so the safe pattern is separate URLs per locale with hreflang annotations.

TL;DR

Use separate URLs per locale (/en/, /de/, /ja/ or en.example.com, de.example.com), annotate them with rel="alternate" hreflang="...", and never silently redirect first-time visitors based on Accept-Language or GeoIP. Send Content-Language on every response so crawlers and assistive tech know which language they got. AI crawlers, like Googlebot, often send no Accept-Language at all per Google Search Central.

Scope

This specification covers the Accept-Language request header and the Content-Language response header for sites that publish multilingual content and want every locale to be citation-eligible by AI search engines. Audience: i18n engineers and platform leads.

It does not cover translation quality, Right-to-Left rendering, or font-loading strategies for non-Latin scripts.

How AI crawlers send Accept-Language

Googlebot by default sends no Accept-Language header and crawls from US-based IPs. Google introduced locale-aware crawl configurations for sites that adapt content by locale, but only after detecting the adaptation pattern (Google Search Central, 2025).
GPTBot, ClaudeBot, PerplexityBot typically send no Accept-Language or Accept-Language: en in practice. None advertise locale-specific crawl pools.
Real-time retrieval bots (OAI-SearchBot, ChatGPT user fetches) may inherit the user's locale, but this is not guaranteed and varies by vendor.

The practical implication: relying on Accept-Language to choose what to render means AI crawlers will index whatever your default-locale fallback is. Your other locales will not enter AI citation pools.

Chrome's Reduce Accept-Language proposal further reduces fingerprinting by trimming the header to a single language for most users, making Accept-Language an even less reliable signal going forward.

Recommended pattern: separate URLs + hreflang

Google's documented best practice is to use separate URLs per locale and annotate them with rel="alternate" hreflang:

<link rel="alternate" hreflang="en" href="https://example.com/en/article" />
<link rel="alternate" hreflang="de" href="https://example.com/de/article" />
<link rel="alternate" hreflang="ja" href="https://example.com/ja/article" />
<link rel="alternate" hreflang="x-default" href="https://example.com/en/article" />

Use BCP 47 language tags (ISO 639-1 language plus optional ISO 3166-1 alpha-2 region, e.g., en-US, pt-BR).
Always include x-default pointing to your fallback locale.
The hreflang set should be reciprocal: /en/article should hreflang to /de/article and vice versa.
Pair every page with the Content-Language HTTP response header (or ).

URL strategies

Strategy	Example	Pros	Cons
Subdirectory	example.com/en/, example.com/de/	Single domain authority; easy hosting.	Edge routing complexity for some stacks.
Subdomain	en.example.com, de.example.com	Region-specific hosting and analytics.	Each subdomain needs hreflang reciprocity.
ccTLD	example.de, example.fr	Strongest local-market signal.	Distributed authority; multiple SEO setups.
URL parameter (?lang=de)	example.com/article?lang=de	Simple to implement.	Crawlers often de-duplicate parameters; weak SEO signal.

Subdirectory and subdomain strategies dominate; ccTLD requires real per-region operations to be worthwhile.

Anti-pattern: auto-redirect by Accept-Language

A 302 Found from /article to /de/article based on the Accept-Language header silently breaks AI crawlers in three ways:

Googlebot, GPTBot, and ClaudeBot send no Accept-Language, so they're redirected to your default locale. Other locales never enter their index.
When Google detects locale adaptation, it switches to locale-aware crawling for some pages but indexing remains incomplete.
Real users who land on the wrong locale for their preferences must navigate manually, hurting UX without improving SEO.

MERJ's analysis confirms that Accept-Language redirects actively block AI crawlers and search engines that would otherwise crawl every locale (MERJ, 2025).

Better pattern: show a banner or prompt suggesting the user switch locale, but render the URL they requested by default.

Anti-pattern: GeoIP redirect by IP

Similar failure mode. Googlebot crawls predominantly from US IPs; redirecting US visitors to /en/ and EU visitors to /de/ based on IP means:

Googlebot only sees /en/ content.
ClaudeBot and PerplexityBot operate from various IP ranges; behavior is unpredictable.
Cloudflare and other CDNs add more variance because their POPs span continents.

Framework recipes

Next.js i18n routing

Use the built-in i18n config with subdirectory routing. Disable localeDetection so first-time crawler visits don't redirect. Add hreflang tags via the component on every page.

// next.config.js

module.exports = {

i18n: {

locales: ['en', 'de', 'ja'],

defaultLocale: 'en',

localeDetection: false

}

Astro

Use astro-i18n or routing under src/pages/[locale]/. Render hreflang from the locale list. Avoid middleware redirects keyed on Accept-Language for first-visit traffic.

Hugo

Use defaultContentLanguage and per-locale URL prefixes. Hugo's built-in i18n exposes a translations list per page; render hreflang tags from it directly in your layout.

Content-Language response header

Always send Content-Language on locale-specific responses:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Language: de-DE
Vary: Accept-Encoding

This tells crawlers, screen readers, and translation pipelines what language to expect even if the URL or HTML doesn't make it obvious.

Common pitfalls

Auto-redirect on first visit — silently blocks AI crawlers from non-default locales.
Missing x-default — leaves Google guessing which locale to surface for unmatched users.
Asymmetric hreflang — /en/article references /de/article but /de/article references nothing back; Google ignores the relationship.
Wrong language codes — de_DE (underscore) is invalid BCP 47; use de-DE. en-UK is wrong; use en-GB.
Vary: Accept-Language without negotiation — fragments cache by header without serving different content. Add only when negotiation is real.
mismatch — if Content-Language says de-DE but , crawlers and assistive tech disagree.
Translated content via Accept-Language only — produces a single URL with multiple bodies; crawlers index one and miss the rest.

FAQ

Q: Does Googlebot send Accept-Language?

Not by default. Per Google's documentation, Googlebot sends HTTP requests without setting Accept-Language. For sites that adapt content by locale, Google introduced locale-aware crawl configurations, but the default behavior remains "no header".

Q: Should I redirect users based on browser language?

Not on first visit and not silently. Show a banner offering to switch locale, but render what the user requested. Auto-redirect breaks AI crawlers and frustrates users who specifically navigated to a non-default locale.

Q: Is hreflang required?

Yes for any site with content in multiple languages. Without hreflang, Google may merge or deduplicate locale variants, and AI engines may pick the wrong locale to cite.

Q: What language codes should I use in hreflang?

BCP 47 (RFC 5646) tags: lowercase ISO 639-1 language plus optional uppercase ISO 3166-1 alpha-2 region, joined by a hyphen. Examples: en, en-US, pt-BR, zh-Hant.

Q: Do AI crawlers respect Content-Language?

They use it as a signal alongside and the URL structure. None of these alone is authoritative; consistency across all three produces the strongest extraction.

Q: Can I use a single URL with Accept-Language negotiation?

Technically yes, with Vary: Accept-Language and proper Content-Language. In practice, Google warns this leads to incomplete indexing, and AI crawlers typically only see one variant. Separate URLs are safer.