AI Visibility Experiments: A/B Testing for GEO

TL;DR

GEO experiments must control more variables than traditional A/B tests — LLM responses are stochastic, multi-engine, and time-varying. The reliable design pattern: pin a prompt portfolio, sample across engines and runs, isolate the page change, and track citation share with confidence intervals. Run for at least two weekly cycles before declaring a result.

Why GEO experiments are different

Classical A/B testing splits users into variants and measures user behavior. AI visibility splits queries against engines and measures citation behavior. Three core differences:

1. Stochastic outputs. The same prompt yields different answers across runs. Measurement must average over many runs.
2. Multi-surface. Citations differ across ChatGPT search, Claude, Perplexity, Gemini, and AI Overviews. A win on one engine is not a win overall.
3. Crawl latency. Page changes take 1-3 crawl cycles to influence citations. Tests need a warm-up window.

Ignoring these gives you noisy, unreproducible numbers that lead to bad decisions.

Experiment design lifecycle

1. Hypothesis. State the change and its expected directional effect.
2. Variant scope. Define what changes (one page, a section, a template) and what stays fixed.
3. Prompt portfolio. Compile 30-80 prompts representative of the target query space.
4. Engine matrix. Choose 3-5 engines to measure.
5. Pre-period baseline. Run the prompt portfolio against the engine matrix N times (N=5 minimum).
6. Apply change. Implement the page change. Document timing.
7. Warm-up window. Wait 7-14 days for crawl, refresh, and reindex.
8. Post-period measurement. Re-run the portfolio with the same N.
9. Analyze. Compare citation share, position, and snippet match.
10. Decide. Ship, refine, or revert.

Picking the prompt portfolio

• 60% target query phrasings the page should win.
• 25% adjacent or follow-up queries (test for spillover).
• 15% unrelated controls (test for broad regressions).
• Cover the user intent taxonomy for the topic.
• Lock the portfolio for the duration of the experiment. Adding prompts mid-test invalidates results.

Variables to control

• Engine version. GPT-4 vs GPT-4o behave differently; pin the model when possible.
• Locale. Run from the same country, language, browser profile.
• Auth state. Always signed-out for retrieval-only behavior.
• Time of day. Schedule runs at the same hour windows.
• Cache. Clear ChatGPT memory; use new sessions per run.
• Crawler completion. Confirm via Search Console that the page was recrawled.

Uncontrolled variables become noise; noise hides effects.

Sample size and significance

GEO citation share for a typical prompt is between 0% and 30%. To detect a meaningful 5-point lift:

• Minimum runs per prompt per engine: 5.
• Minimum prompts per portfolio: 30.
• Minimum total observations: 30 prompts × 4 engines × 5 runs = 600 per period.
• Statistical test: two-proportion z-test on citation share, or paired comparison of prompt-level scores.
• Confidence target: p < 0.05 and the effect must persist across engines.

If an effect is significant on one engine but not the others, treat the result as engine-specific.

Measurement metrics

• Citation share. Percentage of runs where the target page is cited.
• Citation rank. Position among cited sources (1 is best).
• Snippet alignment. Whether the cited snippet matches the page's answer block.
• Spillover. Effect on adjacent prompts.
• Regression. Effect on unrelated control prompts.

Report all metrics; cherry-picking citation share alone hides regressions.

Common variants worth testing

• Rewriting the lead paragraph as an answer block.
• Adding FAQ schema and a 5-question FAQ section.
• Adding sameAs schema for the primary entity.
• Adding 2-3 internal links from the relevant hub page.
• Updating publication date and refreshing data.
• Tightening title and description.

Keep changes atomic. Bundling four changes obscures attribution.

Reporting template

A strong report includes:

• Hypothesis and variant.
• Prompt portfolio (counts and sample).
• Engine matrix.
• Pre-period and post-period dates.
• Citation share, rank, snippet alignment per engine.
• p-value and effect size.
• Spillover and regression analysis.
• Decision and follow-up plan.

Store reports next to the experiment configuration so future audits can replicate.

Pitfalls

• Single-run measurement. A single Perplexity query is not a measurement.
• Mid-test prompt edits. Invalidates the comparison.
• No crawl confirmation. You measured a change that was not yet indexed.
• Ignoring engine version drift. A model upgrade between periods can swamp the effect.
• No control prompts. You miss broad regressions.
• Comparing pre-test to post-test on different days of week. Cyclical seasonality matters.

Tooling

• See AI rank tracking tools for instrumenting prompt portfolios at scale.
• Build internal scripts for engine-specific endpoints when reproducibility matters.
• Cache and version control prompts and responses.

FAQ

Q: Can I A/B test by alternating page versions over time?

You can, but only with care. Use a switchback design with paired periods, and ensure crawl cadence allows propagation between switches. For most teams, parallel page tests on similar topics work better.

Q: How long does an experiment run?

Minimum 14 days post-change for retrieval-time engines; 30 days when training-time citations are part of the hypothesis. Shorter windows hit crawl-latency noise.

Q: How do I handle stochastic LLM outputs?

Average across 5+ runs per prompt per engine. Report both mean and standard deviation. Treat single-run differences as noise.

Q: Should I test multiple changes at once?

Not in the same variant. Run them sequentially or use multivariate designs only when the prompt portfolio is large enough (200+ prompts).

Q: What if a model upgrade lands mid-experiment?

Reset the experiment. Model changes can shift citation behavior more than any on-page change. Document the model version in the experiment config to detect drift early.