What Is LLM Evaluation for Search?

TL;DR

LLM evaluation for search measures whether an AI search system retrieves the right documents, grounds its answers in those documents, answers the user's actual question, and cites the right sources. The canonical metric set is the RAG triad—context relevance, groundedness, answer relevance—plus faithfulness and context precision/recall from RAGAS, layered on top of classical IR metrics like NDCG and recall@k from benchmarks like BEIR. Frameworks like RAGAS, TruLens, OpenAI Evals, and DeepEval provide LLM-as-a-judge implementations so teams can run evaluations continuously rather than relying on manual review. For GEO and AEO practitioners, evaluation is what separates measurable improvement from "vibes-based" optimization.

Definition

LLM evaluation for search is the discipline of quantitatively assessing the quality of LLM-powered search and answer engines along four dimensions: (1) retrieval quality—are the right documents being retrieved for the query? (2) groundedness/faithfulness—is the generated answer factually consistent with the retrieved context? (3) answer relevance—does the answer address the user's question? and (4) citation accuracy—do the cited sources actually support the claims attributed to them? It applies to traditional retrieval-augmented generation (RAG) systems, to public AI search engines (ChatGPT, Perplexity, Gemini, Google AI Overviews), and to private enterprise search.

The field emerged because traditional NLP metrics like BLEU and ROUGE "fall short in capturing the nuances of factual accuracy, context relevance, and support in RAG applications" (Dhanakotti, 2024). RAGAS introduced a reference-free evaluation framework specifically for RAG pipelines, with metrics that decompose performance into retrieval and generation dimensions (Es et al., 2023). TruLens introduced the RAG Triad as a parallel formulation focused on hallucination detection along each edge of the RAG architecture (TruLens, 2024).

Why LLM evaluation for search matters

RAG and AI search systems have several silent failure modes that traditional engineering metrics miss. The retriever can return irrelevant chunks, but the LLM still produces a plausible-sounding answer. The retriever can return the right chunks, but the LLM ignores them and hallucinates. The answer can be perfectly grounded but unhelpful for the user's actual intent. The system can cite a source that doesn't actually contain the cited claim. None of these failures show up in latency dashboards, error logs, or user-satisfaction surveys until trust is already lost.

RAGAS framing puts it directly: "You can't improve what you don't measure. Metrics are the feedback loop that makes iteration possible" (RAGAS, 2024). OpenAI's evals positioning is similar: "If you are building with LLMs, creating high quality evals is one of the most impactful things you can do. Without evals, it can be very difficult and time intensive to understand how different model versions might affect your use case" (OpenAI Evals, 2025). Evaluation is how teams turn upgrades into known-quality changes instead of gambles.

For GEO and AEO practitioners, the stakes are higher than for internal RAG systems because the AI search engine is outside the team's control. Evaluation becomes the lens through which content teams understand which assets are being cited correctly, which are being misquoted, and which are being ignored entirely.

How LLM evaluation for search works

A modern evaluation pipeline has five stages: dataset construction, system execution, metric computation, aggregation, and regression detection. Most teams build the pipeline once and run it on every model upgrade, retriever change, or prompt revision.

flowchart LR
    A["Eval dataset:
queries + golden contexts
+ reference answers"] --> B["Run system:
retriever + reranker
+ LLM"]
    B --> C["Capture artifacts:
retrieved chunks,
generated answer,
citations"]
    C --> D["Compute metrics:
retrieval (NDCG, recall@k)
+ generation (faithfulness,
answer relevance)"]
    D --> E["Aggregate scores
by slice
(query type, domain)"]
    E --> F["Compare to baseline:
regression? promotion?"]
    F -->|Iterate| B

Stage 1: dataset construction

The evaluation dataset is the foundation. It contains representative user queries paired with either reference answers (reference-based eval) or just retrieved-context expectations (reference-free eval). RAGAS supports both modes; the reference-free mode uses an LLM to compare retrieved contexts against the response (RAGAS, 2024). Most production teams maintain a curated dataset of 100-1,000 queries spanning the most important query types, with periodic refreshes.

Stage 2: system execution

The full system runs on each query, capturing the retrieved chunks, the generated answer, and any citations. For comparison runs, the same queries run against a baseline system (e.g., the previous model version or a different retriever).

Stage 3: metric computation

Metrics fall into two families: classical IR metrics computed against the retrieval stage, and LLM-as-a-judge metrics computed against the generation stage. Classical metrics include NDCG (normalized discounted cumulative gain), recall@k, MRR (mean reciprocal rank), and MAP (mean average precision)—the same metrics used to evaluate retrievers on the BEIR benchmark (Thakur et al., 2021). LLM-as-a-judge metrics include faithfulness, answer relevance, context precision, and context recall.

Stage 4: aggregation

Scores are aggregated globally and by slice—query type, domain, difficulty, language. Aggregating only globally hides regressions that affect specific user segments. Slice analysis is what turns a flat "answer quality went down 3%" into "answer quality went down 12% on multi-hop reasoning queries while up 1% elsewhere."

Stage 5: regression detection

The pipeline compares the new run to a baseline (typically the last shipping version) and flags regressions. Mature teams gate deploys on regression checks the same way they gate on unit-test failures.

Core metrics for LLM search evaluation

The canonical metric set, drawn primarily from RAGAS and TruLens, decomposes search-system quality along the retrieval-and-generation seams.

Faithfulness (a.k.a. groundedness)

Faithfulness measures "how factually consistent a response is with the retrieved context" (RAGAS, 2024). The standard computation extracts each claim from the answer, checks each claim against the retrieved context, and returns the fraction supported. DeepEval's faithfulness metric uses LLM-as-a-judge with a self-explaining output (DeepEval, 2025). IBM's documentation describes faithfulness as the "answer quality metric that measures how grounded the model output is in the model context" (IBM, 2025). Low faithfulness is the canonical hallucination signal.

Answer relevance

Answer relevance asks: does the generated answer actually answer the user's question, regardless of whether it is grounded? An answer can be faithful (every claim supported by retrieved context) but still irrelevant (every claim about the wrong topic). Answer relevance and faithfulness must be measured together.

Context precision

Context precision measures whether the retrieved chunks are relevant to the question, with extra weight on top-ranked chunks because LLMs disproportionately attend to early context (DeepEval, 2025). It is the LLM-as-a-judge analog of precision@k from classical IR.

Context recall

Context recall measures whether the retrieved chunks contain all the information needed to answer the question. Low context recall is a retriever problem, not a generator problem.

Citation accuracy / attribution

For AI search engines that emit explicit citations, citation accuracy measures whether each cited source actually supports the claim attributed to it. This is conceptually distinct from faithfulness: a system can be faithful (claims supported by retrieved context) but still cite the wrong source for a given claim.

Classical IR metrics

NDCG@k, recall@k, MRR, and MAP measure the retrieval stage in isolation. BEIR is the canonical zero-shot retrieval benchmark with 18 datasets across QA, fact-checking, biomedical retrieval, and more (Thakur et al., 2021). BEIR remains the standard for comparing dense, sparse, late-interaction, and reranking architectures (Thakur et al., 2021).

Comparison: traditional IR evaluation vs. LLM evaluation for search

Classical information retrieval evaluation focuses entirely on retrieval quality. Did the system rank the relevant document above the irrelevant ones? It assumes a fixed corpus, a fixed query set, and binary or graded relevance judgments. Metrics are deterministic and reproducible.

LLM evaluation for search retains classical IR evaluation as a sub-component (the retrieval stage) but adds the generation stage. Generation evaluation is harder because outputs are open-ended, multiple correct answers exist, and ground truth is often unavailable. The dominant pattern—LLM-as-a-judge—uses a strong LLM as the evaluator, which introduces its own biases: position bias, length bias, self-preference bias when evaluating outputs from the same model family.

Mature teams treat LLM-as-a-judge as a noisy but useful signal: calibrate the judge against human-labeled samples, monitor inter-rater agreement between the judge and humans, and re-calibrate when the judge model changes. Snowflake's eval-guided optimization of LLM judges illustrates the pattern: tune the judge prompt against ground-truth labels before trusting the judge at scale (Snowflake, 2025).

Frameworks for LLM evaluation for search

RAGAS

RAGAS is the most widely cited open-source framework, introduced in 2023 as a reference-free evaluation framework for RAG pipelines (Es et al., 2023). Its canonical metrics—faithfulness, answer relevancy, context precision, context recall—decompose RAG quality along retrieval and generation seams. RAGAS also supports test-data generation, which lets teams bootstrap an evaluation dataset from a corpus.

TruLens

TruLens, originally from TruEra and now part of the Snowflake ecosystem, formalized the RAG Triad: context relevance, groundedness, answer relevance (TruLens, 2024). Each metric is computed by an LLM-as-a-judge prompted to score along that single dimension. TruLens emphasizes tracing—capturing the full execution flow of an agent or RAG—so per-record diagnosis is possible.

OpenAI Evals

OpenAI Evals is OpenAI's open-source framework plus registry of benchmarks for evaluating LLMs and LLM-based systems (OpenAI, 2024). It supports custom evals (write your own task) and model-graded evals (use an LLM as the judge). Evals can now be configured directly in the OpenAI Dashboard (OpenAI Evals, 2025), reducing the friction for teams that don't want to maintain a custom evaluation harness.

DeepEval

DeepEval (Confident AI) bundles five RAG metrics (contextual relevancy, contextual precision, contextual recall, answer relevancy, faithfulness) plus six agentic metrics (task completion, argument correctness, tool correctness, step efficiency, plan adherence, plan quality) and the more general G-Eval and DAG metrics (Atlan, 2026). It is the closest open-source framework to a one-stop suite for both RAG and agent eval.

BEIR

BEIR is not an evaluation framework but a benchmark. It provides 18 datasets, a standard evaluation protocol, and a reference leaderboard for zero-shot retrieval (Thakur et al., 2021). Use BEIR to evaluate retrievers in isolation; use RAGAS/TruLens/DeepEval to evaluate the full RAG pipeline.

Practical applications

1. Pre-deploy regression gating

Run the eval suite in CI on every retriever, reranker, prompt, or model change. Fail the build on faithfulness drops above a threshold (e.g., 2 points) on any tracked slice. This is the single highest-leverage application because it prevents silent regressions from shipping.

2. Model-upgrade comparison

When a new foundation model is released, run the eval suite on the new model with the existing retriever and prompt. Compare faithfulness, answer relevance, and latency. Decide whether the upgrade is a net positive on your task.

3. Retriever selection

Use BEIR-style retrieval evaluation (NDCG@10, recall@k) on a representative subset of your domain to choose between dense, sparse, hybrid, or late-interaction retrievers. Then run the full RAG eval on the top candidates.

4. Prompt iteration

Most prompt changes feel like improvements until measured. Use answer relevance and faithfulness scores to pick between candidate prompts.

5. Citation-quality monitoring for GEO

For content teams optimizing for AI search citation, run an eval that samples target queries across ChatGPT, Perplexity, Gemini, and Google AI Overviews and scores citation accuracy and brand-mention share over time. This is the GEO equivalent of a rank-tracking dashboard.

Common mistakes

Evaluating only globally. Aggregating across all queries hides slice-level regressions. Always cut by query type, domain, and difficulty.

Trusting LLM-as-a-judge without calibration. LLM judges have position bias, length bias, and self-preference bias. Calibrate against human-labeled samples before trusting the score at scale (Snowflake, 2025).

Optimizing one metric in isolation. A system can improve faithfulness while degrading answer relevance—"the LLM is answering the question correctly but ignoring the provided context" (Kinde, 2025). Always measure faithfulness, answer relevance, and recall together.

Stale eval datasets. Production query distributions drift. Refresh the eval dataset quarterly and verify it still reflects what users actually ask.

Skipping classical IR metrics. LLM-as-a-judge metrics describe end-to-end quality but obscure where the failure is. Keep NDCG@k and recall@k on the retriever so retrieval regressions are visible immediately.

Evaluating only on success cases. Include adversarial queries, ambiguous queries, and queries that should not be answered. A system that gracefully refuses to answer when context is insufficient is more trustworthy than one that hallucinates confidently.

How LLM evaluation for search relates to GEO and AEO

Generative Engine Optimization (GEO) and Answer Engine Optimization (AEO) practitioners optimize content for AI engines they don't control. Evaluation is the bridge: it turns "is our content getting cited?" from a vibes question into a measurable one. The same metric vocabulary applies—faithfulness, citation accuracy, answer relevance—but the unit of analysis shifts from a single RAG system to a portfolio of public AI engines querying public content. Teams that run weekly evaluation jobs against a tracked query set across ChatGPT, Perplexity, Gemini, and AI Overviews can detect content-quality regressions, model-upgrade impacts, and competitor citation gains the same way RAG teams detect retriever regressions.

FAQ

Q: What's the difference between faithfulness and groundedness?

They are largely synonymous. RAGAS uses "faithfulness" (RAGAS, 2024); TruLens uses "groundedness" as part of the RAG Triad (TruLens, 2024). Both measure whether the answer's claims are supported by the retrieved context. Some frameworks use "groundedness" for the general property and "faithfulness" specifically for the RAG-pipeline metric, but the underlying computation is the same.

Q: Is LLM-as-a-judge reliable enough to trust?

With calibration, yes; without calibration, no. Calibrate by hand-labeling a sample (e.g., 100 query-answer pairs), running the LLM judge on the same sample, and measuring agreement. If agreement is high (>80%), the judge is usable as a continuous signal. If it is low, tune the judge prompt or pick a stronger judge model. Snowflake's eval-guided optimization is the canonical pattern (Snowflake, 2025).

Q: When should I use BEIR vs. RAGAS?

BEIR for retrievers, RAGAS for full RAG pipelines. BEIR's 18 datasets are designed for zero-shot retrieval evaluation with NDCG@10 (Thakur et al., 2021); it tells you whether your retriever generalizes. RAGAS evaluates the end-to-end pipeline including the generation stage. Most teams use both: BEIR for retriever selection, RAGAS for production monitoring.

Q: Do I need a reference (golden) answer for every query?

No. Reference-free metrics (RAGAS faithfulness, TruLens groundedness) work without golden answers because they compare the answer against the retrieved context, not against a fixed reference. Reference-based metrics (factual correctness, semantic similarity) are stronger when you can afford the labeling cost.

Q: How big should my eval dataset be?

100-1,000 queries is a reasonable starting range. Below 100 and slice-level signals get noisy; above 1,000 and per-run cost (especially with LLM-as-a-judge) becomes painful. Grow the dataset by capturing real production failures and adding them to the set.

Q: How do I evaluate AI search engines I don't control (ChatGPT, Perplexity, Gemini)?

Maintain a tracked query set, query each engine through its public surface, capture the answers and citations, and run the same faithfulness and citation-accuracy metrics. The retriever metrics (NDCG, recall@k) are not directly applicable because you don't see the underlying retrieval, but answer-level metrics work fine. This is how GEO citation tracking is operationalized.

Q: How often should I run the eval suite?

On every change (CI), nightly (drift detection), and weekly (slice analysis). Continuous evaluation catches silent regressions; periodic deep dives catch slow drift.

Q: What's a passing faithfulness score?

There is no universal threshold. A reasonable starting target is faithfulness ≥ 0.9 on production queries, with regression alerts on drops greater than 0.02 between runs. The exact threshold depends on the task: a customer-support assistant tolerates lower faithfulness than a medical or legal RAG system.