Agent Tool Latency Disclosure Specification: Documenting Response Time SLAs for AI Agent Tools

TL;DR

AI agents call tools blind to expected latency. The result is timeouts, retry storms, and wasted tokens. This specification defines a latency_disclosure block — designed to live alongside any tool manifest (MCP, OpenAI function calling, OpenAPI, proprietary) — that declares p50/p95/p99, timeout policy, idempotency, and behavior under degradation. Tool publishers populate it from production telemetry; agents read it before each call to plan deadlines, fan-out, and fallback paths.

1. Why latency disclosure matters

Tool manifests today describe parameter schemas and return shapes in detail, but say almost nothing about timing. The Model Context Protocol specification defines inputSchema, outputSchema, and structured annotations, but has no mandatory field for response time. OpenAI and Anthropic function calling have the same gap. The result is that an agent picking between two equivalent tools — a fast in-memory lookup and a slow downstream API — has no way to know which is which.

This shows up as concrete pain in production:

• OpenAI's Responses API has been observed to time out long-running MCP tool calls beyond about 60 seconds with no agent-side configuration exposed.
• Microsoft Foundry's agent monitoring guidance flags any tool latency above 10 seconds as a signal of throttling, complex tool calls, or network issues. However, the agent only learns this after the call.
• Practitioners routinely report function-calling completions that intermittently take 20 to 40 seconds even with simplified schemas.
• The AgentSLA proposal (arXiv 2025) argues that agent SLAs need machine-readable quality models drawing on ISO/IEC 25010, but stops short of standardizing the manifest-side disclosure format.

Latency disclosure closes this gap. It treats response time as a first-class part of the tool contract, not an emergent property the agent must rediscover at every invocation.

2. Scope and non-goals

In scope:

• Per-invocation response time of a single tool call.
• Server-imposed timeout semantics.
• Streaming time-to-first-byte.
• Degradation modes that change the latency profile.
• Methodology and freshness of the disclosed numbers.

Non-goals:

• Availability or uptime SLAs (covered by a separate availability disclosure spec).
• Output quality SLAs, accuracy bounds, or hallucination rates.
• Cost or token disclosure.
• End-to-end agent latency, which is the sum of model, tool, and orchestration latency.

3. The latency_disclosure block

The specification is a single object. Tool publishers MAY include it inline in a manifest or reference it via URL. All time fields are integers in milliseconds.

latency_disclosure:
version: "1.0"
observed_at: "2026-04-22T00:00:00Z"
measurement_window_days: 7
sample_size: 12483
region: "us-east-1"
p50_ms: 180
p95_ms: 720
p99_ms: 1850
cold_start_ms: 1200
timeout_ms: 30000
idempotent: true
retry_safe: true
streaming: false
time_to_first_byte_ms: null
degradation_modes:
- trigger: "queue_depth>50"
behavior: "queue and serve"
expected_p95_ms: 4500
- trigger: "downstream_dependency_unavailable"
behavior: "fail fast with 503"
expected_p95_ms: 200
methodology_url: "https://example.com/latency-methodology"

3.1 Field reference

• version (string, required) — Semver of the disclosure schema. This document defines 1.0.
• observed_at (ISO-8601 datetime, required) — The end of the measurement window.
• measurement_window_days (integer, required) — Width of the rolling window. RECOMMENDED values: 7 or 30.
• sample_size (integer, required) — Number of invocations the percentiles were computed over. MUST be at least 1000 for p99_ms to be considered binding.
• region (string, optional) — A region or zone identifier when latency varies materially by region.
• p50_ms, p95_ms, p99_ms (integer, required) — Median, 95th, and 99th percentile response time in milliseconds, measured server-side from request receipt to last byte of response.
• cold_start_ms (integer, optional) — Median additional latency on a cold start. Omit if the tool has no cold-start path.
• timeout_ms (integer, required) — The hard server-side cutoff. Calls exceeding this MUST be aborted by the publisher and returned as a typed timeout error.
• idempotent (boolean, required) — Whether repeating the call with identical input produces the same effect.
• retry_safe (boolean, required) — Whether the agent MAY retry on transport failure without semantic risk. May differ from idempotent for read-after-write tools.
• streaming (boolean, required) — Whether the tool returns streaming output.
• time_to_first_byte_ms (integer, conditionally required) — Required when streaming: true. The p95 latency from request to first response byte.
• degradation_modes (array, optional) — Documented states in which the latency profile changes. Each entry has trigger, behavior, and expected_p95_ms.
• methodology_url (URL, optional) — A page describing how the numbers are computed, including filtering rules and exclusions.

3.2 Validation rules

• p50_ms <= p95_ms <= p99_ms <= timeout_ms.
• p99_ms 1.2 SHOULD be less than timeout_ms so the publisher's timeout is not within normal variance.
• Disclosures with observed_at older than 90 days from the agent's current time MUST be treated as missing.
• If streaming: true, both time_to_first_byte_ms and p95_ms MUST be present and time_to_first_byte_ms <= p95_ms.

4. Embedding the block in existing tool surfaces

The block is transport-agnostic. Adopters embed it where the tool itself is described.

4.1 Model Context Protocol

MCP defines annotations on each tool. Add the block as a structured annotation. The MCP server returns it in the tools/list response.

{
"name": "search_docs",
"description": "Search documentation for a query.",
"inputSchema": { "...": "..." },
"annotations": {
"latencyDisclosure": {
"version": "1.0",
"p50_ms": 180,
"p95_ms": 720,
"p99_ms": 1850,
"timeout_ms": 30000,
"idempotent": true,
"retry_safe": true,
"streaming": false,
"observed_at": "2026-04-22T00:00:00Z",
"measurement_window_days": 7,
"sample_size": 12483
}
}
}

This is consistent with MCP's existing annotation pattern and requires no protocol change.

4.2 OpenAI / Anthropic function calling

Function-calling schemas are JSON Schema. Use a vendor extension key.

{
"type": "function",
"function": {
"name": "search_docs",
"parameters": { "...": "..." },
"x-latency-disclosure": {
"version": "1.0",
"p95_ms": 720,
"...": "..."
}
}
}

Vendor extensions are ignored by clients that do not understand them, which keeps the disclosure backward compatible.

4.3 OpenAPI

For HTTP-fronted tools, attach the block under x-latency-disclosure at the operation level. This sits beside any existing performance hints in x- fields.

4.4 HTTP response header

For lightweight cases where a manifest is impractical, the same JSON object MAY be returned in a base64-encoded X-Tool-Latency-Disclosure response header on each call. This permits dynamic updates without re-fetching the manifest.

5. How agents consume the disclosure

A consuming agent SHOULD use the block in four ways.

Routing. When multiple tools satisfy the same intent, prefer the one with lower p95_ms. For latency-sensitive workflows the agent may also exclude tools whose p99_ms exceeds the user-facing budget.

Retry budgeting. Set the per-call deadline at `p99_ms 1.2 rather than the publisher's timeout_ms. Allocate retry budget only when retry_safe: true. Avoid unbounded retries against tools that report idempotent: false.

Parallelization. When fanning out independent tool calls, use p95_ms to estimate worst-case fan-in time. Anthropic and others recommend running independent tools in parallel; the disclosure is what makes that estimate quantitative rather than guessed.

User-facing latency hints. Chat or agent UIs MAY surface the expected wait based on p95_ms to set user expectations.

6. Measurement methodology

Publishers MUST compute the percentiles from production telemetry, not synthetic benchmarks. Recommended methodology:

1. Capture server-side wall-clock from request receipt to last response byte.
2. Use a rolling window of 7 or 30 days, ending at observed_at.
3. Require at least 1,000 samples for the disclosure to be valid; otherwise raise the window or omit p99_ms.
4. Exclude calls aborted at timeout_ms from p95_ms and p99_ms. Disclose the abort rate in the methodology document if non-trivial.
5. Exclude client-side network time. The disclosure is a publisher-side commitment; client transit is the agent's concern.
6. Segment by region when cross-region p95 spread exceeds 50%. Publish one disclosure per region.

Publishers SHOULD link to a stable methodology_url describing these choices, including any exclusions or filtering. The AgentSLA paper's quality-model framing offers a useful template for documenting methodology.

7. Degradation mode taxonomy

The degradation_modes array makes second-order behavior visible. Recommended trigger vocabulary:

• queue_depth>N — Concurrent invocations exceed a threshold.
• downstream_dependency_unavailable — A required upstream service is unhealthy.
• rate_limit — The publisher's rate limiter has engaged.
• cold_start — The serving instance is starting from cold.
• large_payload — Input or output exceeds a documented size threshold.

For each mode, declare the behavior (queue, fail fast, degrade quality, fall back) and the expected_p95_ms while in that mode. This is the field that lets agents reason about retries during a partial outage.

8. Versioning, freshness, and trust

• version follows semver. Breaking changes (renamed or removed fields) MUST bump the major version.
• Publishers SHOULD re-publish disclosures at least every 30 days. The spec defines a hard freshness ceiling of 90 days, after which agents MUST treat the block as missing.
• Disclosures SHOULD be signed or otherwise attested when they appear in security-sensitive routing decisions, though signing is out of scope of this version.
• Agents that observe sustained real-world latency more than 50% above the disclosed p95_ms over a meaningful window SHOULD penalize that tool in routing until a fresher disclosure arrives.

9. Relationship to existing standards

This specification deliberately reuses, rather than replaces, neighboring work.

• ISO/IEC 25010 — Performance efficiency. The block populates the time-behaviour subcharacteristic with concrete metrics.
• OpenAPI vendor extensions. The OpenAPI mechanism for vendor extensions is the natural transport for HTTP tools.
• MCP annotations. MCP already permits arbitrary structured annotations on tools, so no change to the protocol is required.
• AgentSLA quality model. AgentSLA defines higher-level quality characteristics; this spec is the on-the-wire implementation of one of them.
• OpenTelemetry / observability conventions. Observability vendors already capture per-tool latency at the agent side. The disclosure block lets publishers and consumers compare numbers using the same percentile vocabulary.

10. Adoption checklist

For tool publishers:

• Ship a latency_disclosure block on every public tool.
• Compute percentiles from at least 7 days and 1,000 samples of production traffic.
• Set timeout_ms to at least p99_ms 1.2.
• Document degradation modes for queue depth, rate limit, and cold start at minimum.
• Re-publish on a cron at least every 30 days.

For agent and orchestrator authors:

• Read the disclosure on tool discovery, not on every call.
• Use p99_ms 1.2 as the per-call deadline.
• Honor retry_safe and idempotent flags before any retry.
• Treat disclosures older than 90 days as missing.
• Surface p95_ms in user-facing latency hints when relevant.

FAQ

Q: Why disclose p50, p95, and p99 instead of an average?

Averages hide the long tail that breaks agents. p95 captures the experience of one in twenty calls; p99 captures the call that triggers a timeout cascade. Median (p50) is included for human-readable expectations, but routing decisions should use p95 or p99.

Q: Where do these numbers come from?

Production telemetry on the publisher side. Synthetic benchmarks under-represent real conditions like queue depth, cache misses, and noisy neighbors. Publishers SHOULD link a methodology page so agents can audit the choice of window, exclusions, and sample size.

Q: How does this interact with the Model Context Protocol?

The disclosure ships as an MCP annotation on each tool. MCP's annotation mechanism is already structured and forward-compatible, so adoption requires no protocol change.

Q: What if a tool has no meaningful latency variance?

Publish p50_ms, p95_ms, and p99_ms` anyway. For tools whose latency is essentially constant, the three values will cluster together, which is itself a useful signal to agents.

Q: Does this replace an availability or uptime SLA?

No. Availability is a separate disclosure. A tool can be highly available but slow, or fast when up but frequently down. Agents need both signals; they answer different routing questions.

Q: How often should publishers update the block?

At least every 30 days, and immediately after any change to infrastructure that materially shifts the percentiles. Disclosures older than 90 days are treated as missing by the spec.