Google ADK Live Evaluation: Test Voice Agents Before Shipping
On August 24, 2026, Google added native live evaluation to the Agent Development Kit (ADK), giving teams a repeatable way to test voice agents with spoken, multi-turn conversations before those agents reach users. The new loop can drive an agent with synthesized audio, score its replies and tool calls, and preserve transcripts and audio for debugging. That matters because a voice-agent demo proves almost nothing about the next prompt change, model revision, interrupted turn, or missed tool call.
The practical shift is simple: voice QA can now live beside text-agent tests instead of in a separate manual ritual. Google's launch walkthrough shows the same eval cases running in standard text mode or live audio mode. The valuable part is not synthetic speech by itself. It is the ability to turn a fuzzy conversation into a versioned test artifact with explicit pass/fail criteria.
What ADK Added
ADK's live evaluation path combines four pieces that were usually stitched together by hand:
- A live agent under test. The official sample chains three task-mode agents into a graph: greeting, date-of-birth verification, and appointment guidance.
- An eval case. A case can be a fixed conversation or a scenario with a goal and a user persona.
- An audio user simulator. A language model decides the next user turn, while a separate Gemini TTS model renders that turn as speech.
- Evaluation criteria. Teams can grade the whole trajectory, individual replies, and tool use with thresholds and natural-language rubrics.
The audio stream stays open across the sample workflow, while session state and conversation history move from one stage to the next. The official live workflow sample is important evidence here: it includes the agent graph, eval set, test configuration, and exact command used to run the test.
This is more useful than a one-shot transcript comparison. Spoken replies can be valid in many forms, so ADK's rubric-based criteria evaluate intent and behavior across a trajectory. The criteria reference also exposes deterministic tool-trajectory checks when the exact call path matters.
Scenario Tests Beat Golden Transcripts
A fixed transcript is still useful for a known regression. If a previous build disclosed account data before identity verification, script that path exactly and keep it forever.
But a live agent also needs to survive variation. ADK's conversation scenarios describe a user goal and persona, then let the simulator improvise turns. In Google's example, the simulated novice shares only high-level goals and waits for the agent to ask for details. That tests whether the agent can lead the conversation instead of succeeding only when the user follows a perfect script.
The user-simulation documentation makes the boundary clear: test authors define the plan, persona, and session input; the simulator generates the interaction. A maximum-invocation limit keeps dynamic conversations from running indefinitely.
For a production suite, use both styles:
| Test type | Best use | What it catches |
|---|---|---|
| Fixed conversation | Known incident or compliance path | Exact regression in a repeatable sequence |
| Scenario + persona | Natural variation | Weak prompting, poor recovery, missing questions |
| Tool trajectory | High-risk action flow | Missing, extra, or misordered calls |
| End-to-end rubric | Conversation quality | Goal completion, disclosure order, tone, handoffs |
The mistake is treating one score as the product. The suite should answer several narrower questions: Did the agent complete the task? Did it call the right tool? Did it protect restricted information? Did it recover after the user changed direction?
A Minimal Live Evaluation Configuration
Google's example separates reasoning from speech synthesis. The model field drives the simulated user's decisions; audio_model turns those decisions into audio. That separation lets a team vary voice and language without changing the conversation plan.
{
"criteria": {
"rubric_based_multi_turn_trajectory_quality_v1": {
"threshold": 0.7,
"judge_model_options": {
"judge_model": "gemini-3.7-flash"
},
"rubrics": [
{
"rubric_id": "verifies_identity_first",
"rubric_content": {
"text_property": "The agent verifies identity before disclosing appointment details."
}
}
]
}
},
"live_model_config": {
"timeout_seconds": 300
},
"user_simulator_config": {
"type": "llm_audio",
"model": "gemini-3.7-flash",
"max_allowed_invocations": 10,
"audio_model": "gemini-3.1-flash-tts-preview"
}
}
The model identifiers, threshold, timeout, and invocation cap above come directly from the August 24 example. They are a starting point, not universal defaults. Your pass threshold should reflect the consequence of failure, and your rubric should describe observable behavior rather than vague quality.
With the eval extras installed, the official sample runs through this command:
uv pip install -e ".[eval]"
uv run adk eval \
contributing/samples/live/live_workflow \
contributing/samples/live/live_workflow/live_workflow.evalset.json \
--config_file_path contributing/samples/live/live_workflow/test_config.json
The sample requires credentials for both the Live API and Gemini TTS. ADK can also invoke the same pipeline programmatically through AgentEvaluator, which is the path to regression checks in CI rather than occasional manual runs.
Build a Voice-Agent Test Matrix
Do not begin with dozens of happy paths. Start with the failures that would force a rollback.
1. Protect state transitions. If the agent must authenticate before reading an account balance or booking an appointment, encode that order as a rubric and verify the required tool trajectory.
2. Vary the speaker, not just the wording. Change voice and language settings, then add personas that are terse, uncertain, or incomplete. The goal is to expose prompts that depend on one conversational style.
3. Exercise interruptions. The ADK Live toolkit supports low-latency bidirectional interaction and user interruption. Add cases where a user corrects a name, changes a date, or interrupts during confirmation.
4. Separate semantic and operational checks. Use rubrics for meaning and conversation quality. Use exact or ordered tool checks for actions that must happen in a specific sequence.
5. Keep the artifacts. ADK Web reconstructs the live stream into a transcript and retains playable audio per turn. When a test fails, review both. A correct transcript can still hide poor timing or an awkward handoff.
A compact suite covering authorization, interruption, tool failure, and task completion will tell you more than a large collection of polite happy-path calls.
Limitations and Tradeoffs
Native evaluation removes plumbing, not judgment.
First, the simulated user and rubric judge are themselves model-driven. A passing score is evidence of behavior under the configured models and prompts; it is not a mathematical proof. High-risk workflows still need deterministic assertions, sampled human review, and production monitoring.
Second, synthesized TTS is not field audio. The official configuration can vary voices and language codes, but a clean generated waveform does not recreate every microphone, background-noise, packet-loss, or latency condition. Keep device and network testing in the release plan.
Third, natural-language rubrics can be too forgiving. Write them around observable ordering and outcomes. “The agent is helpful” is weak. “The agent confirms name and date of birth before disclosing appointment details” can be audited.
Finally, live evaluation adds model calls and therefore time and cost. Reserve the full audio matrix for release candidates and high-risk changes; run fast text and deterministic tool tests on every commit.
The Bottom Line
Google ADK's live evaluation is valuable because it turns voice-agent quality from a demo opinion into a regression suite. Use scenarios for conversational variation, fixed cases for known incidents, ordered tool checks for risky actions, and rubrics for end-to-end behavior. If a voice agent cannot survive that matrix, it is not ready for a user with a microphone.
