How we test a voice agent before it talks to a patient

In a 2025 study in Communications Medicine, six large language models were given 300 physician-validated clinical cases, each carrying one planted false detail, and repeated or built on that detail in 50% to 82% of cases. A prompt written to prevent it lowered the mean rate from 66% to 44%. A team that wants to test a voice agent for a patient line has to test every change on simulated calls before it ships.

That study covered clinical decision support, and a front-desk call is a narrower job: book, reschedule, answer a policy question, route an urgent call to a person. The failure carries over. An agent that offers an opening that does not exist, confirms a booking the practice system never accepted, or answers a question it should have routed is making the same error, a model filling a gap in what it was given.

Health systems will be asking about this at HLTH USA in Las Vegas this month. The Joint Commission and the Coalition for Health AI published joint guidance on responsible AI use in September 2025 that covers local validation and monitoring, and buyers now ask vendors how they validated their agent. Below is how we answer that for 2care's voice agents: what runs before a release, what blocks one, and what watches a version once it is live.

1. CandidateA change to the prompt or tools becomes a candidate version
2. SimulateEach scenario runs three times against the candidate with a simulated caller
3. GradeEvery run is graded, and one critical failure fails the release
4. CompareThe regression gate checks every case the live version passed
5. PromoteThe new prompt is written, read back and saved as a numbered version

A prompt change meets a simulated caller before it meets a patient

Every change to an agent's prompt or tools starts as a candidate. The test harness builds the agent from that candidate text, the version about to be written to the live line, and puts it on a call with a simulated caller. The caller is a separate model with its own message history.

Each scenario gives it a persona, a goal and an expected outcome, such as a returning patient who wants a follow-up with her usual doctor, expected result booked. A scenario can also carry the details a real call would, like the patient's name and upcoming appointments.

The caller's words pass through simulated speech-recognition errors before the agent reads them: a surname mangled, a date of birth heard as words. Names and numbers are where transcription goes wrong most often, and the errors fall unevenly. A 2020 PNAS study of five commercial speech recognition systems found an average word error rate of 0.35 for Black speakers and 0.19 for white speakers. The agent has to read back and confirm what it heard, and the scenarios check that it does.

The agent calls the real tool routes, so availability and booking rules behave as they do in production. The final write to the practice system is mocked, so a test never books, cancels or moves a real appointment. When a scenario needs an existing appointment to cancel or move, the harness seeds one first. Calls run up to 12 turns by default.

Every scenario runs three times because one run is one sample

The agent under test runs at the temperature it uses on live calls, so the same prompt can take a different path through the same conversation. The simulated caller also runs above zero, so the three runs of a scenario are three different conversations. A prompt that passes once has shown that one path works.

Testing at temperature zero would make runs repeatable, and it would test an agent that never takes a patient's call. It also leaves the underlying error in place. The Communications Medicine study ran its six models under default settings, a mitigation prompt and temperature 0, across 5,400 outputs, and reported that "temperature adjustments offer no significant improvement."

The mitigation prompt did more, lowering the rate from 66% to 44%, and the authors concluded that prompt engineering reduces errors but "does not eliminate them."

So each scenario runs three times by default, and the default bar is that all three pass. A team can lower the threshold for a test group, and that choice is recorded on the run. Each run is bounded by turns, tool calls and time, and a call that stalls halfway is graded on what it said before it stopped.

One critical failure blocks a release, whatever the pass rate

Each run is graded against a rubric. Every criterion has a code, a category and a severity: critical, major or minor. A critical criterion covers a failure a person at the practice would want to hear about every time, such as an urgent call that did not reach a human, or a booking confirmed before the practice system accepted it.

A failed critical criterion in any single run fails the whole release, whatever the overall pass rate. Two clean runs cannot outvote one run where the agent kept booking while a caller described chest pain.

For urgent calls, 2care's lanes are fixed: routine requests are resolved on the call, emergencies go to a human in 3 seconds, and clinical questions go to a nurse or provider queue in 4 seconds. The agent routes these calls and gives no clinical advice. The critical criteria check the route, the timing and that nothing clinical was said. The handover itself is set out in what an escalation contains.

Criterion codes are append-only. When the meaning of a check changes, the old code is retired and a new one is added, so a pass rate from March and one from November measure the same thing.

A new version must match the live one on every case the live one passed

Passing the harness certifies a candidate on its own terms. The regression gate asks a second question before anything is promoted: did the candidate do worse on a case the live version passed? It compares every scenario and criterion with the stored results of the test run that justified the version now in production, and it refuses the promotion if any case got worse. The gate reads stored results and runs no model, so it adds no cost and is on by default for every promotion.

Promotion re-reads the live prompt, writes the new one, reads it back to confirm the write took, and saves a numbered version. Each live version can be traced to the test run that approved it, and rolled back to the one before.

This is the order of work that NIST's AI Risk Management Framework sets out in its MEASURE function: "AI systems should be tested before their deployment and regularly while in operation." The framework is voluntary and carries no certification. It is the clearest public statement of what a buyer can ask a vendor to show.

Live calls are graded by an AI judge, with human corrections kept apart

A test set is a sample of the calls a practice gets, so a deployment can also switch on automatic grading of live calls. Each completed call of 20 seconds or more, with enough transcript to judge, is graded once against the clinic's rubric. A separate check measures, from the call's own clock, how long the caller waited between finishing a sentence and hearing a reply. Latency is measured by code and never estimated by a model. When a critical criterion fails, a second pass looks for the root cause.

The judge is a model, and model judges have known biases. In a NeurIPS 2023 study of LLM judges, Zheng and colleagues found strong judges reached "over 80% agreement" with humans and showed "position, verbosity, and self-enhancement biases." So the judge grades the full line the agent delivered, including words a caller talked over, and a human correction is stored beside the judge's verdict and never written over it. That keeps the judge's agreement with people measurable.

Two more parts are opt-in. A post-release watch compares a new version's live pass rate with its predecessor's once it has 20 automatic reviews. A nightly loop drafts a fix for repeated failures, runs it through the harness, and queues it for a person to approve. The one change it makes on its own is rolling back a version that regressed.

Each check, in order:

Check	When it runs	What fails it	What happens next
Simulated call	Every candidate change	The expected outcome is missed	The run is graded as failed
Repeated runs	Three runs per scenario	Any run fails (default bar)	The release is held
Critical criterion	Every run	One critical failure in any run	The release fails outright
Regression gate	Before promotion	Worse than live on a case live passed	Promotion is refused
Promotion check	At promotion	The new prompt does not read back	Promotion stops
Live-call grading (opt-in)	After each eligible call	A criterion fails	Critical failures get a root-cause pass
Post-release watch (opt-in)	From 20 automatic reviews	Pass rate falls below the predecessor's	A person is alerted and can roll back

What a 2care release goes through before your line hears it

Every change to a 2care agent, whether a new provider, a new visit type or a reworded policy, takes the same path. The harness runs the practice's scenarios against the candidate three times each, with simulated callers whose words arrive with transcription errors. Any critical failure stops the release, and the critical set includes the urgent-call lanes: emergencies to a human in 3 seconds, clinical questions to a nurse or provider queue in 4. The regression gate then checks the candidate against every case the live version passed, and promotion confirms the new prompt is what the line is running.

Practices that switch on live grading also get each eligible call graded against their own rubric, with a person approving any fix the nightly loop drafts. The rest of the system, from write-back to languages, is on the platform page, and the agents are on our voice agents page.

Which practices this suits

This way of testing suits groups that change their agent often: adding providers, opening a site, changing a cancellation policy or a visit type. Each change becomes a candidate, and each candidate is tested before it goes live. It suits practices that can describe their hard calls as scenarios, with a persona, a goal and the outcome they expect, because those scenarios become the test set. It also suits organisations that want a named person to approve every prompt change, and a record linking each live version to the test run that justified it.

Frequently asked questions

Does a test run ever write to our practice system?

No booking, cancellation or reschedule is written during a graded test. The agent calls the real tool routes so availability and booking rules behave as in production, and the final write is mocked. When a scenario needs an existing appointment to act on, the harness seeds one first, so the test has a real appointment to cancel or move.

Does the testing cover accents and misheard names?

It covers them as text. The simulated caller's words pass through speech-recognition errors before the agent reads them, so a surname arrives mangled or a date arrives as words, and the scenarios check that the agent reads details back and confirms them. The harness plays no audio, so accent handling as sound is assessed on live calls.

How is the wait between turns checked if the simulation has no audio?

It is checked on live calls, from the call's own timing records. A code check measures how long the caller waited between finishing a sentence and hearing the agent reply, and lists the slowest waits first. The simulation tests words and decisions, and timing is measured where it happens, on the phone line.

Who approves a change to the agent's prompt?

A person. The harness and the regression gate decide whether a candidate may be promoted, and a member of the team makes the promotion. Where a deployment switches on the nightly loop, it drafts fixes and verifies them through the harness, then leaves each one in an approval queue. The one thing it does without approval is roll back a version that regressed.

What happens if a new version does worse after it goes live?

Once the new version has 20 automatic reviews, the post-release watch compares its pass rate with the version it replaced and alerts a person when it drops. If the rubric changed between the two, that is flagged, because a stricter rubric can lower a pass rate by itself. A regressed version can be rolled back.

Your next step

Pick ten calls from last month that went badly at your desk: the caller who needed a person sooner, the booking that landed in the wrong slot, the question nobody could answer. Write each one as a persona, a goal and the outcome you expected. Take the list to the vendors you meet at HLTH or on your next demo, ask each to run every call three times, and ask to see the transcripts and the grades, failures included.

To see our harness run against your own list, book a demo and bring the ten scenarios. We will run them on a candidate agent and walk through every transcript with you.

Sources

• Omar et al., Communications Medicine, Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support, 2 August 2025.
• Koenecke et al., PNAS, Racial disparities in automated speech recognition, 23 March 2020.
• Coalition for Health AI, Joint Commission and CHAI release initial guidance, 17 September 2025.
• National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), January 2023.
• Zheng et al., NeurIPS 2023 Datasets and Benchmarks, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, June 2023.