Vapi: test difficult calls before your AI answers customers

Give an application a voice and the ability to act

Vapi is a platform for building voice AI assistants. It connects speech recognition, a language model, speech output and tool calls. Developers can build telephone reception, support or booking assistants: the job is to connect a conversational model to a voice interaction, rather than merely generate a recording.

Consider a photography studio that wants its assistant to answer inquiries, ask for a preferred time and check availability through a booking tool. One successful test call does not establish that the next prompt revision will work. Callers change their minds or speak ambiguously, and the booking service can fail.

Define a result that can be checked

Imagine testing appointment changes for a photography studio. The caller needs a clear new time, and the assistant must not claim success when the booking service fails. Turn these outcomes into success conditions, then configure the caller in Simulations. A scenario describes the goal and known information; Personality describes communication style. Related simulations can form a suite, a reusable test collection.

Prepare a separate test assistant and controlled tool environment, then confirm the version under test. This avoids exposing unverified changes to real callers. The production guide explains that simulations target the latest published assistant version, not an editing draft. Editing a draft and rerunning tests may still test earlier behavior.

Control the inputs and tools before running the conversation

1. Choose what to inspect. The voice-testing guide distinguishes checking an individual tool decision, running a full text conversation and testing an audio conversation. Start with task logic, then test important voice questions in audio. Text success does not establish audio quality.
2. Configure caller behavior. The same rescheduling goal can involve a patient caller, incomplete information or a change of mind. Check whether the assistant can still clarify and complete the task.
3. Control tool results. Use a mock—a preset tool response—to reproduce no availability or a service error. It replaces a specified call, not the entire environment: unmocked tools still execute.
4. Run and inspect evidence. Read turns and condition results, and listen to recordings for audio tasks. Identify where misunderstanding or an incorrect promise occurred.

Simulation is useful for deliberately creating difficult situations, not merely repeating a natural-sounding greeting.

Retest the same tasks, then verify real outcomes

Check changes with a separate test assistant and controlled tools first. Do not publish an unverified version directly to an assistant connected to a real number. The official production guide states that once a number is connected, publishing a version immediately changes what handles live calls. Publish the target version of the test assistant, rerun the saved suite and compare evidence; arrange production publication after those checks pass.

Conditions that are too broad or narrow can mislead, so explain score changes through actual conversations. The quickstart distinguishes an assistant saying a booking exists from an entry actually appearing in the calendar. A mock success establishes behavior under that response only. Real phone lines, noise and external-system results still need checks. Together, simulation and real integration checks establish whether the assistant both communicates correctly and completes the task.

Separate the task from the caller’s manner

The studio needs to know whether rescheduling works, not whether the assistant can repeat a fixed script. Separating the goal from the communication style lets the same task meet different callers: one has all the facts, another needs repeated explanations, and another changes the requested time.

That combination tests a product question: does the task still succeed when the conversation changes? Success criteria should allow more than one reasonable path. Requiring an exact script can reject effective communication.

The user’s goal stays stable; the path to it can vary.

Turn failure into a condition you can deliberately test

A booking service working today does not mean it will work at the next release. Mocking “no availability” or a service error lets a team examine the assistant’s explanation, follow-up questions and possible false claims of success.

Two layers matter: what the tool returns, and how the assistant communicates that result. Control the first to examine the second. “No booking was created, but the assistant said it was booked” becomes a concrete failure rather than a vague impression that AI is unreliable.

Keep reviewable dialogue beside the score

A pass rate can reveal change without explaining its cause. A broad criterion can accept a bad conversation; a narrow one can punish another valid response. Dialogue and recordings let people inspect whether scoring reflects the real task.

The useful pattern connects criteria, observable evidence and correction. Keep the same cases before and after a change to see what the correction fixed and whether it introduced another problem.