Vapi: test difficult calls before your AI answers customers
Vapi connects speech recognition, model responses, voice output and business tools into voice assistants. Simulated callers help teams check booking and support tasks after changes and inspect dialogue and recordings when something fails.
Form your own approach first, then explore the reasoning to check trade-offs and validation you may have missed. There is no single right answer.
What should staff see before releasing a voice assistant?
Scenario
Design a pre-release checking page for your photography studio’s voice booking assistant. It can look up slots, create appointments and send confirmation messages. Staff need to understand what the tests prove, rather than see only a green pass rate. This is an invented design exercise, not a recreation of Vapi’s interface.
Known conditions
- Use three scenarios: ordinary booking; no availability at the requested time; and an explicit failure from the appointment-creation tool. Test each with patient and impatient callers. Every tool uses a sandbox or a specific mocked response; no real customers are contacted.
- Record assistant version, scenario, personality, text/voice mode, actual tool isolation, dialogue and judgment for each test. Answers may vary under the same configuration, so repeat every combination at least three times.
- “The assistant said it booked” and “the correct booking exists in the sandbox” are separate conditions. A mocked success cannot prove sandbox state. An unchecked condition must say unverified: it is neither a failure nor a pass.
- Staff have changed the prompt, but the latest results belong to the previous version. The page cannot use old results as evidence for releasing the current version. Design the information and confirmation flow, not a production release backend.
Example and deliverables
In one failure-scenario conversation the assistant incorrectly announces a successful booking. The other 17 tests behave correctly. Sketch the overview, exception detail and state after a version change. Write two key success criteria and one counterexample, and explain what staff should inspect next. Design two observation tasks that test whether a high pass rate hides a critical failure; do not invent experiment results.
Explore the reasoning
- Show coverage by task combination. Three scenarios × two personalities × three repetitions means 18 tests. Keep counts, versions and modes visible. More ordinary calls do not cover missing failure cases.
- Expose a critical failure separately. “Do not claim a booking succeeded after its tool failed” can block release. A 17/18 pass rate does not clear that counterexample. Link directly to its dialogue and tool response.
- Distinguish evidence layers. Mocked success checks conversational behavior. Sandbox state needs its own status and evidence. Show unverified when it has not been inspected.
- Invalidate current eligibility when the version changes. Retain old results for comparison, but identify the current version as unchecked. Retest affected combinations and other critical cases after correction; do not merely hide the previous failure.
- Observe decisions. Give staff the 17/18 case and an all-pass case from an old version. Ask whether they would release, what evidence they used and what they would do next. Use observed score-only or version-blind decisions to improve the hierarchy.