aioff.
中文
Product design
PRODUCT DESIGN·9 min

Vapi: test difficult calls before your AI answers customers

Vapi connects speech recognition, model responses, voice output and business tools into voice assistants. Simulated callers help teams check booking and support tasks after changes and inspect dialogue and recordings when something fails.

Break down the problem and try your own design and implementation first. Then explore the reasoning to check states, edge cases, trade-offs, and ways to test your approach. The guide offers a reasoning path, not a complete implementation.

01
CODE EXERCISEAdvanced

How should duplicate, versioned test results determine readiness?

Scenario

Design a pure in-memory release-check summarizer. Given a test plan and results, it decides whether evidence is complete for each combination and whether the process may continue to human release confirmation. It runs no model and calls no deployment API. These rules are invented for the exercise, not Vapi’s built-in algorithm.

Input contract and decisions

  1. plan contains a nonempty string targetVersion and a nonempty cases array of at most 10,000 items. Each has a unique, nonempty caseId and an integer minRuns from 1 to 100. A case is one fixed scenario/personality/mode combination. The plan is already validated and read-only.
  2. results has at most 200,000 items. Each must be a non-null object containing nonempty strings resultId, caseId, version and iterationId, plus outcome equal to pass, fail or unknown. Result objects are untrusted; missing or mistyped fields fail the entire batch. Ignore other fields. Let L be the total length of the listed strings.
  3. Repeated resultId values with identical listed fields are duplicate deliveries, counted once. A differing field is a batch error that locates both entries. Different resultIds sharing (version, caseId, iterationId) also conflict, even if outcomes agree. Check the entire batch, including old versions and unplanned cases, before filtering.
  4. After validation, count only results whose version strictly equals targetVersion and whose caseId belongs to the plan. Count the remaining unique results as ignored. For each case count pass, fail and unknown: any fail means blocked; otherwise any unknown or fewer passes than minRuns means incomplete; otherwise ready. Extra passes cannot cancel an unknown.
  5. Overall status is blocked if any case is blocked; otherwise incomplete if any case is incomplete; otherwise ready. Output every case’s counts and status in plan order, the ignored unique-result count and overall status. A case with no results still appears with zero counts and incomplete status. Never mutate inputs. Errors return no partial summary.

Trace the example

Target V2 has normal requiring two runs and tool-error requiring one. Results are: normal/V2/i1 pass; normal/V2/i2 pass; tool-error/V2/i1 unknown; tool-error/V1/i1 pass; and an identical delivery of the first result under its original resultId. Give the four distinct results different IDs. Calculate each case’s state, ignored and overall status.

Try these changes independently: (a) add tool-error/V2/i2 pass; (b) change the unknown outcome to fail; (c) add a different resultId with normal/V2/i1 and pass; (d) change the target to V3, keeping all results. Distinguish valid but unready evidence from conflicting input.

Deliverables

Describe validation, both identity checks, version filtering, three-state reduction and all-or-nothing output. Explain why averaging pass rates is insufficient. Give at least four invariants and time/space analysis in terms of n results, p planned cases and L. Discuss JS Map or Rust collections if useful, but provide no complete implementation or line-by-line pseudocode.

Explore the reasoning
  1. Validate everything before deciding readiness. Check fields and outcomes while retaining input indices. Index resultId to distinguish duplicates from contradictions. Unknown fields do not affect equality. Filtering old versions first would hide invalid data.
  2. Protect iteration identity separately. Use nested indexes or structured keys for version, caseId and iterationId, avoiding delimiter collisions. Remove exact repeated resultIds before checking whether different IDs occupy the same iteration.
  3. Count only for the current plan. Initialize all cases to zero. Filter validated unique results by version and planned case; count other results as ignored. Preserve unknown counts so that missing verification differs from explicit failure.
  4. Apply the priority rules. The example gives normal 2/0/0 and ready; tool-error 0/0/1 and incomplete; ignored 1; overall incomplete. In (a), unknown remains, so incomplete. In (b), fail makes it blocked. In (c), conflict means no summary. In (d), all four unique results are ignored, both cases have zero results, and overall is incomplete.
  5. Check stability and cost. Reordering or adding identical duplicates preserves the result; old versions cannot improve current readiness; extra passes cannot offset failure; unverified evidence cannot become a pass; errors yield no partial output. Under average constant-time hashing there are O(n+p) collection operations; account separately for string hashing and equality using L. An O(n+p+L) space bound covers indexes. Plan-order output needs no extra sort.