Every change you make to an agent, a new instruction, a rewritten personality, a knowledge base update, can shift how it responds. The Testing tab is how you catch that before a real caller does: you script the conversations you care about, run them against the agent, and get a score back instead of guessing.
What is a test suite?
A test suite is the set of test scenarios you’ve built for that agent. Open the Testing tab on an agent that doesn’t have one yet and you’ll see No test suite yet with a Generate test suite button. Select it and Speak AI reads the agent’s own instructions and knowledge base and writes you a starting set of scenarios with an LLM call, so you aren’t building one from scratch.
Scenarios and criteria
Each scenario is one scripted mini-conversation, made of:
- A Name and a Category: Greeting, KB retrieval, Off topic, Edge case, or Custom, so you can see at a glance what kind of situation it covers.
- User messages, one per line, sent to the agent in order.
- Criteria, the checks the run scores that scenario against.
A criterion is one of four types. LLM judged asks an LLM judge to evaluate an evaluation prompt you write, such as “Did the agent answer the user’s question accurately?” Response length fails if any of the agent’s replies go over a maximum word count you set. Regex match fails if none of the replies match your pattern, or, with Must match turned off, if any of them do. Tool called fails if the agent never calls the tool name you specify. Every criterion also has a Weight from 1 to 10, for how much it counts toward the scenario’s score, and a Critical flag you can turn on for the ones the agent has to get right.
You can add scenarios and criteria by hand with + Add scenario and Add criterion, edit the generated ones, or select Regenerate suite to have Speak AI write a fresh set. Regenerating replaces the whole scenario list, so anything you’ve hand-edited goes with it. Select Save configuration to keep your changes, along with a Budget per run ($) cap and two switches, Auto-run on KB update and Auto-run on instruction save, so the suite can re-run itself the moment you touch the knowledge base or the agent’s instructions instead of waiting for you to remember to check.
Run a test
Once the suite has at least one enabled scenario, select Run tests. Only one run can be active on an agent at a time, so wait for an existing run to finish, or cancel it, before starting another.
While a run is in progress, a status bar shows Queued, then Running scenario by scenario, then Completed, along with how much of your budget it has spent so far and how long it has been running. If you’re an Owner or Admin, you also get Pause, Resume, and Cancel controls. A cancelled run can’t be resumed. A run can also end as Failed, if a step couldn’t finish, or Budget exceeded, if it would have spent more than the cap you set.
If the agent runs on a Live (speech to speech) model, a notice on this tab tells you tests run on that model’s text side and live audio isn’t simulated.
Read the results
Once a run completes, Test results shows its overall score as a percentage, how many scenarios passed and failed, what the run cost, and what triggered it, manual, a knowledge base update, an instruction save, or a schedule. If the score fell below the agent’s best previous run, a regression notice tells you and points at the scenarios that failed.
Select a scenario to expand it and see each of its criteria on its own, marked passed or failed, with the judge’s reasoning underneath where there is one. That’s where you actually debug a failing scenario: the top-level score tells you something regressed, the expanded criteria tell you what.
Run history and score trend
Run history, below the current results, lists every past run for the agent with its date, trigger, score, change from your baseline, cost, and status. Select a row to open that run’s own results in the same view above, with a banner reminding you it’s an older run and a link back to the latest one.
Score trend charts your completed runs’ scores over time, oldest to newest, with your baseline, the agent’s best completed run so far, drawn in as a reference line. History and trend together are how you tell a one-off bad run from a real regression: a single dip that recovers next run is noise, a trend that keeps sliding after an instruction change is worth a closer look.
Apply a recommendation
A completed run can come back with Recommendations: suggestions grouped by severity, high, medium, or low, and by what they touch, Instructions, Personality, Knowledge base, Topics to avoid, Custom vocabulary, Welcome message, or Response length. Filter by severity to focus on the high ones first.
For a recommendation about the agent’s instructions, personality, or topics to avoid, select Apply fix and it’s appended to that setting right away, no need to go find it in another tab. For one about the knowledge base or vocabulary, you’ll see Copy fix and a note to apply it in the Knowledge / Vocabulary tab instead, so copy the suggested text and paste it in there yourself. Either way, an applied recommendation gets an Applied badge with the date, and you can Dismiss any recommendation you don’t want to act on.
Who can do this
Anyone with access to the agent can open the Testing tab and read its results, history, and trend. Generating or editing the suite, starting, pausing, resuming, or cancelling a run, and applying a recommendation all need the Owner or Admin role on your Speak AI account.
Related pages
- Run scripted tests against a Speak AI voice agent: the API reference for the same suites, runs, and recommendations, useful if you want to trigger a run from your own backend or CI.
- Voice and avatar: how an agent’s model, including a Live model, is chosen on the Details tab and how that changes what this tab can test.