Assessment Tests & Evaluations: Automated Quality Scoring for Your Agents
Run structured evaluation suites against your agents, score responses against predefined scenarios, and track performance over time to catch regressions before users do.
How do you know if your agent is getting better or worse after a change? If the answer is “we’ll notice if users complain,” there’s a better way.
Assessment Tests give you a systematic, automated way to evaluate your agents — before changes reach production.
How It Works
Define scenarios — A scenario is a predefined conversation: a question or sequence of messages, plus a description of what a correct response looks like. You build a suite of these scenarios covering the behaviors that matter most for your agent.
Run evaluations — Trigger an evaluation run from the Evaluations area. Agentwise executes each scenario against the live agent, collects the responses, and scores them automatically.
Review results — Results are presented clearly: which scenarios passed, which failed, and how scores have changed compared to previous runs. A regression is immediately visible.
Track over time — Every evaluation run is stored. You can compare the current agent against any previous version — after a knowledge update, a system prompt change, or a model upgrade.
Why This Matters
Agent quality is hard to maintain without measurement. A knowledge base update might fix one problem while breaking something else. A model upgrade might improve general capability while degrading specific behaviors your users depend on. Assessment Tests catch this before it affects real conversations.
This is especially important for agents in high-stakes contexts — IT support, compliance, customer service — where a wrong answer isn’t just unhelpful, it’s a problem.
Getting Started
Assessment Tests are available in the Evaluations area of any agent. Start with a small suite covering your most common and most critical scenarios. Even five well-chosen test cases will tell you more than running blind.