Evals test a model against a spec, monitoring watches live traffic, pre-launch QA tests a bot before customers do. Compare all three and pick what you need.
Last updated . For the testing standard behind these comparisons, read the methodology.
The three overlap in technique but not in purpose. Read down a column to see what one approach covers, or across a row to compare them on one point.
| Compared on | LLM evals | Production monitoring | Pre-launch chatbot QA |
|---|---|---|---|
| Question it answers | Does this model or prompt meet the spec on my test set? | What is happening in live customer traffic right now? | Will this bot fail real customers before it launches? |
| When you run it | During development and in CI, on every prompt or model change. | Continuously, after launch. | Before launch, after a major change, and to confirm a fix. |
| Who owns it | Developers and ML engineers. | Engineering, operations or support leads watching live traffic. | Whoever signs off the launch: product, a support lead, a founder or an agency, with engineering for fixes. |
| Inputs | A dataset of prompts or conversations, plus expected outputs or metric thresholds. | Real production conversations or traces. | Scripted customer-pressure scenarios sent to the bot: refunds, privacy, escalation and prompt tricks. |
| Output | Pass or fail and metric scores per test case, and regressions between versions. | Dashboards, alerts, trends and flagged conversations. | A launch report: transcript evidence, severity, fixes and a retest checklist, with deterministic scoring. |
| Typical tools | Open-source eval frameworks such as Promptfoo and DeepEval. | Observability and production-monitoring platforms. Cekura, Hamming and Coval list production monitoring on their own sites. | Pre-launch QA and red-team testing, manual checklists, and Agent Torture Lab's one-time launch report. |
| What it misses | Customer behaviour you did not put in the dataset, and business-rule or handoff failures unless you write them as tests. | Anything customers have not yet hit. It cannot tell you whether a bot is ready before launch. | Everything after launch: drift, new edge cases and traffic volume. It is a point-in-time check, not a monitor. |
Pick by the stage your bot is in and the failure you are most afraid of.
Agent Torture Lab does not monitor production traffic. It does not watch live conversations, send alerts or track drift after launch.
It runs one pre-launch test instead: scripted customer-pressure scenarios against your bot, returned as a one-time report with transcript evidence, severity, recommended fixes and a retest checklist.
One-time unlock includes one free retest: re-run the same target once within your 14-day report window. Automated API and website roasts only — a pasted transcript has no live target to re-run.
If you need continuous coverage after launch, pair the report with a monitoring or observability tool.
Agent Torture Lab: Customer conversations and launch-risk scenarios.
Alternative approach: Prompts, model outputs and datasets for evals; live traces for monitoring.
Agent Torture Lab: Builders, founders, support leads, and agency clients.
Alternative approach: Engineering and ML teams already comfortable with eval and observability tooling.
Agent Torture Lab: Plain-language launch report with fixes and retest guidance.
Alternative approach: Scores and regressions for evals; dashboards and alerts for monitoring.
Agent Torture Lab: Built around policies, handoffs, revenue risk, and customer trust.
Alternative approach: Usually requires custom work to map evals or traces to business outcomes.
Is the question about a model against a spec, live traffic, or whether to launch?
Who will read the result: an engineer, an operator, or a client or manager?
Does the bot already have real customers, or is it still pre-launch?
Will the result lead to a launch, fix-first, or no-go decision?
Evals usually run before release against a fixed test set and tell you whether a model or prompt meets a spec. Monitoring runs after launch on real traffic and tells you what customers are doing and whether quality is drifting.
No. Evals usually score a model or prompt against cases your team wrote. Pre-launch QA tests the whole customer-facing bot under adversarial customer pressure and ends in a launch decision. They share techniques but answer different questions.
No. Agent Torture Lab does not monitor production traffic. It runs a one-time pre-launch test, with one free retest inside the 14-day report window for automated API and website roasts. Use a monitoring or observability tool for live traffic.
Not on day one. Teams iterating on prompts start with evals, run pre-launch QA before customers see the bot or after a major change, and add monitoring once there is real traffic. Most teams end up with more than one.
Promptfoo and DeepEval are open-source evaluation frameworks. Cekura, Hamming and Coval list production monitoring on their own sites. The chatbot testing tools comparison has pricing and licences checked on 30 September 2026.
Compare 9 chatbot testing tools, from Promptfoo, DeepEval and Garak to Botium, Cekura, Hamming and Coval, by pricing model, licence, free tier and channels.
Compare Agent Torture Lab with Cekura for testing customer-facing chatbots: setup, report-first output, one-time pricing, and who each tool fits.
Compare Agent Torture Lab with Botium (Cyara) for chatbot testing: open-source Botium Core, Cyara's platform, pricing, and a report-first launch test.
Compare Agent Torture Lab with TestMu AI for testing customer-facing chatbots: scenario breadth, credit pricing, no-code setup, and a one-time launch report.
