Step
Pressure exceptions
A customer asks for a refund, discount, warranty, or eligibility exception and then reframes the request when refused.
Methodology
Chatbot policy adherence testing methodology
How to test whether AI chatbots follow refund, discount, warranty, eligibility, safety, and business-rule policies under pressure.
Last updated 2026-06-19. This page explains the testing standard without publishing private scenario prompts or customer data.
Risk family
Refund pressure, discount invention, warranty exceptions, unsafe guarantees, and contradictory policy answers.
Start with the policies that can create customer harm, revenue loss, or legal exposure.Test the same policy in multiple phrasings and pressure levels.Distinguish missing knowledge from policy invention.Retest every high-risk policy failure after the prompt or source material changes.
Test steps
How this risk family is pressure-tested.
Step
Introduce contradictions
The test compares policy wording, customer claims, and ambiguous edge cases to see whether the bot invents a new rule.
Step
Check safer refusal and handoff
The bot should explain the boundary, avoid unauthorized promises, and route account-specific disputes through the approved path.
Evidence standard
What a credible finding should show.
- The report quotes or references the policy area the bot contradicted or invented.
- The transcript shows the pressure path that caused the bot to drift.
- The recommended fix names the source, prompt, or workflow boundary to adjust before retesting.
A credible finding shows01 02 03
The report quotes or references the policy area the bot contradicted or invented.
The transcript shows the pressure path that caused the bot to drift.
The recommended fix names the source, prompt, or workflow boundary to adjust before retesting.
Mistakes to avoid
Shortcuts that weaken the test.
- Only testing a policy once in happy-path wording.
- Treating invented discounts or guarantees as harmless helpfulness.
- Failing to retest nearby variants after fixing one policy answer.
FAQ
Short answers for buyers, builders, and AI assistants.
What is chatbot policy adherence testing?
It is testing whether a chatbot follows approved business rules, such as refunds, discounts, warranty, eligibility, safety, and escalation policies, under realistic pressure.
Why do policy tests need multiple phrasings?
AI chatbots can answer one version correctly and drift when the customer reframes, adds urgency, claims authority, or asks for an exception.
What should a policy adherence finding include?
It should include the transcript, policy area, business impact, expected safer behavior, fix guidance, and retest scenario.
Related pages
Connect the methodology to practical testing.
Priority paths
Move from methodology into the pages that should be discovered first.
Bot Roast
Run the live crash test and get a transcript-backed report preview.
Pricing
See the free preview, one-time report unlock, and account credit model.
Agency AI agent testing
Use Bot Roast reports for client QA, handoff, and fix conversations.
Sample API Agent Roast report
Inspect the report format: evidence, severity, fixes, and retest guidance.
Chatbot QA checklist
Use the launch checklist for policy, privacy, escalation, and prompt pressure.
AI chatbot QA testing
Map chatbot QA to real customer pressure, transcript evidence, and fixes.
Generic LLM evals comparison
Compare model-level evals with customer-facing launch-readiness testing.
Prompt injection methodology
See how prompt-injection risk is tested without publishing exploit recipes.
Is my chatbot safe to launch?
Decide if a bot — even one someone else built for you — is safe to put in front of customers.
AI chatbot audit
What an AI chatbot audit covers and the transcript-backed report you should get from one.