Updated October 13, 2026
The difference between an AI agent and a chatbot is not the model, the tone, or the resolution rate. It is whether something changes in one of your systems after the conversation. A chatbot answers. An agent acts and leaves an auditable record. These twelve questions test for that.
Key takeaways
Only 5 of the 12 questions test capability; the other 7 test whether the vendor can prove the capability in your systems.
Resolution rate, deflection rate, and CSAT fail the test: a chatbot with a good knowledge base can score well on all three.
The single most revealing question is #4: "Show me the record this created in the customer's own system."
A vendor who answers all 12 with a demo but none with a log is selling a chatbot.
Why the label on the box doesn't tell you anything
Every vendor in 2026 calls its product an "AI agent." The term has stopped carrying information. What still carries information is a simple test: could a chatbot with a well-built knowledge base do this too? If yes, it is a knowledge feature. If no, it is an agent feature. Apply the test to every claim in the deck.
The test matters because the two products are sold on the same slides but bought for different reasons. A chatbot lowers the cost of answering. An agent lowers the cost of doing: the return that gets booked, the appointment that gets scheduled, the policy endorsement that gets filed. If your problem is the second kind, a very good chatbot is still the wrong purchase.
Here is how the twelve questions split.
# | Question | What it tests | Chatbot can pass? |
1 | What can it change in our systems, not just read? | Write access | No |
2 | Which of those actions require a human approval step? | Guardrails on money and data | No |
3 | What happens when the action fails halfway? | Transaction integrity | No |
4 | Show me the record this created in our system, not your dashboard | Verifiable outcome | No |
5 | What is the metric you will report, and where is the denominator? | Honest measurement | Partly |
6 | Why did the last handoff happen: the agent couldn't, the policy said no, or the customer asked? | Handoff taxonomy | Partly |
7 | Can we see the diff between version 13 and version 14 of the agent? | Versioning | No |
8 | Can we roll back a change in under a minute? | Rollback | No |
9 | Can we test a change against last month's conversations before it goes live? | Simulation | No |
10 | Which model runs it, and what happens when we switch? | Model dependency | Partly |
11 | Where does the conversation data live, and who can read it? | Deployment and perimeter | Yes |
12 | Who from your side is in our workflow after go-live, and for how long? | Operating model | Yes |
Caption: The right-hand column is the test. Questions 11 and 12 are important, but a chatbot vendor can answer them well, so they don't help you tell the two apart.
Questions 1–4: does anything actually change?
These four are the core. An agent is defined by its side effects: a record created, a status updated, a refund issued. If the vendor can't show you a change that happened outside their own product, you are looking at a chatbot with a confident interface. Ask for the record, not the transcript.
1. What can it change, not just read? Reading an order status is retrieval. Changing a delivery address is an action. Ask for the full list of writes the agent can perform on day one, and which systems they land in.
2. Which actions sit behind a human approval? Anything that moves money or touches personal data should have a designed approval step, not a hope. On CXOS, tools that move money are wrapped so the parameters are bound from verified context and a person approves the call. Ask how the vendor's equivalent works, and ask to see it fire.
3. What happens when the action fails halfway? A refund that is issued but not logged is worse than no refund. Ask what the agent does when the API times out on step two of three.
4. Show me the record in our system. This is the question that ends most demos. A Turkish mobility operator running CXOS has the agent handling 80% of its support volume; the proof of that number is in the operator's own ticketing system, not in the vendor's panel. Whatever vendor you pick, the arbiter should be a system neither of you controls.
Questions 5–6: is the metric honest?
Resolution rate is the industry's headline number, and it is mostly a measure of silence: the customer didn't write again after the last answer. Customers also go quiet when the answer was wrong. Ask instead for the share of conversations that ended with a completed transaction in your system, and ask how the denominator was chosen.
5. What metric, and where is the denominator? "72% resolution" means nothing until you know what the 100% was. Conversations that never needed an action should not be in the denominator of an action metric. Ask the vendor to show you the split.
6. Why did the handoff happen? Handoffs come in three kinds: the agent couldn't do it, a policy rule said a person must, or the customer asked for a human. Only the first kind is something the vendor can improve. If they can't break handoffs into those three buckets, they can't tell you what their ceiling is.
Questions 7–9: can you change it safely?
An agent that touches your systems will be edited weekly: new products, new policies, a new return window before peak season. The question is not whether it can be changed but whether the change can be seen, tested, and undone. This is where most products that demo well fall apart in month three.
7. Show me the diff. A prompt edited in a text box with no history is an incident waiting to happen. Ask to see version 13 next to version 14, with a description of what changed and who changed it.
8. Roll it back. If a change breaks the return flow on a Friday evening, how long until the previous version is live? On CXOS this is a published-version switch; ask the vendor to do it in front of you and time it.
9. Test against real traffic before go-live. Simulation means replaying last month's conversations against the new version and reading the differences before customers do. If the answer is "we'll monitor it closely after launch," the customers are the test.
Questions 10–12: what are you actually depending on?
The last three are less about the agent and more about the arrangement around it. They are the questions your security team and your CFO will ask in month two, so ask them in week one. A chatbot vendor can answer them well too, which is why they come last, not first.
10. Which model, and what happens when we switch? Model quality moves every quarter. If the agent is welded to one provider, every improvement elsewhere is one you can't use. Ask whether switching the engine is a setting or a rewrite, and whether your keys can be used instead of theirs.
11. Where does the data live? Conversations, logs, and documents should sit in a perimeter you define: global cloud, in-region cloud, on-premises, or on your own model keys. If a regulator asks, the answer should be a diagram, not a conversation.
12. Who is in our workflow after go-live? The demo is built by the vendor's best engineer. Ask who is in your systems in week six, and for how long. A forward deployed engineer who stays until the workflow runs is a different purchase from a login and a documentation link.
How to score the answers
Give one point per question answered with evidence you can check in your own systems, and zero for anything answered with a slide. A vendor scoring 10 or more is selling an agent. A vendor scoring under 6 is selling a chatbot, however the product is named, and you should price it as one.
The scoring has a second use. Take the questions where the vendor scored zero and put them into the contract as acceptance criteria. That converts a buying checklist into a delivery obligation, which is the only place a checklist earns its keep.
Related guides
Frequently Asked Questions
What is the difference between an AI agent and a chatbot?
A chatbot retrieves and answers. An AI agent can take an action in another system, such as booking an exchange or updating a policy, and leave a verifiable record of it. The test: could a chatbot with a good knowledge base do this? If yes, it is a chatbot feature.
Is resolution rate a good way to compare vendors?
Not on its own. Resolution rate usually means the customer didn't write again, which also happens when the answer was wrong. Compare the share of conversations that ended with a completed transaction in your own system instead.
Do I need an AI agent, or is a chatbot enough?
If most of your volume is questions with answers, a chatbot with a strong knowledge base is the cheaper, correct buy. If most of it is requests that need something to happen, you need an agent.
What is a human handoff supposed to look like in an AI agent?
A designed step, not a fallback. The conversation and the reason for the handoff should arrive with the person, and the reason should be one of three kinds: the agent couldn't, a policy required a human, or the customer asked.
Why does versioning matter for an AI customer service agent?
Because the agent will be edited constantly. Without a diff, a test run, and a rollback, every edit is a live experiment on your customers.
Can I run an AI agent on-premises?
Some platforms, including CXOS, run in four modes: global cloud, in-region cloud, on-premises, or on your own model keys. Ask any vendor for the diagram, not the promise.
