How to Test an AI Chatbot: Build a Thai Test Set
A practical way to test an AI chatbot before launch: build a Thai and mixed-language test set, score answers, read failures and keep testing after go-live.
To test an AI chatbot, build a test set of real questions with agreed correct answers, run every question through the chatbot, and grade each reply against a simple scorecard. Do this before launch, after every change, and on a schedule. For a Thai business, the test set must include Thai, mixed Thai-English and badly typed questions, because that is how your customers and staff actually write.
This guide is for operations, customer service and IT leads who need to decide whether a chatbot is ready. It shows how to build the test set, how to score it, and when to stop and fix the content instead of the chatbot.
Why a demo is not a test
A demo shows the questions the vendor or builder chose. Those questions are clean, well formed and usually covered by the documents. Real users ask half-finished questions, use nicknames for products, and ask things nobody planned for.
A test set replaces opinion with evidence. Instead of "it seemed good in the meeting", you can report how many questions it answered correctly and how many it rightly refused. Those figures come from your own run, not from a benchmark.
The same test set also protects you later. When someone updates a document, swaps the model or changes the search settings, you rerun it and see what moved.
How do you build a Thai test set?
Start from real traffic, not imagination. Pull questions from LINE chats, email, call notes, web forms and the questions staff ask each other. Remove names, phone numbers and any personal data before you save them. For the privacy side, see our guide to the Personal Data Protection Act (PDPA) and where your data can live.
Aim for thirty to fifty questions for a first pass. That is enough to expose patterns and small enough for one expert to grade in an afternoon.
Group the questions so you can see where it fails:
- Plain questions. Clear, in scope, answered directly by one document.
- Multi-step questions. The answer needs two passages, such as a rule and its exception.
- Thai variants. The same question in formal Thai, casual Thai, with spelling slips, and with product names typed in English letters.
- Mixed Thai-English. Staff often write in both languages in one sentence.
- Out-of-scope questions. Things the documents do not cover. The right answer is a polite refusal and a route to a person.
- Trap questions. Questions with a false assumption, such as asking about a discount that does not exist.
- Sensitive questions. Anything about health, money, legal position or personal data.
Write the correct answer, and the source document and page, next to each question before you run anything. If you grade after you see the reply, you will be biased toward accepting it.
What does a good test set check?
A good chatbot test set checks five things for every question: whether the answer is correct, whether it is grounded in an approved source, whether it cites that source, whether it refuses when the documents are silent, and whether it replies in the language the user wrote in. Grade each question pass or fail on each check, with a short note on why.
Include at least a few questions the chatbot should refuse, a few with a false assumption, and a few in Thai, English and mixed form. Save the run date, the model, the document version and the settings so you can compare results later. Keep the test set separate from anything used to tune the chatbot, otherwise it will score well on questions it has effectively already seen. Have an expert in the subject, not the builder, do the grading.
How to score the answers
Keep the scorecard small. A long rubric slows graders down and they start guessing. Use a pass, partial or fail grade for each of the five checks, then add a severity tag to any failure.
Severity matters more than the raw pass rate. A wrong opening-hours answer is annoying. A wrong refund rule, medical instruction or legal statement is serious. Decide in advance which topics are high severity, and set a rule that any failure there blocks launch.
Count refusals as a result in their own right. A chatbot that answers every question is not good, it is overconfident. For more on this, read our piece on how to stop an AI assistant making things up.
Read the failures before you tune anything
Sort the failed questions by cause. Most causes fall into four groups.
- The document is wrong or missing. The chatbot repeated an outdated policy or had nothing to quote. Fix the document.
- Search did not find the right passage. The answer exists but was not retrieved, often with Thai spelling variants or product nicknames. Fix the search and chunking.
- The model ignored the passage. The right text was retrieved but the reply drifted from it. Tighten the instructions and require citations.
- The question was ambiguous. A human would ask a follow-up. Let the chatbot do the same.
Fixing documents usually helps more than changing the model. Our explainer on retrieval-augmented generation, or RAG, covers how retrieval and generation fit together.
Test beyond the first run
A single pass proves little. Keep testing in three ways.
First, rerun the full set after every change to documents, prompts or models. Second, add every real failure reported by a user to the set, so it grows from live traffic. Third, sample a handful of live conversations each week and grade them with the same scorecard.
For chatbots on LINE, add channel checks: message length, buttons, rich menus and what happens when a user sends an image or a voice note. Our guide on building a LINE AI assistant that answers from your own content goes through the channel side. For shops, the AI customer service guide for Thai online stores lists the questions that tend to come up most.
When a test set is not enough
A test set tells you how the chatbot behaves on the questions you thought of. It cannot promise it will handle every question.
Do not rely on a chatbot alone when the answer must be exact, such as an account balance, an order status or a legal filing. Fetch those from the system of record. Keep a person in the loop for anything that affects money, health or legal position.
Also do not start testing if nobody owns the documents. If no one can say which policy is current, the test will only measure the confusion in your files.
If a vendor shows you a demo, ask them to run your test set live, with your documents, in front of you. Any UPSPIRE solution demo on this site is a concept built on synthetic data, not a record of a client result, so judge ours the same way.
How to start
- Choose one chatbot use case and one document owner.
- Collect thirty to fifty real questions and remove personal data.
- Write the expected answer and source for each before running.
- Run the chatbot, grade with the five checks, and tag severity.
- Fix the documents first, rerun, and record the result.
- Add a weekly sample of live conversations and grow the set.
If you want help building a Thai test set or reviewing results, our team can work through it with you. Book a consultation.
Frequently asked questions
How many questions should a chatbot test set have?
Start with thirty to fifty real questions. That is enough to show patterns and small enough for one expert to grade in an afternoon. Add every real failure from live use so the set grows over time.
Who should grade the chatbot answers?
A subject expert from your own team, not the person who built the chatbot. Write the correct answer and source before you run the test, so you are not swayed by a fluent reply.
Should the test include questions the chatbot cannot answer?
Yes. A good chatbot declines politely and routes to a person when the documents are silent. Include out-of-scope questions and questions with a false assumption to check this.
How often should we retest?
Rerun the full set after any change to documents, prompts or models, and sample live conversations weekly with the same scorecard. Add new failures to the set as they appear.
Do we need Thai questions in the test set?
Yes. Include formal and casual Thai, spelling slips, mixed Thai-English and product names typed in English letters, because that is how real users write.
Ready to put this to work?
Turn this playbook into a working system
From a first agent to a national platform, the same team plans it, builds it and runs it with you.
