Up to 70% off kits30% off services
LLM Evaluation Template
Build a golden test set for an AI feature before it reaches users. Write real inputs with the behavior you expect, mark the cases that must never fail, then score two runs side by side, such as the current prompt against a new one. A run is ready when every case is marked, the score meets your bar and every critical case passes.
Run each input through your feature, read the answer, and mark it against the expected behavior. Changing the preset replaces the cases below.
- Case 1
- Case 2
- Case 3Critical
- Case 4Critical
- Case 5Critical
- Case 6Critical
- Case 7
- Case 8Critical
- Case 9Critical
- Case 10Critical
10 cases · 7 critical. 30 to 50 cases from real user questions is a good first set.
Current prompt
Not marked
0 of 10 marked · critical 0 of 7 passed
Not ready yet
- 10 of 10 cases not marked yet.
- 7 critical cases not passed. Critical cases must all pass.
New prompt
Not marked
0 of 10 marked · critical 0 of 7 passed
Not ready yet
- 10 of 10 cases not marked yet.
- 7 critical cases not passed. Critical cases must all pass.
Everything stays in this tab. Nothing is sent to us.
Want the test set built and run for you?30% OFF
Every AI feature we build ships with a test set of real cases and a pass bar agreed in writing. It starts in our $349$499 AI Feature Discovery, which tests your best idea on 30 real examples and comes off the build.
A simple LLM evaluation framework for product teams
You don't need a research lab to test an AI feature. You need a fixed set of real cases, a clear idea of the right behavior for each, and the habit of running it before every change.
- Collect real inputs. Pull 30 to 50 questions or requests from support tickets, search logs or user interviews. Real wording beats made-up examples.
- Write the expected behavior, not the exact words. "Gives the steps and links the article" is testable. A word-for-word answer is not, because models phrase things differently each time.
- Cover the four outcomes. Cases it should answer, refuse, hand to a person, and turn into an action after the user confirms.
- Mark the critical ones. Pricing, refunds, personal data, safety and anything your content can't answer. These must pass every time.
- Compare runs, then keep growing the set. Score the current and new version side by side. Every real mistake a user reports becomes a new case, so it can't come back.
Not sure which feature to test first? Rank your ideas with the AI use case scorecard, and check what it would cost per user with the AI app cost calculator.
Frequently asked questions
Golden datasets, what to test, metrics, scoring and privacy.
A golden dataset, or test set, is a fixed list of real inputs for an AI feature, each with the answer or behavior you expect. You run every version of the feature on it and compare scores, so you know whether a prompt or model change made things better or worse before users find out. Start with 30 to 50 cases from real user questions, and add every real mistake as a new case.
The common questions your users actually ask, the awkward ones (typos, vague requests, two questions in one), the ones it should refuse or hand to a person, anything involving money, personal data or safety, and questions your content cannot answer, where the right behavior is to say so. Mark the ones that must never fail as critical.
For a product feature, start with the simple ones: did it give the right answer or take the right action, did it refuse or hand over when it should, and did every critical case pass. Then add the ones your users feel: is the answer grounded in your content with the source shown, how long it takes, and what it costs per use. Automated scores like similarity or a second model acting as judge help at scale, but check them against a person's marks first.
A pass counts 1, a partial pass counts 0.5 and a fail counts 0, averaged over the cases you have marked. A run is ready to go live only when every case is marked, the score meets your bar and every critical case is a full pass. Change the bar to suit how costly a mistake is for your feature.
Comparing two versions on the same cases: the current prompt against a new one, a small model against a larger one, or before and after adding retrieval. If run B scores higher and passes every critical case, ship it. If it scores higher but breaks a critical case, it does not ship.
No. Everything stays in this browser tab and nothing is sent to us. Download the CSV to keep it, then import it into a spreadsheet or your own test runner.