Studio
Black Friday

Up to 70% off kits30% off services

Ends in
All Tools
AI product · Free template

LLM Evaluation Template

Build a golden test set for an AI feature before it reaches users. Write real inputs with the behavior you expect, mark the cases that must never fail, then score two runs side by side, such as the current prompt against a new one. A run is ready when every case is marked, the score meets your bar and every critical case passes.

Run each input through your feature, read the answer, and mark it against the expected behavior. Changing the preset replaces the cases below.

  1. Case 1
    Current prompt
    New prompt
  2. Case 2
    Current prompt
    New prompt
  3. Case 3Critical
    Current prompt
    New prompt
  4. Case 4Critical
    Current prompt
    New prompt
  5. Case 5Critical
    Current prompt
    New prompt
  6. Case 6Critical
    Current prompt
    New prompt
  7. Case 7
    Current prompt
    New prompt
  8. Case 8Critical
    Current prompt
    New prompt
  9. Case 9Critical
    Current prompt
    New prompt
  10. Case 10Critical
    Current prompt
    New prompt
% overall, and every critical case passes

10 cases · 7 critical. 30 to 50 cases from real user questions is a good first set.

Current prompt

Not marked

0 of 10 marked · critical 0 of 7 passed

Not ready yet

  • 10 of 10 cases not marked yet.
  • 7 critical cases not passed. Critical cases must all pass.

New prompt

Not marked

0 of 10 marked · critical 0 of 7 passed

Not ready yet

  • 10 of 10 cases not marked yet.
  • 7 critical cases not passed. Critical cases must all pass.

Everything stays in this tab. Nothing is sent to us.

Want the test set built and run for you?30% OFF

Every AI feature we build ships with a test set of real cases and a pass bar agreed in writing. It starts in our $349$499 AI Feature Discovery, which tests your best idea on 30 real examples and comes off the build.

A simple LLM evaluation framework for product teams

You don't need a research lab to test an AI feature. You need a fixed set of real cases, a clear idea of the right behavior for each, and the habit of running it before every change.

  1. Collect real inputs. Pull 30 to 50 questions or requests from support tickets, search logs or user interviews. Real wording beats made-up examples.
  2. Write the expected behavior, not the exact words. "Gives the steps and links the article" is testable. A word-for-word answer is not, because models phrase things differently each time.
  3. Cover the four outcomes. Cases it should answer, refuse, hand to a person, and turn into an action after the user confirms.
  4. Mark the critical ones. Pricing, refunds, personal data, safety and anything your content can't answer. These must pass every time.
  5. Compare runs, then keep growing the set. Score the current and new version side by side. Every real mistake a user reports becomes a new case, so it can't come back.

Not sure which feature to test first? Rank your ideas with the AI use case scorecard, and check what it would cost per user with the AI app cost calculator.

Frequently asked questions

Golden datasets, what to test, metrics, scoring and privacy.