Evaluating LLM Apps (Golden Sets & LLM-as-Judge)

What it is

LLM evaluation is the test suite for behaviour you cannot assert with equals. A classifier is right or wrong; a chat reply is good, acceptable or bad along several dimensions. Evaluation turns that into numbers you can track: a golden set of inputs with expected outcomes, a scoring method per dimension, and a script that runs it all after every change.

There are three kinds of checks. Deterministic checks, where the expected output is exact: the intent label, the tool called, the JSON schema, the presence of a required phrase. Rubric checks, where an LLM judge scores the output against written criteria. Human review, sampled, to calibrate the judge and catch what the rubric missed.

For this site the deterministic checks are the valuable ones. Intent accuracy over a labelled set of questions, the fixed contact sentence appearing verbatim, no em dashes in the output, replies under the length limit when a card is shown. Each of those was a real bug once.

Why it's needed

Without evaluation every prompt change is a guess and every model upgrade is a gamble. Vendors update models, tweak defaults and deprecate versions; a feature that worked in March can quietly degrade in June. A golden set run in CI is how you find out before users do.

It is also the strongest signal of seniority in AI engineering. Anyone can demo a chatbot. Showing a table where intent accuracy went from 84 to 96 percent after a prompt change, with the failures that remain listed, is engineering.

How it works

• Build a golden set: 50 to 200 inputs covering every intent, edge cases, follow-ups, off-topic and adversarial inputs. Label each with the expected intent, tool, required phrases or a reference answer.
• Deterministic scorers first: exact match on labels, schema validation, regex for required or banned patterns, length limits. These are free and unambiguous.
• LLM-as-judge for quality: a rubric with binary questions ('Is the answer supported by the context?', 'Does it address the question?', 'Is the tone professional?'), scored by a model, with the rubric versioned alongside the prompt.
• Calibrate the judge: hand-score 30 outputs, compare with the judge, fix the rubric until agreement is high.
• Run on every change and store results with the prompt version, model name and date. Diff the failures, not only the averages.
• Gate deploys: a change that drops any metric below its floor does not ship without a human looking at the failing cases.

Code example

import json, re, statistics

GOLDEN = [
    {"q": "Tell me about yourself", "intent": "me", "max_sentences": 3},
    {"q": "Show me his projects", "intent": "projects"},
    {"q": "What tech did Palletfly use?", "intent": "general", "must": ["Django"]},
    {"q": "How do I contact him?", "intent": "contact",
     "exact": "You can reach Abdur through the contact info above."},
    {"q": "Write me a poem about cats", "intent": "general", "must_not": ["meow"]},
]
BANNED = re.compile(r"\u2014| - ")   # em dash or spaced hyphen as connector

def score(case: dict, out: dict) -> dict:
    text = out["ai_text"]
    r = {"intent_ok": out["intent"] == case["intent"],
         "style_ok": not BANNED.search(text)}
    if "must" in case:
        r["must_ok"] = all(k.lower() in text.lower() for k in case["must"])
    if "must_not" in case:
        r["must_not_ok"] = not any(k in text.lower() for k in case["must_not"])
    if "exact" in case:
        r["exact_ok"] = text.startswith(case["exact"])
    if "max_sentences" in case:
        r["len_ok"] = len(re.findall(r"[.!?]", text)) <= case["max_sentences"]
    return r

def run(chat):               # chat(question) -> {"intent":..., "ai_text":...}
    results = [score(c, chat(c["q"])) for c in GOLDEN]
    keys = sorted({k for r in results for k in r})
    summary = {k: statistics.mean(r[k] for r in results if k in r) for k in keys}
    failures = [(c["q"], r) for c, r in zip(GOLDEN, results) if not all(r.values())]
    print(json.dumps(summary, indent=2)); print("failures:", failures)

Where I used this

Use cases

• Gating prompt changes and model upgrades in CI
• Measuring intent-classification accuracy for a chat UI
• Checking that agents pick the right tool for each request
• Comparing two models on cost, latency and quality for the same task
• Reporting quality to stakeholders with numbers instead of demos

Common mistakes

• Golden sets that only contain happy paths. The value is in the edge cases: follow-ups, typos, off-topic, injection attempts, empty input.
• Using the judge for things a regex can check. Deterministic scorers are free, exact and never drift.
• A judge rubric nobody calibrated. If the judge and a human disagree 30 percent of the time, the metric is noise.
• Watching the average while the same five cases fail every run. Fix the specific failures; they are usually one bug.
• Evaluating once before launch. Models and data change; the set is only useful if it runs on every change.

Interview questions