LLM evaluation dataset: Build one from real runs

An LLM evaluation dataset for a shipped application is a frozen set of recorded cases from your own production runs, distinct from a public benchmark. Each row holds an input, an expected behaviour or reference, and metadata. You use it for regression checks and for comparing versions, scoring with deterministic checks, a judge model and human review as three separate signals.

You change a prompt or swap a model, and the agent starts answering differently. A public benchmark or a hand-written list of questions can’t tell you whether the change helped, because neither contains your users’ real traffic.

An LLM evaluation dataset made from your own runs closes that gap. You’ll learn to build an LLM evaluation dataset from recorded runs, pick scoring methods, check agent behaviour, compare two versions and watch a score over time.

Key Takeaways

• A row built from a recorded run carries the exact prompt, completion, tool calls and cost, so the payload is the evidence.
• Averaging a machine score with human feedback hides disagreement. Every score should declare the scale it was measured against.
• Redact before storage, and a dataset built from production runs never held the secret.
• For agents, assert the route as well as the answer: tool calls, loops, cost and latency.

What is an LLM evaluation dataset, and how is it different from a benchmark?

An LLM evaluation dataset tests one application on cases taken from its own traffic, while a benchmark tests a model on a public task its authors defined. You use the dataset for two jobs: rerunning the same rows after a change as a regression check, and running two prompts or models on identical inputs to compare them.

Each row has three parts. The input is what the application received, and for an agent it can be a whole conversation. The expectation is a reference answer or a behaviour, such as “calls the refund tool once”.

The metadata holds the source run id, the date, tags, and the model and prompt version. This is what lets you trace a case back to the run it came from.

Because a row comes from a recorded run, its fields map to span attributes. That’s what lets an LLM evaluation dataset point each row back at the span it came from.

A public benchmark starts from the opposite end, with a task its authors chose. The GSM8K paper introduces “a dataset of 8.5K high quality linguistically diverse grade school math word problems” (hosted as openai/gsm8k).

The MMLU paper describes a test that “covers 57 tasks including elementary mathematics, US history, computer science, law, and more” (hosted as cais/mmlu).

The HumanEval paper describes a new evaluation set “to measure functional correctness for synthesizing programs from docstrings”.

None of those tasks covers your agent, and none of them is an LLM evaluation dataset built for it. GSM8K accuracy can’t tell you whether your support agent still refuses out-of-policy refunds after a prompt edit. A case built from one of that agent’s recorded runs can.

Libraries name the parts differently. The deepeval documentation says: “an evaluation dataset, or just dataset, is a collection of goldens. A golden is a precursor to a test case.” Before running evals, you convert goldens to test cases.

How do you build an LLM evaluation dataset from production traffic?

You build an LLM evaluation dataset from production traffic by taking a complaint to the one run behind it, reading that run, redacting it and freezing it as a case. Any framework that exports OpenTelemetry spans can feed this loop, so you adopt no package and no vendor platform first.

1. Send runs to an endpoint. Point your exporter at the hosted version (mcpd.sh/app) or at a build from source at github.com/loruorg/mcpd, with a project key as described on the OpenTelemetry docs page. A new run appears in the interface after your next agent call.

export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.mcpd.sh
export OTEL_EXPORTER_OTLP_PROTOCOL=http/json

2. Pick runs by signal: errors, user complaints, low scores, and runs with surprising cost or latency. Add a sample of ordinary runs too, because a dataset made only of failures can’t show whether a change broke the cases that already worked.
3. Search by what the model said. mcpd’s home page says: “Search finds a run by what the model said, and every log line links back to the step that wrote it.” Paste the odd phrase from the complaint and open the matching run.
4. Open the run and read the exact prompt, the exact completion and any tool calls. Write down what a correct answer would have been, or the case will have no expected behaviour.
5. Turn on redaction before you build the dataset. mcpd’s privacy docs state: “Redaction runs before the write, because a secret deleted afterwards was still written down.”
6. Add the run to a dataset. The evaluation docs describe datasets built from production runs. Adding a run freezes its exact payload as a case, and you can open the dataset to check that the prompt and completion match what you read.

A refund agent that calls the same lookup tool again and again in one conversation shows the loop end to end. A user writes in to complain, you search for an odd phrase the model produced, and the run shows the repeated calls.

You redact the customer’s details and add the run to a dataset. From then on, every prompt or model version you test against that case can be checked for repeated tool calls, and any version that still loops fails.

Deterministic checks, LLM-as-judge or human review: which scoring method fits?

Deterministic checks fit any rule you can write down, a judge model fits open-ended quality, and human review fits calibrating the judge and settling disagreements. Each method answers a different question about the same run.

A rule-based check returns the same result every time you run it on the same output, and it costs nothing in model calls. It can’t judge tone or relevance, because neither fits in a rule.

A judge model covers what rules can’t, such as whether a summary is faithful or a reply is helpful. Zheng et al., in the MT-bench and Chatbot Arena paper, examined “position, verbosity, and self-enhancement biases, as well as limited reasoning ability”.

The same paper reports that strong judges such as GPT-4 “can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.”

That agreement was measured on the paper’s tasks. A judge can stand in for review on your task only after you’ve checked its verdicts against human labels on your own LLM evaluation dataset.

Keep the two outputs apart. mcpd’s docs put it this way: “A score is what a machine thought. Feedback is what a person thought.”

If a judge averages 0.9 on a case and reviewers give it 0.4, a merged figure reads 0.65 and looks mediocre. The real finding is that the judge and your reviewers disagree, and that case is the one to open.

Every score also declares the scale it was measured against. A result of 0.7 on a 0 to 1 scale differs from 7 on a 10-point scale, so never compare scores without their scales.

Work in this order:

1. Put every rule you can state as a condition (format, containment, budgets, error rules) into a deterministic scorer.
2. Use a judge model for open-ended quality at the volume reviewers can’t read, once it has been checked against human labels.
3. Use people to calibrate the judge and to rule on cases where the signals disagree.

How do you evaluate an agent, not only its final answer?

An agent evaluation dataset checks the route an agent took as well as its answer, because an agent can reach the right answer by a wasteful or unsafe route. A case built from a recorded run holds the tool calls, retrieval spans, tokens and cost, so each property of the route becomes an expectation on data the case already holds.

The runs documentation shows how steps and spans appear in a run. These are the properties to assert:

1. Tool calls: which tool ran, how many times and with which arguments.
2. Retrieval: what was retrieved and whether the answer used it. A judge model or a reviewer reads the retrieval span’s returned documents against the completion.
3. Loops: the no_tool_loop scorer checks a run for a tool called over and over.
4. Cost is priced once at ingest, per step, from a price book. An unknown model has no cost, not a cost of zero, so a missing price never reads as a free run.
5. A judge model or human review covers quality, since no deterministic check can.

A correct answer reached after repeated tool calls passes an answer-only check and fails a budget. That gap is the reason the dataset asserts the route.

Suppose a support agent answers a refund question correctly but calls an order-lookup tool far more often than it should. exact_match on the answer passes. Frozen as a case with a cap on calls to the lookup tool, the same run fails.

The count comes from the recorded tool spans, so nothing depends on parsing the answer text. After a prompt change, the experiment shows which cases now break the call, token or cost budget.

How do you curate an LLM evaluation dataset over time?

You curate an LLM evaluation dataset over time by treating it as a versioned record: add cases on purpose, label each one, freeze every version and retire cases when the product changes.

1. Take cases from failures, surprising runs and ordinary traffic. Failures and surprises cover the edges, and ordinary runs keep the set close to what users do. Without them, a dataset only ever measures the last incident.
2. Tag cases by failure type and read scores per tag. A set dominated by one failure type rewards fixing that failure and hides regressions elsewhere, because the overall score moves while the other tags slip.
3. Give every case an expected behaviour or a reference, since a case without one can be read but not scored. Settle disagreements about the expected answer through a review queue.
4. Record the human verdict as feedback next to the machine score. mcpd’s evaluation docs say it “keeps them as two signals and never averages one into the other,” so the cases where they disagree stay visible.
5. Freeze each version and add new cases as a new version. A frozen case never changes, so two experiments on the same dataset version compare like with like.
6. Link prompt versions through mcpd prompts, where each step links to the prompt version that wrote it. A score change then traces to a prompt change or a dataset change, never an unknown mix of both.
7. Refresh from production and re-check the judge. Add cases from new production failures, and retire cases for features that no longer exist, since they score behaviour nobody ships.
8. Re-check judge scores against human review on a sample. A judge that drifts from your reviewers still produces confident numbers.

How do you keep secrets and personal data out of an evaluation dataset?

You keep secrets and personal data out of an LLM evaluation dataset by redacting every run before it is stored, so the dataset never holds them.

Production runs carry customer messages, email addresses and API keys pasted into prompts. If redaction runs after storage, the secret sits in storage until someone deletes it.

Four modes cover most fields: drop, mask, partial and keyed hash.

• drop removes the field.
• mask replaces the value with a placeholder.
• partial keeps a fragment, such as the last characters.
• keyed hash replaces the value with a token that repeats for the same input, so you can still group cases by customer.

Platform teams also need cases to leave. Retention is set in days, so old cases age out on their own. Erasure requests name a person, so their data can be removed on request.

With a default configuration nothing leaves the machine, which covers update checks, usage reporting and telemetry. The mcpd security documentation covers the controls, and the source build at github.com/loruorg/mcpd runs on your own network.

How do you go from one bad run to a monitored regression check?

You go from one bad run to a monitored regression check by freezing the run as a case, scoring it, comparing two versions in an experiment and putting a monitor on the score.

1. Send runs to mcpd. Use the hosted version at mcpd.sh/app, or build from source at github.com/loruorg/mcpd with make build, run ./target/release/mcpd demo and open http://localhost:4317. The quickstart walks through the first trace.
2. Find the bad run by searching for the words the model said, such as the wrong refund policy a customer quoted. Read the exact prompt, completion and tool calls before you keep it.
3. Add the run to a dataset. The run becomes a case with its recorded input and metadata. Write the expected behaviour in your own words, since a copied bad answer would pass forever.
4. Attach deterministic scorers such as contains or json_valid first, because rule-based checks don’t need a model call. Add a judge model only for open-ended quality, and record human verdicts as feedback in the review queue.
5. Run the dataset against the old and the new prompt version and compare the scores side by side. Change one thing between versions, or you can’t tell which change moved the score.
6. Set a monitor on the case’s mean score, with the threshold taken from the passing version’s result in the experiment. The monitor watches production runs, so a regression that the dataset missed still surfaces.

The loop is complete when the original complaint is a case that fails on the old version and passes on the new one, and a monitor watches the same score on live traffic.

Frequently asked questions

What is the difference between an LLM evaluation dataset and a benchmark?

A benchmark such as GSM8K or MMLU measures a model on a fixed public task, as its authors defined it (see the llm_benchmarks collection). An evaluation dataset measures your application on cases from your real inputs, so it shows whether this version of your prompt, tools and model still behaves correctly for your users.

How many rows does an LLM evaluation dataset need?

No published figure fits every application. Start with the failures you have seen plus a sample of ordinary traffic, then add one case for each new failure so a fixed bug can’t return unnoticed.

Can an LLM judge replace human review?

A judge model shows known biases, so check it against human labels before you rely on it. The MT-bench and Chatbot Arena study names position, verbosity and self-enhancement biases. Keep judge scores and human feedback as separate signals.

Can I build a dataset without storing customer data?

Yes. In mcpd, redaction runs before storage, so redacted fields were never written. A case frozen from those runs holds only the masked or dropped values, and nothing needs deleting afterwards.

Do I need to install anything to start?

You don’t need to install anything. Point any OpenTelemetry exporter at https://api.mcpd.sh (hosted at mcpd.sh/app), or build from source at github.com/loruorg/mcpd.

Next steps: build your first dataset this week

Today: set OTEL_EXPORTER_OTLP_ENDPOINT=https://api.mcpd.sh in your agent’s environment and get runs flowing. Any framework that emits gen_ai.* attributes works, as described in the OpenTelemetry ingestion docs.

This week: freeze your first failing or surprising runs as cases, with redaction on, and attach deterministic scorers such as contains and json_valid. Add a case each time a new failure turns up.

This month:

• Add a judge model and check it against cases a person has reviewed.
• Compare two versions in an experiment.
• Set a monitor on the mean score so a regression shows up before a user reports it.

Start free at mcpd.sh/app and send your first runs. If your data can’t leave the network, build from source at github.com/loruorg/mcpd and point the exporter at your own daemon.

Sources

Training Verifiers to Solve Math Word Problems · openai/gsm8k · Datasets at Hugging Face · Measuring Massive Multitask Language Understanding · cais/mmlu · Datasets at Hugging Face · Datasets | DeepEval - The LLM Evaluation Framework · Privacy · mcpd · Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · Scores and feedback · mcpd. Every other source is linked where the article cites it.