Skip to content
mcpd
no daemon
Source

Scores and feedback

A score is what a machine thought. Feedback is what a person thought. mcpd keeps them as two signals and never averages one into the other.

Two signals, not one number

A score is a machine evaluation: a judge model, a regular expression, a unit test, anything that can produce a number about a run. A feedback record is a person's judgement: a thumbs-down in your product, a reviewer's rating, a corrected answer.

Both attach to a trace_id or a span_id, both arrive as signals in their own right, and both are stored in their own table. They are never combined. A judge scoring 0.9 for relevance and a customer pressing thumbs-down are both true statements, and the disagreement between them is the most useful thing on the run. An average would erase exactly that, and leave you with a number nobody can act on.

  • Spans

    What ran, nested, with timings

  • Logs

    Lines the run emitted

  • Metrics

    Numbers, with exemplars back to a run

  • Scores

    Machine evaluation

  • Feedback

    What a person thought

  • Aspect notes

    A sentence a model wrote about the run

trace_id Store Search Export

A score declares its scale

min and max are part of the score. Without them a value is not interpretable: 0.86 could be a good result on a 0 to 1 scale, a poor one on a 0 to 5 scale, or a distance where lower is better. Six months later, with two scorers in the same project, nobody can tell which. The spec defaults them to 0 and 1, and every SDK requires them at the call site so that a default is never accepted by accident.

FieldWhat it is for
scorer_id Which scorer produced this, such as answer_relevance. Required.
scorer_version A score compares only to another score from the same version. Change the prompt of a judge and you have a new scorer, not a continuation of the old one.
value The number.
min, max The declared scale the number sits on.
label A categorical result, when the scorer produces one alongside the number.
reason Why. For a judge model this is its explanation, and it is what makes a score arguable instead of magic.
source Live, offline, experiment or manual.
score_trace_id The trace of the scoring run itself, so a suspicious score can be opened and read like any other run.
cost What producing the score cost.

Recording one

The SDK helper queues a score the same way it queues a span, so the call does not block and cannot throw. With no traceId given it attaches to the active span's trace.

score and feedback 

// A machine evaluation of the run. `min` and `max` are required: 0.86 on an
// undeclared scale cannot be compared to anything.
mcpd.score({
  traceId,
  scorerId: 'answer_relevance',
  value: 0.86,
  min: 0,
  max: 1,
  reason: 'Answered the question and gave the delivery date.',
})

// A human evaluation. Kept separate from a score, and never averaged with
// one, because the two answer different questions.
mcpd.feedback({
  traceId,
  kind: 'thumbs',
  value: 'up',
  comment: 'Exactly what the customer asked for.',
  authorKind: 'end_user',
})

// Anything you already count can travel with the trace.
mcpd.metric('support.tickets_handled', 1, 'counter', { labels: { queue: 'tier-1' } })

Over HTTP that is POST /v1/api/scores, taking a Score as the body. A score must name the trace_id or the span_id it is about; a score attached to nothing is refused with 400 rather than stored where it can never be found. Send score_id as an empty string and the server generates one.

Where a score came from

source is separate from scorer_id because the same scorer runs in more than one context, and the contexts are not comparable. Re-scoring last week's traffic with today's judge is not evidence that quality improved.

SourceMeaning
liveProduced online, as the traced run happened.
offlineProduced later, by scoring stored traces.
experimentProduced by an experiment run against a dataset.
manualEntered by a person, through the API or the interface.

A score written through an SDK defaults to live. Set it yourself for the other three.

Scoring costs money, so it is recorded

A judged evaluation is usually another model call. Scoring every production run with a large judge can cost more than serving the traffic did, and that is a bill people discover late. A Score therefore carries a cost in the same shape a span does, with a currency, an input and output split, and the price book that produced the figure. Set it when you write the score through the HTTP API or a native ingest batch.

score_trace_id is the other half of the same idea. Trace the scoring run itself and the id goes on the score, so a judge that started marking everything down can be opened, read, and shown to be broken.

Human feedback

Feedback is written from your own product. Call it from the control the person actually pressed and the judgement lands on the exact trace that produced the answer they were reacting to. kind decides the shape of value.

Kindvalue
thumbsup or down.
ratingA number, on a scale the producer declares.
correctionThe right answer, as text.
tagA label chosen from a set.
commentFree prose, with no score attached.

a thumbs-down from your product 

// No traceId: it attaches to the active span's trace.
mcpd.feedback({
  kind: 'thumbs',
  value: 'down',
  comment: 'Named the wrong depot.',
  authorKind: 'end_user',
})

Redaction matches on key names, and it runs over the metadata you attach to a judgement. author_id is stored as you send it, so send a pseudonymous id rather than an email address. See Redaction and retention.

Scoring what the run did, not what it said

Most scorers read a run's answer. An agent that finishes with I ran all the tests and there were no regressions satisfies every one of them, and satisfies a judge model too, because the sentence is well formed and says the right thing. The sentence is the agent's account of its own work, and an account is not evidence. If the test command never ran, no scorer that reads the answer can tell you so.

Two built-in scorers read the trace instead. They ignore the answer entirely and ask what the run recorded.

ScorerScores 1 whenConfigured with
span_present The run recorded at least one span of the required kind or name. span_kind, span_name, or both together. Set both and both must hold.
tool_succeeded A tool span with that name ran, and at least one call of it ended ok. tool_name.

Both need configuration, so neither runs in the default set. A scorer that passed because nobody told it what to look for would report every run as having done work no one checked, which is the failure it exists to catch. Name them, and give them their settings: 

{
  "trace_ids": ["4bf92f3577b34da6a3ce929d0e0e4736"],
  "scorers": ["tool_succeeded", "span_present"],
  "config": {
    "tool_name": "run_tests",
    "span_kind": "tool_call",
    "span_name": "run_tests"
  }
}

span_name and tool_name are compared exactly, and case_sensitive does not loosen them. A span name is an identifier your code chose rather than prose, and a loose comparison would let run_tests_dry satisfy an assertion about run_tests, on precisely the run the check was written for. tool_succeeded accepts any of the four tool span kinds, tool_call, client_tool_call, provider_tool_call and mcp_tool_call, because which one a run records says where the tool lives, not whether it ran.

Two ways to score zero, and they are different bugs

A tool that never ran and a tool that ran and failed are both a zero, and fixing them are opposite jobs: one is a step the agent skipped, the other a step that broke. The reason separates them, and carries the recorded error so the score is actionable without opening the run.

What happenedreason
The tool never ran. "run_tests" never ran; the run records 2 tool call(s), named: lookup_order, send_email
It ran and failed. "run_tests" ran 1 time(s) and none ended ok; the last ended error: pytest exited 1: 3 failed
It failed, was retried, and worked. "run_tests" ran 2 time(s), 1 ended ok and 1 did not

span_present does the same thing for its own zero: it names what it looked for and then what the run actually holds, because a wrong name and absent instrumentation are indistinguishable from a bare not found and have nothing in common as fixes.

What these scorers prove, and what they do not

They prove an action was recorded. They do not prove it was correct. A run_tests span that ended ok says the tool returned without raising, not that it ran the right suite, not that it ran the whole suite, and not that anything read the result. Treating a green tool_succeeded as proof the code works is the same mistake as trusting the sentence, one step further along.

They also measure the trace rather than the world. A span exists because your process recorded one. Work that happened inside a step you never instrumented is invisible here, and reads exactly like work that never happened, which is the one way these scorers report a failure that is not one. When one of them scores zero, check the instrumentation before you conclude the agent lied.

Both hold even when payload capture is off. Neither reads an input, an output or a completion: they read span kind, span name and span status, which the daemon holds whatever your capture settings are.

The query that finds bad answers

This is what the scores are for. has_score names a scorer and score_below sets the bar, and the trace list returns the runs that scorer put underneath it. The comparison is strict, so score_below=0.5 excludes a run that scored exactly 0.5.

find the runs that scored badly 

// The same client reads. This is the query that finds bad answers: every run
// that a scorer put below the bar.
const suspect = await mcpd.listTraces({
  hasScore: 'answer_relevance',
  scoreBelow: 0.5,
  limit: 10,
})
console.log(`runs scoring under 0.5: ${suspect.traces.length}`)

const recent = await mcpd.listTraces({ limit: 5, sort: 'start_time', order: 'desc' })
for (const trace of recent.traces) {
  const seconds = ((trace.durationNs ?? 0) / 1e9).toFixed(2)
  console.log(
    `${trace.traceId.slice(0, 8)}  ${trace.name.padEnd(16)} ${trace.status.padEnd(6)} ` +
      `${seconds}s  ${trace.spanCount} spans  $${(trace.cost?.totalCost ?? 0).toFixed(6)}`,
  )
}

// One run, with every span in it, ordered so a waterfall renders directly.
const detail = await mcpd.getTrace(traceId, { include: ['scores', 'feedback'] })
for (const span of detail.spans) {
  console.log(`  ${span.kind.padEnd(18)} ${span.name}`)
}

score_below only means something with has_score beside it: a bar with no scorer named is a bar on nothing. Both are parameters of GET /v1/api/traces, so they combine with every other filter there. Bad answers from one model, in production, in the last day, is one request.

Reading them back

GET /v1/api/scores lists machine evaluations, filtered by scorer_id, source, trace_id, experiment_id, min_value, max_value and a time window, with cursor paging. source takes live, offline, experiment or manual, which is how you read back the judge's own scores without the ones a person typed. GET /v1/api/feedback takes a trace_id and returns the judgements on that run. Listing a project's feedback without naming a run is not implemented in this build, and the endpoint says so: with no trace_id it refuses with a 400 naming the parameter. It used to answer an empty list, which reads as though nobody had judged anything.

The score list takes no order: it is newest first, and its cursor compares one way only. The run and span lists do take one, so it is refused here by name rather than accepted and dropped. The same applies to the kind, from, to and cursor filters that several SDK clients will let you set on a feedback query. None of them was ever applied, and all four are now refused.

To read a run and everything attached to it at once, ask for the extras by name: GET /v1/api/traces/{trace_id}?include=scores,feedback, or include=all for the logs as well.

What the trace list carries, and what it does not

Each stored trace keeps two rollups so the list needs no join. One is the mean value per scorer: an average within answer_relevance, and a separate average within groundedness. The other is a count of feedback per kind: three thumbs, one correction. Counts, not a mean, because the mean of a thumbs-up and a correction is not a quantity.

There is no overall quality score, and there will not be one. A single figure combining a judge model, a regular expression and a customer's thumbs-down would be a number with no unit and no way to argue with it. mcpd shows you the parts and lets you decide.

Next