Agent workflow state tracking: Observe any agent

Agent workflow state tracking records four kinds of state: the cursor (where the run is), retries and timers, waits for external events, and step payloads. You observe them by exporting every run as an OpenTelemetry trace, whose gen_ai.* spans can be read from outside the agent in any framework. Durable state, which the daemon holds between steps so the run can continue, is kept separate from that trace.

An agent passes on your laptop, then in production it loops, stalls on a wait or hands a wrong value to the next step. The only record is scattered logs. The problem belongs to the one-engineer team hunting one bad run among thousands, and to the platform team that needs retention and redaction.

mcpd watches any agent in any framework or language, with nothing to adopt. After reading, you can name each kind of state, observe it from outside the agent, make it survive a crash, and trace a failure to the run behind it.

Key Takeaways

• Give the cursor and the payloads different lifetimes: the cursor must survive a crash, while completions should expire.
• Replaying a trace must never write to durable state, or model calls and tool calls run twice.
• Use monitors that latch, so a quiet hour with no runs never clears a fired monitor.
• Leave an unknown model’s cost empty, so a missing price never passes a cost budget as zero.

What is agent workflow state tracking?

Agent workflow state tracking means recording everything a workflow must remember to continue, so a run can be inspected, resumed or explained. That state comes in four kinds, and each behaves differently:

• The cursor: which step the run is on and which steps are finished. Lose it and a restart repeats work or skips it.
• Retries and timers: how many attempts a step has used, when the next attempt is due, and when a sleep ends.
• Waits for external events: a run paused until an approval, a webhook or a reply arrives. The wait can last hours or days.
• Payloads: the prompt sent, the completion returned and the tool result. These are the largest values and the most sensitive.

The cursor changes at every step. A wait can sit untouched for days. A payload is written once and read later by a person, so it can’t share the cursor’s rules.

One framework’s key-value answer

Microsoft Agent Framework’s state documentation says that “State allows multiple executors within a workflow to access and modify common data.” Its Python API is WorkflowContext.set_state() and get_state(), and its. NET API is QueueStateUpdateAsync and ReadStateAsync. That key-value store answers where shared data lives.

If you keep all four kinds as keys in one such store, they’ll share one lifetime, one write path and one redaction rule. A cursor that must survive a crash sits beside a completion that should be deleted after 30 days. Nothing in the store says which key is which kind, so agent workflow state tracking can’t tell them apart.

Where each kind lives in mcpd

mcpd keeps the kinds apart. mcpd’s workflow documentation puts it this way: the cursor, retries, timers and waits live in the durable run, which the daemon holds between steps.

Payloads live on spans, the records of each model call, tool call and retrieval that make up the trace. The concepts page defines traces and spans in full.

For agent workflow state tracking, the split gives two kinds of state. Observed state is what happened, and traces record it. Durable state is what the daemon holds between steps so the next step can run, and replaying a trace must never write to it.

How do you track agent workflow state from outside the agent?

You track agent workflow state from outside the agent by exporting each run as an OpenTelemetry trace, so the span tree becomes the state history. The cursor is the span you are inside, retries are repeated sibling spans, and payloads are span attributes. Nothing in the agent’s framework or language changes.

1. Create a project and a key. Sign up at mcpd.sh/app, create a project and copy its key. To self-host, build from source at github.com/loruorg/mcpd instead. A key from one project writes only to that project, so use a separate project for each environment.

2. Set the exporter variables in the agent’s environment, then restart the process, because most SDKs read them once at start-up.

export OTEL_SERVICE_NAME=refund-agent
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.mcpd.sh
export OTEL_EXPORTER_OTLP_PROTOCOL=http/json

OTEL_SERVICE_NAME labels every run so you can filter by agent, and OTEL_EXPORTER_OTLP_ENDPOINT is where spans are sent. The OpenTelemetry exporter specification lists the protocol value as http/json, which selects the JSON encoding.

Add the project key to the exporter’s headers as mcpd’s OpenTelemetry guide describes. Leave any trailing path off the endpoint: the exporter appends its own signal path.

3. Emit the span and attribute names that carry state.

Model calls carry gen_ai.request.model. Most framework instrumentation libraries already write these, so you add no code.

4. Accept both conventions. OpenInference publishes a separate set of conventions, such as llm.token_count.prompt, in its semantic conventions. mcpd reads the OpenTelemetry gen_ai.* names and the OpenInference llm.* names, so existing instrumentation works unchanged.

5. Run the agent and open the trace. Within a minute or two of a run finishing, it appears in the interface as a waterfall on one clock, with timings, tokens and cost per step. A healthy run shows a root span with the agent’s steps nested under it, each showing its prompt and completion.

A run that’s still going shows the spans it’s finished so far. That’s how you inspect a stuck one.

If a custom exporter’s runs never appear, check the IDs first. The OTLP specification says trace and span IDs in JSON are hex strings and are “not base64-encoded like is defined in the standard Protobuf JSON Encoding”.

The integration guide compares this route with the HTTP API and the SDKs. To try it, start free at mcpd.sh/app.

Observed state or durable state: what survives a crash?

Durable state survives a crash: it is what a workflow holds between steps so it can continue, while observed state is the trace of what happened.

Mixing the two breaks agent workflow state tracking in specific ways. A trace used as a resume point replays model calls and tool calls that already ran. A checkpoint used as a debugging record has no timings, tokens or cost, so it can’t tell you why a run looped.

Microsoft’s checkpoint documentation describes checkpoints that save the state of a workflow at specific points during its execution so it can resume from those points later.

In that model, your process and the framework share one runtime, and you configure the checkpoint store yourself.

mcpd splits the roles. The workflow definition uses nine constructs: task, agent, sleep, sleep_until, wait_event, parallel, foreach, loop and subworkflow. The daemon holds the cursor, retries, timers and waits for an event that may take a week.

A worker claims one step at a time under a lease, and never holds the record of progress itself.

Every durable run is also a trace, so the same run gives you the observed history.

During a wait of hours or days, the run stays at its wait_event step in the daemon until the event arrives or a sleep_until timer fires. Then a worker claims the next step.

A crash in each model

In Microsoft Agent Framework, a workflow can be used as an agent. Its workflows-as-agents documentation says a serialized session carries the inner workflow checkpoint, and inner agents must keep the same id and name when rehydrated. Rename an inner agent between deployments and the saved state no longer matches it.

In mcpd, the equivalent failure is a worker killed halfway through a step. The worker never held the record of progress, so another worker claims the step once the lease expires, and the cursor is unchanged.

Microsoft Agent Framework suits teams that build the whole workflow in that framework and want recovery inside it. mcpd suits agents in any language or framework, where the daemon should hold the cursor and long waits while your code stays unchanged.

How do you trace a looping or stuck agent to the run behind it?

You trace a looping or stuck agent to its run by letting a monitor or scorer flag the deviation, searching for what the model said, and reading that run’s span tree.

1. Set a step-count divergence monitor. The alerting docs describe a monitor that is evaluated every 60 s, latches, fires once, resolves only on a real reading, and calls your webhook. Silence is not recovery: a quiet hour with no runs does not clear it.

Point the webhook at a channel someone reads.

2. Score every run with no_tool_loop. A run that repeats the same tool call fails it, which gives you a per-run flag to filter on, where the monitor gives you a rate.

3. Search by what the model said. mcpd’s search finds a run by what the model said, so take a phrase from the complaint or from the agent’s last message, such as “let me check that order again”.

Searching labels alone usually misses the run, because a loop rarely carries one.

4. Read the span tree. Open the run in the runs view. A loop appears as repeated sibling spans, and a stuck wait as a span that never closes. For a wrong value, find the first span whose output holds it, then read the next step’s input to confirm it was handed forward unchanged.

5. Keep the run as a test. The testing guide covers turning a production run into a regression case, so the fix is checked against the run that exposed it.

A worked example of agent workflow state tracking

If a refund agent calls lookup_order nine times for one customer, the divergence monitor fires and no_tool_loop marks that run as failed. A search for the agent’s final sentence returns that one run.

Its tree shows nine sibling tool spans with the same input. An empty order ID in the third span’s output, retried by every later call, is the cause to fix.

Hand agent workflow state tracking to a coding or support agent

Run mcpd mcp and the same investigation is available to another agent.

The Model Context Protocol specification says servers use tools so that “language models” can invoke them, so a coding assistant can search for the failing phrase and fetch the run itself. A key with read scope can look but not change anything.

What does agent workflow state tracking cost, and how do you keep it private?

Agent workflow state tracking costs tokens and holds personal data, because the payloads of each step (prompts, completions and tool results) are the state.

In agent workflow state tracking, tokens grow when you pass the full history forward, because each step resends everything before it. Step 2 sends its own prompt plus the prompt and answer of step 1. Step 3 carries steps 1 and 2 as well.

The input of a late step is every earlier step plus its new content, so input tokens climb at each step even when each step adds little. A large tool result is the usual cause, since every later step sends it again.

mcpd’s cost documentation says mcpd prices each step once, at ingest, from a price book of per-million-token rates. The book handles cached tokens, long-prompt tiers and service tiers, and you can supply your own. Reading cost per step shows which step the growing history made expensive.

An unknown model has no cost, not a cost of zero. A zero would pass a cost budget and hide the gap. An empty value tells you the model is missing from the book.

Privacy follows the same payloads. Redaction runs before the write, because a secret deleted afterwards was still written down. mcpd’s privacy documentation lists four modes, applied before storage on every signal:

• drop removes the value.
• mask replaces it with a placeholder.
• partial keeps only part of the value.
• hash replaces it with a hash made with a secret key, so equal values still match across runs.

Retention is set in days, so old runs expire without anyone deleting them by hand. When a person asks you to erase their data, mcpd handles erasure requests about a person.

The mcpd security documentation covers keys, scopes and who can read a payload.

Frequently asked questions

How do I track agent workflow state in any framework?

Emit OpenTelemetry spans using the gen_ai.* conventions and point the OTLP exporter at a collector such as mcpd. Each run becomes a span tree with timings, tokens and cost, so you read state from the outside. It works with any framework or language that emits OpenTelemetry.

What is the difference between observed state and durable state?

Observed state is the trace of what happened. Durable state is what the daemon holds between steps: the cursor, retries, timers and any wait_event. The next step reads durable state, and replaying a trace never writes to it.

What happens to a workflow when a worker dies mid-step?

The step’s lease expires, and another worker claims the step from the run the daemon still holds. A wait_event stays with the daemon too, so a worker dying doesn’t lose a wait of hours or days.

How do I find the run where my agent looped?

Attach the no_tool_loop scorer and a step-count divergence monitor, then search for what the model said. The monitor latches, fires once and resolves only on a real reading. Open the flagged run and look for repeated sibling spans in its tree.

Does state tracking store prompts and secrets, and how do I redact them?

It stores prompts, completions and tool results, because they are the payloads of each step. Redaction runs before storage (drop, mask, partial or hash), so a secret that was never written can’t leak later. Retention, set in days, limits how long the rest stays.

Track your first workflow run in five minutes

Add agent workflow state tracking in layers, starting with the trace.

• In five minutes: set OTEL_EXPORTER_OTLP_ENDPOINT=https://api.mcpd.sh and OTEL_EXPORTER_OTLP_PROTOCOL=http/json, add the key from your project, and run the agent once. Open the first span tree and find the cursor, the retries and the payload of each step.
• In an hour: add a step-count divergence monitor, so a run that takes far more steps than usual raises a flag. Attach the no_tool_loop scorer to catch repeated tool calls.
• In a week: choose redaction modes and a retention period before sensitive payloads arrive.

Two start routes work today: the hosted version, or a build from source at github.com/loruorg/mcpd. Start free at mcpd.sh/app; you adopt no framework, and your agent code stays as it is. If you’d rather read first, the quickstart walks through the same first trace.

Sources

mcpd · The unopinionated agent observatory. · Microsoft Agent Framework Workflows - State | Microsoft… · Workflows · mcpd · OpenTelemetry Protocol Exporter · OpenInference semantic conventions · OpenTelemetry · mcpd · mcpd cost documentation · mcpd privacy documentation. Every other source is linked where the article cites it.