Internal technical note · agent-server & canvas
Streaming text leaves the agent on two paths that know nothing about each other. Only one of them is tracked, nothing orders them, and the browser puts them back together by comparing strings. Seven open issues come out of that single gap.
When an agent produces a message, that message reaches the browser twice. Once as a durable event — appended to the conversation log, written to disk, and published to every subscriber. And once as a stream of token deltas, emitted while the model is still generating so the user sees text appear.
Those are two representations of the same thing. Nothing in the system says so.
Three details make this worse than it first looks, and every one of them is load-bearing.
on_token directly and never enters the LLM layer, so anything fixed there does not apply to it.acp_agent.py:1305–1307Here is the entire payload of a streamed token, minus the boilerplate every event carries:
StreamingDeltaEvent carries content, reasoning_content, a randomly generated id and a wall-clock timestamp. There is no stream identity, no sequence number, no attempt counter, and no pointer to the message that will supersede it.
So a client receiving one of these cannot answer any of the four questions it needs to answer:
It answers all four by inspecting the text. That is what the 448 lines of handle-event-for-ui.ts are for — protocol repair in the view layer, complete with prefix matching, an ordered-segment search, a whitespace-trimmed candidate set, and a sniff for a <function= marker to detect a raw tool call that leaked into the visible text.
None of that is bad code. It is careful code compensating for information that was thrown away upstream.
This is not an unsolved problem. It is a solved problem we solved differently. Field lists below come from the installed packages and from source, not from documentation.
| System | What a delta says about itself | How a message starts and ends |
|---|---|---|
| pi | messageId · contentIndex · kind | item_started → item_updated → item_finished |
| OpenAI Responses | item_id · output_index · content_index · sequence_number | output_item.added → .done |
| Anthropic Messages | index — the message is implicit, one per stream | content_block_start → _stop |
| Agent Client Protocol | nothing | agent_message_chunk, in arrival order |
| Ours | nothing | nothing — and on a second, unordered path |
Every one of the other four puts an identity on the increment and a lifecycle around it, and delivers both on a single ordered stream. That combination is what makes their clients simple: a delta names its target, so the client updates a keyed slot instead of guessing.
ACP is the interesting near-miss. Its ContentChunk is _meta plus content and nothing else, so it has no identity either — and because our ACP bridge sits on top of it, we inherit that gap on that path regardless of what we do elsewhere. _meta is ACP's sanctioned extension point, so identity can be added there without forking the protocol.acp/schema.py:4289
For a sense of the difference in cost: pi's entire client-side stream reconciliation is 101 lines. It keeps a disposable overlay keyed by item id and throws the whole overlay away whenever authoritative state arrives, because a delta that names its target never has to be matched to anything.
These are not seven independent bugs. They are seven symptoms of one missing relationship.
| Issue | What actually happens | What's missing |
|---|---|---|
| #15433split text | A user message arrives mid-stream, so the next delta no longer sees a delta last in the array and opens a second bubble. On finalize only the later half is superseded — the first half survives next to the finished message. | A turn-scoped slot. |
| #15432message ignored | A single lock guards the event log, the agent, execution status, secrets, stats and the conversation head. Message intake contends with the run loop at every step boundary, and whether it gets picked up depends on a lock race plus a re-arm flag. | Separate locks. |
| sdk#4077six defects | Deltas fire inside the retry boundary, so a retry re-streams everything. An empty stream returns None and the following assertion raises an error the retry logic does not catch. Delta futures are discarded, so the drain barrier cannot see them. State updates are a bare await that can hang the run. The resume cursor compares naive local timestamps. | An attempt number, one tracked path, a real cursor. |
| #15720security, P1 | The ACP path masks each chunk on its own, so a secret split across two chunks matches neither piece. On the standard agent path it is worse: agent.py never calls mask_secrets_in_output at all — the only places that mask model-adjacent output are the terminal and MCP tool results. | One redaction boundary both paths cross. |
| #15493 | Text is committed to the view at whatever granularity the network happened to deliver it, so rendering stutters. | A render clock. |
| #15511 | The compose form forces streaming on, and the server then independently re-decides whether it is available. | One owner for the flag. |
| #16331not pinned | Serving a workspace file goes through a process-wide lock that is also held across conversation start, fork, delete and eviction — so a file preview can block behind unrelated lifecycle work. | Per-conversation locking. |
Every token delta is POSTed to configured webhooks. Deltas are published into the same fan-out the webhook subscriber listens on, nothing filters by event kind, and the webhook buffer flushes every five events by default. That is roughly one HTTP request per five tokens, per configured webhook, for the entire duration of a response.
conversation_service.py:2233 · config.py:75
The shape follows from the diagnosis rather than from taste. Three changes, and the first one is what makes the other two available.
Today the socket frame and the disk record are literally the same class, serialized the same way and validated by the same strict validator. That is a single decision with a long tail: it is why a delta cannot carry identity without a storage migration, why an additive field fails closed on older clients, and why the transient delta had to be modelled as a durable event class in the first place — it is never persisted, it is only in that hierarchy because the socket speaks nothing else.
Introduce a new endpoint speaking an envelope that is not an event. The event itself does not change at all — same class, same validator, same bytes on disk. It rides inside the envelope as a payload. Everything the protocol needs now lives on the envelope, where adding a field costs nothing and breaks nobody.
The old endpoint is frozen and deprecated as it stands; a client speaks one or the other. No handshake, no version field, no dual-decode path. This is worth stating plainly because it is the cheapest possible way to get versioning we currently do not have: pi's session files are on format v4 while its wire protocol is still on v1 — three storage rewrites, zero wire breaks.
The cursor comes free. A monotonic per-conversation index is already durable in every event filename, and the writer has it in hand at write time. Returning it from the append call and putting it on the envelope adds nothing to disk, so there is no migration and no compatibility break.
Today the delta exists before the durable event does, so nothing can name the relationship. Invert it: allocate the event id up front, and construct the finished message with that id.
A client holding an open slot for item_id sees the durable frame arrive carrying event.id == item_id, and swaps the slot for the real message. That is the entire close protocol — one equality test, on a field that already exists.
Gone with it: the close frame, the terminal marker, the "has the closing event arrived yet" bookkeeping, and every line of text comparison in the browser. What remains is one integer that says where in the transcript to place the slot, which exists only to stop a mid-stream user message from splitting the bubble.
A stream that dies without ever producing a message still needs a terminal frame — that is what ItemAborted is for, and §06 covers what guarantees it always fires.
Admission becomes a synchronous check that never blocks the producer: if a write would exceed the connection's byte budget it is rejected immediately, rather than queued or waited on. The budget is derived — a small multiple of the largest legal frame — rather than picked out of the air. Ordering comes from a single write chain, which is one writer per connection in a handful of lines.
A consumer that cannot keep up is disconnected, not buffered. That is only safe because the durable side is replayable: the client reconnects with its cursor and loses nothing. Disconnection is the backpressure mechanism, and it is also why one socket is now sufficient where a second one seemed necessary before.
None of these are streaming work, all three stand alone, and each removes a mechanism the problems above depend on: split the overloaded lock into its separate concerns, which resolves the dropped-message issue by itself; make the lifecycle lock per-conversation rather than process-wide; and drop the full directory listing that runs on every single append while the lock is held.
Four frames on one ordered channel. Two of them carry durable state; two are progress hints the client is free to lose.
| Frame | Carries | When it is sent |
|---|---|---|
| Durable | seq · event | Once per durable event, after it is safely on disk. The event payload is the existing JSON, unchanged. |
| ItemStarted | item_id · attempt · anchor_seq | Once per attempt, before the first token. A higher attempt supersedes a lower one — that is how a retry announces itself instead of silently re-streaming. |
| Delta | item_id · attempt · order · kind · content | Per masked chunk. kind separates visible text from reasoning, the way all three vendor protocols do it. |
| ItemAborted | item_id · attempt · reason | Only when a stream ends without ever producing a message. |
order marks the slot lossy; the authoritative text arrives with the message regardless.ItemStarted is retired by exactly one durable frame or one ItemAborted — including on cancellation, provider failure, policy rejection and unhandled exception.ItemStarted frames arrived in.Today the socket subscribes and then replays history, so live events interleave with the backlog and the client sees them out of order. Four steps close that window:
pi does the equivalent in about twenty lines — subscribe, snapshot, then flush while still buffering so reentrant events keep their order. Ours needs the page loop only because our history lives on disk rather than in memory.
A stream context wraps each attempt in try/finally: opening mints the id and reads the anchor, streaming masks and stamps each chunk, and the message is built with the minted id. If no message was produced, the finally emits exactly one ItemAborted. It has to sit at four call sites — both agents, on both their synchronous and asynchronous paths — because the LLM-layer call context cannot reach conversation state or the secret registry. The retry wrapper contributes exactly one thing, the attempt number, and that is the one thing it owns.
This is not hypothetical. The asynchronous step function alone has three except … return branches that emit an error message and return without ever touching the stream, leaving the client's slot open forever. pi enforces the same invariant one level lower: its stream function may not throw, and failures must come back as terminal events inside the stream. A stream that cannot raise cannot be stranded; our try/finally is that rule retrofitted onto code that can.
Redaction. The masking helper resolves secret sources before it does anything else — its own comment warns that resolution "may do blocking network I/O" — and then runs one string replacement per secret across the whole text. On the synchronous path that happens while the state lock is held. On the stream path, replace it with a snapshot compiled once when the stream opens: resolved values only, a single-pass matcher, no lock and no I/O while streaming. The trade-off, stated rather than buried: a secret registered after the stream opened will not be masked in that stream's deltas, but it is masked in the durable message, which is the authoritative one.
Publish after persist. Today the publishing callback runs before the persisting one. Swap them, so publication happens on a successful append and only then — both under the lock the run loop already holds, so the two cannot be separated. If the append raises, nothing is published.
Worth recording, because the earlier drafts of this design accumulated machinery that the substrate change simply deletes.
| Previously needed | Now |
|---|---|
| A second socket dedicated to the stream | One endpoint — the byte budget plus the cursor make a slow consumer safe to drop |
| A parallel frame hierarchy sitting alongside the event socket | One envelope, with the event riding inside it |
| Three stream frames plus a close reason | Two, retired by the durable message itself |
| A closing sequence pointer, a final-order counter, and a watermark comparison to answer "has it closed yet?" | event.id == item_id |
| A stream identifier unrelated to any durable identity | The item id is the message id, minted at open |
| Seven named queue capacities, none of them sized | One byte budget per connection, derived from the largest legal frame |
| Protocol negotiation, transient-event filtering, an unknown-kind gate | Structural — a new endpoint has no legacy clients, and deltas never reach the event bus at all |
The stream context at four entry points with its try/finally; the attempt number from the retry wrapper; snapshot masking with a carry-over tail; masking the durable message too; non-blocking admission with resumable disconnect; one writer per connection with a deadline; the acceptance barrier; publish-after-persist; the atomic replay boundary; and turning the fan-out from a callback list into a real delivery component.
That last one is the actual new machinery here, and no amount of substrate change avoids it.
Steps 1 through 3 are independently shippable and independently valuable. Nothing before step 5 is visible to a user.
| Step | Work | Needs |
|---|---|---|
| 0 | Reproductions and baselines. A delta-flood stress test, a retry-injecting fake model, and measurements of lock contention, append cost against conversation length, fan-out cost against connection count, and the frame-size distribution the byte budget is derived from. Without these, the third change above is unverified. | — |
| 1 | The three standalone repo fixes, redaction of the durable message, the retryable-error parity fix, and giving the token callback the accumulated text alongside each increment so a failed stream can still emit what it has. All independent of the protocol. | — |
| 2 | The append call returns its sequence number; publication moves after persistence. Invisible on the wire. | — |
| 3 | The new endpoint and envelope, carrying durable frames only. Cursor, paged replay, atomic reconnect boundary. Both clients switch endpoints and behaviour does not change. | 2 |
| 4 | Byte-budget admission, one writer per connection, the acceptance barrier. | 3 |
| 5 | The stream context mints the id, messages are constructed with it, and the three progress frames go out across all four entry points. | 3 |
| 6 | Client turn model — keyed slots, retirement by id, and the render clock that makes streaming smooth. | 5 |
| 7 | Stop emitting the old delta event; freeze the old endpoint and announce its removal. Nothing decodes wrong — it simply stops being offered. | 6 |
The current streaming test covers deltas being emitted and suppressed, and nothing else. Six are missing: retry against identity, reconnect against slot discard, replay against live interleave, ACP against a split-chunk secret, error-after-tokens against exactly one retirement, and a wedged connection against a publisher that must not block. All six fail today.
event.model_dump(...), the log persists event.model_dump_json(...), and the Python client decodes with Event.model_validate — the same class, end to end.sockets.py:489 · event_store.py:217 · remote_conversation.py:239openai, anthropic and acp packages and from pi's source, not from documentation.testing/. It is specified and tested, and its client half is real, but it is a clean-room answer to the same question rather than a battle-hardened one.