The Interface Is the Bottleneck Now — Streaming Structured Output Becomes the Default
With output tokens in the $4-6 range after July's price war, inference stopped being the thing users wait on. The frontier of LLM product design moved to the interface layer, where structured output that renders field by field is quietly becoming the norm — and the unglamorous plumbing underneath is what decides whether it feels good.
For three years the defining constraint on an LLM feature was how long the user had to wait for the model to think. Product design bent around that constraint: the streaming chat bubble exists because watching text arrive token by token is less painful than staring at a spinner for the same total time. The whole visual language of AI products — the typing indicator, the shimmer placeholder, the progressive reveal — was invented to make latency tolerable.
That constraint is loosening fast. After July's compressed run of releases pushed flagship output token prices into the four-to-six-dollar range per million, and with the newest lightweight models answering structured queries in well under a second, the inference step is no longer the slowest part of most interactions. The bottleneck has moved. It is now the interface — specifically, how quickly and how gracefully an application can turn a model's output into something on screen that a person can act on. And that shift is dragging a formerly-obscure technical problem into the center of LLM product work: parsing structured output while it is still streaming.
From prose to structure
The move from chat to structure has been underway for a while. Serious LLM features rarely ask the model for free text anymore; they ask for a JSON object with a known shape — a triage decision, an extracted record, a set of tool arguments, a plan — and render the fields. Structured output is what makes model responses composable with the rest of an application instead of being a wall of text a human has to read.
The tension is that structure and streaming pull against each other. Streaming
text is trivial: each token is more text, you append it, done. Streaming a JSON
object is not, because a JSON object is not usable until it is syntactically
complete. JSON.parse throws on every incomplete prefix. So the pragmatic
default for most teams shipping structured output has been to not stream it at
all: buffer the entire response, parse once, render once. The user waits behind
a spinner for the full generation even though the first field was decided almost
immediately.
That was an acceptable trade when inference was slow enough that a few hundred extra milliseconds of buffering disappeared into the noise. It is not acceptable now. When the model returns in 600 milliseconds, a UI that waits for the closing brace before showing anything feels broken in a way it did not a year ago, because the latency it is hiding is no longer there to hide.
The plumbing decides the feel
The fix is a tolerant partial-JSON parser — a small component that takes any prefix of a JSON document and returns the best-effort object parsed so far, closing whatever structure is still open. Feed it a half-arrived object and it hands back the fields it can already see, so the interface can render them and fill in the rest as more tokens land. It is not conceptually hard, but it is easy to get subtly wrong, because JSON's structural characters — braces, brackets, commas, colons, quotes — are all also legal characters inside strings, and the naive approaches (count the braces, append the missing ones) break the moment a brace appears inside a string value.
CrashBytes published a full build of one of these parsers today, with a runnable TypeScript companion project and 49 offline tests, in the tutorial Parse Streaming JSON From an LLM. The interesting part is not the code volume — it is a few hundred lines — but the design decisions the problem forces. A partial string value has a truthful prefix and gets kept; a partial object key does not and gets dropped; an unambiguous partial literal resolves while an ambiguous partial number is withheld rather than guessed. Every one of those choices is really a product decision about what the interface is allowed to show before it is certain, dressed up as a parsing rule.
That is the pattern worth noticing. As the interface becomes the bottleneck, the components that decide how output reaches the screen stop being incidental plumbing and start being where the product's quality actually lives. The same story played out with the token-level plumbing of cost and reliability — the prompt-caching discipline that decides the bill, the retry and failover logic in a resilient multi-provider client that decides whether the feature stays up. None of it is glamorous. All of it is load-bearing.
What to watch
The near-term signal to watch is the SDK layer. Right now, tolerant partial-JSON
parsing is something application teams roll themselves or pull from a small
grab-bag of community libraries of varying quality. The moment the major model
SDKs ship first-class streaming structured-output parsing as a supported
primitive — a streamObject-style call that yields progressively-completed
typed objects rather than raw text deltas — is the moment this stops being a
thing every team reinvents and becomes an assumed capability of the platform.
That is the tell that structured streaming has fully arrived as the default, the
way token-level text streaming did years ago.
Until then, the advantage sits with the teams that treat the interface layer as seriously as they once treated the model layer. Inference got cheap and fast. The part users actually feel — output turning into interface, field by field, without a stall — is now the part worth engineering.