Repository navigation
perf(executor): stop re-parsing the whole structured output on every streamed token - #10087
Merged
waleedlatif1 merged 1 commit intoNov 16, 2026
Conversation
…streamed token When a chat selects a response-format field, StreamingResponseFormatProcessor re-ran JSON.parse on the whole buffer for every token delta, then scanned it twice more with brace-count regexes, so CPU per response grew with the square of the output length. Text can only parse to an object once it ends in `}` or `]`, so parse only then. The brace-count branch re-parsed the same `buffer.trim()` that had just failed or yielded a non-object, so it could never return a result and is removed.
|
@b84330808 is attempting to deploy a commit to the Sim Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
[Medium impact] This PR appears safe to merge. Summary
Reviews (1) · Last reviewed commit: "perf(executor): stop re-parsing the whol..." · Reviewed by Greptile |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A chat or run can select a response-format field of an Agent block (an output like
agent1_answer). When the agent streams,StreamingResponseFormatProcessorcallsJSON.parse(buffer.trim())on the whole buffer for every token delta. When that fails, it also counts braces with two global regexes over the whole buffer. CPU per response grows with the square of the output length.Text can only parse to an object once it ends in
}or]. The processor now tracks the last non-whitespace character of the stream and only parses then. The brace-count branch re-parsed the samebuffer.trim()that had just thrown or produced a non-object, so it could never return a result. I removed it. The emitted output does not change.Agents with tools send each turn's text as one chunk, so this mostly affects structured-output agents without tools, which stream token by token.
Type of Change
Testing
executor/utils.test.tspasses unchanged. Biome is clean.}inside strings, arrays, primitives, invalid JSON, trailing whitespace or text, and multibyte characters. Each was split into random 1 to 12 character chunks, and the old and new processors emitted identical streams.processStreamover about 4-character chunks (like token deltas), pretty-printed JSON. Node 24.15 on Windows 11, 7 alternating rounds, median (min to max):This CPU time is spread over the stream rather than added to latency, since the model's token rate dominates. It still runs on the event loop that serves other requests.
Checklist