Skip to content

perf(executor): stop re-parsing the whole structured output on every streamed token - #10087

Merged
waleedlatif1 merged 1 commit into
simstudioai:stagingfrom
b84330808:perf/response-format-stream-parse
Nov 16, 2026
Merged

waleedlatif1 merged 1 commit into
simstudioai:stagingfrom
b84330808:perf/response-format-stream-parse

Conversation

@b84330808

@b84330808 b84330808 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A chat or run can select a response-format field of an Agent block (an output like agent1_answer). When the agent streams, StreamingResponseFormatProcessor calls JSON.parse(buffer.trim()) on the whole buffer for every token delta. When that fails, it also counts braces with two global regexes over the whole buffer. CPU per response grows with the square of the output length.

Text can only parse to an object once it ends in } or ]. The processor now tracks the last non-whitespace character of the stream and only parses then. The brace-count branch re-parsed the same buffer.trim() that had just thrown or produced a non-object, so it could never return a result. I removed it. The emitted output does not change.

Agents with tools send each turn's text as one chunk, so this mostly affects structured-output agents without tools, which stream token by token.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Other: performance

Testing

  • executor/utils.test.ts passes unchanged. Biome is clean.
  • I ran a throwaway fuzz with 2000 random outputs: nested objects with } inside strings, arrays, primitives, invalid JSON, trailing whitespace or text, and multibyte characters. Each was split into random 1 to 12 character chunks, and the old and new processors emitted identical streams.
  • Timing setup: processStream over about 4-character chunks (like token deltas), pretty-printed JSON. Node 24.15 on Windows 11, 7 alternating rounds, median (min to max):
output chunks before after
2 KB 516 10.8 ms (7.9 to 11.3) 1.6 ms (1.1 to 1.9)
10 KB 2,563 102.4 ms (96.6 to 105.3) 5.3 ms (4.6 to 13.1)
50 KB 12,814 1683.5 ms (1599.9 to 1753.5) 34.2 ms (28.6 to 39.3)

This CPU time is spread over the stream rather than added to latency, since the model's token rate dominates. It still runs on the event loop that serves other requests.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…streamed token

When a chat selects a response-format field, StreamingResponseFormatProcessor
re-ran JSON.parse on the whole buffer for every token delta, then scanned it
twice more with brace-count regexes, so CPU per response grew with the square
of the output length. Text can only parse to an object once it ends in `}` or
`]`, so parse only then. The brace-count branch re-parsed the same
`buffer.trim()` that had just failed or yielded a non-object, so it could never
return a result and is removed.
@vercel

vercel Bot commented Nov 16, 2026

Copy link
Copy Markdown

@b84330808 is attempting to deploy a commit to the Sim Team on Vercel.

A member of the Team first needs to authorize it.

@greptile-apps

greptile-apps Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] This PR appears safe to merge.

Summary

StreamingResponseFormatProcessor now skips whole-buffer parsing unless the last non-whitespace character is } or ].

  • Response-format streams wait for a closing bracket before parsing.

Reviews (1) · Last reviewed commit: "perf(executor): stop re-parsing the whol..." · Reviewed by Greptile

@waleedlatif1
waleedlatif1 merged commit 318f1d5 into simstudioai:staging Nov 16, 2026
78 of 80 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants