Repository navigation
Description
Summary
Streaming chat.completions against the Provider API intermittently ends the SSE stream mid-tool-call while the model is generating a large tool-call JSON object. The stream stops cleanly — no [DONE], no finish_reason chunk, no error — with the tool-call arguments still incomplete. A client cannot distinguish this from a network drop.
Correlated with:
- Large tool-call argument payloads (~10k–50k chars, e.g. a
write_file/terminalcall carrying a long body) - Large prompt context (~50k+ tokens)
Small tool-call arguments, plain text output of any length, and the same request in non-streaming mode all complete reliably. This looks like a streaming-path-specific drop in the upstream SSE implementation, not a hard output-token cap.
Expected Behavior
The SSE stream should run to completion ([DONE], finish_reason: "tool_calls") with complete JSON tool-call arguments ending in }, regardless of argument size.
Actual Behavior
The stream ends after a few thousand chunks with no terminator of any kind — no [DONE], no finish_reason chunk. The tool-call arguments are truncated mid-string. Total delivered arguments ~20–50KB (roughly 5–9k completion tokens), far below the requested max_tokens.
Steps to reproduce the issue
Send a streaming chat completion that forces the model to emit one big tool call:
{
"model": "deepseek/deepseek-v4-flash",
"messages": [
{
"role": "user",
"content": "Call write_file with path '/tmp/big.txt' and content being a 20,000-character essay about Vietnamese coffee culture. The ENTIRE content must be inside the tool call arguments. Do not abbreviate."
}
],
"tools": [
{
"type": "function",
"function": {
"name": "write_file",
"description": "Write a file to disk",
"parameters": {
"type": "object",
"properties": {
"path": {"type": "string"},
"content": {"type": "string"}
},
"required": ["path", "content"]
}
}
}
],
"tool_choice": {"type": "function", "function": {"name": "write_file"}},
"max_tokens": 131072,
"stream": true
}Python repro (stdlib only):
import json, urllib.request, os
key = os.environ["COMMANDCODE_API_KEY"]
url = "https://api.commandcode.ai/provider/v1/chat/completions"
body = {
"model": "deepseek/deepseek-v4-flash",
"messages": [{"role": "user", "content": "Call write_file with path '/tmp/big.txt' and content being a 20,000-character essay about Vietnamese coffee culture. The ENTIRE content must be inside the tool call arguments. Do not abbreviate."}],
"tools": [{"type": "function", "function": {"name": "write_file", "description": "Write a file to disk", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}],
"tool_choice": {"type": "function", "function": {"name": "write_file"}},
"max_tokens": 131072,
"stream": True,
}
req = urllib.request.Request(url, data=json.dumps(body).encode(),
headers={"Content-Type": "application/json", "Authorization": "Bearer " + key})
with urllib.request.urlopen(req, timeout=300) as r:
chunks = 0
for raw in r:
line = raw.decode("utf-8", "replace").strip()
if line.startswith("data: "):
line = line[6:]
if line == "[DONE]":
break
chunks += 1
print(line[:200])
print("chunks:", chunks)Expected: stream continues to [DONE] with finish_reason: "tool_calls", complete JSON args ending in }.
Observed (when it fails): stream simply ends after a few thousand chunks — no [DONE], no finish_reason chunk, tool arguments truncated mid-string.
Command Code Version
Provider API (gateway) — api.commandcode.ai/provider/v1/chat/completions, model deepseek/deepseek-v4-flash
Operating System
Linux
Terminal/IDE
OpenAI-compatible client (raw SSE reader)
Shell
No response
Session file (optional)
No response
Fix prompt (optional)
No response
Additional context
- The non-streaming equivalent of the same request completes fine — a 190KB tool-call payload via
stream: falsesucceeds. So any cap is in the streaming path only. - The failure reproduces through a plain OpenAI-compatible client; no client SDK or router involved.
- Frequency appears to increase with prompt context size; the smallest repro used an ~8k-token prompt.
- This is distinct from Error: 500 Upstream stream ended before terminal chunk #628 and CLI reports a fabricated HTTP 500 when the response stream drops mid-flight #755 (both CLI-side handling of a mid-flight drop that already ended with an error). Here the stream ends silently with no terminator at all.
Happy to run additional repros (specific context sizes, different models in the catalog, non-SSE streaming) on request.
Activity
seanpham99 commentedon Sep 1, 2026
Follow-up: direct repro passes today — updating with honest findings
I ran the exact repro from this issue today against
api.commandcode.ai/provider/v1/chat/completions(modeldeepseek/deepseek-v4-flash,stream: true, 20k-char tool-call request, browser UA):Result: PASS — no drop.
finish_reason: "tool_calls",[DONE]received, ~18.5KB of tool-call arguments delivered, complete.Also tested via an OpenAI-compatible relay (the environment where the drop was originally observed):
finish_reason: "tool_calls", complete args — no drop.finish_reason: "tool_calls", complete args — no drop.What this means
The silent mid-tool-call stream end is intermittent / environment-dependent, not a deterministic failure of a big tool-call request. The original observations came from a production agent harness at 53–64k prompt tokens, with multi-turn tool-call history (assistant tool_calls → tool results → next assistant turn), not from a single-shot request. I have not yet reduced it to a minimal standalone repro.
What I suspect (unverified)
tool_choice(that's the deepseek-v4-flash thinking-mode constraint, separate from this stream-drop issue).[DONE]and nofinish_reason— indistinguishable from a network/upstream cut, which is the hardest part to debug.Request to maintainers
If you have internal visibility into
deepseek/deepseek-v4-flashstreaming (or upstream DeepSeek SSE), a likely culprit is the SSE/streaming path dropping the connection when reasoning + tool-call generation runs long at high context. Anything you can share about known streaming stability limits (context size × reasoning × tool-call interplay) would help.I'll keep this open and will reduce to a minimal repro (large context + multi-turn tool loop) if I can catch it again; happy to provide trace IDs from the harness when it recurs.
seanpham99 commentedon Sep 1, 2026
Stronger repro found: drop correlates with prompt context size, not tool-call size
Catching up on the earlier thread: the silent mid-stream drop reproduced again in a live agent session today, and the trigger is now clearer.
What happened (production harness, model = deepseek/deepseek-v4-flash via an OpenAI-compatible relay):
Relay usage history for the failed calls (all status "ok", i.e. no HTTP error, stream just ended):
Correlation:
So the drop appears tied to context length, not output/tool-call size. DeepSeek's own context window for this model is large (claims 128k+), so 156k may exceed the effective upstream limit — the relay may be sending a request past the model's real window and getting a near-empty stream back instead of a 400.
Ask: could CommandCode clamp/reject requests that exceed the upstream model's actual context window instead of streaming 1 token and ending silently? A clear
400 context_length_exceeded(or afinish_reason/[DONE]with an error payload) would be dramatically easier for clients to handle than an indistinguishable stream cut.I'll keep the earlier repro too — both are useful. Happy to add exact request headers/timestamps if you want them.
arasovic commentedon Sep 2, 2026
Another data point for this, from a streaming client hitting the Provider API over one working day. Same signature, but the correlation looks wider than large tool-call arguments.
deepseek/deepseek-v4-flashdeepseek/deepseek-v4-flashdeepseek/deepseek-v4-flashdeepseek/deepseek-v4-flash500(body below)deepseek/deepseek-v4-flashmoonshotai/Kimi-K2.7-CodeThe 14:08:50 one returned a real error body rather than a silent stop:
{"success":false,"error":{"code":"INTERNAL_SERVER_ERROR","status":500, "message":"An unexpected error occurred. Please try again later.", "docs":"https://commandcode.ai/docs/reference/errors/internal_server_error"}}Three things that differ from the report above:
1. It is not confined to large context. Prompt sizes on the immediately preceding successful request in the same conversation were 2,423 / 19,594 / 28,489 / 29,105 / 89,933 tokens. One failure follows a 2.4k-token turn. The issue text points at ~50k+; we see it across two orders of magnitude, including well under 10k.
2. It is not confined to large tool-call payloads. Completion sizes on those same preceding requests were 2 / 81 / 48 / 30 / 81 tokens — small turns, not 20-50KB argument generation. Large outputs do occur elsewhere in these sessions (max 21,082 completion tokens) and complete fine, so size does not separate the failures from the successes in our data.
3.
reasoning_effortwashighormaxon every one. We have no instance of this at a lower effort. That may be the sharper correlate than payload size — worth checking against your own traces.A second model shows what may be the same truncation surfacing differently:
moonshotai/Kimi-K2.7-Codedelivered tool-call arguments that were malformed rather than stopping silently.In every case usage is unreported, so these responses carry no token accounting and do not appear in usage reporting at all — which also makes them invisible from the billing side while a client has already paid for the whole conversation up to that point.
Client-visible effect matches the original report: the stream ends with no terminal chunk, the client sees an empty or truncated assistant turn, and a provider fault is indistinguishable from a network drop.
Coding-Dev-Tools commentedon Sep 6, 2026
I isolated the current client’s handling of this failure using Command Code 1.49.1 on Windows, with an owned localhost OpenAI-compatible endpoint. Installed dependencies were
@ai-sdk/openai-compatible2.0.74 andai6.0.277. No production gateway requests or paid inference were used; this does not reproduce the gateway’s intermittent drop.There is a concrete diagnostic problem worth fixing:
finish_reasonor[DONE].run_error:Stream failed: Response stream ended without a finish reason.Error: Unable to connect to the API. Please check your network connection.The connection succeeded and data arrived, so the final message hides the useful diagnosis. This also reproduced with complete and incomplete tool-call JSON following visible text: both runs exited with an error before any tool-running/completed event or tool result. The current client is detecting the incomplete response in these tests.
Fix pointer: preserve the stream error’s category, message and cause through
transportErrorToApiError/APIError.generateandclassifyPrintModeError. The status-0 conversion drops the original message, then the formatter replaces it with generic connection advice. Changing only the formatter is insufficient. Keep actual HTTP status distinct from stream-completion status.Acceptance: final JSON/stderr retain the missing-finish explanation; real connection failures retain appropriate guidance; incomplete tool calls remain unexecuted. Keep missing
finish_reasonseparate from missing[DONE]alone.For the original gateway failure, the next useful evidence is a failed/successful request pair with correlated gateway/upstream IDs, route, reasoning settings, actual HTTP status, and terminal-event state at each hop. The discussion does not yet establish a fixed context/output-size trigger; counts from preceding successful requests should remain labeled as such.
Download the reproduction scripts, results and evidence ZIP (44,995 bytes; sanitized captures and documented test limitations).
ahmadbilaldev commentedon Sep 17, 2026
Taking a look