Skip to content

Provider API: SSE stream ends silently mid-tool-call during large tool-call argument generation (no [DONE], no finish_reason) #785

Open

Description

@seanpham99

Summary

Streaming chat.completions against the Provider API intermittently ends the SSE stream mid-tool-call while the model is generating a large tool-call JSON object. The stream stops cleanly — no [DONE], no finish_reason chunk, no error — with the tool-call arguments still incomplete. A client cannot distinguish this from a network drop.

Correlated with:

  • Large tool-call argument payloads (~10k–50k chars, e.g. a write_file/terminal call carrying a long body)
  • Large prompt context (~50k+ tokens)

Small tool-call arguments, plain text output of any length, and the same request in non-streaming mode all complete reliably. This looks like a streaming-path-specific drop in the upstream SSE implementation, not a hard output-token cap.

Expected Behavior

The SSE stream should run to completion ([DONE], finish_reason: "tool_calls") with complete JSON tool-call arguments ending in }, regardless of argument size.

Actual Behavior

The stream ends after a few thousand chunks with no terminator of any kind — no [DONE], no finish_reason chunk. The tool-call arguments are truncated mid-string. Total delivered arguments ~20–50KB (roughly 5–9k completion tokens), far below the requested max_tokens.

Steps to reproduce the issue

Send a streaming chat completion that forces the model to emit one big tool call:

{
  "model": "deepseek/deepseek-v4-flash",
  "messages": [
    {
      "role": "user",
      "content": "Call write_file with path '/tmp/big.txt' and content being a 20,000-character essay about Vietnamese coffee culture. The ENTIRE content must be inside the tool call arguments. Do not abbreviate."
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "write_file",
        "description": "Write a file to disk",
        "parameters": {
          "type": "object",
          "properties": {
            "path": {"type": "string"},
            "content": {"type": "string"}
          },
          "required": ["path", "content"]
        }
      }
    }
  ],
  "tool_choice": {"type": "function", "function": {"name": "write_file"}},
  "max_tokens": 131072,
  "stream": true
}

Python repro (stdlib only):

import json, urllib.request, os

key = os.environ["COMMANDCODE_API_KEY"]
url = "https://api.commandcode.ai/provider/v1/chat/completions"
body = {
    "model": "deepseek/deepseek-v4-flash",
    "messages": [{"role": "user", "content": "Call write_file with path '/tmp/big.txt' and content being a 20,000-character essay about Vietnamese coffee culture. The ENTIRE content must be inside the tool call arguments. Do not abbreviate."}],
    "tools": [{"type": "function", "function": {"name": "write_file", "description": "Write a file to disk", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}],
    "tool_choice": {"type": "function", "function": {"name": "write_file"}},
    "max_tokens": 131072,
    "stream": True,
}
req = urllib.request.Request(url, data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json", "Authorization": "Bearer " + key})
with urllib.request.urlopen(req, timeout=300) as r:
    chunks = 0
    for raw in r:
        line = raw.decode("utf-8", "replace").strip()
        if line.startswith("data: "):
            line = line[6:]
        if line == "[DONE]":
            break
        chunks += 1
        print(line[:200])
    print("chunks:", chunks)

Expected: stream continues to [DONE] with finish_reason: "tool_calls", complete JSON args ending in }.
Observed (when it fails): stream simply ends after a few thousand chunks — no [DONE], no finish_reason chunk, tool arguments truncated mid-string.

Command Code Version

Provider API (gateway) — api.commandcode.ai/provider/v1/chat/completions, model deepseek/deepseek-v4-flash

Operating System

Linux

Terminal/IDE

OpenAI-compatible client (raw SSE reader)

Shell

No response

Session file (optional)

No response

Fix prompt (optional)

No response

Additional context

Happy to run additional repros (specific context sizes, different models in the catalog, non-SSE streaming) on request.

Activity

  1. seanpham99 commented on Sep 1, 2026

    @seanpham99
    Author

    Follow-up: direct repro passes today — updating with honest findings

    I ran the exact repro from this issue today against api.commandcode.ai/provider/v1/chat/completions (model deepseek/deepseek-v4-flash, stream: true, 20k-char tool-call request, browser UA):

    Result: PASS — no drop.

    • 3,973 chunks, finish_reason: "tool_calls", [DONE] received, ~18.5KB of tool-call arguments delivered, complete.

    Also tested via an OpenAI-compatible relay (the environment where the drop was originally observed):

    • Same 20k-char tool-call request, ~7k chunks, finish_reason: "tool_calls", complete args — no drop.
    • Same request with a ~50k-token context prefix: 6,504 chunks, finish_reason: "tool_calls", complete args — no drop.

    What this means

    The silent mid-tool-call stream end is intermittent / environment-dependent, not a deterministic failure of a big tool-call request. The original observations came from a production agent harness at 53–64k prompt tokens, with multi-turn tool-call history (assistant tool_calls → tool results → next assistant turn), not from a single-shot request. I have not yet reduced it to a minimal standalone repro.

    What I suspect (unverified)

    • The drop may require a combination of: large context + multi-turn tool history + reasoning/thinking mode (deepseek-v4-flash has thinking on by default).
    • I did hit a related but distinct 400 on the same model: "Thinking mode does not support this tool_choice" when sending an explicit tool_choice (that's the deepseek-v4-flash thinking-mode constraint, separate from this stream-drop issue).
    • In the failing harness, the drop appeared as the stream ending with no [DONE] and no finish_reason — indistinguishable from a network/upstream cut, which is the hardest part to debug.

    Request to maintainers

    If you have internal visibility into deepseek/deepseek-v4-flash streaming (or upstream DeepSeek SSE), a likely culprit is the SSE/streaming path dropping the connection when reasoning + tool-call generation runs long at high context. Anything you can share about known streaming stability limits (context size × reasoning × tool-call interplay) would help.

    I'll keep this open and will reduce to a minimal repro (large context + multi-turn tool loop) if I can catch it again; happy to provide trace IDs from the harness when it recurs.

  2. seanpham99 commented on Sep 1, 2026

    @seanpham99
    Author

    Stronger repro found: drop correlates with prompt context size, not tool-call size

    Catching up on the earlier thread: the silent mid-stream drop reproduced again in a live agent session today, and the trigger is now clearer.

    What happened (production harness, model = deepseek/deepseek-v4-flash via an OpenAI-compatible relay):

    • Session context reached ~150k–169k prompt tokens per request.
    • Two API calls at 166k/169k in → 3.7k/4.2k completion tokens: OK.
    • Immediately after, three consecutive streaming calls each emitted ~1 completion token and the stream ended with no finish_reason / no [DONE] (logged as "Stream ended with no finish_reason ... treating as a mid-stream drop").
    • The client fell back to another provider and the turn eventually completed.

    Relay usage history for the failed calls (all status "ok", i.e. no HTTP error, stream just ended):

    • prompt=156150, completion=1
    • prompt=156210, completion=1
    • prompt=156270, completion=1

    Correlation:

    • Same model, same client, same tool-call shape — the only material difference between working and failing calls was prompt context size.
    • The earlier 20k-char tool-call repro at ~8k context passes cleanly (recorded above).
    • Failing calls were at ~156k+ prompt tokens. Working calls were at ≤169k in the same session, then it flipped to 1-token drops.

    So the drop appears tied to context length, not output/tool-call size. DeepSeek's own context window for this model is large (claims 128k+), so 156k may exceed the effective upstream limit — the relay may be sending a request past the model's real window and getting a near-empty stream back instead of a 400.

    Ask: could CommandCode clamp/reject requests that exceed the upstream model's actual context window instead of streaming 1 token and ending silently? A clear 400 context_length_exceeded (or a finish_reason/[DONE] with an error payload) would be dramatically easier for clients to handle than an indistinguishable stream cut.

    I'll keep the earlier repro too — both are useful. Happy to add exact request headers/timestamps if you want them.

  3. arasovic commented on Sep 2, 2026

    @arasovic

    Another data point for this, from a streaming client hitting the Provider API over one working day. Same signature, but the correlation looks wider than large tool-call arguments.

    time (UTC) model reasoning_effort first byte total upstream symptom
    11:39:38 deepseek/deepseek-v4-flash high 1778 ms 58.9 s connection closed
    12:25:48 deepseek/deepseek-v4-flash max never 1.0 s stream ended before terminal chunk
    12:40:15 deepseek/deepseek-v4-flash high 1537 ms 11.4 s stream ended before terminal chunk
    14:08:50 deepseek/deepseek-v4-flash max never 3.0 s hard 500 (body below)
    14:09:21 deepseek/deepseek-v4-flash max 1635 ms 134.4 s stream ended before terminal chunk
    14:15:22 moonshotai/Kimi-K2.7-Code — 8290 ms 12.4 s malformed tool-call arguments

    The 14:08:50 one returned a real error body rather than a silent stop:

    {"success":false,"error":{"code":"INTERNAL_SERVER_ERROR","status":500,
     "message":"An unexpected error occurred. Please try again later.",
     "docs":"https://commandcode.ai/docs/reference/errors/internal_server_error"}}

    Three things that differ from the report above:

    1. It is not confined to large context. Prompt sizes on the immediately preceding successful request in the same conversation were 2,423 / 19,594 / 28,489 / 29,105 / 89,933 tokens. One failure follows a 2.4k-token turn. The issue text points at ~50k+; we see it across two orders of magnitude, including well under 10k.

    2. It is not confined to large tool-call payloads. Completion sizes on those same preceding requests were 2 / 81 / 48 / 30 / 81 tokens — small turns, not 20-50KB argument generation. Large outputs do occur elsewhere in these sessions (max 21,082 completion tokens) and complete fine, so size does not separate the failures from the successes in our data.

    3. reasoning_effort was high or max on every one. We have no instance of this at a lower effort. That may be the sharper correlate than payload size — worth checking against your own traces.

    A second model shows what may be the same truncation surfacing differently: moonshotai/Kimi-K2.7-Code delivered tool-call arguments that were malformed rather than stopping silently.

    In every case usage is unreported, so these responses carry no token accounting and do not appear in usage reporting at all — which also makes them invisible from the billing side while a client has already paid for the whole conversation up to that point.

    Client-visible effect matches the original report: the stream ends with no terminal chunk, the client sees an empty or truncated assistant turn, and a provider fault is indistinguishable from a network drop.

  4. Coding-Dev-Tools commented on Sep 6, 2026

    @Coding-Dev-Tools

    I isolated the current client’s handling of this failure using Command Code 1.49.1 on Windows, with an owned localhost OpenAI-compatible endpoint. Installed dependencies were @ai-sdk/openai-compatible 2.0.74 and ai 6.0.277. No production gateway requests or paid inference were used; this does not reproduce the gateway’s intermittent drop.

    There is a concrete diagnostic problem worth fixing:

    1. Return HTTP 200, send partial text, then end SSE without finish_reason or [DONE].
    2. The CLI correctly emits run_error: Stream failed: Response stream ended without a finish reason.
    3. Its final JSON error and stderr instead say: Error: Unable to connect to the API. Please check your network connection.

    The connection succeeded and data arrived, so the final message hides the useful diagnosis. This also reproduced with complete and incomplete tool-call JSON following visible text: both runs exited with an error before any tool-running/completed event or tool result. The current client is detecting the incomplete response in these tests.

    Fix pointer: preserve the stream error’s category, message and cause through transportErrorToApiError / APIError.generate and classifyPrintModeError. The status-0 conversion drops the original message, then the formatter replaces it with generic connection advice. Changing only the formatter is insufficient. Keep actual HTTP status distinct from stream-completion status.

    Acceptance: final JSON/stderr retain the missing-finish explanation; real connection failures retain appropriate guidance; incomplete tool calls remain unexecuted. Keep missing finish_reason separate from missing [DONE] alone.

    For the original gateway failure, the next useful evidence is a failed/successful request pair with correlated gateway/upstream IDs, route, reasoning settings, actual HTTP status, and terminal-event state at each hop. The discussion does not yet establish a fixed context/output-size trigger; counts from preceding successful requests should remain labeled as such.

    Download the reproduction scripts, results and evidence ZIP (44,995 bytes; sanitized captures and documented test limitations).

  5. ahmadbilaldev commented on Sep 17, 2026

    @ahmadbilaldev
    Contributor

    Taking a look

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions