Troubleshooting OpenAI Compatible API Streaming Output (stream) Failures

Identify the Symptom First

Four symptoms point to different root causes:

• No output at all: The request did not include stream: true, or the client failed to send it correctly.
• Content outputs all at once: The server returned data in streaming mode, but an intermediate layer buffered and merged the chunks before forwarding (common with reverse proxies or Serverless platforms that have buffering enabled).
• Stream cuts off midway: The connection was closed prematurely due to timeout, network jitter, or upstream limits.
• Client parsing error: The client did not read SSE line by line, did not handle data: [DONE], or tried to parse each incremental chunk as a complete JSON object.

Use curl to Verify Whether the Server Is Actually Streaming

Run the following command (-N disables buffering):

curl -N -X POST https://api.treenew.online/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-xxx" \
  -d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"test"}],"stream":true}'

If data: lines appear progressively in the terminal, the 「树新在线」 gateway and upstream are functioning normally. The problem lies with the client or any self-built intermediate layer.

If You Have Deployed a Reverse Proxy

Nginx must disable buffering; otherwise “output all at once” will occur:

location /v1/ {
    proxy_pass https://api.treenew.online;
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 300s;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
}

proxy_buffering on (the default) accumulates SSE events and returns them in a single batch, producing symptom (b).

Client Parsing Essentials

• Read the response line by line and skip empty lines.
• Each line begins with data: ; strip the prefix before parsing.
• Terminate when data: [DONE] is received.
• Each chunk contains a delta incremental field, not a complete message object.

Stream Cuts Off Midway

Increase the client timeout (recommended 300s+). Implement exponential backoff reconnection: wait 1s on the first retry, then 2s, 4s, with a maximum of 3 retries.

See also

• OpenAI-compatible setup
• Timeouts and 502