Figures on this page were verified 31 August 2026 against the providers' own documentation. Pricing, context windows and rate limits change without notice, so confirm any number against the provider before you rely on it. Tell us if something here is out of date.
A stream that dies mid-response is usually killed by something between you and the provider, not by the model. Proxies, load balancers and serverless platforms enforce idle and total-duration timeouts that a long generation quietly exceeds. The text you already received is valid; the job is to detect that it is incomplete and decide whether to resume or restart.
Why streams get cut
| Cause | Typical symptom | Where to look |
|---|---|---|
| Idle timeout on a proxy | Dies during a long thinking pause, before the first token | nginx proxy_read_timeout, load balancer idle timeout |
| Total request duration cap | Dies at the same elapsed time every run | Serverless function limits, CDN caps |
| Response buffering | Nothing arrives, then everything at once | proxy_buffering, CDN compression |
| Client read timeout | Client raises while the server is still sending | Your HTTP library timeout settings |
| Genuine provider interruption | Sporadic, no time pattern | Provider status page |
The distinguishing question is whether the failure happens at a consistent elapsed time. A stream that always dies at 30 or 60 seconds is hitting a configured limit. One that dies at random points is a network or provider event. That single observation removes most of the guesswork.
Detect an incomplete stream
The dangerous failure mode is silent: your loop ends, you have text, and nothing raised. A stream that finished properly emits a terminal event. A stream that was cut simply stops.
complete = False
text = []
try:
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
text.append(delta)
if chunk.choices[0].finish_reason is not None:
complete = True # terminal event seen
except (ChunkedEncodingError, ReadTimeout, ConnectionError):
pass # cut mid-flight
if not complete:
raise IncompleteStream("".join(text)) # partial: do not treat as successAnthropic makes this explicit with a message_stop event; OpenAI signals it with a non-null finish_reason and a [DONE] sentinel on the raw SSE endpoint. Track the flag deliberately rather than assuming a clean exit from the loop means a clean finish.
Fix the infrastructure first
Retrying does not help if a proxy will cut the next attempt at the same second. Raise the timeouts on every hop in the path.
location /api/chat {
proxy_pass http://app;
proxy_read_timeout 600s;
proxy_send_timeout 600s;
proxy_buffering off; # without this, SSE arrives in one lump
proxy_cache off;
add_header X-Accel-Buffering no; # do not buffer downstream either
}On the client, set a generous read timeout while keeping the connect timeout short. Those are different settings, and conflating them is a common cause of self-inflicted cuts.
client = OpenAI(timeout=httpx.Timeout(connect=5.0, read=600.0,
write=10.0, pool=5.0))Serverless platforms often impose a hard ceiling you cannot raise. If your generations can exceed it, streaming from a request handler is the wrong architecture: move generation to a worker, write chunks to a store or a pub/sub channel, and have the browser subscribe to that instead.
Resume rather than restart
Restarting a long generation from zero discards everything you already paid for. Because the partial text is valid output, you can feed it back as the beginning of an assistant turn and ask the model to continue.
def resume(messages, partial):
return client.messages.create(
model="claude-sonnet-5",
max_tokens=4096,
messages=messages + [
{"role": "assistant", "content": partial}, # prefill
],
)Two caveats. Resuming re-sends the whole prompt plus the partial answer, so it is not free and it counts against the context window. And for structured output, resuming mid-JSON is fragile; prefer restarting with a smaller task. If the thing cut short was a tool call, see tool call arguments that will not parse.
Retry safely
A dropped connection is a transport failure, so retrying is legitimate, but with the same discipline as any transient error: exponential backoff with jitter and a hard attempt cap. Never retry in a tight loop. A stream that dies after 30 seconds of generation has already cost you 30 seconds of tokens, and an unbounded loop multiplies that bill quickly. The backoff pattern is set out in retrying a 529 without making it worse.
Related errors
If the response ended cleanly but early, it was not a dropped connection: see stop_reason max_tokens. The full map of failures is in the LLM API error reference.
Frequently asked questions
How do I tell a timeout from a provider problem?
Check whether the failure happens at a consistent elapsed time. A stream that always dies at 30 or 60 seconds is hitting a configured limit on a proxy, load balancer or hosting platform. One that dies at unpredictable points is a network or provider event.
How do I detect that a stream was cut short?
Track a completion flag rather than trusting a clean exit from your loop. A stream that finished properly emits a terminal event: a non-null finish_reason on OpenAI, or message_stop on Anthropic. A stream that was cut simply stops, leaving you with partial text and no exception.
Can I resume a cut stream instead of starting again?
For prose, yes. Send the partial text back as the beginning of an assistant turn and ask the model to continue. It re-sends the whole prompt so it is not free, and it is fragile for structured output, where restarting with a smaller task is safer.
