Prompt caching charges repeated input at a fraction of the normal rate, and for any workload with a long fixed system prompt it is the single largest cost reduction available....
Four things drive an LLM bill, and the price per token is the least interesting of them. Resent conversation history, output tokens costing several times more than input, retries, and...
Vision requests fail for two unrelated reasons that look similar: the HTTP request body is physically too big, or the images consume more of the context window than you have...
A stream that dies mid-response is usually killed by something between you and the provider, not by the model. Proxies, load balancers and serverless platforms enforce idle and total-duration timeouts...
When a tool call arrives with arguments you cannot parse, the model is almost never inventing malformed JSON at random. Three causes account for nearly all of it: the arguments...
An insufficient_quota error is a billing problem wearing a rate-limit costume. It arrives as HTTP 429, the same status as an ordinary rate limit, but it means your account has...
Usually the JSON is not invalid, it is incomplete or wrapped in prose. Regex repair is a symptom of not using constrained output.
If the reply stops mid-sentence the model did not fail, it hit your output ceiling. The response is a 200, so most code never notices.
A 529 is not your fault and there is nothing in the request to fix. It is retryable, but retrying badly makes an overloaded service worse.
A 401 is almost never a wrong key. It is usually a stale environment variable, whitespace, the wrong header, the wrong organisation, or a rotated key.