Calling an LLM API from Python sounds simple: send a list of messages, get text back. Once LLM APIs ship in a product, three questions show up right away. How do users see words appear instead of waiting ten seconds? How do you turn the output into data your code can parse? And how do you keep the end-of-month bill from being a surprise? This article answers them with code that actually ran: streaming, structured output with Pydantic, timeouts, retries and cost control for LLM APIs.
It continues Systematic prompt engineering: roles, examples and prompt testing and How LLMs work: tokens, context, sampling and limitations. Environment: Apple M4, 16 GB, Python 3.12, the openai SDK 1.51.0 and Pydantic 2.9.2. The code is written against the OpenAI SDK but runs on Ollama 0.40.15 (qwen2.5:1.5b) through its compatible endpoint http://localhost:11434/v1, so no real key is needed. Note that json_schema and include_usage on Ollama's /v1 only exist in the December 2024 releases (v0.5.x); on an older Ollama, run those two parts against a commercial API. The authoritative source for how LLM APIs behave is the OpenAI documentation; Ollama is only the vehicle for the experiments.
Calling LLM APIs: one request, one response and the fields to read
The entry point of chat-style LLM APIs is a messages list with the roles system, user and assistant. client.py reads its configuration from environment variables so the key never lives in source code:
import os
from openai import OpenAI
MODEL = os.environ.get("LLM_MODEL", "qwen2.5:1.5b")
def make_client(**kw):
return OpenAI(base_url=os.environ.get("OPENAI_BASE_URL", "http://localhost:11434/v1"),
api_key=os.environ.get("OPENAI_API_KEY", "ollama"), **kw)Changing OPENAI_BASE_URL and the key switches you to another provider; structured output needs extra care when you do (see below). The result of basic.py:
[Plain call]
finish_reason: stop
usage : 56 input tokens, 24 output tokens, total 80
total latency: 0.40sThree fields are worth reading on every call to LLM APIs. finish_reason is stop when the model ended on its own and length when it was cut off at max_tokens; ignore it and a truncated answer becomes a bug that is hard to trace. usage gives input, output and total tokens, the basis of every cost calculation. Measure latency on your side, because it includes the network, the queue and the time LLM APIs spend generating tokens.
Streaming with LLM APIs: return words as soon as the model produces them
The model generates one token at a time (see How LLMs work), so LLM APIs do not have to wait for the full answer before responding. With stream=True, they return a stream of server-sent events, and each chunk carries a delta fragment. Two numbers matter: time-to-first-token (TTFT, the time until the first token) and total time. The parameter stream_options={"include_usage": True} makes the last chunk carry usage:
stream = client.chat.completions.create(model=MODEL, messages=msgs, temperature=0, max_tokens=60,
stream=True, stream_options={"include_usage": True})
first = usage = None
for chunk in stream:
if chunk.usage: usage = chunk.usage # last chunk
if chunk.choices and chunk.choices[0].delta.content:
if first is None: first = time.perf_counter() - t0 # TTFT
text.append(chunk.choices[0].delta.content)[Streaming]
time-to-first-token: 0.02s | total: 0.38s | chunks: 29
usage in last chunk: 56 in, 30 outTwo calls with the same prompt and temperature=0 produced 24 and 30 tokens with different wording: temperature=0 does not guarantee identical output across runs or call modes (batching, floating-point arithmetic). Treat it as "nearly deterministic".
With a short prompt, the first token arrives after 0.02 seconds. But TTFT depends heavily on prompt length, because the model must process the whole prompt first. The next experiment uses a prompt of more than 2,000 tokens (120 review lines), each run with a different prefix so the server's prompt cache is not reused:
[Long prompt, at most 250 output tokens]
read to the end prompt=2394 | chunks=127 | first token after 3.15s | total 5.02s
user clicks stop prompt=None | chunks= 15 | first token after 3.11s | total 3.42sThe 2,394-token prompt pushed TTFT past 3 seconds, and generating 127 tokens took about 1.9 seconds more: with a long prompt, most of the wait is prompt processing. Streaming with LLM APIs also allows early cancellation: when the user clicks "Stop" after 15 chunks, we call stream.close() and finish at 3.42 seconds instead of 5.02. The usage in the last chunk never arrives if you close the stream early (the prompt=None column), so count what you received yourself.
Keep three things in mind when running LLM APIs in production. The openai-python README says streams are not retried automatically, so if the connection drops midway you must handle the part already received. Proxies must turn buffering off (nginx: proxy_buffering off). Parse JSON only after you have received all of it.
Structured output from LLM APIs: the output is data, not prose
Applications usually need LLM APIs to return data for code, for example to classify intent and extract an order ID. There are three guarantee levels, each stronger than the last. Level one: ask for JSON in the prompt, with no guarantee at all. Level two: JSON mode (response_format={"type": "json_object"}), which guarantees valid JSON but not the right keys or types. Level three: Structured Outputs with json_schema. The OpenAI Structured Outputs guide states the difference concisely: both guarantee valid JSON, but only Structured Outputs guarantees schema adherence. The feature launched in August 2024 with gpt-4o-2024-08-06, uses strict: true and adds a refusal field for when the model declines.
Strict mode has conditions: per the docs, every object must set additionalProperties: false and every field must be listed in required; optional fields are declared as a union with null. The Pydantic class below meets those conditions thanks to extra="forbid" and an order_id field with no default value (so it stays in required but allows null):
class Ticket(BaseModel):
model_config = ConfigDict(extra="forbid") # additionalProperties: false
intent: Literal["delivery", "refund", "payment", "other"]
order_id: str | None = Field(description="Order ID if present, otherwise null")
urgent: bool
schema = {"type": "json_schema",
"json_schema": {"name": "ticket", "strict": True, "schema": Ticket.model_json_schema()}}
r = client.chat.completions.create(model=MODEL, temperature=0, response_format=schema, messages=msgs)
ticket = Ticket.model_validate_json(r.choices[0].message.content) # re-validate with the same classThe schema printed by model_json_schema() has "additionalProperties": false and a required list with all three keys. One class both generates the schema you send and validates the result you get back. With real LLM APIs there is a simpler route, the SDK helper client.beta.chat.completions.parse(response_format=Ticket) (documented in openai-python's helpers.md), which converts the Pydantic class to a strict schema, parses the reply and returns the object in message.parsed. Sending the schema of a class without extra="forbid" to a commercial API gets rejected for missing additionalProperties: false; Ollama is more lenient and does not catch this error.
The experiment sends 10 Vietnamese messages through three modes at temperature=0 with two prompt styles. The "vague" prompt only says "return JSON with the keys intent, order_id, urgent" and deliberately does not list the labels. The "complete" prompt lists the four labels, says order_id is null when there is no ID, and urgent is true when the customer says it is urgent. Pydantic validates every output:
vague prompt prompt_only valid JSON 10/10 | schema-correct 0/10
vague prompt json_mode valid JSON 10/10 | schema-correct 0/10
vague prompt json_schema valid JSON 10/10 | schema-correct 10/10 | intent 6, order_id 6, urgent 10
complete prompt prompt_only valid JSON 1/10 | schema-correct 1/10 | intent 1, order_id 1, urgent 1
complete prompt json_mode valid JSON 10/10 | schema-correct 10/10 | intent 8, order_id 10, urgent 10
complete prompt json_schema valid JSON 10/10 | schema-correct 10/10 | intent 8, order_id 10, urgent 10Read this carefully (the intent, order_id and urgent columns count the messages with correct content among the schema-correct ones). With the vague prompt, the model invented its own label "request_delivery", so 0/10 schema-correct in the two unconstrained modes; that is a consequence of a prompt missing information, and does not prove JSON mode is "worse". Only json_schema forces the labels through constrained decoding, but the model cannot see the schema, so it guesses the labels: intent was right 6/10. Once the prompt lists the labels, JSON mode reaches 10/10 schema-correct, on par with json_schema. A prompt that only asks for JSON in words returned pure JSON just 1/10 times: checking the 10 messages by hand showed 6 replies wrapped in a ```json block and 3 written as prose. So json_schema guarantees the shape; correctness is still decided by the prompt.
Schema-correct output from LLM APIs is not the same as true. With the vague prompt, the model once filled in order_id='123456' for a message with no order ID, and '' instead of null in another. With the complete prompt, intent was still wrong on two messages, for example "card charged twice" was labeled refund; the output of the parse helper:
[SDK parse helper, complete prompt]
intent='refund' order_id='55102' urgent=True | expected: ('payment', '55102', True)Three lessons. (1) Structured output solves shape, not correctness; you still need a test set like the one in the previous article. (2) Allow null, say in the prompt when to use it, and verify with code (the order ID matches a regular expression or exists in the database). (3) Use Literal to close the set of values and list them in the prompt. With a commercial API, check refusal and finish_reason before parsing; the schema supports only a subset of JSON Schema.
Timeouts and retries when calling LLM APIs
Every one of the LLM APIs is a network service, so it will sometimes be slow, overloaded or failing. According to the openai-python README, the SDK retries twice by default with exponential backoff on connection errors, 408, 409, 429 and 5xx, and the default timeout is 10 minutes; both are tuned with max_retries and timeout. We check this with a fake server built on httpx.MockTransport (no network needed), counting how many requests are actually sent:
def make_fake(statuses):
calls = {"n": 0}
def handler(request):
calls["n"] += 1
s = statuses[min(calls["n"] - 1, len(statuses) - 1)]
if s == 200: return httpx.Response(200, json=OK) # OK: a fake chat.completion JSON
return httpx.Response(s, json={"error": {"message": "simulated"}}, headers={"retry-after-ms": "10"})
return OpenAI(api_key="fake", base_url="http://fake.local/v1", max_retries=2,
http_client=httpx.Client(transport=httpx.MockTransport(handler))), calls[Timeout 5 ms, max_retries=0] APITimeoutError after 0.03s
[SDK default retry, max_retries=2, fake server]
429, 429, 200 -> success requests actually sent: 3
429 forever -> RateLimitError (429) requests actually sent: 3
400 (client error) -> BadRequestError (400) requests actually sent: 1When the server returns 429 twice and then 200, the SDK retries on its own and succeeds after 3 requests; with endless 429s it raises RateLimitError only after 3 requests (1 original, 2 retries); a 400 sends just 1 request because it is a client-side error (so is 401, see the full list in reliability_output.txt). The fake server sends the header retry-after-ms: 10 (a non-standard extension the SDK reads before the seconds-based Retry-After), so the wait is only about 10 ms; in real life the backoff is much longer. Rules for calling LLM APIs: set timeout to match your SLA instead of 10 minutes; retries multiply cost and load; honor Retry-After and add jitter; always set max_tokens.
Cost control when using LLM APIs
Most LLM APIs charge by token, with separate input and output prices. Cost is a function of prompt length (including examples and history resent on every turn), output length and the number of calls. The price table in this article is an assumption (0.50 and 1.50 USD per million input and output tokens), for illustration only; replace it with current pricing.
Count tokens before calling LLM APIs
To block prompts that are too long, you must estimate before sending. tiktoken does this for OpenAI models; for other models it is only an approximation because tokenizers differ. We compare the cl100k_base estimate with the real usage of Qwen2.5:
[tiktoken cl100k_base vs real Qwen2.5 usage]
0 examples: estimate 95 | real 74 | output tokens 5
4 examples: estimate 204 | real 165 | output tokens 5For LLM APIs, the estimate runs 24 to 28% above reality on Vietnamese with this tokenizer pair; other pairs will deviate differently. It is fine as a blocking threshold, not for billing: the amount you pay comes from the usage the API itself reports. The table also shows the price of few-shot prompting: four examples take the prompt from 74 to 165 tokens.
LLM API cost scales with volume
At 100,000 messages per day:
[Cost at 100,000 calls/day, ASSUMED price 0.5/1.5 USD per million input/output tokens]
0 examples: 74 input tokens, 5 output -> 4.45 USD/day
2 examples: 124 input tokens, 3 output -> 6.65 USD/day
4 examples: 165 input tokens, 5 output -> 9.00 USD/dayGoing from 0 to 4 examples doubles the daily cost, from 4.45 to 9.00 USD. If those four examples raise accuracy, they may still be worth it; what matters is that you can calculate it. Levers for cutting the cost of LLM APIs: shorten the system prompt, cap the history you resend, use a small model for easy tasks, set max_tokens, and use the provider's prompt caching where available.
Cache LLM API requests by content
Many requests repeat exactly, such as frequently asked questions. The Cached class hashes (model, messages, parameters) with SHA-256 and calls LLM APIs only when no result is stored. A cache means "same input, reuse the saved result"; it does not promise the model would always give the same answer, and it only fits when the answer does not depend on data that changes over time:
class Cached:
def __init__(self): self.store = {}; self.hits = self.calls = self.tokens = 0
def ask(self, messages):
key = hashlib.sha256(json.dumps([MODEL, messages, 0], ensure_ascii=False, sort_keys=True).encode()).hexdigest()
if key in self.store: self.hits += 1; return self.store[key]
r = client.chat.completions.create(model=MODEL, messages=messages, temperature=0, max_tokens=8)
self.calls += 1; self.tokens += r.usage.total_tokens
self.store[key] = r.choices[0].message.content.strip(); return self.store[key][Cache: 40 requests, only 3 distinct questions]
real API calls: 3 | cache hits: 37 | tokens consumed: 447Out of 40 requests with only 3 distinct questions, we made 3 real API calls and hit the cache 37 times. Set an expiry on the cache and never store personal data in a shared cache.
A hard budget for LLM APIs
The Budget class blocks calls to LLM APIs when the estimated total tokens exceed the limit, and adds the real usage after each call:
class Budget:
def __init__(self, max_tokens): self.max, self.used = max_tokens, 0
def charge(self, usage): self.used += usage.total_tokens
def check(self, est):
if self.used + est > self.max:
raise RuntimeError(f"budget exceeded: used {self.used}, need ~{est} more, limit {self.max}")[Budget 1,000 tokens] sent 6 requests then stopped: budget exceeded: used 907, need ~191 more, limit 1000With LLM APIs in practice, set the budget per customer or feature, store it in Redis or a database, and alert at 80%. A budget also guards against error loops (an agent calling itself thousands of times) and abuse.
Security when calling LLM APIs
Keep LLM API keys only in environment variables or a secret store: never commit them and never put them in browser-side code. Mask phone numbers, card numbers and addresses before they go into a prompt. Logs should record only the model, token counts, latency and finish_reason.
When not to call LLM APIs
Tasks with clear rules that LLM APIs would only complicate (validating an order ID format, looking up a price table) are cheaper and more deterministic in plain code; data that is too sensitive should go to an in-house model such as Ollama.
Common mistakes with LLM APIs
- Not reading
finish_reason: an answer cut off bylengthis treated as complete. - Sending a schema without the strict conditions: forgetting
additionalProperties: falseor leaving a field out ofrequiredgets it rejected; useextra="forbid"or theparsehelper. - Blaming JSON mode when the prompt lacks information: without the labels, the model has to guess in any mode; a complete prompt moved JSON mode from 0/10 to 10/10 schema-correct.
- Assuming a correct schema means correct content: the model fills in fields even when there is no information; allow
nulland verify with code. - Leaving the LLM API timeout at the 10-minute default: the user left long ago while the worker is still held.
- Using token estimates to bill LLM APIs: the error can be large, such as 24 to 28% for Vietnamese in this experiment.
Checklist
- Configure the LLM API client through environment variables; set
timeout,max_retriesandmax_tokensexplicitly. - Read
finish_reasonandusageon every LLM API call. - Use LLM API streaming for user interfaces; measure TTFT and total time; support cancellation.
- Use
json_schemawith a Pydantic type (extra="forbid", enums, allownull), list the values in the prompt and verify content with code. - Count tokens before sending, set a budget per customer or feature, and cache repeated requests.
The code blocks are shortened versions of six scripts that really ran (client.py, basic.py, stream_demo.py, structured.py, reliability.py, cost.py) with openai==1.51.0, httpx==0.27.2, pydantic==2.9.2, tiktoken==0.7.0. The output is real output (a few lines trimmed for brevity); latency and token counts vary slightly between runs. The data is simulated.
Next article
Running a local LLM with Ollama and llama.cpp: model choice, quantization and performance. This article already used Ollama as an OpenAI-compatible server for LLM APIs; the next one goes inside: choosing a model, quantization and measuring performance when you run the LLM yourself.
