Streaming
Set stream: true and the reply arrives in small pieces as it is generated, instead of all at once at the end.
Last updated
Analogy
Server-sent events for text. The ChatGPT typing effect isn't theatre; it's the response arriving chunk by chunk.
Code example
const stream = await client.chat.completions.create({ model: "gpt-4o-mini", messages, stream: true,}); for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content ?? "");}In an agent
Agents can be slow. Streaming the final answer (and step updates) lets the user see progress immediately.
Practice
Flashcards
- What is streaming?Sending tokens to the UI as they are generated, instead of all at the end.
Interview questions
- How do you reduce cost and latency?
- Small models for routing and simple steps; big models only for hard reasoning.
- Caching and short tool results.
- Parallel tool calls and streaming.
- Track cost per request from usage tokens.
Questions people ask
How do I stream an OpenAI response in Node?
Pass stream: true to chat.completions.create, then loop over the result with for await. Each chunk carries a small piece of text in choices[0].delta.content, which may be empty. Write each piece to the page or terminal as it arrives, and the user sees the answer being typed out.
Why should agents stream their answers?
Agents can take many seconds, because they call tools and the model several times. Streaming the final answer, and short status updates for each step, shows the user progress straight away instead of a frozen screen. The total time is the same, but it feels much faster and users trust it more.