What consumes the context budget

The context window is the maximum token sequence a model can process for a request under its documented rules. Think of it as a workbench, not permanent memory. Everything placed on the bench competes for space.

A simplified constraint is:

system and developer instructions
+ conversation history
+ current user message
+ tool definitions and tool results
+ retrieved documents
+ generated output
<= available context

Some APIs specify input and output limits separately; others describe a combined limit with additional caps. The exact rule is model-specific. Read the documentation and test boundary behavior. “128K context” is not enough information to design a reliable 128K input.

More context is also not automatically better. Irrelevant text can distract the model, bury important evidence, increase latency, and raise cost. The useful goal is enough high-quality context for the task with explicit room for the answer.

System and developer instructions may define the assistant’s role, policies, response format, and safety boundaries. They are often invisible to the end user but still occupy tokens. Repeating the same rule in several layers wastes budget and can create conflicts. Keep stable instructions clear, consolidated, and tested.

Chat history grows on every turn if the application resends the full conversation. A ten-turn exchange is not priced as ten isolated user messages: later requests may include earlier questions and answers again. A verbose assistant response can therefore increase both its current output cost and future input cost.

Choose a history policy. Recent turns may remain verbatim; older turns can be summarized; durable facts can move into structured state; irrelevant branches can be dropped. Summaries must preserve commitments, unresolved questions, names, numbers, and user preferences needed later. A summary that says “discussed budget” is not a substitute for “approved ceiling is $4,500 before tax.”

Tool-capable models need descriptions of available operations, argument schemas, and sometimes examples. Twenty elaborate tool definitions can consume substantial input before the user asks anything. Give the model only tools relevant to the current route when your framework supports that safely.

Tool results also re-enter context. A database query returning 10,000 rows should not be pasted wholesale if the task needs three aggregates. Perform deterministic filtering or aggregation in code, then provide the result plus enough provenance to verify it. Preserve exact records when exact comparison is required.

Agent loops multiply this effect. Each step may include prior tool calls and results. Set limits on steps, result size, and repeated failures. A context budget is not only a document limit; it is an operational control against loops that grow until they fail.

Retrieval-augmented generation, or RAG, selects external passages and inserts them into the request. Retrieval does not give the model infinite memory. Every selected chunk uses tokens.

Suppose retrieval returns twelve chunks of 700 tokens:

12 × 700 = 8,400 tokens

If only four chunks are strongly relevant, sending all twelve adds 5,600 tokens of weak evidence. Improve retrieval, rerank results, remove duplicate overlap, or adapt the number of chunks to the query. Keep source labels and boundaries so the model can cite and distinguish evidence.

The best chunk count is not a universal “top three” or “top five.” A narrow fact lookup may need one passage; a comparison across policies may need several. Evaluate whether the answer is supported, not merely whether the request fits.

Assume a teaching scenario with a 16,000-token combined allowance and a desired maximum output of 1,500 tokens. An initial budget could be:

Instructions and wrappers       1,100
Recent history                  2,400
Current user request              500
Tool definitions and results    1,500
Retrieved evidence              8,000
Output reservation              1,500
Operational headroom            1,000
Total                          16,000

Headroom covers count variance, conditional tools, and formatting. It is not permission to ignore measurement. If actual wrappers use 1,400 rather than 1,100 tokens, reduce another category before launch.

When over budget, prioritize deliberately: preserve governing instructions, the current request, and evidence necessary for correctness. Summarize older history, select retrieval more carefully, compact tool results, and remove duplication. Do not silently truncate the end of a policy or the middle of a code file; arbitrary cuts can remove the exact condition needed to answer.