How generation unfolds

At one generation step, the model outputs a number called a logit for every token in its vocabulary. Larger logits indicate more favored continuations in the current context. A mathematical function called softmax converts them into probabilities that add to 100 percent.

For a tiny imaginary vocabulary, the distribution might be:

• cereal: 70%
• oats: 20%
• coffee: 7%
• keyboard: 3%

Real vocabularies contain many thousands of tokens, and the distribution changes at every position. After choosing cereal, the next likely tokens may include a period, before, or and. If the system instead chooses oats, a different path opens.

These probabilities describe the model’s next-token distribution, not calibrated confidence that a whole answer is factually correct. A high-probability phrase may repeat a common misconception. A low-probability phrase might be the precise specialist answer.

A decoding strategy turns the distribution into a token. Common approaches include:

• Greedy decoding: always choose the highest-probability token. This is predictable but can become repetitive or miss a better sequence whose first token ranked second.
• Sampling: randomly choose according to probabilities. Likely tokens win more often, but alternatives remain possible.
• Top-p sampling: keep the smallest set of tokens whose combined probability reaches a threshold, then sample from that set. This can remove a long tail of unlikely choices.

Some products also use top-k limits, repetition penalties, stop sequences, constrained decoding, or search methods. The interface may expose only a few controls even when the serving system applies several.

Deterministic settings improve reproducibility but do not make an output truthful. They merely make the same likely path easier to repeat. System updates, numerical differences, or provider changes can still alter results.

Temperature changes how sharp or flat the distribution is before sampling. Lower temperature concentrates probability on already likely tokens. Higher temperature spreads probability more broadly.

Using our imaginary breakfast example:

• At a low temperature, cereal might dominate, producing stable conventional wording.
• At a medium temperature, oats and other sensible choices appear more often.
• At a high temperature, unusual choices gain enough probability to create surprising or incoherent text.

Temperature does not add knowledge, creativity, or reasoning capacity. It changes selection among possibilities the model already scores. Low temperature is often useful for extraction, classification, and strict formats. Moderate variation may help brainstorming or stylistic alternatives. Extremely high settings can damage coherence.

Not every API defines the scale identically, and some advanced models manage sampling internally. Treat temperature as an empirical control: test it on the actual model and task.

Generation is autoregressive: each selected output token joins the context used to predict the next one. Consider:

1. Prompt: The capital of France is
2. Select: Paris
3. New context: The capital of France is Paris
4. Select: .
5. New context: The capital of France is Paris.

An early choice can steer the rest of the answer. If the model starts a numbered list, list markers become likely. If it invents a wrong name in the first sentence, later tokens may build a consistent story around that name. Generation does not normally pause to revise earlier text unless the application explicitly asks for critique, editing, or another pass.

This path dependence is why prompts that establish constraints early can help. It is also why requesting a structured outline before a long answer sometimes improves organization: the outline becomes context for later tokens.

The context window is the maximum amount of tokenized material a model can process in one request, including instructions, examples, retrieved documents, conversation history, and often the generated output. When content exceeds the limit, an application must reject it, truncate it, summarize it, or choose what to retain.

A large context window is useful, but capacity is not the same as attention quality. A model can miss one sentence inside a huge document or confuse similar passages. Placing critical instructions clearly, removing irrelevant material, and evaluating long-context tasks remain important.

The model also does not carry an unlimited private scratchpad across conversations. If an application appears to remember you, the product may be storing and reinserting selected information. That product memory is separate from the model’s one-step generation mechanism.

Generation needs an end condition. It may stop after:

• producing a special end token;
• matching a configured stop sequence;
• reaching a maximum output-token limit;
• completing a constrained data structure;
• being interrupted by the user or application.

A cutoff mid-sentence may mean the output limit was too low, not that the model “forgot.” For machine-readable tasks, schemas or constrained decoding can limit which tokens are valid next. This greatly improves syntax, but valid JSON can still contain wrong values. Structure and truth require different checks.

Streaming interfaces display tokens or chunks as they arrive. Streaming feels faster because the user sees progress, although the same sequential generation is occurring underneath.