When fluency fails

Next-token training rewards prediction, not direct verification against reality. Many true statements are likely continuations, so prediction learns substantial factual structure. But the objective also rewards language that fits a pattern. When evidence is missing, a fabricated citation can look more like the expected answer than “I do not know.”

Several conditions raise the risk:

• The prompt assumes a false premise.
• The requested fact is rare, ambiguous, private, or newer than training.
• The answer demands exact names, dates, quotations, or calculations.
• Relevant context is absent, buried, or contradictory.
• The prompt pressures the model to answer instead of abstain.
• Earlier generated text introduced an error that later tokens continue.

Hallucination is not fixed by telling the model to “be accurate.” Clear uncertainty rules, supplied sources, retrieval, tools, and output verification can reduce risk. None removes it universally.

An LLM’s weights reflect patterns available during training and post-training. They do not automatically update when the world changes. Even if a product connects the model to search, the model must select, interpret, and cite useful results.

The model also lacks automatic access to:

• your private files or accounts;
• live prices, weather, and inventory;
• events after its knowledge cutoff;
• the exact provenance of every learned association;
• hidden organizational policies;
• facts never represented adequately in its training or context.

Products can provide some of these through retrieval and tools. Always distinguish “the model knows” from “the application supplied.” A confident answer claiming “I found this in my training data” is not reliable source tracing.

False facts are only one failure type. LLMs may also:

• follow irrelevant patterns or misunderstand instructions;
• lose track of details in long contexts;
• reproduce social biases present in data and feedback;
• produce unstable answers when wording changes;
• make arithmetic, counting, or logical errors;
• expose sensitive input if an application handles data poorly;
• accept malicious instructions embedded in retrieved text;
• generate insecure or nonexistent code APIs;
• over-agree with a user’s mistaken assumption.

Some risks belong to the surrounding system rather than the base model. A secure model endpoint can still be placed in an application that logs secrets. A capable model can still receive stale documents. Diagnose the full pipeline: input handling, retrieval, model, decoding, tools, validation, and user interface.

People naturally use tone as a trust signal. LLMs weaken that signal because fluent wording is their core product. Hedging can accompany a correct answer, while certainty can accompany a false one.

Token probabilities are not a simple truth meter. A model can assign high probability to a familiar but false sentence. Self-reported confidence is generated text too. Asking “Are you sure?” may prompt a correction, but it may also produce a more elaborate defense.

Better evidence comes from outside the claim:

• a quotation from an authoritative supplied source;
• a calculator result;
• executable tests for code;
• agreement with a maintained database;
• independent review by a qualified person;
• repeated task-level evaluation on labeled examples.

Match verification effort to consequences. A playful name suggestion needs little checking. A dosage, legal deadline, payment, or security change requires authoritative validation and accountable human judgment.

A practical reliability pattern has three parts.

Ground the answer in appropriate evidence. Supply the relevant document, retrieve current records, or call a suitable tool. Ask the model to distinguish source content from its own synthesis.

Constrain the task. Define allowed sources, output schema, uncertainty behavior, and forbidden actions. Request not found when evidence is missing. Keep untrusted retrieved text separate from instructions.

Verify the result. Check citations against passages, validate data types and ranges, run generated code in a sandbox, compare calculations, and route consequential decisions to a person.

For example:

Using only the policy excerpts below, answer the question and cite the excerpt number. If the excerpts do not contain the answer, say “Not supported by the supplied policy.”

This prompt cannot guarantee compliance, but it creates an answer that is easier to inspect and a safer fallback than forced improvisation.

An LLM is a poor default when a deterministic method already solves the task reliably. Use a database query for an account balance, a calculator for exact arithmetic, a schema validator for syntax, and established business rules for fixed eligibility criteria.

An LLM may still provide a language interface around those systems, but it should not invent their outputs. It can translate “How much do I owe?” into a permitted account query, then explain the returned value. The authoritative value comes from the system of record.

Avoid autonomous deployment where errors could cause severe harm and cannot be detected or reversed. If uncertainty cannot be bounded, use the model for drafting or recommendation while preserving human control.