Reference · Glossary
Transformer
Last updated
A **transformer** is a neural network architecture that uses **attention** to weigh how much each token (or patch) should influence every other one. It powers most modern LLMs and many multimodal models.
#When to use
Sequence tasks where context matters: chat, translation, summarization, code, and many vision/language models. Prefer transformers when you need long-range dependencies and parallel training.
#When not to
Ultra-low-latency tiny devices where a smaller specialized model fits better, or problems that are pure tabular regression with few features.
#Mental model
1. Embed tokens as vectors
2. Attention mixes information across positions
3. Stacked layers deepen the representation
4. A head predicts the next token (or class / embedding)
#Failure modes to watch
- Context window limits truncate important evidence
- Attention does not equal “understanding” or truth
- Long prompts raise cost and latency
#Learn next
- Lesson: `transformers-plain`
- Related: attention mechanism, LLM