Reference · Glossary
KV cache
Explaining why long contexts cost more and why caching prefixes helps latency.
When to use
Explaining why long contexts cost more and why caching prefixes helps latency.
When not to
As a user-facing product feature name — say “prompt caching” instead.
Example
After the system prompt is processed once, later tokens in the same session reuse cached KV blocks.