Reference · Glossary
Calculus and gradients
Last updated
A **gradient** is the direction and steepness of a loss function's slope — training nudges model weights downhill along that gradient to reduce error.
#When to use
Understanding how neural nets and LLMs learn during fine-tuning, why learning rates matter, and why "loss going down" means the optimizer found a useful direction.
#When not to
Day-to-day prompt engineering or RAG retrieval tuning — you rarely compute gradients by hand in application code.
#Example
Loss = (prediction - target)²
Gradient points toward increasing loss
Optimizer step: weights -= learning_rate × gradient
Repeat until loss plateaus**Partial derivatives** tell each weight how much it contributed to the error.