AnyoneLearnAI
Start learning

Chapter D · 8 pages · ~112 min

Local model lab

Serve an open-weight model locally with Ollama, measure tokens/sec and memory, and expose a policy-controlled cloud fallback when the machine cannot meet the envelope.

Start the first lesson →Chapter index

What you will be able to do

  • Explain local inference tradeoffs for privacy, latency, and hardware limits
  • Implement a local chat adapter with timeouts and a fallback policy
  • Evaluate quality, tokens/sec, and memory against a small golden set
  • Ship a router that prefers local models and fails over under explicit policy

Lessons in this topic

  1. 01Frame the local Ollama router experiment14m
  2. 02Define model tags, timeouts, and fallback policy14m
  3. 03Build the first working local chat adapter14m
  4. 04Measure quality, speed, and memory envelope14m
  5. 05Debug hangs, OOMs, and accidental cloud calls14m
  6. 06Instrument tokens/sec and fallback reasons14m
  7. 07Set privacy and network egress boundaries14m
  8. 08Ship and explain the local-first router14m