Production AI Architecture

SystemAroundModelSim turned layers on and off — gateway, cache, retrieval, guardrails, monitor, fallback — and showed what breaks when the “architecture around the model” is missing.

A common request path is:

1. Application receives the user’s goal and identity
2. Policy layer checks permissions, risk, and input limits
3. Orchestrator chooses a model, prompt, tools, or retrieval path
4. Data layer fetches approved context and records source versions
5. Model service generates a structured response
6. Validator checks format, citations, and safety rules
7. Application returns the answer or asks a human to review
8. Observability layer records quality, latency, cost, and failures

Keep components replaceable. A model gateway can route a simple request to a fast model and a hard request to a stronger one. Caches, queues, retries, and timeouts help with traffic and provider failures.

Design failure behavior on purpose: show a clear error, use a safe fallback, or hand off to a person. “Try forever” is not a production strategy.

• Turning a successful AI demo into a customer-facing product
• Combining RAG, tools, and multiple model providers
• Meeting privacy, reliability, or audit requirements
• Finding which component caused a bad answer

Every added layer has a cost. Start with the smallest architecture that meets the risk, then add complexity because a measured need demands it.

Never let the model be the sole authority for access control, payments, deletion, or other irreversible decisions. Enforce those rules in ordinary software too.

Use the Try yourself RAG pipeline above. Place authentication before retrieval, validation after generation, and monitoring around the whole path.

Draw the request path for one AI feature you use. Circle where it could fail and write the fallback beside each circle.