Ship and explain the tool registry runtime

A shippable lab artifact includes: how to run it, the metric result (allowlist reject count; schema validation failures; successful typed results), the failure you can still reproduce (string-eval calculator, or lookup that returns entire DB on missing id), the security gate for registering shell-exec as a 'helper' tool, and a rollback note. The user decision it supports remains: route model tool requests through a registry that validates names and args.

RUN_LIVE_EVALS=1 npm run test:live

Expected evidence: **duplicate names, stale schemas, timeouts, oversized results, and unsafe side effects**. Store this beside the fixture version so scores remain meaningful after content changes in `tools-in-code`.

State operating limits for the tool registry runtime in plain language: fixture size, offline vs live dependencies, and what would require a new eval set. Shipping tools-in-code is honest scoping, not maximal confidence language.

Fill this table in your notes for the tool registry runtime:

• Proved on registry with calc and kb.lookup: …
• Unproved beyond the fixture: …
• Metric that blocks release: allowlist reject count; schema validation failures; successful typed results
• Failure still reproducible: string-eval calculator, or lookup that returns entire DB on missing id
• Security gate: registering shell-exec as a 'helper' tool
• Rollback: …

Ship the narrative only when the unproved list is honest. Reviewers trust narrow claims that support route model tool requests through a registry that validates names and args more than maximal language that collapses under the first production oddity.

Hand your ship note to a peer and ask them to recreate a proved/unproved ship note with rollback without watching you type. If they cannot, your evidence is still tribal knowledge. Tighten the run command and the metric line until a stranger can validate the tool registry runtime against registry with calc and kb.lookup.

At the mastery and shipping stage for tools-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about registry with calc and kb.lookup that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: inline if/else tool routing without schemas.

For this page specifically, success looks like a proved/unproved ship note with rollback while still centering the user decision to route model tool requests through a registry that validates names and args. If you cannot point to a file, command, or assertion that proves that for the tool registry runtime, stay on this page instead of advancing.

Glossary: tool · Glossary: structured output · Cheatsheet: production ops signals

Previous