Instrument the tool registry runtime

Instrument the calculator + knowledge lookup tool registry so a run records enough structure to debug offline: counts, latency if relevant, pass/fail of allowlist reject count, and a stable stage name. Redact secrets and raw credentials from every event.

const event={stage:'tool-registry', unknown_rejects:1, schema_fails:0, ok:1};
console.log(JSON.stringify(event));

Expected evidence: registry telemetry. Prefer JSON or structured text you can grep in CI over prose logs for tools-in-code.

Turn one historical failure—especially string-eval calculator—into a test that fails if the signal disappears for the tool registry runtime. Observability without a failing test is optional decoration; observability with a test is part of the tools-in-code artifact.

Draft a three-field event for the tool registry runtime: stage, ok, and one domain field derived from allowlist reject count; schema validation failures; successful typed results. Add fixture_id or docs_version when content can change. Explicitly list fields that must never appear (tokens, passwords, raw prompts) because registering shell-exec as a 'helper' tool is in scope for this lab.

Wire one assertion that fails if the tool registry runtime event is missing after a run. Observability that cannot fail a test will not survive contact with a busy tools-in-code repository.

Imagine a teammate opens only your event stream after a bad deploy. Could they tell whether registry with calc and kb.lookup was wrong, whether string-eval calculator, or lookup that returns entire DB on missing id returned, or whether registering shell-exec as a 'helper' tool slipped through? If not, rename fields until those three stories are distinguishable.

At the testing and observability stage for tools-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about registry with calc and kb.lookup that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: inline if/else tool routing without schemas.

For this page specifically, success looks like a structured event schema locked by a test while still centering the user decision to route model tool requests through a registry that validates names and args. If you cannot point to a file, command, or assertion that proves that for the tool registry runtime, stay on this page instead of advancing.

Glossary: tool · Glossary: structured output · Cheatsheet: production ops signals

Previous · Next