Add observability and tests

Emit one structured record per run plus focused events for external attempts. Useful fields include operation, request ID, duration, attempt or step count, terminal status, model/tool name, and token or byte totals when available. Do not use model text or raw prompts as metric labels.

Test at three levels: pure validators and classifiers, adapter behavior with a scripted fake, and one opt-in smoke test against the configured service. Quality evals need stable fixtures and task-specific assertions rather than exact matching of naturally variable prose.

The primary eval is: registry contract tests exercise valid input, invalid input, timeout, and cancellation. Add a fake clock and scripted adapter so timing and attempts are deterministic. Unit-test the schema refinements and failure classifier. Integration-test the orchestration with realistic wire-shaped fixtures. Keep one credentialed smoke test behind RUN_LIVE_EVALS=1; ordinary CI must not require paid services.

Emit counters for started, succeeded, rejected, retried, cancelled, and exhausted runs. Track p50/p95 duration and work units such as bytes, tokens, tool calls, or steps. Attach a correlation ID across boundaries. For quality, save expected properties, not brittle exact prose. A regression should tell the team whether transport, contract, policy, or task quality changed.

it("satisfies the fixture contract", async () => {
  const result = await run(fixture.input, fakeDependencies(fixture));
  expect(result.value).toMatchObject(fixture.expected);
  expect(result.durationMs).toBeGreaterThanOrEqual(0);
});

it("rejects malformed external data", async () => {
  await expect(run(badInput, malformedDependencies())).rejects.toMatchObject({
    code: "invalid_output"
  });
});

The code is intentionally provider-neutral at the adapter boundary. AI_BASE_URL and model names are environment placeholders, not invented services. If the topic uses MCP, install the current official SDK and verify its exported paths against its release documentation. If a provider offers an official SDK, it can replace fetch inside execute without changing the domain contract. Always inspect the real provider’s documented request and response fields before connecting credentials.

Debug from deterministic code outward. First print the parsed configuration with secret values replaced by [set]. Next run the parser against a local fixture. Then run the orchestration with a fake dependency. Only after those pass should you inspect the real boundary. Capture status, content type, request ID, elapsed time, and a redacted response shape. Do not “fix” malformed data with a permissive cast; save it as a failing fixture and decide whether the contract or adapter is wrong.

For this chapter, the fastest diagnostic is to assert that tool descriptions, validators, and implementations agree on names and shapes. When that assertion fails, stop before downstream work. Expected behavior is concrete: calculate({expression:"6*7"}) returns 42; an unknown tool is rejected without execution. A different but plausible sentence is not enough if the structural and policy checks fail.

• Pure, integration, and opt-in live tests are separated.
• Logs use correlation IDs and omit sensitive bodies.
• A task-specific quality assertion protects behavior.
• Run npm run typecheck and npm test without provider credentials.
• Record expected terminal behavior beside the fixture so future changes remain reviewable.

A passing test is necessary but does not establish production quality. Review the fixture set for realistic distributions, stale assumptions, tenant boundaries, and expensive edge cases. When outputs are probabilistic, assert schemas, citations, chosen capabilities, stop reasons, and task rubric scores. Reserve exact string assertions for deterministic adapters and protocol framing.