Measure whether the research-critique coordinator works

Translate the claim into assertions or a tiny eval harness. The metric to protect is: veto rate; unsupported claim count after merge; total step budget. Always record the denominator (how many cases) beside any rate. A percentage without a denominator is marketing, not measurement.

const parsed = OutputSchema.safeParse(raw);
if (!parsed.success) {
  throw new ContractError("invalid_output", parsed.error.issues.map(i => ({
    path: i.path.join("."), code: i.code
  })));
}
return parsed.data;

Expected evidence: Research returns two sourced claims; critique rejects one unsupported claim; synthesis keeps one.. A passing assertion proves only the behavior it names; broader usefulness still needs the chapter’s full limits.

Be explicit: beating the baseline (single agent without critique pass) on this fixture does not prove behavior under workers sharing mutable state, or coordinator ignoring critique veto. Label observations separately from conclusions so the next page inherits honest evidence about the research-critique coordinator.

Compute veto rate; unsupported claim count after merge; total step budget with the denominator written beside the rate every time. For this chapter, the evaluation set is intentionally tiny; that is allowed only if you say so in the evidence. Compare against single agent without critique pass before celebrating.

Add one negative case aimed at workers sharing mutable state, or coordinator ignoring critique veto. A suite with only happy cases cannot protect the research-critique coordinator when the characteristic failure appears in review.

If a check is expensive or flaky, shrink it until it is deterministic on two workers with separate step budgets. Flaky green builds teach the team to ignore gates. Record what this page does not prove so security-ops and mastery-ship inherit honest limits.

At the evaluation stage for multi-agent-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about two workers with separate step budgets that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: single agent without critique pass.

For this page specifically, success looks like metrics with explicit denominators and a negative case while still centering the user decision to delegate research and critique to two bounded workers, then merge under a coordinator policy. If you cannot point to a file, command, or assertion that proves that for the research-critique coordinator, stay on this page instead of advancing.

Glossary: tool · Glossary: structured output · Cheatsheet: production ops signals

Previous · Next

For multi-agent-in-code, write a transfer example that differs in one constraint from the chapter scenario. Keep the quality bar fixed. Explain which check still applies.

For multi-agent-in-code, write a transfer example that differs in one constraint from the chapter scenario. Keep the quality bar fixed. Explain which check still applies.

For multi-agent-in-code, write a transfer example that differs in one constraint from the chapter scenario. Keep the quality bar fixed. Explain which check still applies.