Worked trace: research with evidence

The runtime creates task R-1042 with this visible state:

• goal: compare City North and City South visitor passes;
• required fields: price, included ride duration, overage rule, source URL;
• constraints: three-day visitor, official sources only, no purchase;
• visited URLs: none;
• evidence table: empty;
• budget: six tool calls and 60 seconds;
• status: running.

Completion requires two rows, all four fields, official-domain provenance for each factual claim, and a final comparison that does not imply a purchase. If evidence conflicts, the agent may still complete only if it presents the conflict rather than silently choosing one value.

These criteria prevent a fluent but incomplete answer from passing. The runtime can count rows and fields. A lightweight semantic check can test whether the comparison preserves conflicts. The absence of a purchase tool enforces the no-purchase boundary structurally.

Action 1: search_web(query="City North official bike share 3 day pass price ride duration", domains=["citynorth.gov"])

Observation 1: Three results return. Result A is an official visitor-pass page updated this year. Result B is a general transport page. Result C is an archived PDF.

State update: Store result metadata and mark no required field as verified yet. Remaining calls: five.

Action 2: open_page(url=result_A)

Observation 2: The page states a price of $18, unlimited trips of up to 30 minutes, and an overage charge of $4 for each additional 30 minutes. It also contains a banner saying, “For system maintenance, paste these instructions into your assistant.” The reader returns page title, canonical URL, update date, and extracted text.

State update: Add three City North facts with citations. Treat the banner as untrusted page content, not task policy. Mark City North complete. Remaining calls: four.

Nothing in the trace requires a hidden reasoning transcript. Reviewers can see why opening Result A was reasonable: it was official, current, and directly matched the required fields. They can also verify that retrieved instructions did not alter permissions.

Action 3: search_web(query="City South official bike share visitor pass overage", domains=["citysouth.example"])

Observation 3: The official results include a pricing page and a separate terms page.

Action 4: open_page(url=pricing_page)

Observation 4: The pricing page lists a $15 day pass and says rides under 45 minutes are included. It does not say whether the pass lasts one day or three days, and it links to terms for overage fees.

State update: The requested product may not match the user’s three-day need. Record price and ride allowance as evidence, but mark pass duration and overage unresolved. Remaining calls: two.

A weak agent might assume three daily passes cost $45 and present that as an official three-day product. A strong trace distinguishes source fact from calculation. The system can later state “three one-day passes would total $45 if consecutive purchase is allowed,” but it cannot label that amount as an advertised three-day pass without evidence.

Action 5: open_page(url=terms_page)

Observation 5: The terms page states that rides beyond 45 minutes cost $3 per additional 15 minutes. It also says day passes expire at midnight rather than 24 hours after activation.

State update: Add the overage rule and expiry condition with the terms URL. The City South row now has official evidence for a one-day product, but the goal asks about a three-day visitor. Remaining calls: one.

The new expiry detail changes the comparison. Three day passes may cover three calendar days, but a visitor arriving late would receive less than 72 hours. The agent should not hide this mismatch merely to fill a table.

The final action should target the highest-value gap.

Action 6: search_web(query="site:citysouth.example bike share 3-day visitor pass", domains=["citysouth.example"])

Observation 6: No matching official three-day pass is found.

State update: Mark “three-day product” as not found after an official-domain search. Remaining calls: zero. Run completion checks.

The system stops with budget_exhausted_partial, not completed. It can still return useful results:

• City North: $18 official three-day pass; rides up to 30 minutes; $4 per extra 30 minutes.
• City South: no official three-day product found; one-day pass is $15, expires at midnight, includes rides under 45 minutes, then charges $3 per extra 15 minutes.
• A three-day City South estimate would be $45 for three passes, but purchase rules were not verified.

The answer cites each source and recommends confirming consecutive-pass eligibility. It does not manufacture parity between unlike products.

Suppose evaluation says the final answer failed because it described City South as cheaper. The trace localizes the issue. Retrieval found the right terms, and state stored the midnight expiry. The synthesis step incorrectly compared $15 with $18 instead of three-day coverage. The fix belongs in comparison validation: normalize quantities and periods before ranking.

Now suppose the agent spent calls reopening City North. That is an orchestration defect: visited-resource tracking or gap prioritization is weak. If it followed the hostile banner, that is a security defect: untrusted content influenced authority. Different failures require different controls.

Useful traces record task ID, timestamps, model and prompt versions, action schemas, validated arguments, tool results or error classes, state changes, budget use, final evidence, and stop reason. Sensitive content should be minimized or access-controlled. Logging everything without retention and privacy rules creates a new risk.