Instrument the score labeler

Instrument the threshold score labeler (scores.py) so a run records enough structure to debug offline: counts, latency if relevant, pass/fail of boundary labels at 0.49/0.50/0.51 and empty-list positive count 0, and a stable stage name. Redact secrets and raw credentials from every event.

import json

def make_event(labels, threshold):
    return {
        "stage": "label_scores",
        "input_count": len(labels),
        "threshold": threshold,
        "positive_count": labels.count("yes"),
        "ok": True,
    }

event = make_event(["no", "yes", "no"], 0.5)
print(json.dumps(event, sort_keys=True))

Expected output:

{"input_count": 3, "ok": true, "positive_count": 1, "stage": "label_scores", "threshold": 0.5}

The event records counts, not raw scores or learner identifiers. A validation failure should emit a separate safe event such as {"stage":"validate","ok":false,"error_type":"TypeError","field":"score[0]"}—never the rejected value.

Turn one historical failure—especially string scores that compare lexicographically—into a test that fails if the signal disappears for the score labeler. Observability without a failing test is optional decoration; observability with a test is part of the python-only-what-you-need artifact.

Draft a three-field event for the score labeler: stage, ok, and one domain field derived from boundary labels at 0.49/0.50/0.51 and empty-list positive count 0. Add fixture_id or docs_version when content can change. Explicitly list fields that must never appear (tokens, passwords, raw prompts) because eval() on pasted score text or logging raw learner identifiers beside scores is in scope for this lab.

Wire one assertion that fails if the score labeler event is missing after a run. Observability that cannot fail a test will not survive contact with a busy python-only-what-you-need repository.

Imagine a teammate opens only your event stream after a bad deploy. Could they tell whether scores=[0.2,0.9,0.4], threshold=0.5 was wrong, whether string scores that compare lexicographically, or IndentationError that hides a wrong cutoff returned, or whether eval() on pasted score text or logging raw learner identifiers beside scores slipped through? If not, rename fields until those three stories are distinguishable.

Assert event["input_count"] == 3, event["positive_count"] == 1, and that forbidden keys scores, learner_id, and email are absent. Then design the corresponding safe validation-failure event.

Instrument a pass/fail attendance counter. Log aggregate counts and a stage name while proving that student names never appear in the event.

ML Python starter

Previous · Next