PROJECT FOUR - EVALUATION AND OBSERVABILITY

EXAMPLES Example 1: Tracing every call. A learner adds code that records a span, with start, end, input, output, and status, for every model call and tool call, then groups them into one trace per task. In the lab, tracing.py and the with span lines do exactly that. Example 2: A one-glance dashboard. The learner builds a table or chart of success rate, average cost, and average delay across the last twenty runs. In the lab, dashboard.py prints the same summary for the last twenty-five. Example 3: An evaluation set with golden answers. The learner writes fifteen test tasks with golden answers covering normal, edge, and at least one adversarial case, reusing the injection example from Project Three. That is eval_set.json. Example 4: Evaluation-driven improvement. After noticing three of fifteen tasks failing because of a context-selection bug, a learner fixes the context builder and re-runs the whole suite, confirming all fifteen now pass. In the lab, the same loop is how you proved the repeated-call guard: the empty-file item failed, you fixed the harness, and you re-ran all fifteen items.