THE EVALUATION HARNESS

CHECK YOUR UNDERSTANDING • What is the difference between a span and a trace? ◦ Answer: A span is one small unit of work, while a trace is the complete record of a run made up of many spans. • Why might an agent pass an outcome check but still deserve concern? ◦ Answer: Because it may have reached the right result by a wasteful or risky path, which only span-level or trace-level checks would reveal. • Name one strength and one weakness of curated datasets. ◦ Answer: A strength is control and repeatability, and a weakness is that they can drift away from what real users actually do. • Why is online evaluation needed if offline evaluation already passed? ◦ Answer: Offline tests only cover situations the designers thought of, while live traffic contains surprises. • Why decide the rollback threshold before a release? ◦ Answer: So the team can act quickly and calmly during an incident instead of arguing about what number counts as bad.