PREPARING FOR CERTIFICATION

SAMPLE SCENARIOS Scenario 1: Refund approvals. An agent handling refund requests begins approving refunds above its authorized limit. Using the layers and patterns of this book, name at least three independent safeguards that should have prevented this and explain why each would have caught it. Model answer: First, a permission or policy check in the harness that compares the amount with the agent's limit before the refund tool runs, because the limit then holds even if the model reasons badly. Second, a human approval hook for any refund above a threshold, because a person reviews the unusual cases. Third, a budget or cap on the total value the agent may refund per run or per day, because it bounds the damage of any single failure. Fourth, a validation of the tool's inputs, such as rejecting amounts above the limit, so a bad argument is refused at the tool itself. Fifth, monitoring and an alert on unusual approval amounts, which would reveal the problem within hours instead of weeks. Sixth, an evaluation case with an over-limit request in the test set, which would have caught the behavior before launch. Each safeguard sits in a different layer, which is the point of defense in depth. Scenario 2: The forgetful agent. An agent on a long document task works well for twenty steps and then repeats work it already completed. Diagnose the likely layer and the fix. Model answer: This is probably a context-management problem, since early results have likely fallen out of the context. Check the trace at the step of the repeat, to see what the model was shown. If the earlier result is absent, the context builder or compaction dropped it. Fix by flagging such results as important, copying them word for word into summaries, and recording progress in an external notes file. Then add an evaluation case with a long task, so the problem is caught next time.