REPLAY AND DEBUGGING

A WORKED DEBUGGING EXAMPLE A user reports that the agent gave a wrong total for an invoice. You find the run in the logs and replay it with the recorded tool results. The replay fails in the same way, so the cause is not random. You open the trace and see that the file-reading tool returned only the first six thousand characters of a long invoice, and the total sat on the last page. This is a harness problem and not a reasoning problem, because the model reasoned correctly with the text it was given. The fix is to make the tool report clearly when it has cut the text short, and to let the agent ask for the next part. You add the invoice to the evaluation set as a test, run the set, and confirm that the agent now reads the whole invoice or says plainly that it has seen only part of it. That single bug has now improved both the tool and the test suite for good. DEBUGGING BY COMPARISON One powerful technique is to compare a good run with a bad run of the same task. Line up their traces side by side and look for the first point at which they diverge. The difference at that point, such as a different tool result or a different retrieved document, is usually the cause or very close to it. When no good run exists, build one by hand, writing down what the correct steps would have been, and compare against that.