REPLAY AND DEBUGGING

WHAT A REPLAY NEEDS Replay only works if enough was recorded. At minimum, record the exact input to each model call, the exact output, and the exact result of each tool call, along with the settings used, such as the model name and the temperature. With these records, a replay tool can feed the saved tool results back to the agent instead of calling the real tools again. This means you can re-run a failed run without touching the outside world, which is safe, fast, and free. It also means that a bug found on Tuesday can still be studied on Friday, even though the live web page the agent read has since changed. WHY MODELS MAKE REPLAY TRICKY Models can give different answers to the same input, a property called non-determinism. Lowering the temperature, which is the setting that controls how adventurous the model's word choices are, makes answers more repeatable, but it does not always make them identical. There are two common ways to cope. In exact replay, you reuse the recorded model outputs and only test your own harness code, which is perfect for finding harness bugs. In live replay, you send the recorded inputs to the model again and compare the new answer with the old one, which helps you judge whether a prompt or model change would have helped. Use both, and be clear about which one you are using.