RELIABILITY AND ERROR RECOVERY

EXAMPLES Example 1: Retry with exponential backoff An agent calls a data service that sometimes returns "temporarily overloaded." The engineer sets a policy: try once, and on a retryable failure wait 1 second and retry, then 2 seconds, then 4, then 8, for a maximum of 4 retry attempts. Each wait also has up to 20 percent random jitter. If the third attempt succeeds, the agent carries on with only a delay of about 7 seconds in total. If all attempts fail, the harness stops retrying and moves to the fallback. The old behavior was an immediate retry in a tight loop, hundreds of times a minute, which made the overload worse and consumed the budget. Example 2: Infinite-loop detection An agent is asked to find a document and calls search_files with the query "Q3 budget" three times in a row, receiving "no results" each time. The harness keeps a short history of recent actions. When it sees the identical tool and identical arguments three times in a row with no new information, it intervenes. It inserts a message: "The same search has failed three times. Try a different query, a different tool, or tell the user that you could not find it." The agent then tries a broader search with different words and finds the document. If the agent had repeated the failing search again, the harness would have ended the run at its step limit and reported the problem. Example 3: Graceful degradation A job-matching agent normally uses a premium database of job postings. One morning, the premium database is unreachable. The harness uses a fallback chain. Level one is the premium database, which fails. Level two is a free public job feed, which is lower quality but works. The agent uses it and says to the user: "I used a backup data source because the primary one was unavailable, so results may be less complete." If the free feed also failed, level three would show recently cached results with their age, and level four would apologize and offer to notify the user when service returns. The user is never left wondering what happened, and never fooled. Example 4: Balancing reliability against cost A team builds two agents. The first is an internal tool that drafts meeting summaries. They decide that three retries at normal cost is enough, with no special fallback, because a failure just means someone clicks the button again. The second is a customer-facing payment agent. A failure here could mean a double charge or a missed payment. They choose five retries with idempotency keys, a second verification step before any money moves, and a human-escalation path when retries fail. This costs more to build and run, but the team judges that it is worth it given the harm a failure could cause. Both choices are correct for their situations.