RELIABILITY AND ERROR RECOVERY

RETRY STRATEGIES AND BACKOFF Some failures are temporary. A service may be overloaded for a few seconds, or a network connection may blip. A retry means trying the same action again. Retries are powerful, but done badly they cause harm. If a service is overloaded and a thousand clients retry instantly and repeatedly, they pile more load onto the struggling service and make recovery impossible. Backoff means waiting longer between repeated attempts. Exponential backoff doubles the wait each time: 1 second, then 2, then 4, then 8. It is like knocking on a door. If nobody answers, you do not hammer the door continuously; you wait a bit and knock again, then wait longer. Good retry design includes the following. Retry only retryable errors. A temporary "service busy" error may succeed later, but a "you are not allowed" error or a "bad request" error will fail every time, so retrying it only wastes money. Set a maximum number of attempts, so retrying cannot go on forever. Add jitter, which is a small random variation in the wait time, so that many clients do not all retry at exactly the same moment. Cap the maximum wait, so backoff does not grow to absurd lengths. Make actions safe to repeat. An idempotent action is one that has the same effect whether done once or many times, such as "set the status to approved." A non-idempotent action, such as "charge the customer 50 dollars," could cause harm if repeated, so it needs a unique request key that lets the receiving system recognize and ignore duplicates. Honor signals from the service. Some services say how long to wait before trying again; respect that.