How to handle unreliable networks in Clojure with timeout discipline, idempotent retries, backoff with jitter, and circuit-aware client design.
Handling network errors and retries is less about writing a loop and more about deciding which failures are safe to repeat, how long to wait, and when to stop making a bad situation worse. Good retry policy protects the caller, the downstream dependency, and the rest of the platform.
The most common mistake is to treat every failure as retriable. That creates duplicate writes, retry storms, and self-inflicted incidents. A stronger design starts by classifying failures and by making idempotency an explicit part of the protocol.
Not every failure deserves another attempt.
Usually retriable:
429502, 503, and 504Usually not retriable without additional logic:
The key question is not “did the request fail?” It is “is another attempt likely to help without causing incorrect side effects?”
Retries are safest when the operation is idempotent. Reads usually are. Writes often are not unless you design them that way.
Typical approaches:
If you cannot explain why a repeated request is safe, do not casually add retries around it.
A retry policy without timeout policy is incomplete. You need at least:
Without those limits, a dependency failure turns into thread starvation, queue buildup, and user-facing latency spikes.
Backoff reduces synchronized retry storms. Jitter keeps clients from retrying in lockstep.
The policy below stays library-agnostic so the retry behavior is easy to test:
1(ns acme.http.resilience)
2
3(def retryable-statuses #{429 502 503 504})
4
5(defn retryable-result? [{:keys [status transport-error?]}]
6 (or transport-error?
7 (contains? retryable-statuses status)))
8
9(defn delay-ms [base-ms cap-ms attempt]
10 (let [exp-delay (min cap-ms
11 (long (* base-ms (Math/pow 2 (dec attempt)))))
12 jitter (rand-int 250)]
13 (+ exp-delay jitter)))
14
15(defn with-retries [request-fn {:keys [max-attempts base-ms cap-ms]
16 :or {max-attempts 3
17 base-ms 200
18 cap-ms 2000}}]
19 (loop [attempt 1]
20 (let [result (request-fn)]
21 (if (or (not (retryable-result? result))
22 (= attempt max-attempts))
23 result
24 (do
25 (Thread/sleep (delay-ms base-ms cap-ms attempt))
26 (recur (inc attempt)))))))
This pattern is intentionally small. In production, the policy usually also needs:
When a write is safe to retry, make that safety explicit:
1(ns acme.orders.client
2 (:import [java.util UUID]))
3
4(defn create-order-request [order]
5 {:method :post
6 :uri "https://orders.internal.example/orders"
7 :headers {"content-type" "application/json"
8 "idempotency-key" (str (UUID/randomUUID))}
9 :body order})
That does not magically make the workflow safe. The downstream service still has to honor the key and deduplicate repeated attempts. But the caller is now participating in a real retry protocol instead of merely hoping duplicate POSTs will be harmless.
Retries are helpful only while the dependency still has a chance to recover. When failure is sustained, callers should stop amplifying it.
That is where circuit breaking and load shedding matter:
The important idea is not a specific library. It is recognizing when “retry harder” becomes the wrong move.
If you cannot answer these questions, your retry policy is under-instrumented:
A retry policy without metrics and structured logs is hard to tune and hard to defend in an incident review.
flowchart TD
A["Call Dependency"] --> B{"Success?"}
B -- Yes --> C["Return Response"]
B -- No --> D{"Retryable and Safe?"}
D -- No --> E["Fail Fast with Context"]
D -- Yes --> F{"Retry Budget Left?"}
F -- No --> E
F -- Yes --> G["Backoff with Jitter"]
G --> H{"Circuit Open?"}
H -- Yes --> E
H -- No --> A
The key thing to notice is that retry is only one branch. Safety, budget, and circuit state all come first.