Timeouts, Retries & Connection Pools

Modules 01–05 taught you what crosses the wire when everything works. This module is about the only question production cares about: what does your code do when the other side doesn’t answer? The settings in this module are the difference between “one dependency got slow” and “the whole service went down at 2am” — and they’re the questions that separate juniors from engineers in every backend interview.

The Goal

By the end of this module you can:

• State the iron law — every network call gets a timeout — and walk the cascade that kills a service when one doesn’t
• Choose a connect timeout and a read timeout separately, and say why they’re sized differently
• Decide when a retry is safe and when it double-charges a customer
• Explain exponential backoff with jitter, and why retrying instantly makes outages worse
• Size a HikariCP pool and predict exactly what happens when it runs dry
• Set the actual Spring properties — not describe them, type them

The Lesson

The iron law: every network call gets a timeout

Here is the fact most tutorials never mention: the default timeout in most HTTP clients is infinity. A plain new RestTemplate() with no configuration will wait forever for a server that never answers. Not 30 seconds. Forever.

“So one request hangs, who cares?” Walk the cascade:

1. Your Spring Boot app runs on Tomcat with a worker pool — 200 threads by default. Every incoming request borrows one thread until the response goes out.
2. One downstream dependency — a payments API, a partner service — gets slow. Not down. Slow.
3. Every request that touches it grabs a Tomcat thread and parks it, waiting on a read that never returns. No timeout means no release.
4. Traffic keeps arriving. In minutes, all 200 threads are parked waiting on the same slow dependency.
5. Now /actuator/health has no thread to run on. Every endpoint — including ones that never touch the slow dependency — stops answering.

flowchart LR
    A["One slow dependency"] --> B["Calls wait with no timeout"]
    B --> C["All 200 worker threads parked"]
    C --> D["Healthy endpoints starve"]
    D --> E["Whole service reported down"]

One slow dependency took down your entire service, and nobody even called the broken parts. This is why an API that hangs is worse than one that fails: a failure returns immediately and frees the thread. A hang holds the thread hostage. Fast failure is a feature.

The two timeouts that matter

“Timeout” is two different decisions, and confusing them is a classic review comment:

The connect timeout is about the network: an unanswered SYN means a firewall, a dead host, or a wrong IP, and second 30 of waiting tells you nothing second 3 didn’t. The read timeout is about the server: it accepted your connection and is (supposedly) working. If an endpoint’s worst honest case is 800ms, a 3-second read timeout is generous. A 30-second one means you’ve decided to park a thread for 30 seconds before admitting defeat.

Retries and their dark side

A timeout fired. Just retry, right? Stop. Burn this in: a retry is a NEW request. The server doesn’t know it’s a retry. And here’s the trap — a read timeout doesn’t mean the request failed. It means you stopped waiting for the answer. The server may have finished the work and sent a response that died on the way back.

Now make it concrete with money, on your own API. A client calls POST /api/v1/groups/1/expenses on splitease-api to record a ₹2,000 dinner. The insert commits — then the response is lost (connection reset, a proxy gave up, anything). The client sees a timeout, “helpfully” retries, and the dinner now exists twice. Replace expense with payment and you’ve built the double-charge that ends up in a fintech postmortem.

The rule, straight from module 04’s idempotency table:

• Safe to retry blindly: idempotent operations — GET, PUT, DELETE. Doing them twice lands in the same state.
• Never retry blindly: POST. Each one creates a new thing.
• The escape hatch: an idempotency key — the client sends a unique key header with the POST; the server stores it and returns the saved response for any repeat instead of doing the work again. Now retrying POST is safe. Every serious payments API works this way.

No idempotency mechanism, no automatic retry on POST. That’s not a guideline, it’s the line between an incident and a non-event.

Backoff and jitter: don’t kick a server that’s getting up

Even when a retry is safe, when you retry decides whether you help or hurt. A dependency falls over. A thousand clients all retry instantly. It comes back up — and a thousand simultaneous retries knock it straight back down. That’s the thundering herd, and it turns a 10-second blip into an hour-long outage.

The fix is exponential backoff: wait 1s before retry one, 2s before retry two, 4s before retry three, then give up. Each wave gives the recovering server more breathing room.

But backoff alone has a subtle flaw: if all thousand clients failed at the same instant, they all retry at exactly t+1s, t+3s, t+7s — still synchronized waves. Your own clients become a coordinated attack on your own dependency: a self-DDoS. The fix is jitter — add randomness, so “wait 2s” becomes “wait 1.5–2.5s”. The waves smear out into a survivable trickle. Backoff without jitter is half a fix.

Connection pools: why HikariCP exists

You priced this in modules 02 and 05: opening a connection costs a TCP handshake, and to PostgreSQL it also costs authentication and session setup — milliseconds of pure overhead before the first byte of your actual query. Pay that on every request and you’ve doubled your latency for nothing.

So we don’t. A connection pool opens a handful of connections once, keeps them alive, and lends them out. In your stack that’s HikariCP — it’s been inside every splitease-api request you’ve ever made. Each repository call borrows a PostgreSQL connection from the pool and returns it when the transaction ends.

The number that matters is pool size (default 10). And the behavior that matters is what happens when all connections are borrowed: the next request does not fail — it WAITS in a queue for a connection to come back. That wait has its own timeout (connection-timeout, default 30 seconds), and only when that expires do you get an error.

Now you can read the classic incident, the one that fools every junior on call:

1. Someone ships a query that’s slow — usually a missing index (SQL 06 — that’s the module that finds it).
2. Each slow request holds its pool connection for seconds instead of milliseconds.
3. Ten concurrent slow requests and the pool is empty.
4. Now every endpoint that touches the database — including perfectly healthy ones — queues for a connection, waits 30 seconds, and dies with a pool timeout.
5. The on-call engineer sees “the friends endpoint is timing out” and debugs the friends endpoint. Which is fine. The root cause is one slow query three controllers away.

Symptom on the healthy code, cause in the slow code. You’ll reproduce this exact incident in Build This, and stage it again as a mystery in the capstone.

Circuit breakers, honestly

One pattern completes the set. If a dependency has failed 50 times in a row, the 51st call won’t succeed either — it’ll just burn a thread for the length of your timeout. A circuit breaker counts failures, and after a threshold it opens: calls fail instantly, without touching the wire. After a cooldown it goes half-open and lets one probe request through — success closes the circuit and normal traffic resumes; failure keeps it open.

flowchart LR
    A["CLOSED calls flow"] -->|"N failures"| B["OPEN fail instantly"]
    B -->|"cooldown ends"| C["HALF OPEN one probe"]
    C -->|"probe succeeds"| A
    C -->|"probe fails"| B

In Java the tool is Resilience4j — a library of annotations that wrap your client calls with breakers, retries, and rate limits. You don’t need to install it today. You need the concept, so when a job hands you a codebase full of @CircuitBreaker annotations, you already know the state machine. Concept now, library when a job demands it.

Spring’s actual knobs

Everything above is configuration you can type today. The pool, in application.properties:

# HikariCP — the PostgreSQL connection pool
spring.datasource.hikari.maximum-pool-size=10
spring.datasource.hikari.connection-timeout=30000
spring.datasource.hikari.max-lifetime=1800000

maximum-pool-size is how many connections exist. connection-timeout (milliseconds) is how long a request waits for a free one before erroring. max-lifetime retires and replaces connections before the database or a firewall kills them mid-use.

Outbound HTTP calls — RestClient, the modern client:

var factory = new SimpleClientHttpRequestFactory();
factory.setConnectTimeout(Duration.ofSeconds(2));   // handshake budget
factory.setReadTimeout(Duration.ofSeconds(5));      // answer budget

RestClient client = RestClient.builder()
        .requestFactory(factory)
        .build();

Or the older RestTemplate, via the builder:

RestTemplate template = builder
        .connectTimeout(Duration.ofSeconds(2))
        .readTimeout(Duration.ofSeconds(5))
        .build();

The point isn’t the exact API — it’s that you now always look for these two settings in any client you touch. A new RestTemplate() with no timeouts in a code review should make your eye twitch.

Check The Concept

How This Shows Up At Work

• The 2am incident. Every endpoint times out, the service is “down,” and the cause is one slow query holding the pool hostage. The engineer who knows the pool-exhaustion pattern reads the Hikari error and finds it in five minutes. Everyone else restarts the service and waits for it to happen again.
• Code review. A teammate adds new RestTemplate() to call a partner API. Your comment: “No timeouts — this defaults to waiting forever. One slow response from them parks our threads.” That comment is the whole module in one line.
• The interview question. “How do you call a flaky downstream service safely?” The complete answer is this module in order: timeouts on everything, retries only on idempotent operations or with idempotency keys, exponential backoff with jitter, a circuit breaker when failures persist.
• The postmortem you don’t want to star in. A retry loop on a payment POST with no idempotency key, written in a hurry. The fix is one design decision made before the incident — which is exactly when nobody is thinking about it.

Build This

You’re going to cause a real pool-exhaustion incident on your own machine, read the most valuable stack trace in production Java, and then fix it. You need splitease-api and its PostgreSQL running.

1. Shrink the pool to a single connection in application.properties — the whole incident, miniaturized:

spring.datasource.hikari.maximum-pool-size=1
spring.datasource.hikari.connection-timeout=2000

One connection total; anyone waiting for it gives up after 2 seconds.

2. Add a deliberately slow endpoint — your stand-in for “someone shipped a slow query.” In any repository:

@Query(value = "SELECT 1 FROM pg_sleep(15)", nativeQuery = true)
int slowQuery();

And a controller method:

@GetMapping("/api/v1/debug/slow")
public String slow() {
    friendRepository.slowQuery();
    return "finally done";
}

pg_sleep(15) makes PostgreSQL hold the connection for 15 seconds — exactly what a missing-index query does at scale (SQL 06).

3. Restart the app, and grab a JWT the same way as the capstone smoke test. Confirm the slow endpoint works alone — it should take ~15 seconds:

curl.exe -H "Authorization: Bearer $TOKEN" localhost:8080/api/v1/debug/slow

4. Now the incident. Open two PowerShell terminals. In terminal 1, fire the slow endpoint — it occupies the only pool connection for 15 seconds. Within 2 seconds, in terminal 2, hit the completely healthy endpoint:

# Terminal 1 — the villain (holds the one connection)
curl.exe -H "Authorization: Bearer $TOKEN" localhost:8080/api/v1/debug/slow

# Terminal 2 — the victim (a healthy endpoint, immediately after)
curl.exe localhost:8080/actuator/health

5. Watch terminal 2. It doesn’t fail instantly — it waits about 2 seconds (that’s the queue you learned about), then returns 503 with "status":"DOWN". The health check needed a database connection, queued for the pool, and hit connection-timeout. Now read the app log:

java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not
available, request timed out after 2003ms.

Read it like a stack trace from module one of this whole journey: HikariPool-1 — it’s the pool, not the network, not the query you called. request timed out after 2003ms — your connection-timeout, fired. The health endpoint did nothing wrong. Symptom on the healthy code, cause in the slow code. When you see this line in production, you stop debugging the endpoint that errored and start hunting for whatever is holding connections.

6. Break it the other way. Fire the slow endpoint in both terminals. Terminal 2’s request queues 2 seconds, then gets a 500 — same Hikari error, this time on the villain endpoint itself. The pool doesn’t care who you are; it cares who’s holding connections.

7. Feel a client-side timeout too — curl has the same two knobs Spring does:

curl.exe --max-time 3 -H "Authorization: Bearer $TOKEN" localhost:8080/api/v1/debug/slow

After 3 seconds: curl: (28) Operation timed out. That’s a read timeout from the client’s side — and note the server kept running the query anyway. Your timeout stops your waiting, not their work. That asymmetry is the entire retry danger in one command.

8. Restore sanity. Delete the debug endpoint and the slowQuery method, and set production-shaped values:

spring.datasource.hikari.maximum-pool-size=10
spring.datasource.hikari.connection-timeout=3000

Production judgment: pool size ~10 covers a surprising amount of traffic when queries are fast (Hikari’s own docs push back hard on huge pools — more connections than the database has cores mostly adds contention). And a 3-second connection-timeout beats the 30-second default: if the pool is starved, you want fast, loud failures — not 30 silent seconds per request while the incident hides.

Where to Practice

Check Yourself

1. Why is an API call that hangs worse for your service than one that fails immediately?
2. What does the connect timeout limit, what does the read timeout limit, and which is sized to the endpoint’s real latency?
3. A read timeout fired on a POST. Why is “the request failed, retry it” the wrong conclusion?
4. What problem does an idempotency key solve, and where does it live?
5. Why does exponential backoff need jitter on top?
6. The HikariCP pool is exhausted. Describe what a new request experiences, second by second, with default settings.
7. In the classic pool-exhaustion incident, why does the on-call engineer usually debug the wrong code first?
8. A circuit breaker is half-open. What is it doing?

Explain it out loud: Tell the empty chair the story of the pool-exhaustion incident end to end — slow query, connections held, queue, connection-timeout, healthy endpoints dying, the Hikari error line — then explain why the fix is an index and not a bigger pool. If the “why not a bigger pool” part stalls, reread the pool section.

Still Unclear?

I set spring.datasource.hikari.maximum-pool-size=1 and connection-timeout=2000,
held the connection with a pg_sleep query, and watched /actuator/health return 503.
Quiz me: give me three production symptoms one at a time and make me say whether
each one is pool exhaustion or a network problem, and which evidence proves it.

Walk me through sizing timeouts for a real chain: my API calls a partner API whose
p99 is 1.8 seconds, and my own callers time out at 10 seconds. What connect and read
timeouts do I set, how many retries fit in the budget, and where does backoff break it?

Explain idempotency keys by making me design one: what the client sends on a
payment POST, what the server stores, what happens on a retry after a lost response,
and what happens if two retries arrive at the same time. Ask me questions instead of
lecturing.

Why AI Can’t Do This For You

AI will happily generate a retry loop with exponential backoff in two seconds — including around a POST with no idempotency key, because the code looks textbook-correct. Whether that retry is safe depends on what the operation does to money and state, and that judgment lives in your understanding of the system, not in the prompt. The double-charge ships in perfectly clean code.

And when the pool-exhaustion incident hits, the evidence is live: which endpoint is erroring, what the Hikari line says, which query is holding connections right now. AI has never seen your pool, your traffic, or last Tuesday’s deploy. The engineer who has caused this incident on purpose — you, today — recognizes it in one log line. That recognition is the job.

Module done? Add it to today’s tracker