Capstone: Debug The Wire

It’s 2am. The pager says “the API is down.” That’s not a diagnosis — it’s a feeling. Down how? Refused, hanging, resolving to the wrong place, answering 504s? Each one is a different layer, a different cause, a different fix — and five minutes of wire debugging beats an hour of scrolling logs hoping the answer surfaces. This capstone turns modules 01–06 into that five minutes.

The Goal

By the end of this module you can:

• Translate any wire symptom — refused, timeout, reset, NXDOMAIN, 502, 504 — into a layer and a most-likely cause, on sight
• Run the debugging protocol in order, instead of trying commands at random
• Diagnose five staged failures on your own machine from the symptom alone, before reading the reveal
• Read a TLS failure from curl’s error text without googling it
• Say what tool comes after curl -v — and why you don’t need it yet

The Decoder Table

This is the centerpiece of the whole track. Every row is a module compressed to one line. The goal is that this table lives in your head — at 2am there is no time to rebuild it from first principles.

Read it by column once: the symptom is what the pager gives you, the layer tells you which module’s mental model to load, and the confirm command turns a guess into a fact before you act on it.

The Drill Ladder

Five staged failures, easiest to nastiest, all on your machine. For each: read the symptom, decide your diagnosis before scrolling to the reveal, then run the walkthrough. You stage each one yourself — the skill is pretending you don’t know what you did.

Drill 1 — “The API is down”

Stage it: stop splitease-api (Ctrl+C in its terminal). Make chai. Come back as the on-call engineer.

Symptom:

curl.exe localhost:8080/actuator/health

curl: (7) Failed to connect to localhost port 8080 after 0 ms: Connection refused

Diagnose. Refused — and fast. The decoder table says TCP layer: a live machine actively answered “nothing listens here.” So the OS is fine and the network is fine; the question is purely “is the process up?” Ask the machine, not your memory:

netstat -ano | findstr :8080

Empty output. Nothing listens on 8080 — confirmed, not assumed.

Reveal: the app isn’t running. Start it, re-run the curl, get 200. Trivial? Yes — and it’s the most common production page in existence. The win is that you proved it with netstat in ten seconds instead of reading logs for a process that wasn’t there to write any.

Drill 2 — The config that lies

Stage it: none needed — just pretend a teammate’s .env says the API runs on 8081.

Symptom:

curl.exe localhost:8081/actuator/health

curl: (7) Failed to connect to localhost port 8081 after 0 ms: Connection refused

Diagnose. Same refused as drill 1 — but this time the app is running, you can see its terminal. So “refused” doesn’t mean “app down.” It means “nothing on that port.” Find out where things actually listen:

netstat -ano | findstr LISTENING | findstr 808

  TCP    0.0.0.0:8080    0.0.0.0:0    LISTENING    21044

Reveal: there’s the listener — on 8080, not 8081. The app is healthy; the config lies. In production this is the wrong port in an environment variable, a load balancer pointed at the old port, a docker port mapping typo. netstat settles “is it the app or the config” in one line — the two camps that otherwise argue for an hour.

Drill 3 — The DNS “outage”

Stage it: from module 03, you know the hosts file overrides DNS. Open Notepad as administrator, edit C:\Windows\System32\drivers\etc\hosts, and point your API’s local name at an unroutable address:

203.0.113.50    splitease.local

Symptom:

curl.exe http://splitease.local:8080/actuator/health

It hangs… and hangs… then:

curl: (28) Failed to connect to splitease.local port 8080 after 21039 ms: Timed out

Diagnose. A timeout, not refused — the decoder table says packets are vanishing. But hold on: where is splitease.local even pointing? Add -v and read the very first lines:

*   Trying 203.0.113.50:8080...

203.0.113.50?! Ask DNS what it thinks:

nslookup splitease.local

*** UnKnown can't find splitease.local: Non-existent domain

Reveal: the contradiction is the diagnosis. Real DNS has never heard of this name — yet curl resolved it to a concrete IP. Someone on this machine is answering before DNS gets asked: the hosts file. nslookup queries the DNS server directly and skips the hosts file; curl uses the OS resolver, which reads it. When the two disagree, a local override is lying to you. Fix the entry to 127.0.0.1, and curl works. In production, this exact shape appears as stale resolver caches, a poisoned hosts entry from an old debugging session, or split-horizon DNS — the wire tools disagree, and the disagreement points at the liar.

Drill 4 — The hanging dependency

Stage it: re-apply the module 06 incident kit — maximum-pool-size=1, connection-timeout=2000, the pg_sleep(15) debug endpoint — restart, then in one terminal fire the slow endpoint with your JWT. Now forget you did that.

Symptom: monitoring screams that /actuator/health — an endpoint that has worked for weeks — is failing:

curl.exe localhost:8080/actuator/health

A ~2-second pause, then 503 and "status":"DOWN".

Diagnose. Run the protocol in order. Is anything listening?

netstat -ano | findstr :8080

LISTENING — yes. Can you reach it? Test-NetConnection localhost -Port 8080 → TcpTestSucceeded : True. So the wire is fine: the app accepts connections, it just answers badly. Everything below HTTP is exonerated — suspicion moves up the stack, into the app. Now the logs earn their place:

java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not
available, request timed out after 2003ms.

Reveal: the pool-exhaustion incident from module 06, replayed as a mystery. The healthy endpoint starved waiting for a database connection that a slow query was holding. The trap this drill inoculates you against: timeouts on endpoint X do not mean endpoint X is broken — check what it shares with the rest of the app (the pool, the database, the worker threads). The wire tools’ real contribution was negative: they proved in 30 seconds that it wasn’t the network, which is half of every diagnosis. Restore sane settings before drill 5.

Drill 5 — TLS in production

Stage it: nothing to stage — badssl.com keeps permanently broken TLS servers running for exactly this. Hit them cold and diagnose each from the error text alone, decoder table in hand:

curl.exe https://expired.badssl.com/

curl: (60) SSL certificate problem: certificate has expired

curl.exe https://wrong.host.badssl.com/

curl: (60) SSL certificate problem: unable to get local issuer certificate

(or, depending on your curl build, a hostname mismatch error naming the subject)

curl.exe https://self-signed.badssl.com/

curl: (60) SSL certificate problem: self-signed certificate

Diagnose and reveal, per the table: all three are TLS-layer, all three are (60) — verification failures, meaning the handshake mechanics worked but trust failed (module 05). “Expired” = renewal automation failed; the 2am fix is renewing the cert, and the postmortem fix is monitoring expiry. “Wrong host” = the cert is real but for a different name — usually a load balancer serving the wrong cert. “Self-signed” = no chain to a trusted root — common inside corporate networks. The discipline drill: you may add -k to confirm the diagnosis (it skips verification and the request succeeds, proving trust was the only problem) — but -k in committed code or production config is how man-in-the-middle attacks become possible. Diagnose with it; never ship it.

The Debugging Protocol

Five questions, in order. Each one either finds the problem or eliminates a layer. Laminate this:

The order matters: each step is cheaper and more decisive than the one after it, and a pass at step N means you never waste time on the layers below it. The single most common on-call mistake is starting at step 5 — an hour in the logs for an app that wasn’t running, which step 1 proves in ten seconds.

When curl -v isn’t enough — when you need to see the actual packets, retransmissions, and timing gaps inside the connection — the next tool is Wireshark: a packet capture UI that shows every byte on the wire, the ground truth beneath every tool in this track. You don’t need it yet; everything in this track and most production incidents fall to the five steps above. But when a problem lives between the layers — packets sent but never acknowledged, mysterious 200ms gaps — Wireshark is where wire debugging goes next, and the four-layer model from module 01 is exactly the map you’ll read its output with.

Check The Concept

Check Yourself

One question per module, plus two that cut across. Close the decoder table first.

1. A request leaves your code. Name the four layers it descends through and one thing each layer adds (module 01).
2. Connection refused vs connection timeout — what does each one prove, and which layers does each exonerate (module 02)?
3. Your JVM cached a DNS answer and the ops team just moved the server. What does your app experience, and what bounds how long it lasts (module 03)?
4. From memory: the first line of a raw HTTP request and the first line of its response, for a successful POST (module 04).
5. curl exits with error 60. What class of problem is this, what part of TLS worked, and why is -k acceptable in diagnosis but never in code (module 05)?
6. Recite the classic pool-exhaustion incident in four steps, from slow query to wrong-service debugging (module 06).
7. Recite the five-step debugging protocol in order, with the command for each step.
8. Why does running the protocol in order beat jumping straight to the application logs?

Explain it out loud: Give the empty chair the 2am briefing: the pager says “the API is down,” and you narrate exactly what you type, in order, and what each possible output would tell you — refused vs timeout at step 1, the nslookup disagreement, the Hikari line. If you can do that without looking at the protocol table, the track did its job.

Still Unclear?

Run me through a wire-debugging simulation. Invent a production incident with
splitease-api behind a gateway, give me ONLY the first symptom, and make me ask
for each command output one at a time — netstat, Test-NetConnection, nslookup,
curl -v, logs. Do not reveal the cause until I commit to a diagnosis.

Quiz me on the decoder table: give me ten symptoms one at a time, in random order
(connection refused, reset, NXDOMAIN, UnknownHostException, 502, 504, TLS errors,
hangs-then-200, slow-first-request) and check my layer, likely cause, and
confirming command for each. Track my score.

I can run netstat, Test-NetConnection, nslookup, and curl -v on Windows. Show me
what each tool CANNOT see, with one example failure per tool that it would miss
and which other tool or technique catches it.

Why AI Can’t Do This For You

AI has never seen your network. It doesn’t know what’s listening on your server right now, what your hosts file says, which cert is on the load balancer, or that someone changed a port mapping on Tuesday. Every diagnosis in this module came from running a command against a live system and reading what actually came back — evidence AI can reason about only after you’ve gathered it, with tools you had to know to reach for, in an order you had to already have.

And the decoder table is in your head at 2am or it is nowhere. When production is down, there is no leisurely chat session — there’s a symptom, a war room, and the engineer who can say “refused means it’s the process or the port, run netstat” in the first thirty seconds. You built that reflex by staging five failures with your own hands. That’s not something a prompt produces; it’s something reps produce, and you just did the reps.

Module done? Add it to today’s tracker