Relay — A Persistent Task Queue

Every backend eventually needs “do this later, reliably”: send the welcome email after the response returns, retry the failed webhook at 2am, run the settlement report at month-end. Most teams either bolt on @Async and silently lose jobs on every restart, or drag in Kafka for 200 jobs a day. Relay is the third option — the one real companies actually run — and building it teaches you more about concurrency and transactions than any course can.

The Real Problem

A payments startup sends a confirmation email after every successful transaction. The junior engineer wires it inline: the API call doesn’t return until the SMTP server does. Then the email provider has a slow day, every checkout request takes 9 seconds, and the on-call gets paged for “site down” when nothing is down — it’s just waiting on email. The fix attempt is @Async: now responses are fast, but the app restarted during Tuesday’s deploy and forty confirmation emails vanished without a trace. No log, no retry, no record they ever existed. Customer support spends Wednesday explaining to angry users that yes, the payment went through, no, we can’t say why there was no email.

The honest fix is a persistent queue: write the job to the database in the same transaction as the business change, and let a worker pool execute it with retries. This is exactly what Solid Queue (Rails), Oban (Elixir), and pg-boss (Node) do — DB-backed queues running in production at thousands of companies, because for most workloads a broker is overkill. Backend engineers and platform engineers own this problem; at fintech scale there is usually a small team whose whole job is “async infrastructure.” Building Relay means you’ve done their job in miniature.

What You Are Building

A reusable task-queue library plus a worker runtime, backed by PostgreSQL — Java 21, Spring Boot 3.x, Maven, the same stack as splitease-api. Callers enqueue jobs (a type, a JSON payload, an optional run-at time); a pool of workers claims them safely under concurrency, executes registered handlers, retries failures with backoff, and parks permanent failures in a dead-letter table. A small REST admin surface lets you see and rescue jobs.

Non-goals — write these down, they are half the design. No distributed brokers (no Kafka, no RabbitMQ — the entire point is proving Postgres is enough at this scale). No exactly-once execution promises — because exactly-once is impossible here: a worker can finish the work and crash before recording success, and on recovery the job runs again. Any system that claims otherwise is hiding the same window. The honest contract is at-least-once delivery + idempotent handlers, and your README will say so in those words. No UI beyond REST. Scoping is a skill: every cut item is a sentence in your README’s “deliberately out of scope” section, and that section reads as seniority.

Why This Lands Jobs

• A hiring manager opening this repo sees someone who understands transactions, locking, and concurrency at the SQL level — not someone who learned a queue product’s config file. The claim query alone is a better concurrency credential than most CVs carry.
• It feeds the exact interview rounds Indian product companies run: the “design async processing” system-design round, the “explain a race condition” concurrency round, and the “why not Kafka” judgment round.
• The proof artifact is rare and powerful: a reproducible test showing zero duplicate executions across concurrent workers — measured, not claimed.
• Honest limits: this does not prove you can operate Kafka at scale, design partitioned consumers, or handle multi-region delivery. Don’t pretend it does — say “I built the 80% case and I know where the 20% starts,” which is a stronger answer anyway.

Concepts It Cements

This project is the exam for half the site. Each row is a concept you’ve already met — here it stops being theory.

Design It Yourself First

Answer these on paper before reading the reference design. The answers are the project; the code is typing.

1. What columns does the jobs table need so that “claim the next runnable job” is a single atomic operation? List them and say what each one is for.
2. A worker claims a job, starts executing, and the process is killed — no exception, no cleanup. How does the system ever find out? How does the job become runnable again, and after how long?
3. Where does the retry count live, and when is it incremented — when the job is claimed, or when it fails? What goes wrong with each choice when a worker crashes mid-job?
4. Two workers poll at the same instant. Walk through, step by step at the SQL level, why they cannot both claim job 42. What specific mechanism guarantees it?
5. Given your answer to Q2, can the same job ever execute twice in total (not concurrently — in total)? If yes, what does that force on every handler you’ll ever write?
6. The enqueue must join the caller’s transaction (R1). What does that mean for who opens the transaction, and what happens if someone enqueues outside any transaction?
7. On SIGTERM, what exactly should happen, in order? What happens to a job that’s 80% done? What’s the maximum time shutdown may take?
8. What is the difference between a failed job and a dead job, and which one do humans need to look at?

Do not scroll past this without written answers. Wrong answers are fine — they’re what the reference design is for. No answers means you’re copying, not designing.

The Reference Design

The state machine

Every job is in exactly one state, and only these transitions exist. Print this; every bug is a violation of this diagram.

stateDiagram-v2
    [*] --> queued: enqueued in caller transaction
    queued --> claimed: worker claims with SKIP LOCKED
    claimed --> succeeded: handler returns normally
    claimed --> queued: handler throws and attempts remain
    claimed --> queued: claimed until expires worker died
    claimed --> dead: handler throws and attempts exhausted
    dead --> queued: human requeues via admin REST
    succeeded --> [*]

Key decisions

Data model

CREATE TABLE jobs (
    id            BIGSERIAL PRIMARY KEY,
    type          TEXT NOT NULL,              -- which handler runs this
    payload       JSONB NOT NULL,             -- handler input, schema owned by the handler
    status        TEXT NOT NULL DEFAULT 'queued',  -- queued | claimed | succeeded | dead
    attempts      INT  NOT NULL DEFAULT 0,
    max_attempts  INT  NOT NULL DEFAULT 5,
    run_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
    claimed_until TIMESTAMPTZ,                -- the lease; NULL unless claimed
    last_error    TEXT,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_jobs_claimable ON jobs (run_at) WHERE status = 'queued';

That partial index is the heap from heaps-greedy: the smallest run_at among queued jobs is always one index probe away, no matter how many million succeeded rows sit in the table. A separate dead_jobs table is optional — status = 'dead' in place is simpler and fine at this scale; decide and defend it.

The claim query — the heart, line by line

UPDATE jobs
SET    status        = 'claimed',
       claimed_until = now() + interval '60 seconds',
       attempts      = attempts + 1
WHERE  id IN (
    SELECT id FROM jobs
    WHERE  status = 'queued'
      AND  run_at <= now()
    ORDER  BY run_at
    LIMIT  5
    FOR UPDATE SKIP LOCKED
)
RETURNING *;

• SELECT ... WHERE status = 'queued' AND run_at <= now() — only runnable jobs. Future-scheduled jobs are invisible until their time comes; that’s R5 for free.
• ORDER BY run_at LIMIT 5 — oldest-due first, batch of five. The index makes this a cheap top-of-heap read.
• FOR UPDATE — lock the selected rows so no other transaction can touch them until we commit.
• SKIP LOCKED — the new idea, and the whole trick. Without it, worker B’s SELECT waits for worker A’s locks — your pool serializes into a queue at the database. With it, worker B silently skips A’s locked rows and grabs the next five. Workers never block each other and never see each other’s claims. One clause turns “locking” into “concurrent work distribution.”
• UPDATE ... SET status, claimed_until, attempts — claim, lease, and count in the same atomic statement. There is no instant where a job is selected but not claimed.
• RETURNING * — hand the claimed rows straight back to Java. One round trip, no read-after-write race.

Crash recovery — the sweep

A worker that dies mid-job leaves status = 'claimed' and a claimed_until in the past. A scheduled sweeper runs every few seconds:

UPDATE jobs
SET    status = 'queued', claimed_until = NULL
WHERE  status = 'claimed' AND claimed_until < now();

The job re-enters the queue with its attempt already counted (claim-time increment — Q3’s answer). If it keeps killing workers, it burns through max_attempts and lands in the DLQ instead of crash-looping forever.

The two hardest correctness problems — read twice

1. The worker-death window. Between “handler finished the work” and “transaction recorded succeeded” there is a window. Crash inside it and the sweep requeues a job whose work is already done — it will run again. You cannot close this window; you can only move it. This is why exactly-once is impossible and why R-everything depends on idempotent handlers: “send email for order 123” must check whether it already sent before sending. Your README explains this window; your interview story is built on it.

2. The lease-length trade-off. claimed_until = now() + 60s assumes every job finishes in 60 seconds. A slow job that’s still running when its lease expires gets requeued and runs concurrently with itself — the exact thing you built this to prevent. Fixes, in increasing effort: size the lease to your slowest job type, heartbeat the lease from long-running handlers (UPDATE jobs SET claimed_until = ... periodically), or per-type lease durations. Pick one consciously and write the decision down.

Build Plan

Each milestone is 1–2 sittings and ends with something runnable.

M1 — Design review, schema, walking skeleton. Your paper answers reviewed against the reference design (note every place you differed and why — that note is interview material). Schema migrated. An enqueue(type, payload, runAt) service method that participates in the caller’s transaction, plus a naive single worker: poll, execute, mark done — no locking yet. Done when: a demo @Transactional method that enqueues then throws leaves zero rows, and a happy-path job enqueues and executes.

M2 — Safe concurrent claiming, and PROVE it. Replace the naive poll with the claim query. Spin up two worker instances, enqueue 1,000 jobs whose handler inserts a row into an executions table, and let both pools drain the queue. Done when: SELECT job_id, count(*) FROM executions GROUP BY job_id HAVING count(*) > 1 returns zero rows — and you commit that query as the proof script. This number goes on your resume.

M3 — Retries, backoff, dead letters. Failed handlers reschedule with run_at = now() + backoff(attempts) plus jitter; exhausted jobs go dead with last_error captured. Done when: a handler rigged to fail 3 times then succeed shows growing gaps in its execution timestamps, and a permanently failing job lands in the DLQ without clogging the queue.

M4 — Scheduling and graceful shutdown. Future run_at honored (mostly already free from the claim query — verify it). The sweeper running on a schedule. Shutdown hook: stop claiming, await in-flight jobs with a timeout, let the lease recover anything that overruns. Done when: a job enqueued for +2 minutes runs on time, and killing a worker mid-batch loses nothing — every job ends succeeded exactly once in the executions table.

M5 — Admin REST and metrics. GET /admin/jobs?status=... (paginated — you know why), POST /admin/jobs/{id}/requeue for dead jobs, and counters worth graphing: queue depth, jobs/minute, failure rate. Secure it with the JWT setup from security-jwt. Done when: you can find a dead job, read its error, requeue it, and watch it succeed — all over curl.

M6 — Deploy, README, demo. Deploy to a free tier (Railway, Render, or Fly.io — same drill as week 9) with a managed Postgres free instance. README: what it is, the delivery contract (at-least-once, in bold), the claim query explained, how to run the double-run proof. The demo script IS the M2 proof: enqueue 1,000, run two workers, run the SQL, show the zero. Done when: a stranger can clone, run, and reproduce your zero.

Definition of Done

• All seven requirements pass their proving tests, R1’s rollback test and R2’s zero-duplicates query in the repo as runnable scripts
• Admin endpoints JWT-secured; all inputs validated records; no entity crosses the API boundary
• One consistent error shape from every endpoint — the ApiError discipline from the SplitEase capstone
• Every state transition matches the state diagram; no SQL anywhere writes a status outside it
• Backoff timing test-pinned (assert the computed delays, not sleeps)
• Graceful shutdown verified by an actual kill, not by reading the code
• README states the at-least-once contract, explains the worker-death window honestly, and includes a curl + proof-query walkthrough
• Deployed and demoable from a phone browser via the admin endpoints

Stretch Goals

1. Priority column — a priority integer joining run_at in the ordering and the index; forces you to rethink what the composite index-as-heap looks like (heaps-greedy).
2. Postgres LISTEN/NOTIFY wakeups — workers sleep until notified instead of polling; forces real connection-lifecycle handling beyond the pool (data-jpa).
3. Lease heartbeats for long jobs — handlers extend claimed_until while running; forces you to solve hardest-problem 2 properly instead of by configuration.
4. Per-type concurrency limits — at most N concurrent send-email jobs; forces semaphore thinking from concurrency into the claim query itself.

Resume Bullets It Generates

• Built a PostgreSQL-backed persistent task queue in Java 21 / Spring Boot with transactional enqueue, lease-based crash recovery, and exponential-backoff retries with jitter
• Achieved zero duplicate executions across N concurrent workers processing M jobs, verified by SQL audit of an executions log — measure N and M yourself
• Implemented atomic job claiming with SELECT FOR UPDATE SKIP LOCKED, sustaining J jobs/minute on a free-tier instance — measure J
• Designed an at-least-once delivery contract with idempotent handlers and a dead-letter queue, documenting why exactly-once delivery is impossible
• Recovered 100% of in-flight jobs across K forced worker kills via lease-expiry sweep — run the kill test K times and count

Never claim a number you did not measure. Every placeholder above has a script in your repo that produces it — that’s the difference between a bullet and a bluff.

Interview Stories It Seeds

“How would you build async processing without Kafka?” S/T: Needed reliable background jobs at small scale where a broker was operational overkill. A: Built a Postgres-backed queue — transactional enqueue so jobs and business changes commit together, SKIP LOCKED claiming so workers never block each other, lease-based crash recovery. R: At-least-once delivery, proven zero duplicates across concurrent workers; and I can name the scale point — sustained high throughput or multiple consumer services — where I’d reach for a broker.

“Can you guarantee exactly-once delivery?” S/T: I promised at-least-once on purpose and got asked why not exactly-once. A: Showed the window — a worker can complete the work and die before recording success; on recovery the job reruns, and no architecture closes that window, it only moves it. R: Shipped at-least-once plus idempotent handlers as the documented contract — the same honest answer Kafka’s own docs give.

“Tell me about a race condition you actually prevented.” S/T: Two workers polling the same table could claim the same job — duplicate emails, double side effects. A: Made the claim a single atomic UPDATE over a FOR UPDATE SKIP LOCKED subselect, then proved it: 1,000 jobs, two competing workers, a SQL audit showing zero double-runs. R: I don’t say “it’s thread-safe” — I show the query that counts the violations and returns nothing.

“What happens when a worker dies mid-job?” S/T: Killed workers were my main failure mode to design for. A: Every claim carries a lease (claimed_until); a sweeper requeues expired claims, and attempts are counted at claim time so crash-loops burn through to the dead-letter queue instead of spinning forever. R: Verified by literally kill -9-ing workers mid-batch and auditing that every job completed exactly once after recovery.

Project started? Add it to today’s tracker