Dispatch — A Notification Service

“Design a notification service” is the system-design question Indian product companies reach for again and again (Week 11 lists it among the classics). Most candidates answer it from theory. You are going to answer it from a thing you built — one that sends across channels, never double-sends, and survives a flaky provider — by standing it on top of Relay, the queue you already wrote. That is the ladder doing its job: project 3 uses project 2.

The Real Problem

Every product sends notifications: the OTP at login, the payment confirmation, the weekly digest. Send them inline and your checkout API now waits on an email server — one slow provider day and the whole site looks down. Send them carelessly and a retry fires the OTP twice, or the payment “confirmed” SMS goes out three times, and now it is a support ticket and a trust problem. At scale, a small team owns “notifications” as infrastructure: templates, channels, rate limits, delivery tracking, and the iron rule that one event produces exactly one delivery.

The engineers who own this are backend/platform engineers; the problem is deceptively deep because the hard parts — idempotency, rate limiting, retry-without-duplication — are exactly the parts that separate a toy from production. Dispatch makes you build those parts.

What You Are Building

A notification service — Java 21, Spring Boot, PostgreSQL — that accepts notification requests over REST, renders templates, sends across two channels, rate-limits per tenant, tracks delivery, and retries failures through Relay. The centerpiece is an idempotency key: the same key twice produces exactly one send, no matter how many times a nervous caller retries.

Non-goals — the scoping that reads as seniority. No SMS — it costs money, and the channel abstraction means adding it later is one adapter, not a rewrite (say that). No template editor UI — templates live in the DB or config; the product is the API. No read receipts or open tracking — a different, heavier problem. Each cut is a sentence in your README’s “deliberately out of scope” section.

Why This Lands Jobs

• It answers a named interview question with a built system — “design a notification service” stops being theory and becomes “here’s mine, and here’s what I’d change at 100x”.
• It proves the three things that separate production from toy: idempotency (no double-sends), rate limiting (no noisy-neighbour), and retry-without-duplication (flaky providers handled).
• It demonstrates real reuse — built on Relay — which is exactly how senior engineers work: composing their own infrastructure rather than starting from a blank folder.
• Honest limits: single-region, two channels, modest scale. It does not prove multi-region fan-out or millions-per-minute throughput. “I built the core contracts correctly and I know where scale changes them” is the right framing.

Concepts It Cements

Design It Yourself First

Write answers before reading on.

1. Where exactly is the idempotency key checked — at the API layer on arrival, or at the send layer? What race exists if two requests with the same key arrive at the same millisecond, and what closes it?
2. What is the rate-limit key? Per tenant? Per tenant-plus-channel? Per recipient? Your choice changes who can starve whom.
3. What does the webhook receiver need in order to verify your signature, and what must never be in the signed payload?
4. A send succeeds at the provider but your “mark sent” write fails. What state is the notification in, and what happens on retry?
5. Where does template rendering happen — on accept, or at send time — and why does the answer matter if a template changes between the two?

Do not scroll past this without written answers.

The Reference Design

flowchart TD
    A["POST /notifications + idempotency key"] --> B["idempotency check: INSERT ON CONFLICT"]
    B -->|new| C["validate + persist accepted"]
    B -->|duplicate| R["return the original result"]
    C --> D["rate-limit gate per tenant"]
    D --> E["enqueue per-channel send jobs into Relay"]
    E --> F["email adapter"]
    E --> G["webhook adapter, HMAC signed"]
    F --> H["update delivery status"]
    G --> H

The hardest problem, called out: idempotency under concurrent duplicates. Two requests, same key, same instant. Both pass an application-level “have I seen this?” check because neither has committed yet. The DB unique constraint is what actually saves you: one INSERT wins, the other gets a constraint violation your code must catch and translate into “return the original result” — not a 500. Build the test that fires both at once and proves one delivery.

Build Plan

1. Schema + REST + idempotency. Notifications table with a UNIQUE (tenant_id, idempotency_key); the accept endpoint; prove it with a concurrent duplicate test. Done when: two simultaneous identical requests yield one row and one result.
2. Templates. Store templates; render {{vars}} at send time. Done when: a payment template renders with real values.
3. Email channel (local). Send via a local SMTP catcher (Mailpit/MailHog). Done when: a rendered email appears in the catcher’s inbox.
4. Webhook channel + HMAC. POST signed payloads; write a tiny receiver that verifies the signature. Done when: the receiver accepts a genuine signature and rejects a tampered one.
5. Rate limiting + priority. Sliding-window per tenant returning 429; OTP lane ahead of marketing. Done when: a marketing burst does not delay an OTP, and an over-limit tenant is throttled while others flow.
6. Deploy + document. Free tier, README, demo script. Done when: the demo sends across both channels and shows a duplicate being absorbed.

Definition of Done

• Same idempotency key twice = exactly one delivery, proven under concurrency
• Both channels work; webhook signatures verify and tampered ones are rejected
• Per-tenant rate limiting returns 429 without affecting other tenants
• Failures retry via Relay with backoff and land in a dead-letter on exhaustion
• Delivery status is queryable per notification
• README with a demo that includes an absorbed duplicate and a throttled tenant

Stretch Goals

• Add SMS as a third adapter (proves the channel abstraction was real).
• Scheduled notifications (digest at 9am) — Relay already supports run_at.
• Per-template rate limits on top of per-tenant (forces a second limiter key).

Resume Bullets It Generates

Measure before you write.

• Built a multi-channel notification service (email + webhook) with idempotency keys guaranteeing exactly-once delivery, verified under concurrent duplicate requests.
• Implemented per-tenant sliding-window rate limiting and priority lanes so transactional messages (OTP) are never delayed by bulk traffic.
• Secured webhook delivery with HMAC-SHA256 signatures and at-least-once retry via a database-backed queue, with a dead-letter path for permanent failures.
• Composed the service on a self-built task queue, reusing its backoff and dead-letter machinery rather than reimplementing reliability.

Interview Stories It Seeds

• “Design a notification system.” → answer from the reference design above: accept → idempotency → rate-limit → enqueue → channel adapters → delivery tracking, with the trade-offs you actually made.
• “How do you stop double-sending an OTP?” → the idempotency key, and the precise reason the database constraint (not application code) is what makes it true under concurrency.
• “A provider keeps failing — what happens to the message?” → retry with backoff via the queue, dead-letter on exhaustion, and why the handler must be idempotent because the provider might have actually sent before timing out.

Project started? Add it to today’s tracker