Career OS

RupeeRail — A Mini Payment Switch

Strip the marketing off Razorpay, Juspay, or Cashfree and what remains is this: a layer between merchants and banks that accepts payment intents, routes them to bank rails, keeps a ledger that is never wrong, and tells merchants what happened. RupeeRail is an honest miniature of that layer — and an honest miniature of a payment switch is the single strongest portfolio signal you can put in front of an Indian fintech backend interviewer.

If the payments domain itself is still fuzzy — what a PA actually does, why settlement is a thing, where the money physically sits — read Payments 101 first. This doc assumes that vocabulary.

The Real Problem

A merchant’s checkout calls “the payment gateway” and gets back success or failure. That sentence hides the hardest distributed-systems problem in Indian tech. The bank’s API is slow, flaky, and sometimes answers nothing at all — and when it answers nothing, the customer’s money may or may not have left their account. Meanwhile the merchant retries, the customer retries, and every retry is a potential double charge. And at 7am tomorrow, finance will ask whether the company’s internal numbers match the bank’s numbers, to the paisa.

The engineers who own this problem at Razorpay, PhonePe, Juspay, and Cashfree carry titles like Payments Platform Engineer, Backend Engineer — Payment Gateway, Ledger Engineer, and Reconciliation Engineer. They are paid well precisely because “the bank timed out — did the customer pay?” has no easy answer, and getting it wrong costs real money, regulatory attention, and merchant trust. RupeeRail puts you in that seat with simulated banks, so you make every one of those decisions yourself.

What You Are Building

A payment switch as a Spring Boot service on PostgreSQL. Merchants onboard and get API keys. They create payment intents (amount in paise, their own reference id, an expiry). The switch routes each intent to a simulated bank adapter whose behaviour you control — including the evil mode where the bank debits the customer and then fails to respond. Every paisa of movement lands in a double-entry ledger whose postings always sum to zero. Merchants hear about state changes via HMAC-signed webhooks. A scheduled reconciliation sweep resolves the payments the bank left in limbo, and a daily recon report proves the ledger against the adapter records. This is Projects 1–3 — Recon, Relay, and Dispatch — fused into one system with money flowing through it.

#Requirement
R1Merchant onboarding: register a merchant, issue an API key (stored hashed), authenticate every API call with it
R2Create a payment intent: amount in paise (BIGINT), merchant reference with a UNIQUE constraint per merchant, expiry timestamp
R3Simulated bank adapters with configurable behaviour modes: success, decline, timeout, slow, fail-after-debit
R4Strict timeout on every adapter call plus exactly ONE retry — and only on operations proven idempotent
R5Double-entry ledger: every money movement is two postings; balances are derived sums; SUM of all postings = 0, enforced and assertable on demand
R6Payment state machine: created → processing → succeeded / failed / expired, with the ambiguous bank-timeout state resolved by a reconciliation sweep
R7Webhooks to merchants on every state change: HMAC-SHA256 signature, retries with backoff, delivery tracking
R8Per-merchant rate limiting on the public API (sliding window)
R9Observability: request IDs on every log line, /actuator/health, a metrics counter per state transition
R10Daily recon report: adapter records vs ledger, exceptions classified

Non-goals — and why scoping is a skill. No real money and no real bank APIs: the simulated adapters give you more control over failure than real banks would, which is the whole point. No UI beyond, at most, a static status page — the product is the API. And no PCI scope: no card numbers, ever, not even fake ones in test data. That last non-goal is itself domain knowledge worth stating in an interview: the moment a system stores, processes, or transmits card data it falls under PCI-DSS, an audit regime that costs real companies lakhs to maintain. Real payment companies fight to shrink their PCI scope through tokenisation. Knowing what you deliberately kept out of scope, and why, is exactly the judgment hiring managers probe for. RupeeRail moves abstract “customer funds” between ledger accounts — the accounting is real, the instrument is out of scope.

Why This Lands Jobs

  • It is the business. A recruiter at a payments company doesn’t have to translate this project into relevance — it is a simplified version of the product their employer sells. The README screenshot of a sum-zero assertion over thousands of simulated payments speaks their language on sight.
  • It feeds every interview round. The state machine and ledger feed system design rounds; the FOR UPDATE money race and the unique-constraint idempotency feed the SQL and concurrency deep-dives; the timeout/retry/amplification discussion feeds the “tell me about a hard problem” behavioural round.
  • It proves design judgment, not code volume. The decisions documented here — ledger-before-bank ordering, unknown as a first-class state, one retry only where idempotent — are senior-engineer talking points coming out of a junior candidate. That contrast gets remembered.
  • What it does NOT prove: that you can operate at real scale, handle real bank integration quirks (every bank’s API is its own adventure), or survive a real PCI audit. Don’t claim those. “I built a faithful miniature and can tell you exactly where the real version gets harder” is a stronger sentence than any inflated claim.

Concepts It Cements

This is the exam for the whole site. Every track shows up, and most show up in the parts that can go wrong with money attached.

ConceptWhere you learned itWhere it bites in this project
Transactions and row lockingSQL — TransactionsState transition + ledger postings must commit or roll back as one unit; SELECT ... FOR UPDATE on the intent row stops two threads transitioning the same payment
Aggregation and derived valuesSQL — AggregationAccount balances are SUM(amount_paise) GROUP BY account over postings — never a stored column; the sum-zero assertion is one query
Indexes for hot queriesSQL — IndexesThe unique index on (merchant_id, merchant_ref), postings by account, intents by (status, created_at) for the sweep — each one chosen against a real query
IdempotencyProject 1 — ReconThe merchant retries create intent after a network blip; the UNIQUE constraint turns the duplicate into a safe replay of the original response
Timeouts, retries, the double-charge trapNetworking — Timeouts, Retries, PoolsEvery adapter call has a hard budget; the one-retry policy exists only because the simulated bank’s debit call accepts an idempotency key
Auth at the boundarySpring Boot — Security and JWTSame filter-chain pattern, API-key flavour: a filter resolves X-Api-Key to a merchant identity before any controller runs
Service-layer transactionsSpring Boot — Services and TransactionsEvery state machine method is @Transactional; the controller translates HTTP and nothing else
Hash maps as infrastructureDSA — Arrays and HashingThe in-process idempotency cache fronting the unique constraint; the recon report’s match-by-reference join in memory
Sliding windowDSA — Sliding WindowThe per-merchant rate limiter is a sliding window over request timestamps — the week-11 pattern guarding a real API
Webhooks, HMAC, at-least-once deliveryProject 3 — DispatchState changes notify merchants; signatures prove origin; retries mean merchants must dedupe — you now sit on the sender’s side of that contract
Partial failure and unknown stateSystem Design — Scaling and Distributed SystemsThe adapter timeout is the two-generals problem with rupees attached; “unknown” becomes a first-class state with its own resolution path
Reconciliation as a patternProject 1 — ReconThe sweep and the daily report are Recon running in-process: adapter truth vs ledger truth, mismatches classified
ObservabilityWeek 8 — Production Patterns, Week 10 — CI/CD and OpsRequest IDs trace one payment across intent, adapter, ledger, webhook; a counter per state transition turns the state machine into a dashboard
Money as integer minor unitsCore Java — CapstonePaise as long/BIGINT end to end — the rule you set in your first capstone, now enforced where being wrong costs money

Design It Yourself First

Answer these on paper before you scroll to the reference design. These are not warm-ups — questions 2, 4, and 5 are asked, nearly verbatim, in payments-company interviews.

  1. List your entities and what each one owns. Where does the payment’s current state live, where does money live, and why must those be different tables?
  2. The bank says success, then your ledger write fails. Walk the order of operations both ways: ledger first (record intent-to-move, then call the bank) and bank first (call, then record). What does each ordering leave behind when the crash happens at the worst moment? There is no perfect answer — write down which wrong you’d rather be, and what cleans it up.
  3. Where exactly does the idempotency key live, and what is the precise race — two threads, step by step — that the UNIQUE constraint closes which an application-level “check then insert” cannot?
  4. How do you know balances are right — not hope, know? Design the invariant, then design when and how it is checked. What does it mean operationally the day that check fails?
  5. The adapter call times out. What state is the payment in right now? Why must “unknown” be a first-class state in your machine rather than a default to failed — and what, concretely, moves a payment out of it?
  6. Who retries — the merchant’s server, your switch, or both? Sketch the amplification: merchant retries 3 times, your switch retries each call once. How many debit attempts can one customer click become, and what caps it?
  7. What must be atomic in a single state transition? List everything that changes when processing becomes succeeded, and decide what is inside the transaction and what (like the webhook) deliberately is not.
  8. Webhooks arrive out of order — the succeeded delivery retries and lands after the merchant already processed a later event. Whose problem is ordering, and what do you put in the payload so the merchant can resolve it?

Do not scroll past this without written answers. The reference design will make sense either way — but it will only become yours if it confirms or breaks something you committed to first.

The Reference Design

Components

flowchart TD
    M["Merchant server"] -->|"API key over HTTPS"| API["Public API  auth and rate limit"]
    API --> INT["Intent Service  owns the state machine"]
    INT --> RT["Router  picks an adapter"]
    RT --> A1["Bank adapter SIMA  configurable modes"]
    RT --> A2["Bank adapter SIMB  configurable modes"]
    A1 --> AL[("adapter_log")]
    A2 --> AL
    INT --> LG[("ledger_postings")]
    INT --> WD["Webhook Dispatcher  HMAC and retries"]
    WD -->|"signed POST"| M
    RS["Recon Sweep  scheduled"] --> AL
    RS --> LG
    RS --> INT

The Intent Service is the only component allowed to change a payment’s state, and it changes state and writes postings in the same transaction. Adapters never touch the ledger; they only talk to their bank and write their own adapter_log — that separation is what makes reconciliation meaningful, because the recon sweep compares two independently written records.

The state machine

flowchart LR
    C["created"] -->|"adapter call starts"| P["processing"]
    C -->|"expiry passed"| X["expired"]
    P -->|"bank approved"| S["succeeded"]
    P -->|"bank declined"| F["failed"]
    P -->|"adapter timeout"| U["unknown"]
    U -->|"sweep finds a debit"| S
    U -->|"sweep finds no debit"| F

Every transition is one @Transactional method that loads the intent FOR UPDATE, verifies the current state is a legal source for this transition, applies it, writes any postings, and increments the transition’s metrics counter. Illegal transitions throw — a payment can never go failed → succeeded because a stale thread woke up late.

Key decisions

DecisionThe callWhy
Ledger first or bank first?Record a pending hold posting pair before the adapter call; finalise or reverse afterLedger-first means a crash leaves an over-cautious record you can reconcile; bank-first means money moved with no record — silent loss. Choose the recoverable wrong.
Where balances liveNowhere — derived as SUM over postings per accountA stored balance column can drift from its postings; a derived sum cannot disagree with itself. Never UPDATE a balance; only INSERT postings.
Idempotency mechanismUNIQUE (merchant_id, merchant_ref) in the database, hash-map cache in frontOnly the database constraint closes the two-concurrent-inserts race; the cache just makes replays cheap
Timeout handlingAdapter timeout moves the intent to unknown, never to failedThe bank may have debited. Declaring failure invents a fact you don’t have; the sweep finds the truth
Retry policyExactly one retry, only on adapter operations that accept an idempotency keyUnbounded retries amplify; retrying a non-idempotent debit is how double charges are manufactured
Webhook deliveryAt-least-once with HMAC signature; payload carries intent id, state, and a sequence numberExactly-once delivery is impossible; signed at-least-once plus a dedupe key on the merchant side is the industry contract
API key storageHashed at rest, like a password; shown once at creationA leaked database must not leak the ability to create payments

Double-entry in one table — the heart of the project

Every movement of money is two postings: a debit from one account and a credit to another, equal and opposite. Account balances are sums over postings. The whole system’s correctness collapses into one query: the sum of all postings is always exactly zero.

EventPosting 1Posting 2
Intent succeeds (Rs 500)debit customer_clearing 50000 paisecredit merchant_payable 50000 paise
Refund of that paymentdebit merchant_payable 50000 paisecredit customer_clearing 50000 paise

That’s the entire model. A refund is not an UPDATE, not a DELETE, not a status flag — it is a new, reversing pair of postings. History is append-only, which means history is auditable, which is why every real ledger from your bank’s core system to Razorpay’s works this way. The FOR UPDATE discipline from SQL — Transactions guards the writes; the derived balances from SQL — Aggregation read them; and an endpoint (then a scheduled job) asserts SELECT SUM(amount_paise) FROM ledger_postings equals zero — continuously, not hopefully.

Data model

TableKey columns
merchantsid, name, api_key_hash, webhook_url, webhook_secret, created_at
payment_intentsid, merchant_id, merchant_ref, amount_paise BIGINT, status, adapter, expires_at, created_at, updated_atUNIQUE (merchant_id, merchant_ref)
ledger_postingsid, intent_id, account, amount_paise BIGINT signed, created_at — append-only, no UPDATE or DELETE granted
webhook_deliveriesid, merchant_id, intent_id, event, sequence_no, payload, signature, attempts, status, next_attempt_at
adapter_logid, intent_id, adapter, request_at, response_at, outcome, bank_ref — written only by adapters

The three hardest problems

1. The ambiguous timeout — fail-after-debit forces it. In this mode the simulated bank debits the customer, then hangs past your timeout. Your switch sees only silence. If you mark the payment failed, the customer was charged for a “failed” payment — the worst outcome in payments. The solution is structural: timeout → unknown, and a scheduled recon sweep queries the adapter’s status endpoint (banks call this a status check or verify API — every real one has it precisely because of this problem) and resolves unknown to the truth. Until the sweep runs, the API honestly reports processing-equivalent, never a guess.

2. Double-entry discipline. The temptation to add a balance column will arrive around milestone 3 and it will feel reasonable. Resist it. The moment a balance is stored, it can disagree with its postings, and now you have two sources of truth and a reconciliation problem inside your own database. Postings are facts; balances are views of facts. If derived sums get slow later, that’s a materialisation problem with known solutions — it is never a reason to corrupt the model.

3. Webhook ordering. Your dispatcher retries, so a delayed payment.succeeded can arrive after a refund.created the merchant already handled. You cannot promise ordering over at-least-once delivery — nobody can. What you can do: put a per-intent sequence_no in every payload and document that merchants must ignore events older than the latest they’ve processed. Designing the contract you can keep, instead of pretending to one you can’t, is the lesson — the same one Dispatch taught from the other side.

Build Plan

Seven milestones. Each ends runnable. This is a 4–6 week build — the flagship — so protect the order: the ledger exists before webhooks, the sweep exists before deploy.

M1 — Design review, schema, walking skeleton. Written answers to the design questions reviewed against the reference design — note every place you differed and why. Full schema in schema.sql. Spring Boot app with merchant registration, API-key issue (hash stored, key shown once), an API-key auth filter, and POST /api/v1/intents that validates, enforces the UNIQUE constraint, stores a created intent, and returns it. Done when: two curls with the same merchant_ref return the same intent — one insert, no error leaked to the caller — and a request without a valid key gets 401.

M2 — Bank adapters and the call policy. An BankAdapter interface; two simulated implementations whose mode (success / decline / timeout / slow / fail-after-debit) is set per-request via a test header or config. Every call wrapped in a strict timeout and the one-retry policy with an idempotency key passed to the simulated debit. Adapters write adapter_log. Done when: each mode produces the documented outcome, slow completes within budget while timeout does not, and the log shows exactly two attempts max for any intent.

M3 — The ledger. ledger_postings append-only; posting pairs written inside the intent transaction; balances endpoint computing derived sums; GET /internal/assert-ledger returning the global sum and 200 only when it is zero. Seed a few hundred simulated payments and run the assertion. Done when: the assertion holds after a bulk run that includes declines, and a deliberately broken single-posting write (try it, then revert) makes the assertion fail loudly.

M4 — State machine and the recon sweep. All transitions as guarded @Transactional methods with FOR UPDATE; expiry handling; timeout → unknown; a scheduled sweep that queries the adapter status endpoint for unknown intents and resolves them — writing or reversing postings accordingly. Done when: a fail-after-debit run lands in unknown, the sweep resolves it to succeeded with correct postings, and the sum-zero assertion still holds.

M5 — Webhooks. On every state change, enqueue a delivery: HMAC-SHA256 signature over the payload with the merchant’s secret, retries with backoff, webhook_deliveries tracking every attempt, sequence_no per intent. A tiny local receiver script verifies signatures. Done when: a state change reaches the receiver with a valid signature, a deliberately failing receiver shows retries then exhaustion in the table, and replayed deliveries are detectable by sequence number.

M6 — Rate limiting and observability. Sliding-window limiter per merchant returning 429 with a Retry-After header; request-ID filter and %X{requestId} in the log pattern; /actuator/health; a counter metric per state transition. Done when: a burst past the limit gets clean 429s, one payment’s full journey is grep-able by a single request ID, and the transition counters match a hand-counted run.

M7 — Deploy, README, the demo. Deploy to a free tier (Render or Fly.io, with PostgreSQL on Neon’s free tier — same stack as week 9). README: what it is, the architecture diagram, run instructions, endpoint table, key decisions. Then script and rehearse the demo — a narrated curl flow: create intent → bank succeeds → show the two ledger postings → webhook lands at the receiver → recon report clean. Then the second act: a fail-after-debit run, the intent sitting honestly in unknown, the sweep catching it, postings appearing, assertion still zero. That second act is THE interview demo — it shows you engineering the failure, not just the happy path. Done when: both acts run top to bottom against the deployed instance, narrated out loud, under five minutes.

Definition of Done

  • Every endpoint behind API-key auth except registration and /actuator/health
  • Every request body validated; one consistent ApiError JSON shape for every failure; no stack trace ever reaches a merchant
  • Money is paise as long / BIGINT end to end; no double touches an amount anywhere
  • ledger_postings is append-only — no code path UPDATEs or DELETEs a posting, and the DB role doesn’t grant it
  • Sum-zero assertion passes after a bulk simulation that includes every adapter mode, induced timeouts, and sweep resolutions
  • Duplicate merchant_ref requests are replays, not errors and not duplicates — proven by a concurrent test firing the same create twice
  • All five adapter modes covered by tests; fail-after-debitunknown → sweep-resolved is an automated integration test
  • Webhook signatures verify; retries and exhaustion visible in webhook_deliveries
  • Tested: money math units, state machine transitions (legal and illegal), one full intent-to-webhook integration path
  • README with architecture diagram, decisions table, and the two-act curl walkthrough — runs clean against a fresh deploy

Stretch Goals

  1. Refunds as reversing postings — partial and full, with the rule that refunds can never exceed the captured amount; forces you to derive refundable balance from postings alone.
  2. Multi-adapter routing rules — route by amount or simulated success-rate, with per-adapter health derived from adapter_log; a taste of the smart-routing product Juspay sells.
  3. Settlement batching — a daily job that sweeps merchant_payable into per-merchant settlement records with a payout file; forces batch processing and one more double-entry leg (merchant_payablesettlement_pending).

Resume Bullets It Generates

  • Built a mini payment switch in Spring Boot and PostgreSQL with a double-entry ledger whose sum-zero invariant held across N simulated payments including M induced bank failures — verified by a continuous assertion, not by hope
  • Designed a payment state machine treating bank timeouts as a first-class unknown state, resolved by a scheduled reconciliation sweep that caught 100% of K induced fail-after-debit scenarios
  • Closed the duplicate-payment race with database-level idempotency (unique merchant reference), proven under concurrent duplicate requests
  • Implemented HMAC-signed merchant webhooks with at-least-once delivery, per-intent sequence numbers, and full delivery tracking across N events
  • Sustained p95 intent-creation latency of X ms under K concurrent requests behind a per-merchant sliding-window rate limiter

Never claim a number you did not measure. Run the load test, run the failure simulation, write down what actually happened — N, M, K, and X are an evening’s work with a script, and “measured” is the difference between a bullet and a story.

Interview Stories It Seeds

“How do payment systems avoid double charging?” Situation: my switch’s merchants retry on network blips, and my switch retries bank calls — both layers can duplicate. Task/Action: I made the merchant reference UNIQUE per merchant so a duplicate create replays the original response, and allowed exactly one switch-side retry, only on the debit call that accepts an idempotency key — then proved it with a concurrent-duplicates test. Result: I can explain the precise race the unique constraint closes that check-then-insert can’t, and why retry amplification means the policy must be designed top-to-bottom, not per-layer.

“The bank timed out — did the customer pay?” Situation: I built a bank adapter mode that debits and then goes silent, specifically to force this. Task/Action: timeout transitions the payment to a first-class unknown state instead of guessing failed; a scheduled reconciliation sweep queries the bank’s status API and resolves it, writing or reversing ledger postings accordingly. Result: in my induced-failure runs the sweep resolved every ambiguous payment with the ledger invariant intact — and I can walk through why declaring failure on a timeout is how customers get charged for “failed” payments.

“Why double-entry instead of a balance column?” Situation: RupeeRail derives every balance as a sum over append-only postings; nothing ever updates a balance. Task/Action: every money event writes an equal-and-opposite posting pair, refunds reverse rather than mutate, and a global sum-zero assertion runs continuously. Result: the invariant held over my full simulation including induced failures — and I can argue the deeper point: a stored balance can disagree with its history, a derived one cannot, which is why every real ledger is append-only.

“Walk me through a hard design trade-off you made.” Situation: ledger-first vs bank-first ordering when the crash hits between the two. Task/Action: I wrote out both failure modes — ledger-first leaves a cautious record with no money moved; bank-first moves money with no record — and chose ledger-first because a recoverable over-record beats silent loss, with the recon sweep as the cleanup path. Result: I can present a decision where there is no perfect answer, name the wrong I chose to be, and show the mechanism that bounds its cost — which is the actual shape of senior engineering decisions.

Project started? Add it to today’s tracker

Saves your progress on this device.