Scaling & Distributed Systems

Every system that succeeds eventually meets the same enemy: more users than one machine can serve. This page is the story of what you do about that — told from first principles, with no hand-waving. We’ll define the words everyone uses but rarely explains (latency, throughput, statelessness, CAP), and we’ll anchor every idea to how Assure PAT actually runs in production: serverless, on AWS Lambda. By the end you’ll understand why a stateless API can be cloned a thousand times, why Assure’s cron jobs refuse to run on Lambda, and why there is no such thing as a “best” architecture.

⏱ ~22 min · 🔴 Advanced · Prerequisites: pages 1–5 (you should be comfortable tracing a request from browser to database)

1. The problem: one server, too many users

Imagine Assure PAT runs on exactly one computer. It works beautifully for 10 testers. Then Ayris onboards 50 organisations, and one morning 4,000 people open the dashboard at once. The single server starts to choke. To talk about why it chokes — and what to do — we need three precise words.

The three words that describe every performance problem

• Latency — how long one request takes, start to finish. Measured in milliseconds. “When I click Save, the spinner shows for 200 ms.” Lower is better. Think: speed for one person.
• Throughput — how many requests the system finishes per second. Measured in requests/second (req/s). “The server handles 800 req/s before it falls over.” Higher is better. Think: capacity for the crowd.
• Bottleneck — the one slowest piece that caps the whole system. A chain is only as fast as its slowest link; a system’s throughput is capped by its busiest component (usually the database, sometimes the network, sometimes CPU).

These are independent. A system can be fast for one user (low latency) but collapse under a crowd (low throughput), or vice versa. Scaling is the craft of raising throughput without wrecking latency — and finding the bottleneck before your users do.

flowchart LR
    Users(["4,000 users"]) -->|all at once| S["One server<br/>(maxes out CPU)"]
    S -->|every query queues| DB[("Database")]
    S -.->|"requests pile up<br/>latency climbs"| Users

Read it aloud: “Thousands of users hit one server; it can only do so much work per second, so requests queue, and everyone waits longer.” That queue is the latency problem. The fix is to add capacity. There are exactly two ways to do that — and the choice defines everything after.

2. Two ways to scale: up vs out

When one machine isn’t enough, you can make the machine bigger, or you can add more machines.

• Vertical scaling (scale up) — replace your server with a beefier one. More CPU cores, more RAM, faster disk. Same single box, just stronger. Like hiring one super-chef who works twice as fast.
• Horizontal scaling (scale out) — keep the same modest machines but run many of them side by side, splitting the load. Like hiring ten ordinary cooks instead of one super-chef.

Why horizontal wins eventually

Vertical scaling is the right first move because it’s free of engineering work — you just pay for a bigger box. But it has a wall: there is a single largest machine in existence, and once you hit it you’re stuck. Worse, that one machine is a single point of failure — when it dies, everything dies. Horizontal scaling has no wall (add another box) and no single point of failure (lose one, the rest serve). Every system that grows past a point goes horizontal. The whole rest of this page is really about what it takes to go horizontal safely.

Don’t just read the difference — watch it. Toggle the visualizer below between scale-up and scale-out, then trigger a server failure and see why one box dying takes down the vertical setup while the horizontal pool just routes around the dead node through its load balancer.

3. Statelessness — the key that unlocks horizontal scaling

Here’s the question that decides whether you can go horizontal: if I clone this server, do the clones behave identically?

They do — only if the server keeps no important information in its own memory between requests. That property has a name: statelessness.

Stateful vs stateless, in one line each

• Stateful — the server remembers things between requests in its own memory (e.g. “user 42 is logged in, here’s their half-filled shopping cart”). Clone it, and clone #2 has no idea who user 42 is.
• Stateless — the server remembers nothing between requests. Every request carries everything it needs (a JWT token, an ID), and all shared truth lives in the database, not in the server. Any clone can handle any request.

Why does statelessness unlock cloning? Because if no server holds unique memory, the servers are interchangeable. Request 1 can go to clone A, request 2 to clone B, request 3 back to clone A — nobody needs to remember anybody. You can spin up a thousand identical copies and they all just work.

How Assure does this

Assure’s API is stateless by design. Look back at page 4: authentication is a JWT (bhtoken) that the client stores and sends with every request. The server doesn’t keep a session in memory — it reads the token, verifies it, learns who you are and which org you’re scoped to, and answers. All shared truth lives in MySQL. That’s the whole reason the next part is even possible.

Because the API is stateless, Assure runs it on AWS Lambda via a library called serverless-http. Lambda is “serverless” — you don’t manage servers at all. Instead, AWS runs your code on demand and, crucially, spins up as many copies of it as the incoming traffic needs, then tears them down when traffic drops. That’s horizontal scaling, fully automatic, because the code is stateless and clonable.

flowchart TD
    Req(["Incoming requests"]) --> AWS{"AWS Lambda<br/>(auto-scales)"}
    AWS --> C1["API copy 1"]
    AWS --> C2["API copy 2"]
    AWS --> C3["API copy 3 ... N"]
    C1 --> DB[("RDS MySQL<br/>(the one shared truth)")]
    C2 --> DB
    C3 --> DB

The real consequence: where did the cron jobs go?

Statelessness and serverless buy you free scaling — but there’s a catch that bites real teams, and Assure hit it. Serverless means there is no always-on process. A Lambda exists only for the few hundred milliseconds it takes to answer one request, then it vanishes. So scheduled background jobs (cron) have nowhere to live — there’s no long-running process to “wake up every hour and do X.”

Assure handles this honestly. The codebase detects how it was launched using Node’s require.main check: if it’s running as a normal local process (a real, always-on server you started by hand), it starts the cron jobs. If it’s running inside Lambda (imported by serverless-http, not the main entry point), it skips the crons entirely. Scheduled work on the cloud side is triggered a different way — by an external scheduler (e.g. AWS EventBridge) that invokes a Lambda on a timer, rather than a cron loop living inside the app.

The lesson

Going serverless is not “free.” You trade away the always-on process. Anything that assumed “there’s a server sitting here 24/7” — in-memory caches, background loops, cron timers, WebSocket connections — needs a rethink. Assure’s require.main switch is exactly that rethink, made explicit in code.

4. Load balancing — the traffic cop

Once you have N identical app servers, something has to decide which copy gets each request. That something is a load balancer (LB).

Picture a traffic cop at a junction with five open toll booths. Cars arrive; the cop waves each one to whichever booth is free. Drivers don’t choose a booth and don’t even know how many booths exist — they just see “the entrance.” The load balancer is that cop: clients send every request to one address, and the LB quietly fans them out across the pool.

How it decides whom to send where (the algorithm):

• Round-robin — booth 1, booth 2, booth 3, back to 1… simple and fair when all servers are equal.
• Least-connections — send to whichever server is currently handling the fewest requests. Smarter when some requests are heavy.
• IP-hash — always send the same client to the same server (used when you can’t be fully stateless).

Health checks — the LB quietly pings each server (“are you alive?”). If a server stops answering, the LB stops sending it traffic until it recovers. This is why horizontal scaling survives failures: a dead box is simply skipped.

flowchart TD
    Client(["Clients"]) --> LB["Load Balancer<br/>(round-robin + health checks)"]
    LB --> A["App server 1 ✅"]
    LB --> B["App server 2 ✅"]
    LB --> C["App server 3 ❌ (failed check, skipped)"]
    A --> DB[("Shared database")]
    B --> DB

With Assure on Lambda, AWS API Gateway + Lambda do this job for you — there’s no LB box you manage. AWS routes each request to a healthy execution and scales the pool. You get load balancing as a property of the platform, not a server you babysit. But the concept is universal: the moment you have more than one app copy, something must route between them.

5. Caching — don’t do the same work twice

A cache is a small, fast store that keeps a copy of an answer you computed (or fetched) recently, so the next time someone asks the same question you hand back the saved copy instead of doing the work again.

Why bother? Two wins at once:

1. Speed (latency) — reading from memory is ~1,000× faster than a database query over the network.
2. Load relief (throughput) — every request the cache answers is a request the database never sees, so the bottleneck (usually the DB) breathes easier.

Where caches live — there’s a cache at almost every layer of the journey:

The hard part: cache invalidation

A cache holds a copy. The moment the real data changes, every copy is potentially wrong. Deciding when to throw away a stale copy is famously one of the genuinely hard problems in the field:

The famous quote

“There are only two hard things in Computer Science: cache invalidation and naming things.” — Phil Karlton

It’s a joke, but it’s true. A cache that serves stale data is worse than no cache — it shows users wrong answers and is maddening to debug (“but the database says X!”). Every cache needs a clear rule for when an entry expires or is purged.

Where Assure stands today — be honest

Assure’s core path is mostly database-direct: a request comes in, the API queries MySQL, returns the answer. There isn’t a heavy application-layer cache (like Redis) on the hot path today, and that’s a perfectly reasonable place to be — caching is something you add when measurements show the DB is the bottleneck, not before. Adding a small cache for read-heavy, rarely-changing data (e.g. profile templates, an org’s static config) is a classic next step when Assure’s read traffic grows. The right time to add it is when a profiler says so — premature caching just adds the invalidation headache for no proven gain.

6. Scaling the database — the part that’s actually hard

App servers are easy to clone (they’re stateless). The database is not — it holds the one shared truth, and you can’t just run ten copies that each think they’re correct. So databases get their own scaling toolkit, cheapest-first.

6.1 Indexing — the cheapest first win

An index is like the index at the back of a textbook: instead of reading all 900 pages to find “CAP theorem,” you jump straight to page 412. Without an index, the database scans every row to find matches (a “full table scan”). With one, it jumps straight to the rows. Adding the right index can turn a 2-second query into a 2-millisecond one — for the price of a one-line migration. Always the first thing to try when a query is slow.

6.2 Connection pooling — reuse, don’t reconnect

Opening a fresh database connection is expensive (a handshake, authentication, setup). If every request opened and closed its own connection, you’d waste enormous time on setup alone. A connection pool keeps a small set of connections open and ready, and hands one to each request, then takes it back when done — like a tool library instead of buying a new drill for every job.

Assure uses Knex’s connection pools for exactly this. But serverless adds a sharp twist:

The serverless connection-limit gotcha — Assure handles this

RDS (the managed MySQL) allows only a limited number of simultaneous connections. On Lambda, AWS may spin up hundreds of API copies under load — and if each copy opens a full-size pool, they collectively exhaust RDS’s connection limit and the database starts refusing everyone. Assure mitigates this deliberately: the cron Lambda uses a smaller pool than the main API, so the scheduled/background path can’t gobble up the connections the user-facing API needs. Sizing pools per-Lambda-role is a serverless-specific discipline, and Assure does it.

6.3 Read replicas — one writer, many readers

Most apps read data far more than they write it (think: thousands of “show me this profile” for every one “save this profile”). So split the work:

• One primary handles all writes (the single source of truth).
• N replicas are read-only copies that the primary continuously streams its changes to. All the reads fan out across them.

This multiplies read throughput without the impossible problem of multiple writers disagreeing.

flowchart TD
    App["App servers"] -->|writes only| P[("Primary DB<br/>(writes)")]
    P -->|"replicates changes"| R1[("Read replica 1")]
    P -->|"replicates changes"| R2[("Read replica 2")]
    App -->|reads| R1
    App -->|reads| R2

The catch: replication lag

Replicas trail the primary by a few milliseconds (sometimes more). So just after a write, a read from a replica might return the old value — the change hasn’t streamed over yet. This is your first taste of eventual consistency (section 9). It’s usually fine for “show the list,” and a problem for “did my save take?” — so you read that from the primary.

Assure today runs against its RDS MySQL primary directly. Read replicas are a standard growth lever to reach for when read traffic — dashboards, profile lists, audit-trail views — starts to dominate and the primary feels the heat. Not built yet; a clear, well-understood next step.

7. Asynchronous work & queues — stop making users wait

Some work is slow and the user doesn’t need to watch it finish: sending an email, generating a PDF report, processing an uploaded file, encrypting a batch of cards. If you do that work inside the request, the user stares at a spinner for 8 seconds. Bad.

The fix is to decouple the slow work from the request using a queue:

1. The request does the fast part, drops a “please do this later” message onto a queue, and immediately replies to the user (“Report is being generated — we’ll notify you”).
2. A separate worker process picks messages off the queue and does the slow work in the background, at its own pace.

flowchart LR
    User(["User clicks<br/>'Generate report'"]) --> API["API"]
    API -->|"1: drop message"| Q["Queue"]
    API -.->|"2: instant reply<br/>'working on it'"| User
    Q -->|"3: pick up later"| W["Worker"]
    W -->|"4: do slow work"| DB[("DB / S3")]
    W -.->|"5: notify when done"| User

The user gets a snappy response; the heavy lifting happens out of band. Queues also absorb spikes (10,000 reports requested at once just sit in line) and survive worker crashes (the message stays until someone successfully processes it).

Assure’s request path is mostly synchronous today. Queues + workers (e.g. AWS SQS + a worker Lambda) are the standard pattern to reach for when a feature involves genuinely slow work — bulk card generation, large report exports — that shouldn’t block the user. Another well-understood place Assure can grow.

8. The Assure serverless deployment, drawn out

Let’s assemble everything into the real production picture. Assure is serverless on AWS: there’s no server you SSH into; AWS runs the stateless Express API on Lambda, scales it automatically, and surrounds it with managed services.

flowchart TD
    Browser(["🌐 Browser<br/>(React dashboard)"]) -->|HTTPS| GW["AWS API Gateway<br/>(the single entrance)"]
    GW -->|invokes| L["AWS Lambda<br/>(Express via serverless-http)"]
    L -->|SQL over pool| RDS[("Amazon RDS<br/>MySQL")]
    L -->|files / artifacts| S3[("Amazon S3")]
    L -->|"encrypt / decrypt keys"| KMS["AWS KMS<br/>(key management)"]

Reading the boxes:

• API Gateway — the single public entrance. Receives every HTTPS request and invokes a Lambda. Also does routing, throttling, and acts as the “load balancer” you never manage.
• Lambda — your stateless Express app, wrapped by serverless-http. AWS runs as many copies as traffic demands.
• RDS (MySQL) — the managed database; the one shared source of truth.
• S3 — object storage for files and artifacts (uploads, exports, snapshots).
• KMS — Key Management Service; guards the encryption keys used for the card crypto (page on data modeling covered those 35+ encrypted columns).

Two serverless realities to keep in mind:

Cold starts

When a request arrives and no warm Lambda copy is sitting ready, AWS must boot a fresh one — load the code, start Node, connect to the DB. That first request eats an extra few hundred milliseconds to a couple of seconds: a cold start. Subsequent requests hit the now-warm copy and are fast. It’s the tax you pay for “scale to zero when idle.” Mitigations exist (provisioned concurrency, keep-warm pings), each with a cost tradeoff.

The RDS connection-limit gotcha (again, because it matters)

As covered in section 6.2: hundreds of Lambda copies × a full pool each can blow past RDS’s connection cap. This is the classic serverless-meets-relational-database trap. Assure’s per-role pool sizing (smaller pool for the cron Lambda) is a direct, deliberate defence. In bigger systems the heavier hammer is a connection proxy (e.g. RDS Proxy) that pools connections outside the Lambdas.

9. CAP theorem — the law of distributed systems, in plain words

The moment your data lives on more than one machine (read replicas, multiple database nodes), you’re in distributed systems territory, and a hard truth applies. Three properties you’d love to have, all at once:

• Consistency (C) — every reader sees the same, latest data. Write something, and every node immediately agrees on the new value. (Note: a different meaning from database “ACID consistency” — here it means all nodes agree.)
• Availability (A) — every request gets a (non-error) answer, every time, even if some machines are down.
• Partition tolerance (P) — the system keeps working even when the network between machines breaks (a “partition” — nodes can’t talk to each other).

The CAP theorem, stated simply

When a network partition happens (and over enough time, it always does), you can keep only two of the three — and since you can’t wish away network failures, P is mandatory. So the real choice is: during a partition, do you sacrifice Consistency or Availability?

Make it concrete. Two database nodes, the cable between them snaps. A write lands on node A. Node B can’t hear about it. A reader now hits node B:

• Choose Consistency (CP): node B refuses to answer (or errors) rather than risk serving stale data. You stayed correct but gave up availability.
• Choose Availability (AP): node B answers with the old value rather than error out. You stayed up but served stale data.

There is no third door. The network will fail eventually; you only get to pre-decide which way you’d rather fail.

flowchart TD
    P{"Network partition happens<br/>(nodes can't talk)"} -->|"Choose Consistency"| CP["Refuse / error<br/>rather than serve stale<br/>(correct, but down)"]
    P -->|"Choose Availability"| AP["Answer with old data<br/>rather than error<br/>(up, but stale)"]

Eventual consistency — the most common AP-style compromise. The system stays available and serves slightly-stale data now, but promises that once the partition heals, all nodes will converge to the same value. You met this already as replication lag in section 6.3: a read replica that’s a few milliseconds behind is “eventually consistent” with the primary. For “show the dashboard list,” eventual consistency is totally fine. For “did my payment go through,” you want strong consistency. Pick per use case.

You’ve now met the four ideas that trip people up most. Test whether they stuck before you read the framing section.

10. The golden framing: there is no best architecture

If you remember one sentence from this entire page, make it this:

The senior engineer’s mantra

There is no best architecture — only tradeoffs. Every choice on this page buys you one thing by spending another.

Look at what each lever costs:

A junior engineer asks “what’s the best database / framework / architecture?” A senior engineer asks “best for what — what are we optimising, and what are we willing to give up?” Assure’s choices read straight off this table: serverless (trading cold-start latency and the connection gotcha for near-zero idle cost and automatic scaling), stateless (paying the discipline of “no in-memory state, crons can’t live here” to unlock that scaling), and DB-direct-for-now (declining caching’s complexity until measurements justify it). Every one of those is a tradeoff someone chose on purpose — not an accident, and not “the best.”

Recap — the mental model to keep

• Latency = speed for one. Throughput = capacity for the crowd. Bottleneck = the slowest piece that caps both.
• Vertical scaling (bigger box) is simple but has a ceiling and a single point of failure; horizontal scaling (more boxes) wins eventually.
• Statelessness is the key that unlocks horizontal scaling — and the reason Assure runs on Lambda, and the reason its crons can’t run there (require.main switch).
• Load balancers route across clones with health checks; on Assure, API Gateway + Lambda do this for you.
• Caching trades consistency for speed; the hard part is invalidation; Assure is DB-direct today and can add caching when measured.
• Scale the DB cheapest-first: indexing → connection pooling (mind the serverless connection limit) → read replicas.
• Queues + workers decouple slow work so users don’t wait.
• CAP: during a network partition, choose Consistency or Availability — you can’t have both. Eventual consistency is the common compromise (replication lag is a live example).
• There is no best architecture, only tradeoffs: latency vs cost vs complexity vs consistency.

Check Yourself

Cover the answers. Say each out loud before you peek — retrieval is what builds the memory.

Still Unclear?

Paste any of these into Claude to go deeper on the parts that didn’t click:

• “Walk me through what happens to an in-flight user session when a load balancer marks one of my app servers unhealthy mid-request. Use a stateless JWT setup like Assure’s as the example.”
• “I have a Lambda + RDS MySQL setup and I’m hitting ‘too many connections’ errors under load. Explain exactly why this happens with hundreds of Lambda copies, and walk me through per-role pool sizing vs RDS Proxy as fixes.”
• “Give me three real product features and for each tell me whether I should choose CP or AP (strong vs eventual consistency), and why. Then quiz me on three more.”

Why AI Can’t Do This For You

AI can recite the CAP theorem and list the scaling levers — that’s the easy half. What it can’t do is make the judgment call for your specific system: whether this query is the bottleneck worth indexing, whether this feature needs strong consistency or can tolerate replication lag, whether now is the moment to add a cache or whether that’s premature complexity you’ll regret. Every row in the tradeoff table is a decision that depends on what you’re optimising and what you’re willing to give up — and only you know your traffic, your data, your failure tolerance, and your team’s appetite for complexity. The senior engineer’s mantra — “best for what?” — is exactly the question a model can’t answer without the context that lives in your head. Reading this page is how you build the judgment to answer it.

Next, we put all eight pages together on the Case Study page — the entire Assure PAT system, drawn at context, container, and component zoom levels, with every concept from this track labelled in place. 👉

Module done? Add it to today’s tracker