Scaling: Before You Buy More Servers, Read This

Series: Backend Engineering Fundamentals · Post 06 of 07 Level: Beginner-friendly · Read time: ~8 min

"We need to scale" is one of the most expensive sentences in engineering.

It triggers infrastructure discussions, migration projects, and architectural rewrites — often before anyone has looked at whether the current system is actually running at capacity.

Before scaling your infrastructure, understand what you're actually scaling for. Most systems that feel slow are bottlenecked by code problems (N+1 queries, missing indexes, synchronous calls that should be async) — not infrastructure capacity. Scaling a slow system gives you a more expensive slow system.

This post covers the actual mechanics of scaling, the tradeoffs between approaches, and how to think about it before opening a cloud console.

Vertical vs Horizontal Scaling

Vertical scaling (Scale Up): Add more resources to existing servers — bigger CPU, more RAM, faster disk.

Horizontal scaling (Scale Out): Add more servers and distribute the load across them.

Vertical Scaling                    Horizontal Scaling

[Server: 8 CPU, 32GB]      →       [Server: 4 CPU, 16GB] ×3
         ↓                                   ↓
[Server: 32 CPU, 128GB]             [Server: 4 CPU, 16GB] ×10
(one big machine)                   (many smaller machines)

In practice: Start with vertical scaling. It's simpler, faster, and often sufficient. Switch to horizontal when you hit the vertical ceiling or need high availability.

The Stateless Requirement for Horizontal Scaling

Horizontal scaling only works if your application is stateless — each request can be handled by any server, with no local state that makes one server "special."

❌ Stateful — Can't Scale Horizontally

Server 1: User session in memory → [Request for user A] works
Server 2: No session for user A  → [Request for user A] fails

✅ Stateless — Scales Horizontally

Server 1: No local state → reads session from Redis
Server 2: No local state → reads session from Redis
Server 3: No local state → reads session from Redis

Any server can handle any request.
Load balancer distributes freely.

The rule: Move all state out of your application servers and into shared storage (Redis for sessions, S3 for files, your database for persistent data). Your servers should be interchangeable.

# ❌ Stateful — in-memory session
app.sessions[user_id] = {"cart": items}  # Lives on one server only

# ✅ Stateless — session in Redis
redis.setex(f"session:{session_id}", 3600, json.dumps({"cart": items}))

Load Balancers — The Front Door to Your Scaled System

A load balancer distributes incoming requests across your pool of servers.

Internet
   ↓
[Load Balancer]
   ├── Server 1
   ├── Server 2
   └── Server 3

Load balancing algorithms:

Layer 4 vs Layer 7:

• L4 (TCP/UDP): Routes based on IP address and port. Extremely fast, no content inspection. AWS NLB, HAProxy in TCP mode.
• L7 (HTTP): Routes based on HTTP content (URL, headers, cookies). More flexible — route /api to one pool, /static to another. AWS ALB, NGINX, Traefik.

# NGINX: Layer 7 load balancing with upstream pools
upstream api_servers {
    least_conn;  # Least connections algorithm
    server app1.internal:8080 weight=3;
    server app2.internal:8080 weight=3;
    server app3.internal:8080 weight=1;  # Lower weight = less traffic
    
    keepalive 32;  # Connection pool to upstream servers
}

upstream static_servers {
    server static1.internal:8080;
    server static2.internal:8080;
}

server {
    location /api/ {
        proxy_pass http://api_servers;
    }
    location /static/ {
        proxy_pass http://static_servers;
    }
}

Database Scaling — Where It Gets Hard

Application servers are stateless and easy to scale. Databases are stateful and hard.

Read Replicas — The First Move

Most applications are read-heavy. Add read replicas and route SELECT queries there.

Primary DB (writes)
    ↓ replication
Replica 1 (reads)
Replica 2 (reads)
Replica 3 (reads)

Application:
  - INSERT / UPDATE / DELETE → Primary
  - SELECT → Random replica

# Connection routing example
def get_db_connection(read_only: bool = False):
    if read_only:
        return random.choice(replica_connections)
    return primary_connection

Limitation: Replication lag. Replicas are slightly behind the primary (usually milliseconds, but can grow under load). Don't read from a replica immediately after a write if you need the result.

Connection Pooling — Before You Add Replicas

Before adding replicas, make sure you're not wasting connections. Databases have a hard limit on concurrent connections. Without pooling, a spike in traffic can exhaust connections instantly.

# SQLAlchemy connection pool
engine = create_engine(
    DATABASE_URL,
    pool_size=20,          # Normal pool size
    max_overflow=30,       # Extra connections under load
    pool_timeout=30,       # Wait up to 30s for a connection before error
    pool_recycle=3600      # Recycle connections after 1 hour
)

For PostgreSQL at scale, use PgBouncer — a lightweight connection pooler that sits between your app and the database, multiplexing thousands of application connections onto a smaller number of actual DB connections.

Sharding — The Last Resort

When a single primary + replicas isn't enough, you shard: split your data across multiple databases.

User IDs 1–1M   → Database Shard 1
User IDs 1M–2M  → Database Shard 2
User IDs 2M–3M  → Database Shard 3

The costs are real:

• Cross-shard queries (JOINs across shards) become application logic
• Transactions across shards require distributed transaction handling
• Resharding (when a shard gets too large) is painful
• Every query needs shard-routing logic

Sharding adds enormous operational complexity. Exhaust all other options first: indexing, query optimization, read replicas, caching, connection pooling, vertical scaling.

Auto-Scaling — Elasticity, Not Magic

Auto-scaling adds or removes servers based on load. This is valuable for variable traffic patterns (traffic spikes on product launches, Black Friday, etc.).

# AWS Auto Scaling Group (simplified)
AutoScalingGroup:
  MinSize: 2          # Always at least 2 servers
  MaxSize: 20         # Never exceed 20 servers
  DesiredCapacity: 4  # Start with 4

ScalingPolicy:
  ScaleOut:
    Trigger: CPUUtilization > 70% for 2 minutes
    Action: Add 2 instances
  ScaleIn:
    Trigger: CPUUtilization < 30% for 10 minutes
    Action: Remove 1 instance

Auto-scaling pitfalls:

1. Cold start time: If spinning up a new instance takes 3 minutes, it won't help with a traffic spike that peaks in 1 minute. Pre-warm with a higher minimum capacity.

2. Scale-in aggressiveness: Removing servers too aggressively causes thrashing (scale up, scale down, scale up again). Add a cooldown period.

3. Database doesn't scale automatically: Auto-scaling your app tier is useless if your database becomes the bottleneck. Ensure your DB can handle the connection surge from new instances.

4. Stateful sessions: If you forgot the stateless requirement, auto-scaling will cause session loss when a server is removed.

CDN for Static Assets — The Easiest Win

Before spending time on application scaling, ask: how much of your traffic is serving static files (JS, CSS, images)?

A CDN serves these from edge locations close to users, eliminating the load from your application servers entirely.

Without CDN:
User (Tokyo) → [Internet] → App Server (US East) → serve image (300ms)

With CDN:
User (Tokyo) → CDN Edge (Tokyo) → serve cached image (8ms)

This also reduces bandwidth costs, since CDN egress is typically cheaper than cloud server egress.

What to cache on CDN:

• All static assets with content-hash filenames (infinite TTL, cache-busted on deploy)
• API responses that are public and change infrequently (product catalog, pricing)
• Rendered HTML pages for anonymous users (massive scale lever for content sites)

Scaling Checklist — Before Adding Servers

Run through this before any infrastructure change:

• Are queries using indexes? (EXPLAIN ANALYZE your slow queries)
• Is there N+1 query behavior in the application?
• Is connection pooling configured? (PgBouncer, HikariCP, SQLAlchemy pool)
• Are static assets served via CDN?
• Is read traffic separated to replicas?
• Are expensive computations cached?
• Are long-running operations async (queues) instead of blocking request threads?
• Is the application stateless (sessions in Redis, files in S3)?

Tick all of these before scaling horizontally. You'll likely find the bottleneck isn't what you thought.

Key Takeaways

• Scale vertically first — it's simpler and often enough
• Stateless design is the prerequisite for horizontal scaling — move all state to shared storage
• Load balancers distribute traffic; Layer 7 gives you routing flexibility
• Read replicas are the first database scaling move — they solve most read-heavy bottlenecks
• Connection pooling (PgBouncer) often eliminates "database can't scale" problems cheaply
• Sharding is a last resort — the complexity cost is real
• CDN and query optimization have better ROI than new servers in most systems
• Profile first. Most slow systems are code problems, not infrastructure problems.

What bottleneck surprised you most when your system first started struggling under load — was it what you expected?

Next in the series → Post 07: You Can't Manage What You Can't See — The Three Pillars of Observability

You've built and scaled your system. Now: how do you know it's working?