Core Code
Core Code Modern Dev Handbook
SYSTEM-DESIGN Contents
Constraints & Core Metrics
What Is System Design (Production-First) Functional vs Non-Functional Requirements Constraints, Assumptions, and Design Inputs Latency vs Throughput Tail Latency (p95/p99) and Why It Dominates UX SLI, SLO, SLA: The Reliability Contract Capacity Estimation Basics Bottlenecks and Queues (Little's Law Intuition)
Scaling & Traffic Management
Vertical vs Horizontal Scaling Load Balancing Fundamentals Caching Strategies Overview Cache Invalidation and Consistency Rate Limiting at Scale Backpressure: Preventing Overload Timeouts and Retries (Safe Defaults) Graceful Degradation and Traffic Shedding
Data & Consistency
Database Scaling Basics Replication: Read Replicas and Failover Consistency Models (Strong vs Eventual) Sharding Basics Partitioning and Hotspot Avoidance Idempotency and Deduplication Eventual Consistency and UX Schema Evolution and Backward Compatibility
Reliability Patterns
Timeouts: Bounding Damage Retry Strategies (Jitter, Caps, Budgets) Circuit Breaker Bulkhead Pattern Fallbacks and Graceful Degradation Load Shedding and Admission Control Backups and Restore Strategy Disaster Recovery (RPO/RTO)
Observability & Control
Golden Signals: Latency, Traffic, Errors, Saturation Metrics: SLIs and SLO Targets Logging at Scale (Structure and Cost) Correlation IDs and Context Propagation Distributed Tracing Overview Alerting and Runbooks Capacity Planning and Saturation Postmortems and Continuous Improvement
Case Studies
Case Study: URL Shortener Case Study: File Upload and Serving Case Study: Notification System Case Study: Chat System Overview Case Study: Rate Limiter Service Case Study: Social Feed (Fan-out Trade-offs) Case Study: Search Autocomplete Case Study: Analytics Ingestion Pipeline
Distributed Systems Foundations
CAP Theorem Consensus Basics Leader Election Distributed Locking Clock Skew Split Brain Quorum Reads and Writes Eventual vs Strong Consistency (Practical)
Service Communication Patterns
Synchronous vs Asynchronous Request-Response Pattern Message Queues Pub/Sub Pattern Event-Driven Architecture Saga Pattern CQRS Overview Outbox Pattern API Gateway Pattern Backend For Frontend (BFF)
Data at Scale Advanced
Index Design Deep Dive Query Planning Multi-Region Databases Read-After-Write Guarantees Global Databases Data Locality Large Table Strategies Background Jobs Design
Multi-Region Architecture
Multi-Region Basics Active-Active Active-Passive Geo DNS Cross-Region Replication Global Latency Optimization Regional Failover
Cost-Aware System Design
Cost vs Performance Storage Cost Trade-offs Cache Cost Analysis Overprovisioning vs Risk Cloud Pricing Mental Model Right-Sizing Instances
Security Architecture
Zero Trust Service-to-Service Auth Secrets Management Encryption at Rest Encryption in Transit DDoS Protection Multi-Tenant Data Isolation
System Design Interview
Design Twitter Design YouTube Design Uber Design Payment System Design E-commerce Platform Design Distributed Cache

Rate Limiting at Scale

Rate limiting protects availability and cost. Design it with burst behavior, identity strategy, and distributed enforcement in mind.
On this page

What Rate Limiting Protects

Rate limiting protects availability, cost, and downstream dependencies. It is not only an anti-abuse tool; it is a stability mechanism. Without limits, a small spike or a single bad actor can saturate your bottlenecks and collapse tail latency.

Decide What You Rate Limit By

  • IP: simple but noisy (NAT, mobile carriers)
  • User or API key: best for authenticated APIs
  • Tenant: necessary for multi-tenant fairness
  • Route: protect expensive endpoints separately

Core Algorithms

  • Token bucket: allows bursts up to a limit; common for APIs.
  • Leaky bucket: smooths traffic; good for steady processing.
  • Fixed window: simple but can allow burst at boundaries.
  • Sliding window: more accurate; slightly more complex.

Burst Behavior Matters

Production traffic is bursty. If you limit strictly without burst allowance, you may reject legitimate spikes (page refresh storms, deploy-triggered retries). Token bucket is often preferred because it supports controlled bursts.

Distributed Enforcement

With multiple instances, rate limiting must be consistent. You can enforce limits at the edge (gateway/load balancer), or centrally using a shared store (Redis). Edge enforcement is fast; centralized enforcement is more accurate per identity.

What to Return

Rate-limited requests should return HTTP 429 with a stable error code and, when possible, retry guidance.

HTTP 429 example response:
{
  "error": {
    "code": "RATE_LIMITED",
    "message": "Too many requests",
    "retryAfterMs": 1000
  }
}

Production-First Takeaway

Rate limiting is an availability feature. Define the identity (user/tenant), choose an algorithm with the right burst behavior, enforce consistently across instances, and make the response predictable for clients.

Backpressure: Preventing Overload →