🚦 Rate Limiting — System Design

What is Rate Limiting?

Rate limiting is a technique to control how many requests a user (or client) can make to your API within a specific time window. It's like a traffic cop for your servers — it makes sure no single user hogs all the resources.

Why Do We Need It?

• Prevent abuse: Stop malicious users or bots from overloading your servers
• Ensure fairness: One user shouldn't consume all the resources meant for everyone
• Protect revenue: If you have paid tiers, rate limiting enforces usage quotas
• Manage costs: Every API call costs money (compute, bandwidth, etc.)
• Improve reliability: Prevents cascading failures during traffic spikes

Common Use Cases

Rate limiting is used across many different scenarios:

Types of Rate Limiting

There are several ways to identify and limit traffic. Each has its advantages and limitations:

1. IP-Based Rate Limiting

Limits the number of requests from a single IP address within a specified time period.

2. Server-Based Rate Limiting

Controls the number of requests to a specific server within a time period.

3. Geography-Based Rate Limiting

Restricts traffic based on the geographic location of the IP address. Useful for blocking malicious regions.

4. User/API Key-Based Rate Limiting

The most granular approach — limits requests per authenticated user or API key. Essential for SaaS and paid API tiers.

Requirements & Constraints

What We Need to Build

• Per-user limits: Limit each API key to X requests per minute/hour
• Per-IP limits: As a fallback for unauthenticated requests
• Low latency: The rate check should add minimal overhead (target: <2ms)
• High availability: Rate limiting shouldn't become a single point of failure
• Real-time policy updates: Change limits without restarting services

High-Level Design

At a high level, when a request comes in, your API Gateway asks the Rate Limiter: "Should I allow this request?" The Rate Limiter checks a shared datastore (usually Redis) and returns a yes/no decision.

Key Components

Rate Limiting Algorithms

There are three main algorithms for rate limiting. Each has its own trade-offs.

Which Algorithm Should You Use?

Implementation with Redis

Redis is the go-to choice for rate limiting because it's blazing fast, supports atomic operations, and has built-in key expiration.

Data Model

For each user, we store: tokens (remaining) and ts (last refill timestamp).

Key format: rl:user:{user_id}

Lua Script (Atomic Token Bucket)

-- Token Bucket Rate Limiter (Redis + Lua)
local key = KEYS[1]
local rate = tonumber(ARGV[1])           -- tokens added per second
local capacity = tonumber(ARGV[2])       -- max tokens in bucket
local now = tonumber(ARGV[3])            -- current timestamp
local requested = tonumber(ARGV[4])      -- tokens needed

local data = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(data[1]) or capacity
local last_ts = tonumber(data[2]) or now

local elapsed = math.max(0, now - last_ts)
local refill = math.min(capacity, tokens + elapsed * rate)

if refill < requested then
  return 0  -- REJECT
else
  local new_tokens = refill - requested
  redis.call('HMSET', key, 'tokens', new_tokens, 'ts', now)
  redis.call('EXPIRE', key, 3600)
  return 1  -- ALLOW
end

Python Implementation

Now let's implement rate limiting in Python with multiple approaches.

1. Token Bucket (Python + Redis)

import time
import redis

class TokenBucketRateLimiter:
    def __init__(self, redis_client, rate, capacity):
        self.redis = redis_client
        self.rate = rate
        self.capacity = capacity
        self.lua_script = self.redis.register_script("""
            local key = KEYS[1]
            local rate = tonumber(ARGV[1])
            local capacity = tonumber(ARGV[2])
            local now = tonumber(ARGV[3])
            local requested = tonumber(ARGV[4])
            
            local data = redis.call('HMGET', key, 'tokens', 'ts')
            local tokens = tonumber(data[1]) or capacity
            local last_ts = tonumber(data[2]) or now
            
            local elapsed = math.max(0, now - last_ts)
            local refill = math.min(capacity, tokens + elapsed * rate)
            
            if refill < requested then
                return {0, refill}
            else
                redis.call('HMSET', key, 'tokens', refill - requested, 'ts', now)
                redis.call('EXPIRE', key, 3600)
                return {1, refill - requested}
            end
        """)
    
    def allow_request(self, user_id, tokens_requested=1):
        key = f"rl:user:{user_id}"
        result = self.lua_script(keys=[key], args=[self.rate, self.capacity, time.time(), tokens_requested])
        return bool(result[0]), float(result[1])

# Usage
r = redis.Redis(host='localhost', port=6379)
limiter = TokenBucketRateLimiter(r, rate=10, capacity=100)
allowed, remaining = limiter.allow_request("user_123")
print(f"Allowed: {allowed}, Remaining: {remaining}")

2. Flask Decorator Example

from functools import wraps
from flask import Flask, request, jsonify
import redis, time

app = Flask(__name__)
redis_client = redis.Redis(host='localhost', port=6379)

def rate_limit(max_requests, window_seconds):
    def decorator(f):
        @wraps(f)
        def wrapped(*args, **kwargs):
            key = f"rl:{f.__name__}:{request.remote_addr}"
            pipe = redis_client.pipeline()
            now = time.time()
            pipe.zremrangebyscore(key, 0, now - window_seconds)
            pipe.zcard(key)
            pipe.zadd(key, {str(now): now})
            pipe.expire(key, window_seconds)
            results = pipe.execute()
            
            if results[1] >= max_requests:
                return jsonify({"error": "Rate limit exceeded"}), 429
            return f(*args, **kwargs)
        return wrapped
    return decorator

@app.route('/api/data')
@rate_limit(max_requests=10, window_seconds=60)
def get_data():
    return jsonify({"data": "success"})

Scaling & Performance

Edge Caching

Maintain local counters at each edge node. These sync with Redis periodically. Trade-off: slight overshoot during sync delays.

Sharding

Use consistent hashing to distribute keys across multiple Redis nodes.

Batching

Batch counter updates (10-50 requests at once) to reduce Redis write QPS.

Availability & Fault Tolerance

Key Trade-offs

Monitoring & Metrics

• Request rate by key: Who's hitting your API and how much?
• Rejection rate: How often are limits being hit?
• Redis latency: Is the rate limiter adding too much overhead?
• Near-limit alerts: Warn when a customer is at 80% of their quota

Rate Limiting in Different Layers

Real-World Examples

Video Resources

Conclusion

Rate limiting is a critical piece of any production API. It protects your services, ensures fair usage, and gives you control over your system's resources.