Cloud Storage Design (S3)

High-Level Architecture

The core principle is separation of concerns: Metadata (the directory structure) is stored separately from Block Data (the actual file content).

Component Responsibilities

• API Service: Authentication (IAM), Rate Limiting, and Request Orchestration.
• Metadata Store: Stores `Buckets`, `ObjectKeys`, `ACLs`. Needs fast Key-Value lookups.
• Block Store: Stores immutable chunks of data (Blobs). Optimized for throughput (MB/s).

Data Partitioning

You cannot store Petabytes on one machine. We rely on Consistent Hashing to distribute data across thousands of nodes.

Erasure Coding vs Replication

How does S3 achieve 11 9s of durability without bankrupting AWS?

Implementation: Multipart Upload (Python Boto3)

For large files (>100MB), passing a single stream is risky (network failure). Use Multipart Uploads to upload chunks in parallel.

import boto3
from boto3.s3.transfer import TransferConfig

s3 = boto3.client('s3')
config = TransferConfig(
    multipart_threshold=1024 * 25, # 25MB threshold
    max_concurrency=10,
    multipart_chunksize=1024 * 25,
    use_threads=True
)

file_path = '/path/to/large_video.mp4'
key = 'uploads/video.mp4'
bucket = 'my-streaming-bucket'

def upload_file():
    # Boto3 handles the complex logic of:
    # 1. InitiateMultipartUpload
    # 2. UploadPart (in parallel threads)
    # 3. CompleteMultipartUpload
    s3.upload_file(file_path, bucket, key, Config=config)
    print("Upload Complete!")

# Presigned URL for Secure Direct-to-S3 Upload (Client-Side)
def generate_presigned_url():
    return s3.generate_presigned_url(
        'put_object',
        Params={'Bucket': bucket, 'Key': key},
        ExpiresIn=3600
    )

Consistency Models: Strong vs Eventual

Historically, S3 was Eventual Consistency (updates might take seconds to propogate). In 2020, AWS moved to Strong Consistency.