Skip to main content

Design URL Shortener Service

A comprehensive high-level design for building a scalable URL shortening service similar to bit.ly, TinyURL, or goo.gl.


Table of Contents​

  1. Problem Statement
  2. Functional Requirements
  3. Non-Functional Requirements
  4. Capacity Estimation
  5. System APIs
  6. Database Design
  7. High-Level Architecture
  8. Core Components
  9. URL Shortening Algorithm
  10. URL Redirection Flow
  11. Caching Strategy
  12. Analytics & Monitoring
  13. Security Considerations
  14. Scalability & Performance
  15. Trade-offs & Design Decisions

Problem Statement​

Design a URL shortening service that allows users to convert long URLs into short, memorable links. When users access these short links, they should be seamlessly redirected to the original long URL.

Example:

  • Long URL: https://www.example.com/articles/2024/best-practices-for-system-design?utm_source=newsletter
  • Short URL: https://short.ly/abc123

Key Challenges​

  1. Generate unique, short URLs for billions of long URLs
  2. Handle high read-to-write ratio (100:1 or more)
  3. Ensure low-latency redirects (< 100ms)
  4. Provide analytics and tracking capabilities
  5. Scale horizontally to handle millions of requests per second

Functional Requirements​

Core Features​

FR1: Users should be able to create a short URL from a long URL

  • System generates a unique short URL
  • Optional custom alias support (e.g., short.ly/my-custom-link)

FR2: Users should be redirected to the original URL when accessing the short URL

  • HTTP 301 (permanent) or 302 (temporary) redirect
  • Redirect should happen within 100ms

FR3: Short URLs should have a configurable expiration time

  • Default expiration: Never (permanent)
  • Optional: 1 day, 7 days, 30 days, custom

FR4: Users should be able to view basic analytics

  • Total clicks
  • Click timestamps
  • Geographic distribution
  • Referrer information
  • Device/browser information

FR5: Users should be able to delete or update their short URLs

  • Soft delete (mark as inactive)
  • Update destination URL

Optional Features​

FR6: Custom URL aliases (vanity URLs) FR7: QR code generation for short URLs FR8: API rate limiting per user FR9: Link preview generation FR10: Bulk URL shortening


Non-Functional Requirements​

Performance​

NFR1: High Availability - 99.99% uptime

NFR2: Low Latency - < 100ms for redirects

NFR3: High Throughput - Support millions of requests per second

NFR4: Read-heavy system - 100:1 read-to-write ratio

Scalability​

NFR5: Handle billions of URLs

NFR6: Scale horizontally without downtime

NFR7: Support global distribution (multi-region)

Reliability​

NFR8: Data durability - No data loss

NFR9: Fault tolerance - Handle server failures gracefully

NFR10: Consistent URL generation - No duplicate short URLs

Security​

NFR11: Prevent malicious URLs (phishing, malware)

NFR12: Rate limiting to prevent abuse

NFR13: DDoS protection

NFR14: HTTPS for all connections

Maintainability​

NFR15: Monitoring and alerting

NFR16: Easy to debug and trace requests

NFR17: Versioned APIs


Capacity Estimation​

Traffic Estimation​

Assumptions:

  • 500 million new URLs created per month
  • Read-to-write ratio: 100:1
  • Average URL stored for 10 years

Write Requests:

URLs per month: 500M
URLs per second: 500M / (30 days × 24 hrs × 3600 sec) = ~200 URLs/sec
Peak traffic (3x average): 600 URLs/sec

Read Requests:

Redirects per second: 200 × 100 = 20,000 redirects/sec
Peak redirects: 60,000 redirects/sec

Storage Estimation​

URL Storage:

Average long URL size: 200 bytes
Short URL (hash): 7 bytes
Metadata (timestamps, user_id, etc.): 100 bytes
Total per URL: ~307 bytes ≈ 500 bytes (with overhead)

Monthly storage: 500M URLs × 500 bytes = 250 GB/month
10-year storage: 250 GB × 12 × 10 = 30 TB

With 3x replication: 90 TB
With 30% overhead: 117 TB

Analytics Storage:

Click event: ~100 bytes (timestamp, IP, user-agent, referrer)
Clicks per month: 500M × 100 (read ratio) = 50B clicks
Monthly analytics: 50B × 100 bytes = 5 TB/month
Annual analytics: 60 TB/year

Bandwidth Estimation​

Incoming (Write):

200 URLs/sec × 500 bytes = 100 KB/sec = 0.8 Mbps
Peak: 2.4 Mbps

Outgoing (Read):

20,000 redirects/sec × 500 bytes = 10 MB/sec = 80 Mbps
Peak: 240 Mbps

Memory Estimation (Cache)​

Cache Strategy: Cache 20% of hot URLs (80/20 rule)

Daily requests: 20,000 redirects/sec × 86,400 sec = 1.7B requests/day
Unique URLs (assume 80% overlap): ~340M unique URLs/day
Cache 20% of hot URLs: 68M URLs

Memory needed: 68M × 500 bytes = 34 GB
With metadata: ~50 GB per cache server

Summary​

MetricValue
Write QPS200 (600 peak)
Read QPS20,000 (60,000 peak)
Storage (10 years)117 TB
Bandwidth80 Mbps (240 peak)
Cache Memory50 GB/server

Conclusions​

  • 200 writes/sec (600 at peak) is trivial for a single PostgreSQL primary — URL creation will never be the bottleneck in this system, so there's no reason to shard the database for write throughput alone.
  • 20,000 reads/sec (60,000 at peak) is the number that actually drives the architecture. Redirects have to answer in under 100ms at that volume, which is why a cache sits in front of the database instead of serving every redirect from disk.
  • 117 TB over 10 years is cheap to store on commodity disks — the hard part of this system was never capacity, it's serving point lookups by short_code fast enough, which a cache and a simple index already solve.
  • A 50 GB-per-server cache covering just 20% of URLs can absorb 80% of read traffic (the 80/20 rule assumed above), which is what makes 20,000 QPS survivable with a handful of Redis nodes instead of scaling the database to match.

System APIs​

REST API Design​

1. Create Short URL​

POST /api/v1/urls
Content-Type: application/json
Authorization: Bearer <token>

Request:
{
"long_url": "https://example.com/very/long/url",
"custom_alias": "my-link", // optional
"expiration_time": "2024-12-31T23:59:59Z" // optional
}

Response (201 Created):
{
"short_url": "https://short.ly/abc123",
"long_url": "https://example.com/very/long/url",
"created_at": "2024-01-15T10:30:00Z",
"expires_at": "2024-12-31T23:59:59Z",
"url_id": "abc123"
}

Error Response (409 Conflict):
{
"error": "Custom alias already exists",
"code": "ALIAS_EXISTS"
}

2. Redirect to Original URL​

GET /{short_code}

Response (301 Moved Permanently):
Location: https://example.com/very/long/url

Or (302 Found) for temporary redirects

Error Response (404 Not Found):
{
"error": "URL not found or expired",
"code": "URL_NOT_FOUND"
}

3. Get URL Analytics​

GET /api/v1/urls/{short_code}/analytics
Authorization: Bearer <token>

Response (200 OK):
{
"short_code": "abc123",
"total_clicks": 15420,
"created_at": "2024-01-15T10:30:00Z",
"last_accessed": "2024-01-16T14:22:10Z",
"clicks_by_date": [
{"date": "2024-01-15", "count": 1200},
{"date": "2024-01-16", "count": 950}
],
"clicks_by_country": [
{"country": "US", "count": 8500},
{"country": "IN", "count": 3200}
],
"clicks_by_referrer": [
{"referrer": "twitter.com", "count": 5000},
{"referrer": "facebook.com", "count": 3200}
]
}

4. Delete URL​

DELETE /api/v1/urls/{short_code}
Authorization: Bearer <token>

Response (204 No Content)

Error Response (404 Not Found):
{
"error": "URL not found",
"code": "URL_NOT_FOUND"
}

5. Update URL​

PUT /api/v1/urls/{short_code}
Authorization: Bearer <token>

Request:
{
"long_url": "https://example.com/updated/url",
"expiration_time": "2025-12-31T23:59:59Z"
}

Response (200 OK):
{
"short_url": "https://short.ly/abc123",
"long_url": "https://example.com/updated/url",
"updated_at": "2024-01-16T15:30:00Z"
}

Database Design​

Schema Design​

URLs Table (Primary Storage)​

CREATE TABLE urls (
id BIGINT PRIMARY KEY AUTO_INCREMENT,
short_code VARCHAR(10) UNIQUE NOT NULL,
long_url TEXT NOT NULL,
user_id BIGINT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
expires_at TIMESTAMP NULL,
is_active BOOLEAN DEFAULT TRUE,
is_custom BOOLEAN DEFAULT FALSE,

INDEX idx_short_code (short_code),
INDEX idx_user_id (user_id),
INDEX idx_created_at (created_at),
INDEX idx_expires_at (expires_at)
);

Design Decisions:

  • short_code is the unique identifier (indexed)
  • long_url stored as TEXT to support very long URLs
  • expires_at allows NULL for permanent URLs
  • is_active for soft deletes
  • Indexes on frequently queried fields

Analytics Table (Click Tracking)​

CREATE TABLE url_clicks (
id BIGINT PRIMARY KEY AUTO_INCREMENT,
short_code VARCHAR(10) NOT NULL,
clicked_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
ip_address VARCHAR(45),
user_agent TEXT,
referrer TEXT,
country VARCHAR(2),
city VARCHAR(100),
device_type VARCHAR(20),

INDEX idx_short_code (short_code),
INDEX idx_clicked_at (clicked_at),
FOREIGN KEY (short_code) REFERENCES urls(short_code)
);

Design Decisions:

  • Separate table for analytics to avoid bloating URLs table
  • Partitioned by date for efficient queries and archival
  • Time-series data, can use time-series databases (InfluxDB, TimescaleDB)

Users Table (Optional)​

CREATE TABLE users (
id BIGINT PRIMARY KEY AUTO_INCREMENT,
email VARCHAR(255) UNIQUE NOT NULL,
api_key VARCHAR(64) UNIQUE NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
rate_limit INT DEFAULT 1000,
is_premium BOOLEAN DEFAULT FALSE,

INDEX idx_email (email),
INDEX idx_api_key (api_key)
);

NoSQL Alternative (Key-Value Store)​

For extremely high-scale systems, consider NoSQL:

DynamoDB / Cassandra Schema:

// URLs Table
{
partition_key: "short_code", // e.g., "abc123"
attributes: {
long_url: "https://example.com/...",
user_id: "user_12345",
created_at: 1705315800,
expires_at: 1735689599,
is_active: true
}
}

// Analytics Table (Time-series)
{
partition_key: "short_code#YYYYMMDD", // e.g., "abc123#20240115"
sort_key: "timestamp",
attributes: {
ip: "192.168.1.1",
country: "US",
referrer: "twitter.com"
}
}

Benefits:

  • Horizontal scalability
  • Low-latency reads/writes
  • No schema migrations
  • Global distribution

High-Level Architecture​

Each box below is a separate layer because each one fails and scales on its own schedule: a traffic spike should force the load balancer and API tier to add capacity without touching the database, and a database failover shouldn't take the cache down with it. Keeping load balancing, compute, caching, storage, and analytics as independent layers is what lets any one of them grow or break without dragging the others down.

┌─────────────────────────────────────────────────────────────────┐
│ CLIENT LAYER │
│ (Web Browser, Mobile App, API Clients) │
└────────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ LOAD BALANCER │
│ (AWS ALB / NGINX / HAProxy) │
│ - SSL Termination │
│ - Health Checks │
│ - Geographic Routing │
└────────────────────────┬────────────────────────────────────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ API Server │ │ API Server │ │ API Server │
│ Node 1 │ │ Node 2 │ │ Node 3 │
│ │ │ │ │ │
│ - URL Gen │ │ - URL Gen │ │ - URL Gen │
│ - Redirect │ │ - Redirect │ │ - Redirect │
│ - Analytics │ │ - Analytics │ │ - Analytics │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
└────────────────┼────────────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────┐
│ CACHE LAYER (Redis Cluster) │
│ - Hot URLs (20% = 80% traffic) │
│ - TTL-based eviction │
│ - Distributed caching with replication │
└────────────────────────┬────────────────────────────────────────┘
│ Cache Miss
▼
┌─────────────────────────────────────────────────────────────────┐
│ DATABASE LAYER (Primary/Replica) │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Primary │─────▶│ Replica 1 │ │
│ │ (Write) │ │ (Read) │ │
│ └──────────────┘ └──────────────┘ │
│ │ │
│ │ ┌──────────────┐ │
│ └─────────────▶│ Replica 2 │ │
│ │ (Read) │ │
│ └──────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ ANALYTICS & MONITORING │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Message │ │ Analytics │ │ Monitoring │ │
│ │ Queue │─▶│ Service │ │ (Prometheus)│ │
│ │ (Kafka) │ │ (Spark) │ │ │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Architecture Components​

  1. Load Balancer: Distributes traffic across API servers. It exists as its own layer so servers can be added or removed behind it without any client needing to know how many there are or which one answered.
  2. API Servers: Stateless application servers (Node.js, Go, Java). Statelessness is what makes horizontal scaling trivial — because no server holds session data, the load balancer can route a request to any of them, and adding a fourth server under load needs no coordination with the other three.
  3. Cache Layer: Redis cluster for fast lookups. This sits between the API servers and the database specifically because reads outnumber writes 100:1; without it, every one of the 20,000 redirects/sec from the capacity estimate above would hit the database directly.
  4. Database: PostgreSQL/MySQL with read replicas. This is the durable source of truth for URL mappings; replicas take read load off the primary for whatever the cache misses, while all writes still go through a single primary so two requests can never claim the same short code.
  5. Message Queue: Kafka/RabbitMQ for async analytics. Click tracking is decoupled from the redirect path through a queue so a slow analytics write never adds latency to the thing users actually feel — the redirect itself.
  6. Analytics Service: Process click events and generate reports. Running as a separate consumer means it can batch inserts and fall behind during a traffic spike without that backlog ever slowing down a single redirect.
  7. Monitoring: Prometheus + Grafana for metrics. Without this layer, the first sign of a dropping cache-hit ratio or growing replica lag would be user complaints about slow redirects, not an alert.

Core Components​

1. URL Shortening Service​

Responsibilities:

  • Generate unique short codes
  • Validate long URLs
  • Store URL mappings
  • Handle custom aliases
  • Check for duplicates

This logic lives in its own service, separate from the Redirect Service, because the two have opposite traffic profiles: creation runs at 200/sec and can afford a database round-trip to check alias uniqueness, while redirects run at 20,000/sec and must avoid the database whenever possible. Keeping them apart means a slow write path can never throttle the read path that the vast majority of traffic depends on.

Implementation (Node.js):

class URLShortenerService {
constructor(database, cache, generator) {
this.db = database;
this.cache = cache;
this.generator = generator;
}

async createShortURL(longURL, customAlias = null, userId = null, expiresAt = null) {
// Validate URL
if (!this.isValidURL(longURL)) {
throw new Error('Invalid URL');
}

// Check if custom alias exists
if (customAlias) {
const exists = await this.db.checkAliasExists(customAlias);
if (exists) {
throw new Error('Custom alias already exists');
}
shortCode = customAlias;
} else {
// Generate unique short code
shortCode = await this.generateUniqueShortCode();
}

// Store in database
const urlData = {
short_code: shortCode,
long_url: longURL,
user_id: userId,
expires_at: expiresAt,
created_at: new Date(),
is_custom: !!customAlias
};

await this.db.insertURL(urlData);

// Cache the mapping
await this.cache.set(shortCode, longURL, { ttl: 3600 });

return {
short_url: `https://short.ly/${shortCode}`,
long_url: longURL,
short_code: shortCode
};
}

async generateUniqueShortCode() {
let attempts = 0;
const maxAttempts = 5;

while (attempts < maxAttempts) {
const shortCode = this.generator.generate();
const exists = await this.db.checkShortCodeExists(shortCode);

if (!exists) {
return shortCode;
}

attempts++;
}

throw new Error('Failed to generate unique short code');
}

isValidURL(url) {
try {
new URL(url);
return true;
} catch {
return false;
}
}
}

2. Redirect Service​

Responsibilities:

  • Lookup short code
  • Return original URL
  • Handle 301/302 redirects
  • Track click events
  • Handle expired URLs

Because this path carries roughly 100x the traffic of URL creation, it is optimized differently from the shortening service: cache-first lookups keep the database out of the common case, and click tracking (below) is fired off asynchronously rather than awaited, so a slow analytics write never adds to the redirect's response time.

Implementation:

class RedirectService {
constructor(database, cache, analytics) {
this.db = database;
this.cache = cache;
this.analytics = analytics;
}

async redirect(shortCode, request) {
// Try cache first
let longURL = await this.cache.get(shortCode);

if (!longURL) {
// Cache miss - query database
const urlData = await this.db.getURLByShortCode(shortCode);

if (!urlData) {
throw new Error('URL not found');
}

// Check expiration
if (urlData.expires_at && new Date() > urlData.expires_at) {
throw new Error('URL expired');
}

// Check if active
if (!urlData.is_active) {
throw new Error('URL has been deleted');
}

longURL = urlData.long_url;

// Update cache
await this.cache.set(shortCode, longURL, { ttl: 3600 });
}

// Track click asynchronously
this.trackClick(shortCode, request);

return longURL;
}

async trackClick(shortCode, request) {
const clickData = {
short_code: shortCode,
clicked_at: new Date(),
ip_address: request.ip,
user_agent: request.headers['user-agent'],
referrer: request.headers['referer'],
country: this.getCountryFromIP(request.ip)
};

// Send to message queue for async processing
await this.analytics.track(clickData);
}

getCountryFromIP(ip) {
// Use GeoIP service
// Implementation details omitted
return 'US';
}
}

3. Analytics Service​

Responsibilities:

  • Process click events
  • Aggregate statistics
  • Generate reports
  • Store time-series data

This runs as a separate consumer off the message queue rather than inline inside the Redirect Service, so a spike in click volume on one link — say it goes viral — gets absorbed as queue backlog instead of slowing down redirects for every other URL in the system.

Implementation:

class AnalyticsService {
constructor(database, messageQueue) {
this.db = database;
this.queue = messageQueue;
}

async track(clickData) {
// Publish to message queue for async processing
await this.queue.publish('url_clicks', clickData);
}

async processClickEvents() {
// Consume from message queue
this.queue.subscribe('url_clicks', async (clickData) => {
// Batch insert to database
await this.db.insertClick(clickData);
});
}

async getAnalytics(shortCode, startDate, endDate) {
const analytics = await this.db.getClickAnalytics(shortCode, startDate, endDate);

return {
total_clicks: analytics.total,
clicks_by_date: analytics.byDate,
clicks_by_country: analytics.byCountry,
clicks_by_referrer: analytics.byReferrer,
clicks_by_device: analytics.byDevice
};
}
}

URL Shortening Algorithm​

Approach 1: Hash-Based Generation​

MD5/SHA256 Hash + Base62 Encoding

class HashBasedGenerator {
generate(longURL) {
// Generate hash
const hash = crypto.createHash('md5').update(longURL).digest('hex');

// Take first 6-8 characters
const shortHash = hash.substring(0, 7);

// Convert to Base62
const base62 = this.hexToBase62(shortHash);

return base62;
}

hexToBase62(hex) {
const chars = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
let num = parseInt(hex, 16);
let result = '';

while (num > 0) {
result = chars[num % 62] + result;
num = Math.floor(num / 62);
}

return result || '0';
}
}

Pros:

  • Deterministic (same URL = same hash)
  • No database lookup needed during generation

Cons:

  • Collision possible (requires handling)
  • Cannot guarantee uniqueness
  • Same long URL gets same short URL (may not be desired)

Auto-increment ID + Base62 Encoding

class CounterBasedGenerator {
constructor(database) {
this.db = database;
}

async generate() {
// Get next ID from database counter
const id = await this.db.getNextSequence('url_counter');

// Convert to Base62
const base62 = this.toBase62(id);

return base62;
}

toBase62(num) {
const chars = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
let result = '';

if (num === 0) return chars[0];

while (num > 0) {
result = chars[num % 62] + result;
num = Math.floor(num / 62);
}

return result;
}

fromBase62(str) {
const chars = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
let num = 0;

for (let i = 0; i < str.length; i++) {
num = num * 62 + chars.indexOf(str[i]);
}

return num;
}
}

Pros:

  • Guaranteed uniqueness
  • No collisions
  • Predictable length growth

Cons:

  • Requires database lookup for counter
  • Sequential IDs (security concern - can be mitigated with randomization)

Optimization: Use distributed ID generators (Snowflake, Twitter ID)

Approach 3: Random String Generation​

class RandomGenerator {
generate(length = 7) {
const chars = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
let result = '';

for (let i = 0; i < length; i++) {
const randomIndex = Math.floor(Math.random() * chars.length);
result += chars[randomIndex];
}

return result;
}
}

Pros:

  • Simple implementation
  • Non-sequential (more secure)

Cons:

  • Requires uniqueness check
  • Collision probability increases over time

Why Counter-Based Generation Wins​

All three approaches reduce to the same trade-off: avoid collisions by construction, or accept them and pay to detect them. Hash-based generation collides because a 7-character slice of an MD5 digest has far fewer possible values than the full hash it's cut from, and it has a second problem beyond collisions — the same long URL always produces the same short code, so two different users can't get two separately-trackable links to the same page. Random generation avoids that second problem, but its uniqueness-check retry loop gets slower as the system grows: with more codes already taken, a freshly generated random string is more likely to collide with one of them. Counter-based generation sidesteps both issues — every ID is unique by definition, so there's no collision handling and no retry loop — which is why it's the recommended approach above, at the cost of needing a single, always-available source of "the next number" (the reason the Optimization note suggests a distributed ID generator like Snowflake instead of one database counter that would become a bottleneck at scale).

Base62 Encoding Explained​

Why Base62?

  • Uses: 0-9 (10) + a-z (26) + A-Z (26) = 62 characters
  • URL-safe (no special characters)
  • Compact representation

Length Calculation:

62^6 = 56.8 billion combinations
62^7 = 3.5 trillion combinations
62^8 = 218 trillion combinations

For 500M URLs, 6 characters sufficient initially, scale to 7 for long-term.


URL Redirection Flow​

Flow Diagram​

User clicks short URL
│
▼
┌─────────────────┐
│ Load Balancer │
└────────┬────────┘
│
▼
┌─────────────────┐
│ API Server │
└────────┬────────┘
│
▼
Check Redis Cache
│
┌────┴────┐
│ │
Hit ✓ Miss ✗
│ │
│ ▼
│ Query Database
│ │
│ ▼
│ Update Cache
│ │
└────┬────┘
│
▼
Validate URL
(active, not expired)
│
▼
Track Click Event
(async via queue)
│
▼
Return 301/302 Redirect

Implementation​

// Express.js Route Handler
app.get('/:shortCode', async (req, res) => {
try {
const { shortCode } = req.params;

// Redirect service
const longURL = await redirectService.redirect(shortCode, req);

// Return 301 (permanent) or 302 (temporary) redirect
res.redirect(301, longURL);
} catch (error) {
if (error.message === 'URL not found') {
res.status(404).json({ error: 'Short URL not found' });
} else if (error.message === 'URL expired') {
res.status(410).json({ error: 'Short URL has expired' });
} else {
res.status(500).json({ error: 'Internal server error' });
}
}
});

HTTP Redirect Status Codes​

301 (Moved Permanently):

  • Browsers cache the redirect
  • Subsequent requests go directly to long URL
  • Analytics lost after first click (browser doesn't hit server)
  • Use for: Permanent redirects where analytics not critical

302 (Found / Temporary Redirect):

  • Browsers don't cache
  • Every click hits the server
  • Full analytics tracking
  • Use for: Temporary redirects, analytics-heavy use cases

Trade-off: 301 = Better performance, 302 = Better analytics


Caching Strategy​

Cache Layer Design​

Technology: Redis Cluster with replication

Cache Architecture:

┌──────────────────────────────────────────────┐
│ Redis Cluster (3 Masters) │
│ │
│ ┌────────────┐ ┌────────────┐ ┌────────────┐
│ │ Master 1 │ │ Master 2 │ │ Master 3 │
│ │ (Shards │ │ (Shards │ │ (Shards │
│ │ 0-5461) │ │ 5462-10922│ │ 10923-16383│
│ └─────┬──────┘ └─────┬──────┘ └─────┬──────┘
│ │ │ │
│ ▼ ▼ ▼
│ ┌────────────┐ ┌────────────┐ ┌────────────┐
│ │ Replica 1 │ │ Replica 2 │ │ Replica 3 │
│ └────────────┘ └────────────┘ └────────────┘
└──────────────────────────────────────────────┘

Caching Policies​

1. Cache-Aside (Lazy Loading):

async function getURL(shortCode) {
// Try cache
let url = await cache.get(shortCode);

if (url) {
return url; // Cache hit
}

// Cache miss - query DB
url = await db.query('SELECT long_url FROM urls WHERE short_code = ?', [shortCode]);

// Update cache
if (url) {
await cache.set(shortCode, url, { ttl: 3600 }); // 1 hour TTL
}

return url;
}

2. Write-Through:

async function createURL(shortCode, longURL) {
// Write to DB
await db.insert({ short_code: shortCode, long_url: longURL });

// Write to cache
await cache.set(shortCode, longURL, { ttl: 3600 });

return shortCode;
}

Cache Eviction Policy​

Strategy: LRU (Least Recently Used) with TTL

LRU fits this workload because URL popularity is skewed and changes slowly — the links getting clicked this hour are usually the same ones that got clicked last hour. Evicting whatever hasn't been touched recently reliably keeps that hot set in cache, which is exactly the assumption the 80/20 cache-sizing math above depends on.

Configuration:

  • TTL: 1 hour for normal URLs
  • TTL: 24 hours for hot URLs (high traffic)
  • Max memory: 50 GB per Redis instance
  • Eviction: allkeys-lru when memory limit reached

Cache Warming​

Preload hot URLs on startup:

async function warmCache() {
// Get top 1000 most accessed URLs
const hotURLs = await db.query(`
SELECT short_code, long_url
FROM urls
ORDER BY access_count DESC
LIMIT 1000
`);

// Load into cache
for (const url of hotURLs) {
await cache.set(url.short_code, url.long_url, { ttl: 86400 }); // 24h
}
}

Analytics & Monitoring​

Analytics Architecture​

Click Event → API Server → Kafka Topic → Analytics Service → Time-Series DB
│
▼
Aggregation Jobs
│
▼
Analytics DB

Click Tracking Implementation​

Async Event Publishing:

class ClickTracker {
constructor(kafkaProducer) {
this.kafka = kafkaProducer;
}

async track(shortCode, metadata) {
const event = {
short_code: shortCode,
timestamp: Date.now(),
ip: metadata.ip,
user_agent: metadata.userAgent,
referrer: metadata.referrer,
country: metadata.country,
city: metadata.city,
device: this.parseDevice(metadata.userAgent)
};

// Publish to Kafka (fire and forget)
await this.kafka.send({
topic: 'url_clicks',
messages: [{ value: JSON.stringify(event) }]
});
}

parseDevice(userAgent) {
// Parse user agent to determine device type
if (/mobile/i.test(userAgent)) return 'mobile';
if (/tablet/i.test(userAgent)) return 'tablet';
return 'desktop';
}
}

Analytics Aggregation​

Batch Processing (Spark/Flink):

// Pseudo-code for aggregation job
function aggregateClicks() {
// Run every hour
const clicks = kafka.consume('url_clicks', { from: lastHour });

// Aggregate by short_code
const aggregated = clicks.groupBy('short_code').aggregate({
total_clicks: count(),
by_country: groupBy('country').count(),
by_referrer: groupBy('referrer').count(),
by_device: groupBy('device').count()
});

// Store in analytics DB
analyticsDB.insert(aggregated);
}

Metrics & Monitoring​

Key Metrics:

  1. Performance Metrics:

    • Redirect latency (p50, p95, p99)
    • API response time
    • Cache hit ratio
    • Database query time
  2. Business Metrics:

    • URLs created per second
    • Redirects per second
    • Active URLs count
    • Top URLs by traffic
  3. System Metrics:

    • CPU utilization
    • Memory usage
    • Network I/O
    • Disk I/O

Implementation (Prometheus):

const prometheus = require('prom-client');

// Counter for total redirects
const redirectCounter = new prometheus.Counter({
name: 'url_redirects_total',
help: 'Total number of URL redirects',
labelNames: ['status']
});

// Histogram for redirect latency
const redirectLatency = new prometheus.Histogram({
name: 'url_redirect_duration_seconds',
help: 'URL redirect duration in seconds',
buckets: [0.001, 0.01, 0.05, 0.1, 0.5, 1]
});

// Gauge for cache hit ratio
const cacheHitRatio = new prometheus.Gauge({
name: 'cache_hit_ratio',
help: 'Cache hit ratio percentage'
});

// Track metrics
app.get('/:shortCode', async (req, res) => {
const timer = redirectLatency.startTimer();

try {
const url = await redirectService.redirect(req.params.shortCode, req);
redirectCounter.inc({ status: 'success' });
res.redirect(301, url);
} catch (error) {
redirectCounter.inc({ status: 'error' });
res.status(404).send('Not found');
} finally {
timer();
}
});

Security Considerations​

1. Malicious URL Prevention​

URL Validation:

class URLValidator {
async validate(url) {
// Check URL format
if (!this.isValidFormat(url)) {
throw new Error('Invalid URL format');
}

// Check against blacklist
if (await this.isBlacklisted(url)) {
throw new Error('URL is blacklisted');
}

// Check for phishing
if (await this.isPhishing(url)) {
throw new Error('Potential phishing URL');
}

// Check for malware
if (await this.hasMalware(url)) {
throw new Error('URL contains malware');
}

return true;
}

async isBlacklisted(url) {
// Check against Google Safe Browsing API
const response = await fetch('https://safebrowsing.googleapis.com/v4/threatMatches:find', {
method: 'POST',
body: JSON.stringify({
threatInfo: {
threatTypes: ['MALWARE', 'SOCIAL_ENGINEERING'],
platformTypes: ['ANY_PLATFORM'],
threatEntryTypes: ['URL'],
threatEntries: [{ url }]
}
})
});

const data = await response.json();
return data.matches && data.matches.length > 0;
}
}

2. Rate Limiting​

Token Bucket Algorithm:

class RateLimiter {
constructor(redis) {
this.redis = redis;
}

async checkLimit(userId, limit = 100, window = 3600) {
const key = `ratelimit:${userId}`;
const current = await this.redis.incr(key);

if (current === 1) {
// Set expiry on first request
await this.redis.expire(key, window);
}

if (current > limit) {
throw new Error('Rate limit exceeded');
}

return {
allowed: true,
remaining: limit - current
};
}
}

// Middleware
app.use(async (req, res, next) => {
try {
const userId = req.user?.id || req.ip;
await rateLimiter.checkLimit(userId, 100, 3600); // 100 req/hour
next();
} catch (error) {
res.status(429).json({ error: 'Too many requests' });
}
});

3. API Authentication​

API Key Authentication:

async function authenticate(req, res, next) {
const apiKey = req.headers['x-api-key'];

if (!apiKey) {
return res.status(401).json({ error: 'Missing API key' });
}

// Validate API key
const user = await db.query('SELECT * FROM users WHERE api_key = ?', [apiKey]);

if (!user) {
return res.status(401).json({ error: 'Invalid API key' });
}

req.user = user;
next();
}

app.post('/api/v1/urls', authenticate, createURLHandler);

4. DDoS Protection​

Strategies:

  • Use CDN (Cloudflare, Akamai) for DDoS mitigation
  • Implement rate limiting at multiple layers
  • Use Web Application Firewall (WAF)
  • Geo-blocking for suspicious regions
  • Challenge-response (CAPTCHA) for high-risk requests

5. HTTPS Enforcement​

app.use((req, res, next) => {
if (!req.secure && process.env.NODE_ENV === 'production') {
return res.redirect(301, `https://${req.headers.host}${req.url}`);
}
next();
});

Scalability & Performance​

Horizontal Scaling​

Stateless API Servers:

  • No session state stored in servers
  • Any server can handle any request
  • Easy to add/remove servers

Database Scaling:

  1. Read Replicas:

    • Master for writes
    • Multiple replicas for reads
    • Route read queries to replicas
  2. Sharding (Partitioning):

    • Partition by short_code range
    • Consistent hashing for distribution
    • Example: short_code[0] determines shard
class DatabaseRouter {
getShardForShortCode(shortCode) {
// Hash-based sharding
const hash = this.hash(shortCode);
const shardId = hash % this.totalShards;
return this.shards[shardId];
}

async query(shortCode) {
const shard = this.getShardForShortCode(shortCode);
return await shard.query('SELECT * FROM urls WHERE short_code = ?', [shortCode]);
}
}

Geographic Distribution​

Multi-Region Deployment:

┌──────────────────────────────────────────────────────────┐
│ DNS / Route53 │
│ (GeoDNS-based routing) │
└───────────────┬──────────────────┬───────────────────────┘
│ │
┌─────────▼────────┐ ┌──────▼─────────┐
│ US Region │ │ EU Region │
│ │ │ │
│ - API Servers │ │ - API Servers │
│ - Redis Cache │ │ - Redis Cache │
│ - DB Replica │ │ - DB Replica │
└──────────────────┘ └────────────────┘

Benefits:

  • Lower latency for users
  • Higher availability
  • Regulatory compliance (GDPR)

Performance Optimizations​

1. Connection Pooling:

const pool = mysql.createPool({
host: 'localhost',
user: 'root',
database: 'url_shortener',
connectionLimit: 100,
queueLimit: 0
});

2. Async Processing:

// Don't wait for analytics tracking
app.get('/:shortCode', async (req, res) => {
const url = await getURL(req.params.shortCode);

// Track click asynchronously (fire-and-forget)
trackClick(req.params.shortCode, req).catch(err => logger.error(err));

res.redirect(301, url);
});

3. Database Indexing:

CREATE INDEX idx_short_code ON urls(short_code);
CREATE INDEX idx_user_id ON urls(user_id);
CREATE INDEX idx_created_at ON urls(created_at);

4. Response Compression:

const compression = require('compression');
app.use(compression());

5. CDN for Static Assets:

  • Serve static files (landing page, docs) via CDN
  • Reduce load on API servers

Trade-offs & Design Decisions​

1. Short Code Length​

Decision: 7 characters (Base62)

LengthCombinationsUse Case
5916MSmall scale
656.8BMedium scale
73.5TLarge scale ✓
8218TOverkill

Rationale:

  • 7 characters support 3.5 trillion URLs
  • Balance between length and capacity
  • Short enough for easy sharing

Picking 5 characters would work today but caps the system at 916M codes against a target of billions of URLs over a 10-year retention window, forcing a disruptive length migration later. Picking 8 avoids that ceiling but makes every link one character longer than the 7-character option already needs for this scale.

2. 301 vs 302 Redirect​

Decision: Use 302 (Temporary Redirect)

Rationale:

  • Need full analytics tracking
  • Can't track if browser caches (301)
  • Slight performance hit acceptable for analytics value

Alternative: Offer both options, let users choose

3. SQL vs NoSQL​

Decision: Start with SQL (PostgreSQL), migrate to NoSQL if needed

DatabaseProsCons
PostgreSQLACID, relations, matureVertical scaling limits
CassandraHorizontal scaling, high writeEventual consistency
DynamoDBManaged, scalableVendor lock-in, cost

Rationale:

  • PostgreSQL handles billions of rows
  • Easier to query and maintain
  • Can shard if needed
  • Migrate to NoSQL only if necessary

Starting with NoSQL instead would mean giving up ACID transactions and SQL joins on day one in exchange for horizontal write scaling this system doesn't need yet — the capacity estimate above puts writes at 200/sec, nowhere near where a single PostgreSQL primary struggles.

4. Hash vs Counter for Short Code​

Decision: Counter-based with Base62 encoding

Rationale:

  • Guaranteed uniqueness
  • No collision handling
  • Predictable behavior
  • Can add randomization for security

Choosing hash-based generation instead would save one database round-trip per request but reintroduce the collision-retry loop and the "same URL always gets the same code" limitation described in URL Shortening Algorithm — not a good trade here, since write volume (200/sec) is nowhere near the point where that extra round-trip would matter.

5. Cache Expiration​

Decision: 1-hour TTL for normal URLs, 24-hour for hot URLs

Rationale:

  • Balance between freshness and performance
  • Hot URLs rarely change
  • Reduces database load

A shorter TTL would catch an edited destination URL faster but push more lookups to the database; a longer one does the opposite. One hour is short enough that an update propagates within the hour, long enough that a hot URL still serves thousands of requests from cache before any single entry expires.

6. Analytics Granularity​

Decision: Store raw events, aggregate hourly

Rationale:

  • Raw events for flexibility
  • Aggregations for performance
  • Can run custom queries on raw data

Storing only hourly aggregates would be cheaper, but it would permanently close off any question nobody thought to ask in advance — for example, "how many clicks came from a specific IP range last Tuesday" can't be answered from an aggregate that never recorded IP per hour. Keeping the raw events costs more storage upfront but leaves that door open.

7. Expiration Handling​

Decision: Soft delete + periodic cleanup job

Rationale:

  • Immediate marking as expired (soft delete)
  • Batch cleanup to save resources
  • Can restore accidentally expired URLs

A hard delete on expiration would free storage immediately, but it also makes an accidental deletion or a compromised account's deletion unrecoverable. The soft delete buys a window to undo that mistake before the periodic cleanup job removes the row for good.


Summary​

This URL shortener design supports:

  • Scale: Billions of URLs, millions of requests/sec
  • Performance: < 100ms redirect latency
  • Availability: 99.99% uptime with multi-region deployment
  • Analytics: Comprehensive click tracking and reporting
  • Security: Rate limiting, malware detection, DDoS protection

Key Technologies:

  • API Servers: Node.js/Go
  • Cache: Redis Cluster
  • Database: PostgreSQL with read replicas
  • Message Queue: Kafka
  • Analytics: Apache Spark
  • Monitoring: Prometheus + Grafana
  • CDN: Cloudflare