Overview
Curated: · Written: · Reviewed:
URL shortener design
A URL shortener is the system-design question that looks like a hash function and is actually a unique-key store, an HTTP redirect policy, and an abuse problem. This guide treats the product as a map from a short public key to a long URI: how keys are allocated without races, which 30x status you can cache, how clicks leave the GET path, and why an open redirect under your domain is the failure that gets you dropped from Safe Browsing. It assumes the vocabulary of scalability, caching, and REST, and it uses concrete QPS, key lengths, and TTLs rather than a nameless 'high scale' box.
the mapping, not the hash
A URL shortener is a durable map from a short key to a long URI, plus a redirect policy; hashing is one way to mint keys, not the product.
The read path is GET /{key} → 30x Location: {long}. The write path inserts (key, long, owner, expiry) under a unique constraint on key.
The product is two operations and one unique column.
| verb | path | store | response |
|---|---|---|---|
| POST | /v1/links | insert unique key | 201 {key, short_url} |
| GET | /{key} | lookup key | 302 Location + Cache-Control |
| GET | /missing | miss | 404, cache 10 s not 1 day |
Interview trap. Designing the whole system as 'MD5 the URL and take six characters' treats collisions, custom aliases, and updates as afterthoughts.
Engineering practice. Draw the table and the two HTTP verbs first. Choose a key generator only after the uniqueness and update rules are stated.
base62 encoding of a counter
Encoding an autoincrement integer in base 62 ([0-9A-Za-z]) yields short keys whose length grows with the count, not with hash collisions.
Alphabet size 62: 62^6 ≈ 56.8 billion keys, 62^7 ≈ 3.5 trillion. A 64-bit counter encoded without padding is 11 characters at most.
Length vs capacity, and when a 6-char hash actually collides.
| scheme | 6-char space | first expected collision |
|---|---|---|
| base62(counter) | 56.8e9, no collision | never, until the counter wraps |
| random 6-char | 56.8e9 | ~2.4e5 inserts (birthday) |
| sha256[:6] of URL | 56.8e9 | same birthday, plus same-URL duplicates |
Interview trap. Saying a 6-character hash of the long URL is unique because 62^6 is large ignores birthday collisions, which start mattering around sqrt(N) ≈ 238,000 keys.
Engineering practice. If the key may be sequential, encode a unique integer. If the key must not leak creation order, hash or permute the integer after it is allocated.
counters vs hashes vs UUIDs
Autoincrement is unique and short; a content hash is deterministic for the same URL; a UUID is unique without a central counter and is too long to put in a tweet unless you truncate it, which reintroduces collisions.
RFC 9562 UUIDs are 128 bits (36 hex chars with hyphens in the standard text form). Truncating to 48 bits to get a 9-char base62 key is a new identifier, not a UUID.
What the user sees versus what the unique index covers.
-- public_key is what GET /{key} looks up. It must be unique, not a prefix of uuid.
create table links (
id bigint generated always as identity primary key,
public_key varchar(12) not null unique,
destination text not null,
created_at timestamptz not null
);
Interview trap. Storing a UUID v4 and showing the first 7 characters as the public key, then looking up with LIKE, both collides and cannot use a unique index on the public key.
Engineering practice. Allocate a unique integer or a unique random key in the public alphabet, and store the UUID only if you need an internal opaque id separate from the public key.
collision handling on insert
Uniqueness is enforced by the store, not by a SELECT-then-INSERT. On conflict, retry with a new key or return the existing row for the same owner and destination.
PostgreSQL unique indexes reject the second insert. INSERT ... ON CONFLICT (public_key) DO NOTHING RETURNING * tells you whether to retry.
Two concurrent POSTs for different URLs that sampled the same random key.
| t | writer A | writer B | store |
|---|---|---|---|
| 1 | SELECT miss | SELECT miss | empty |
| 2 | INSERT k | INSERT k | unique violation on B |
| 3 | 201 | retry with k2 | A holds k |
Interview trap. Checking existence in the application then inserting is a race: two writers both see a miss and one fails with a 500 the client cannot retry safely.
Engineering practice. Put a unique index on public_key, retry a bounded number of times on conflict for random keys, and treat same-owner same-destination as idempotent.
custom aliases
A user-chosen alias is still a unique public_key, with a reserved-word list and a character allowlist, not a free-form slug.
Allow [0-9A-Za-z_-], reject homoglyphs, reject aliases that match product paths (/api, /health, /v1), and cap length (e.g. 32).
Unicode TR36 mixed-script spoof of a reserved alias.
| requested alias | bytes | decision |
|---|---|---|
| apple | 61 70 70 6c 65 | reject if reserved |
| аррle (Cyrillic а,р) | d0 b0 ... | reject mixed-script |
| my-launch | 6d 79 2d ... | accept if unique |
Interview trap. Accepting any Unicode string as an alias because 'the database is UTF-8' creates /аррle vs /apple phishing (Cyrillic а).
Engineering practice. Normalize to a documented alphabet, compare case-insensitively if you fold case, and keep a blocklist of routes and trademarks.
301 versus 302 versus 307
RFC 9110: 301 and 308 are permanent; 302 and 307 are temporary. 301/302 historically allowed method rewriting; 307/308 preserve the method.
Browsers and CDNs cache 301 aggressively. If the owner later changes the destination, a cached 301 keeps sending users to the old Location.
A destination change after a cached 301.
| status | Cache-Control | owner updates destination at t=60s | user who cached at t=0 |
|---|---|---|---|
| 301 | public, max-age=86400 | DB has new URL | still old Location for up to 1 day |
| 302 | private, max-age=60 | DB has new URL | new Location within 60 s |
Interview trap. Always returning 301 because it is 'better for SEO' makes destination edits and takedowns ineffective for anyone who already resolved the key.
Engineering practice. Use 302 or 307 for a product that allows edits, expiry, and abuse takedowns. Use 301 only when the mapping is contractually immutable.
open redirects
A shortener that will redirect to any http(s) URL is an open redirect: attackers mint keys that 302 to phishing pages under your domain's trust.
OWASP: validate the destination scheme (https only), optionally allowlist hosts for enterprise shorteners, and refuse javascript: and data: URIs (RFC 3986 schemes).
Destinations a naive startsWith('https') check accepts.
https://good.example/ok -> allow
javascript:alert(1) -> reject (scheme)
https://evil.example -> allow only if host allowlisted / scanned
https://good.example.evil.example -> reject if you meant suffix match on good.example
Interview trap. Percent-decoding once then substring-matching the trusted host misses https://trusted.example@evil.example, whose host is evil.example, and nested encodings.
Engineering practice. Parse with a URI library, require https, reject userinfo, and run Safe Browsing or an equivalent lookup before first redirect.
the read path latency budget
Redirect latency is a cache lookup plus a 30x, not a join. p99 should stay under a few tens of milliseconds at the edge.
Redis GET of the mapping is O(1). A miss loads the row, populates the cache, and coalesces stampedes (AWS builders library).
Budgets for 50k redirects/s.
| hop | p50 | p99 |
|---|---|---|
| edge TLS + 302 | 8 ms | 25 ms |
| Redis GET (same AZ) | 0.4 ms | 2 ms |
| Postgres primary on miss | 3 ms | 40 ms |
| join to click_facts on GET | 12 ms | 180 ms (do not) |
Interview trap. Joining owner, tags, and click totals on every GET /{key} puts analytics on the user-visible path and blows p99 when the stats table is large.
Engineering practice. Serve Location from a key-value cache. Write clicks asynchronously. Keep the hot GET handler under 1 ms of application work plus one cache RTT.
click accounting off the GET
A click is an event, not a row update on the mapping. Counting with UPDATE links SET clicks = clicks + 1 on the redirect serializes the hot key.
Emit (key, ts, ua_hash, country) to a queue or log; aggregate later. The mapping row stays read-mostly.
One viral key at 8,000 rps.
| design | lock / hotspot | lost clicks |
|---|---|---|
| UPDATE clicks+1 per GET | row lock on that key | 0, p99 collapse |
| Redis INCR, flush 1 s | one Redis key | ~0, still on GET |
| append-only event, 10 s rollup | none on GET | ±0.3% at flush |
Interview trap. Incrementing a counter on the same row you read for Location creates write contention on the most popular keys, which are exactly the ones you needed to be fast.
Engineering practice. At-least-once click events with idempotent aggregation windows. Accept ±1% on a dashboard, not a locked integer on the request path.
negative caching
Cache misses and 404s, but much more briefly than hits, so a just-created key is not hidden behind a long negative entry.
At an HTTP cache, RFC 5861 stale-if-error can keep a cached redirect usable when an origin dependency fails; it does not justify caching a 404 for 24 h.
Create then immediately GET from another POP.
| t | event | US-east cache | EU-west cache |
|---|---|---|---|
| 0 | GET /abc 404 | NX ttl=15s | empty |
| 1 | POST creates abc | DEL NX | empty |
| 2 | GET /abc from EU | 302 | miss → DB → 302 |
Interview trap. Caching every unknown key as NX for one hour means a user who creates a link then shares it immediately still 404s.
Engineering practice. Negative TTL on the order of 5–30 s. After a successful insert, delete the negative cache entry for that key in the same region.
idempotent create
Retrying POST /v1/links after a timeout must not mint a second key for the same owner and destination unless the client asked for uniqueness of the long URL.
Idempotency-Key header or a unique (owner_id, destination_hash) for the default 'reuse existing' mode.
Client timeout after the insert committed.
| attempt | Idempotency-Key | DB | response |
|---|---|---|---|
| 1 | k-9f | insert abc | timeout, 201 lost |
| 2 | k-9f | see k-9f | 201 abc again |
| 2 without key | — | insert def | two keys, abc unused |
Interview trap. Returning a new random key on every retry makes the first key an orphan and the client's 'copied URL' point at a 404 if the first write actually landed.
Engineering practice. Document two modes: reuse-if-exists (default) and always-new. Both need an idempotency key for the HTTP retry case.
read-your-writes after create
The creator must resolve their new key immediately, even if the replica or remote cache is behind.
Return the key and destination in the 201 body. Route GET /{key} for a few seconds to the primary, or write-through the local cache in the same AZ as the insert.
W=1 on the primary, GET hits a replica 400 ms behind.
| read | creator sees |
|---|---|
| replica, R=1 | 404, retries POST |
| 201 body, no extra GET | key abc, done |
| primary for 2 s | 302 |
Interview trap. Reading the mapping only from a 400 ms lagging replica after insert looks like a failed save and produces duplicate POSTs.
Engineering practice. 201 body is source of truth for the creator. Sticky or primary reads until replica lag is below your SLA.
expiry and 410
Expired keys should stop redirecting. RFC 9110 410 Gone is more honest than 404 when you know the key existed.
Store expires_at. The GET path treats now >= expires_at as gone. Cache 410 briefly, bounded by the product's reactivation and key-reuse policy.
TTL 24 h, cache max-age 6 h, job runs hourly.
| t | DB expires_at | cache | GET |
|---|---|---|---|
| 23 h | future | 302 cached | 302 |
| 24.1 h | past | still 302 if not revalidated | wrong |
| 24.1 h if value carries expiry | past | treat as gone | 410 |
Interview trap. Leaving expired rows in the hot cache as 302 until LRU evicts them keeps phishing and stale campaigns alive.
Engineering practice. Include expiry in the cache value. A background job can delete rows; the GET path must still enforce expires_at even if the job lags.
capacity of the write path
Write QPS is usually tiny next to redirects. Size the unique-id allocator and the primary for creates, and the cache for reads.
1 million new links/day is ~12 inserts/s average, ~200/s peak. 10 billion redirects/day is ~116k/s average.
A launch-week vs a mature consumer shortener.
| stage | creates/s p95 | redirects/s p95 | store |
|---|---|---|---|
| internal tool | 2 | 40 | one Postgres |
| public beta | 80 | 2_000 | Postgres + Redis |
| consumer app | 400 | 80_000 | partitioned KV + edge cache |
Interview trap. Sharding the mapping table on day one because 'shorteners are high scale' when you have 80 creates/s and 2k redirects/s.
Engineering practice. Quote both numbers in the design. Shard or partition when a single primary cannot take the write peak plus compaction, not when the whiteboard looks sparse.
partitioning the map
When writes outgrow one primary, partition by public_key (hash or range), not by destination URL, so GET /{key} is a single partition lookup.
hash(key) % N or a directory. Range-partitioning sequential counters hotspots the latest partition.
Where a GET lands.
| shard by | POST duplicate URL | GET /abc |
|---|---|---|
| hash(public_key) | may be another shard | one shard |
| hash(destination) | same shard | unknown shard, scatter |
| sequential range | latest shard hot | one shard, write hotspot |
Interview trap. Sharding by hash(long_url) so duplicates collocate, then GET /{key} has to broadcast because the key no longer determines the shard.
Engineering practice. The public key is the partition key. Dedup of destinations is a secondary index or an async job, not the shard function.
preview and interstitial
A preview page such as GET /preview/{key} that shows the destination before redirect is an abuse control, not a performance feature.
The interstitial is HTML, noindex, and must not auto-redirect in 0 ms or phishing filters treat it as a 302 with extra steps.
What the interstitial must display.
| field | example | why |
|---|---|---|
| registrable domain | paypal-secure.example | not the full 200-char path |
| scheme | https | http is a warning |
| continue | explicit button | no 0-delay meta refresh |
Interview trap. Putting a 3-second countdown that still always continues trains users to click through and does not change Safe Browsing risk.
Engineering practice. Show the registrable domain prominently, block known-bad destinations, and skip the interstitial for allowlisted enterprise hosts.
malware scanning
Scan destinations at create and periodically after, because a clean URL can be parked then swapped to phishing.
Use Safe Browsing for a non-commercial service, or Web Risk/equivalent for commercial use, on insert and on a crawl interval. On a confirmed hit, stop 302 and serve an interstitial block.
Same key, destination path unchanged, content swapped.
| t | destination | scan | GET |
|---|---|---|---|
| day 0 | https://site.example/x | clean | 302 |
| day 12 | same URL, new HTML | hit | 410 / block page |
Interview trap. Scanning only at create, once, because 'the URL string did not change' misses hosted content that changed behind the same path.
Engineering practice. Re-scan popular keys more often. Treat scanner downtime as fail-closed for new creates and fail-open-with-log for existing GET only if product accepts that risk explicitly.
rate limits on minting
Create is the expensive, abusable verb. Rate-limit by account and by IP; do not rate-limit GET /{key} the same way.
Token bucket per owner: e.g. 30 creates/min, 500/day. RFC 6585 defines 429 Too Many Requests and permits a Retry-After header.
Limits that survive a viral GET storm.
| route | limiter | budget |
|---|---|---|
| POST /v1/links | owner_id | 30 / min |
| POST /v1/links | ip if anonymous | 5 / min |
| GET /{key} | none (cache) | edge capacity |
Interview trap. A global 100 rps cap on the whole HTTP service, which 302 traffic blows through, so creates starve or you open the floodgates.
Engineering practice. Separate limiters. Redirects are cheap and cacheable; creates allocate unique keys and may call the scanner.
HTTPS and mixed destinations
Serve the shortener only on HTTPS. Redirecting an https short URL to http: is a downgrade; prefer to refuse http destinations.
HSTS on the short host. Destination scheme checked after RFC 3986 parse, not with a regex on the raw string.
What the browser sends after 302.
| short URL | Location | Referer / cookies on next hop |
|---|---|---|
| https://s.example/a | https://app.example | HTTPS, Referrer-Policy applies |
| https://s.example/a | http://app.example | downgrade, path on the wire |
Interview trap. Allowing http:// destinations 'for compatibility' turns your 302 into a plaintext leak of the path and cookies on the target.
Engineering practice. Default deny http. If you must support it, warn on preview and never silently rewrite https→http.
key alphabet and copy-paste
Avoid visually ambiguous characters in generated keys (0/O, 1/l/I) if humans will read them aloud; keep the decoder accepting the full base62 if you ever used it.
A 32-char Crockford-like alphabet reduces mis-reads. Changing alphabets later requires supporting old keys forever.
A printed key that used 0 vs O.
| printed | typed | lookup |
|---|---|---|
| ab0cd | abOcd | miss if alphabet distinguishes |
| ab0cd | ab0cd | hit |
| Crockford: no O,I,L,U | fewer misreads | decoder maps o→0 if you allow |
Interview trap. Switching from base62 to a smaller alphabet and re-encoding old ids changes public URLs already printed on packaging.
Engineering practice. Pick the public alphabet once. New encodings can be versioned with a prefix; never mutate existing keys.
analytics privacy
Click logs can be personal data when they include IP addresses or stable device identifiers. Store only what the product needs, pseudonymize where useful, and enforce a retention limit.
Keep day-level counts forever if they are not personal. Keep IP at most days, or hash with a rotating salt.
What to keep after 30 days.
| field | 24 h | 30 d | 1 y |
|---|---|---|---|
| public_key, ts bucket | yes | yes | daily count |
| IP | yes | hash or drop | drop |
| user-agent | coarse | coarse | drop |
Interview trap. Logging full IP, User-Agent, and destination in one row 'for fraud' with no expiry recreates a surveillance table on a link-shortening company.
Engineering practice. Document retention. Fraud features can use short-lived hashes. Dashboards should run on aggregates.
CDN in front of 302
A CDN can cache 302s per key if Cache-Control allows it, but then destination edits and takedowns wait on TTL or purge.
Purge by URL on update/delete. Vary is unused if the response does not depend on cookies.
Takedown at t=0 with a 5-minute CDN TTL and no purge.
| t | origin | CDN POP | user |
|---|---|---|---|
| 0 | 410 | still 302 | phishing |
| 0 with purge | 410 | 410 | blocked |
| 0, max-age=60, no purge | 410 | 302 ≤60 s | bounded |
Interview trap. Setting max-age=31536000 on 302 at the CDN because 'the short URL never changes' after you already shipped editable links.
Engineering practice. Short shared cache (30–120 s) plus purge-on-write. Origin shield to protect Redis on a viral miss storm.
multi-region active-active
Redirects can be served from any region that has a replica of the map. Creates need a uniqueness story across regions: one writer, or CRDT-unfriendly unique keys generated per region with a prefix.
Key prefix e vs w (1 char) partitions the namespace. Or a global identity service (single region) for creates only.
Two regions minting independently.
| design | unique? | GET anywhere |
|---|---|---|
| random 6-char both sides | no | last-write-wins clobber |
| prefix + local counter | yes | replicate map |
| global allocator, local cache | yes | create latency = RTT to allocator |
Interview trap. Active-active inserts of random 6-char keys in two regions without a uniqueness protocol: both succeed, later replication conflicts.
Engineering practice. Either generate keys in a single allocator, or make them unique by construction (region prefix + local counter).
what to draw first in the interview
The whiteboard should start with clients, edge, cache, store of record, async analytics, and the uniqueness rule — not with a hash function.
State QPS, key length, editability, expiry, and abuse requirements as numbers and yes/no before choosing Postgres vs Dynamo vs Cassandra.
Questions that fork the design in the first five minutes.
| question | if yes | if no |
|---|---|---|
| Editable destination? | 302, purge, no 301 | 301 possible |
| Custom aliases? | allowlist, Unicode fold | generated keys only |
| Multi-region creates? | prefix or allocator | single primary |
Interview trap. Opening with 'we MD5 and take 6 chars, then ZooKeeper for locks' spends the interview on a collision protocol you may not need.
Engineering practice. Ask whether aliases, edits, and enterprise host allowlists exist. Those three answers change the design more than the hash.
when not to build one
Most products need a link branded to their own host and a table of (key, url), not a platform shortener with global abuse, CDN, and identity.
An internal tool at 40 rps GET is a Postgres table and a 302 handler. The consumer-scale design is for a different QPS and threat model.
Same feature, two honest designs.
| constraint | internal | public |
|---|---|---|
| creates | 200 / week | 400 / s |
| destinations | corporate allowlist | the internet |
| store | one table | partitioned KV |
| scanner | optional | required |
Interview trap. Copying Bitly's public architecture into a sprint for a marketing team that will mint 200 links a week.
Engineering practice. Scale the uniqueness and abuse controls to the actual create rate and the trust of the destinations. Quote 200/week vs 80k/s before drawing Kafka.