Skip to content
Tech Interview Prep

Top 100 Cloud Architect Interview Questions and Answers

The questions most likely to come up in your Cloud Architect interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.

Curated: · Written: · Reviewed:

Reviewed 100Review pending 3

103 questions

Q&A-56What are the main ways a cloud bill grows without anyone deploying anything new?

Accumulation and traffic shape, mostly.

Storage only grows. Log retention set to never expire, snapshots taken daily and never pruned, orphaned volumes left behind by terminated instances, incomplete multipart uploads that are invisible in the console but billed, and old object versions retained by versioning without a lifecycle rule. Each is small monthly and unbounded over years.

Data transfer scales with usage rather than with deployments. Cross-zone traffic between chatty services, egress to the internet as your user base grows, NAT gateway processing charges on traffic that could have used a private endpoint, and cross-region replication all rise silently.

Autoscaling responds to traffic, so organic growth raises the bill without a deploy. That is expected; the problem is a maximum raised during an incident and never lowered, which turns a temporary ceiling into a permanent floor.

Idle and forgotten resources: non-production environments running at night and at weekends, load balancers with no targets, provisioned capacity for a launch that already happened, and elastic IPs that are billed when unattached.

Committed-use discounts expiring, or reserved capacity lapsing, produces a step change with no infrastructure cause at all.

The systemic fix is lifecycle rules and expiry as defaults rather than as cleanup: retention configured at creation, non-production shut down on a schedule, and a monthly review of the largest line items by tag. Cleanup projects work once; defaults work continuously.

Curated: · Written: · Reviewed:

Q&A-72What is the role of an architecture decision record, and what makes a good one?

Its role is to preserve the reasoning, not the decision. Anyone can read the current system and see what was chosen; almost nobody can reconstruct why, and without the why, teams either cargo-cult a constraint that no longer applies or overturn a decision that is still load-bearing.

A good one is short, a page, and contains five things. The context: what forces were in play, what the constraints were, what was actually being decided. The options considered, including the ones rejected, because the rejected options are the part that is impossible to recover later. The decision itself, stated plainly. The consequences, honestly, including what became harder. And the status, with a link to any record that supersedes it.

The two qualities that separate useful records from ceremony: writing the consequences you dislike, and stating what would change the decision. A record that lists only benefits is marketing. One that says "this makes cross-region reads slower, and we would revisit if read latency outside our primary region became a product requirement" is a decision that can be re-evaluated on evidence years later.

They should live in the repository next to the code, be written at the time of the decision rather than reconstructed, and be immutable: superseded by a new record rather than edited, so the history of the reasoning survives.

The test of the practice is whether a new engineer's question "why is it like this?" has an answer that is not a person.

Curated: · Written: · Reviewed:

Q&A-47What is the practical difference between an SLA, an SLO, and an SLI, and why does the distinction matter to an architect?

An SLI is a measurement: the proportion of requests served successfully under 300ms, for example. An SLO is a target for that measurement over a window, such as 99.9% over 28 days. An SLA is a contract with a customer, with financial consequences, and it should be looser than the SLO so that missing internal targets does not immediately mean paying out.

The distinction matters architecturally for three reasons.

The SLI definition determines what you build. Measuring at the load balancer and measuring from the client produce different numbers and justify different work: one leads to backend investment, the other to CDN and network investment. Arguments about reliability are often really arguments about the indicator.

The SLO sets the architecture. 99.9% and 99.99% are different systems, as discussed, and the SLO is the input that makes multi-region a requirement rather than an aspiration. Agreeing an SLO before designing prevents building the wrong thing in both directions.

The error budget makes the trade-off explicit. The budget: the allowed 0.1%: is a resource that both feature work and reliability work spend. When it is healthy, ship faster. When it is exhausted, reliability work takes priority. That converts an endless argument between product and engineering into a rule that both agreed to in advance, which is the actual value of the whole framework.

Curated: · Written: · Reviewed:

Q&A-51Explain what an availability zone actually is, and what failure it does and does not protect against.

The 90-second version

An availability zone is a fault-isolation domain inside a region: one or more data centres with independent power feeds, cooling, and physical security, joined to the region's other zones by dedicated fibre. AWS documents zones as separated by many kilometres but within 100 km of each other; Azure documents zone-to-zone round-trip latency inside a region as typically under 2 ms. That sub-2 ms RTT is the whole point: it is what makes synchronous cross-zone replication affordable, so a write is acknowledged only after a second zone has it on disk.

What it buys you: a facility-level failure (power, cooling, fire, flood, a failed spine switch) takes out one zone, not the region. What it does not buy you: anything regional (control planes, DNS, IAM, a bad deploy), and anything your own change management breaks.

The numbers

ThingFigureBasis
Zones per region3 in most regions; us-east-1 has 6, GCP us-central1 has 4provider docs
Intra-region inter-AZ RTT~1–2 ms typicalmeasure yours; varies by region and instance type
Cross-region RTTus-east-1 ↔ us-west-2 ~60–70 mstypical measured
EC2 SLA99.99% multi-AZ, 99.5% single-AZAWS Compute SLA
RDS Multi-AZ99.95% SLA; failover typically 60–120 sAWS RDS docs
S399.9% availability SLA, 11 nines durability designAWS S3 docs

99.99% is 4.4 minutes of downtime a month; 99.9% is 43.8 minutes. That gap is roughly the difference between "one zone died and nobody noticed" and "one zone died and you were paged".

Capacity arithmetic

Three zones, even spread, each carrying a third of traffic. Run at 60% of provisioned capacity in steady state. Lose one zone: remaining capacity is 2/3 = 66.7% of provisioned, demand is 60%, so you have 6.7 points of headroom and survive. Run at 70%: you are 3.3 points short, queues build, retries amplify, and the two surviving zones fall over too. Zone redundancy is a capacity plan, not a checkbox: N+1 across three zones means running at or below 66% before you lose anything.

What it does not protect against

  • Regional control plane. EC2's RunInstances API, IAM, Route 53, a regional DNS resolver. These are shared across zones; when they degrade, all three zones degrade together. Zone redundancy is not regional DR.
  • Your own changes. A bad deploy, a schema migration, a certificate expiry, an over-broad IAM policy. Replication copies those to every zone in milliseconds.
  • Zonal dependencies you forgot. An EBS volume lives in one zone, a NAT gateway lives in one zone, an ELB node lives in one zone. An app with a single-AZ NAT gateway is not multi-AZ.
  • Single-writer databases. If the primary is in zone A and you only have read replicas elsewhere, losing A stops writes until failover completes: 60–120 s for RDS Multi-AZ, longer if your connection handling is naive.
  • Two-zone deployments. With synchronous replication and a quorum of 2, losing either zone stops writes. Three zones is the minimum for one-zone fault tolerance.

Detect zone loss with per-AZ error-rate and latency dashboards, not a region-wide average: a region-wide p99 hides a zone that is gone.

Landing

Zones absorb facility failure. Regions absorb regional failure. Backups and change control absorb the failure class that replication faithfully reproduces.

Curated: · Written: · Reviewed:

Q&A-60What is the difference between horizontal and vertical scaling in practice, and why do people reach for vertical first?

The 60-second answer. Vertical scaling makes one machine bigger; horizontal scaling adds machines. Teams reach for vertical first because it is a configuration change with no application change, and for anything stateful it is often the only option that does not require redesigning where state lives. What you buy is headroom; what you pay is a hard ceiling, a restart, and an unchanged blast radius.

VerticalHorizontal
Changeinstance classinstance count
App changenoneworkload must be splittable
Ceilinglargest instance in the familypartitions, coordination, downstream
Availabilitystill one nodesurvives node loss
Downtimerestart or failovernone if rolled

What vertical actually buys. EC2 m7i.large is 2 vCPU / 8 GiB; m7i.24xlarge is 96 vCPU / 384 GiB. That is 48x in one resize with no code change. On-demand list in us-east-1 is roughly $0.10/hr and $4.84/hr: 48x the price for 48x the machine. Within a family the price per vCPU is roughly flat, so the case for vertical is not a cheaper price curve; it is the engineering cost of the alternative.

Where the ceiling bites. 24xlarge is the top of general purpose. Memory-optimized families push the ceiling to multiple TiB, but the ceiling exists, and a class change restarts the instance: with Multi-AZ it is a failover, so it is not an incident response.

The subtler ceiling is memory. Hypothetical: a Postgres primary on db.r6g.4xlarge (16 vCPU, 128 GiB) with shared_buffers at the usual 25% of RAM has 32 GiB of cache. If the hot working set is 50 GiB, about 18 GiB of it is served from storage on every pass; at gp3's 1000 MiB/s throughput ceiling that is roughly 18 seconds of I/O per pass. Moving to db.r6g.8xlarge (32 vCPU, 256 GiB) gives 64 GiB of shared_buffers, the hot set fits, and the misses go to near zero. One config change, one restart, no schema change.

Horizontal's own ceilings. Horizontal is not free capacity. A Kafka topic with 6 partitions supports at most 6 active consumers in a group; the seventh idles. A stateless app tier that scales from 2 pods to 200 hits PostgreSQL's default max_connections of 100 long before it hits CPU: the fix is PgBouncer in transaction mode, which collapses 200 pods × 10 pooled connections into a couple of dozen server connections. And fan-out amplifies tails: if a request reads 10 shards and each has an independent 1% chance of being slow, the chance that at least one is slow is 1 − 0.99^10 ≈ 9.6%, so the request's p99 is worse than any single shard's.

The sequence. Make the app tier stateless and scale it horizontally; scale the data layer vertically while the working set fits in RAM and write throughput fits one machine; shard only when writes or dataset size genuinely exceed the largest instance you can buy. Sharding early costs cross-shard joins and distributed transactions in exchange for capacity you are not using.

Curated: · Written: · Reviewed:

Q&A-85What is the practical difference between a NAT gateway and an internet gateway, and why does it appear on a cost review?

An internet gateway is a route attachment that lets resources with public IP addresses communicate with the internet in both directions. It is horizontally scaled by the provider and it is free: you pay for data transfer, not for the gateway.

A NAT gateway lets resources in private subnets, which have no public addresses, make outbound connections while remaining unreachable from outside. It is a managed appliance and it is billed two ways: an hourly charge per gateway, and a per-gigabyte processing charge on everything that passes through it.

That processing charge is why it appears on cost reviews. A workload pulling container images, downloading packages, shipping logs to a third party, or reading from object storage through the NAT gateway pays per gigabyte for traffic that has no business leaving the provider's network at all. On a busy cluster this is regularly one of the largest surprise line items.

On AWS, the fix is often private endpoints. S3 and DynamoDB gateway endpoints have no hourly or data-processing charge and remove that traffic from the NAT path. Interface endpoints for other services have hourly and per-gigabyte charges; whether they cost less than NAT depends on traffic volume, availability-zone placement, and the number of endpoints, so compare the actual pricing rather than assuming.

The other common cause is deploying one NAT gateway per availability zone for resilience, which is correct, since a single gateway is both a single point of failure and a source of cross-zone transfer charges, and then forgetting that the hourly cost is now tripled across several non-production environments.

Curated: · Written: · Reviewed:

Q&A-97What is the difference between a read replica and a standby, and why does it matter operationally?

A standby exists to be promoted; a read replica exists to answer queries. The replication stream underneath can be identical: the difference is the contract you operate it under, and the defaults that enforce that contract.

PostgreSQL. Streaming replication is asynchronous by default. synchronous_commit defaults to on, but with synchronous_standby_names empty there is no synchronous standby to wait for, so a commit returns once the local WAL is flushed. To make a standby a real failover target you set synchronous_standby_names and pick a synchronous_commit level: remote_write waits until the standby has received the WAL and written it to its file system, remote_apply waits until it has replayed it so queries on the standby can see the commit. wal_level=replica and max_wal_senders=10 have been the defaults since PostgreSQL 10. A standby with hot_standby=on (the default) serves reads, so "standby" does not mean "not readable" in PostgreSQL.

MySQL 8.0. Defaults are binlog_format=ROW, sync_binlog=1, innodb_flush_log_at_trx_commit=1. Semi-synchronous replication is opt-in via the rpl_semi_sync_source plugin, and rpl_semi_sync_source_timeout defaults to 10000 ms. When that timeout fires the source falls back to asynchronous replication: the replica you believed was synchronous is now async, and nothing pages you. That is the most common way an RPO quietly widens.

Standby (failover contract)Read replica (query-serving contract)
PostgreSQL knobssynchronous_standby_names, synchronous_commit=remote_applyhot_standby=on, max_standby_streaming_delay
MySQL knobssemi-sync plugin, rpl_semi_sync_source_timeoutSHOW REPLICA STATUS, Seconds_Behind_Source
Lag signalpg_stat_replication.replay_lag / flush_lagSeconds_Behind_Source, pt-heartbeat
Failure modesilent async fallback → unbounded RPOlong query cancels replay → lag spike

Where the read load bites. On a PostgreSQL standby, max_standby_streaming_delay defaults to 30 s: a long-running query on the standby blocks WAL replay, and after 30 s the server cancels the conflicting query to let replay proceed. So heavy reporting on your failover target does not just add lag: it can cancel queries and delay replay past your RPO. On MySQL, Seconds_Behind_Source is a poor lag signal: it is NULL when the applier thread is idle and it measures the timestamp of the event currently being applied, so it can read 0 while the relay log is far behind. Use pt-heartbeat, or compare Read_Source_Log_Pos against Relay_Log_Pos.

Promotion breaks the topology. Promoting a PostgreSQL standby with pg_ctl promote (or SELECT pg_promote()) leaves every other standby still pointing primary_conninfo at the old primary. They do not follow the new one; you reconfigure them or run Patroni, repmgr or pg_auto_failover. The old primary, when it returns, needs pg_rewind or a fresh base backup. In MySQL 8.0.22+ promotion is STOP REPLICA; RESET REPLICA ALL;, and any replica whose SOURCE_HOST points at the old source needs the same treatment. This is the step that turns a five-minute failover into an hour.

Managed services. Amazon RDS Multi-AZ (DB instance deployment) keeps a standby that does not serve reads and fails over by swapping the DNS CNAME to the standby's endpoint: the application reconnects to the same hostname. RDS read replicas are separate instances you promote manually. Aurora replicas are readable and promotable, which is exactly the case where the two contracts blur and you must decide which node you are protecting.

What to do. Separate the contracts even when one node could satisfy both. Pick the failover target, set its acknowledgement policy to match your RPO, and monitor replay_lag on it. Size query-serving replicas for read load and monitor their lag for read correctness. Do not run reporting on the node you intend to promote.

Curated: · Written: · Reviewed:

Q&A-5Explain the difference between sharding and replication, and describe a situation where adding replicas makes things worse.

They solve different problems. Replication copies the same data to more machines: it buys read capacity, availability, and durability. Sharding splits different data across machines: it buys write capacity and storage beyond one machine.

If your bottleneck is write throughput or dataset size, replicas do not help: every replica applies every write, so the write load per machine is unchanged. If your bottleneck is read throughput, sharding is a much heavier tool than you need.

Replicas make things worse in at least three situations. First, when the application reads its own writes: with asynchronous replication a user updates a profile, is routed to a replica that has not applied the change, and sees stale data. The fix is routing reads that follow a write to the primary, or a session-consistency token, but it must be designed, not discovered.

Second, when write amplification is the real problem: adding replicas increases the total write work in the cluster and the replication traffic, and a primary already saturated by writes gets no relief and some additional load.

Third, when replicas mask a query problem. A missing index turns into "add another replica" because it distributes the pain, and the team pays monthly for what an index would have fixed once.

The diagnostic question is which resource is saturated. Read IOPS and read CPU point to replication; write IOPS, replication lag under normal load, or storage growth point to sharding, which is a much larger commitment, since cross-shard joins and transactions largely stop being available.

Curated: · Written: · Reviewed:

Q&A-39Explain the trade-off between synchronous and asynchronous replication for a database, using numbers.

Synchronous replication commits only after at least one replica has acknowledged the required log position. Your RPO for acknowledged commits can be zero, subject to the system's exact acknowledgement policy, and the cost is that each commit waits for remote coordination.

Within a region, providers engineer availability zones for low-latency connectivity, so synchronous cross-zone replication is often affordable and is a common managed multi-AZ configuration. The actual latency must be measured for the chosen regions, zones, and database rather than assumed from a universal number.

Across regions the numbers change decisively: round trips are commonly tens of milliseconds on the same continent and can exceed 100ms intercontinentally. The replication wait is normally paid when the transaction commits, not once for every SQL statement inside that transaction. Workloads that commit frequently still add an inter-region round trip to each transaction, so latency rises and commit throughput per connection falls.

Asynchronous replication commits locally and ships the log afterward, so write latency is unaffected. The cost is a non-zero RPO equal to the replication lag at the moment of failure, and lag is not constant: it grows under write bursts, during index maintenance, and during long-running transactions on the replica.

So the usual production shape is synchronous within a region for durability and automatic failover, asynchronous across regions for disaster recovery. And whichever you choose, replica lag must be monitored, because with synchronous replication a slow replica becomes a write-latency problem, and with asynchronous replication it silently becomes a data-loss problem.

Curated: · Written: · Reviewed:

Q&A-78Explain the difference between at-least-once, at-most-once, and exactly-once delivery, and which one you can actually buy.

At-most-once means a message is delivered zero or one times: the sender does not retry, so nothing is duplicated and messages can be lost. At-least-once means the delivery protocol retries until acknowledgement, so duplicates can occur; it reduces loss from ambiguous acknowledgements but does not defeat retention expiry, unrecoverable broker failure, or application bugs. Exactly-once means precisely one delivery.

You can buy at-least-once. It is what essentially every broker provides, and it follows from the fact that a sender cannot distinguish a lost message from a lost acknowledgement.

Exactly-once delivery, in the strict sense of the message crossing the network exactly once, is not achievable in a distributed system with failures. What you can achieve is exactly-once processing, and it is built at the consumer: deduplicate on a message identifier, or make the effect idempotent so that repeated application is harmless.

Some systems advertise exactly-once semantics, and what they generally provide is transactional processing within their own boundary: a consume, process, and produce cycle committed atomically inside the same platform. That is genuinely useful and it stops at the boundary: the moment the effect is an external API call or a write to a different system, the guarantee no longer covers it.

So the architectural rule is to assume at-least-once everywhere and design consumers to be idempotent. That assumption costs little when duplicates never arrive and saves you entirely when they do, and the failure it prevents, double-charging or double-shipping, is the kind that is visible to customers.

Curated: · Written: · Reviewed:

Q&A-16Explain isolation levels by describing a bug that read committed allows and repeatable read does not.

A non-repeatable read. Under read committed, each statement sees a fresh snapshot, so two reads of the same row inside one transaction can return different values if another transaction commits between them.

The classic bug: a transfer routine reads an account balance, checks it against the withdrawal amount, and then re-reads the row to build a receipt. Between the two reads another transfer commits, and the receipt reports a balance inconsistent with the decision that was just made.

Repeatable read pins one snapshot for the whole transaction, so both reads return the same value. It does not prevent everything: write skew survives repeatable read. Two transactions each read the same set of rows, each concludes a constraint still holds (say, at least one doctor remains on call) and each writes a different row. Neither sees the other's write, both commit, and the invariant is violated even though every individual read was consistent.

Serializable is what prevents write skew, at the cost of aborted transactions your application must be prepared to retry.

The practical point for an architect is that isolation level is a contract about what the database prevents, and everything it does not prevent becomes application responsibility. Choosing read committed and then writing check-then-act logic without SELECT FOR UPDATE is the single most common source of correctness bugs that only appear under load.

Curated: · Written: · Reviewed:

Q&A-18When is a window function the right tool instead of GROUP BY, and what does that change about the result set?

Use a window function when you need a value computed over a related set of rows and you need to keep those rows. GROUP BY collapses each group to one row; a window function attaches the computed value to every row it was computed from. If the deliverable is one row per group, GROUP BY is the right tool and usually the cheaper one.

Worked example

orders (PostgreSQL 16):

customer_idorder_idordered_atamount
11012024-01-0540
11022024-01-0925
11032024-01-0910
22012024-01-0360
SELECT customer_id, order_id, ordered_at, amount,
       SUM(amount) OVER (PARTITION BY customer_id ORDER BY ordered_at) AS running_total
FROM orders;

Trace, partition by partition. Customer 1's rows in ordered_at order are 101 (Jan 5), then 102 and 103 (both Jan 9). The default frame with an ORDER BY is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and RANGE includes peers, rows tied on the ordering key, as a block:

  • 101: frame is just Jan 5 → 40.
  • 102: frame is everything up to Jan 9, including both peers → 40 + 25 + 10 = 75.
  • 103: same frame → 75.
customer_idorder_idordered_atamountrunning_total
11012024-01-054040
11022024-01-092575
11032024-01-091075
22012024-01-036060

The same data under GROUP BY customer_id returns two rows, (1, 75) and (2, 60), and the individual orders are gone. That is the whole distinction: four rows versus two.

Mechanism and the details that decide it

Evaluation order is FROM → WHERE → GROUP BY → HAVING → window functions → ORDER BY → LIMIT. Windows run after filtering and aggregation, which is why you cannot reference one in WHERE, GROUP BY or HAVING; filter on a rank via a CTE or subquery. It also means you can combine the two: GROUP BY first, then a window over the grouped rows.

Frame choice changes results, not just syntax. Swap in ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW on the example and the ties break by physical order: 40, 65, 75. Arbitrary unless you add a tiebreaker to ORDER BY. Omit ORDER BY entirely and the frame is the whole partition, so every row gets the group total (75, 75, 75, 60): a per-group total repeated, not a running one.

Ranking follows the same shape: ROW_NUMBER is unique but arbitrary on ties, RANK leaves gaps (1, 1, 3), DENSE_RANK does not (1, 1, 2). LAG/LEAD give row-over-row comparison without a self-join.

Trade-offs and failure modes

  • Cost. A window needs its input ordered by (partition, order); without a matching index that is a sort, and large partitions can spill to disk (work_mem in Postgres). GROUP BY can use a hash aggregate with no sort and emits fewer rows downstream. If you only need the aggregate, don't window.
  • Ties. The default RANGE frame silently groups peers, as above. If you want strict row-by-row accumulation, say ROWS and add a deterministic tiebreaker.
  • Non-deterministic ordering. ROW_NUMBER() OVER (ORDER BY score) on duplicate scores is not stable across runs; add a unique column to the ORDER BY.
  • Filtering. WHERE rank = 1 fails to parse; wrap the windowed query in a CTE and filter outside it.

Detect the cost problem with EXPLAIN (ANALYZE, BUFFERS): a Sort node above the scan with disk spills means the index on (customer_id, ordered_at) is missing or unused.

Curated: · Written: · Reviewed:

Q&A-22Explain why normalizing to third normal form can hurt, and how you decide to denormalize.

Normalization optimizes for write correctness: each fact stored once, so no update can leave two copies disagreeing. It costs read work, because reassembling an entity means joins.

It hurts when a hot read path needs many joins to produce one screen, when a query aggregates across a deep join tree, or when the join keys are large and the tables live on different shards: at which point the join stops being a query-planner problem and becomes an application one.

The decision to denormalize should be driven by a measured query, not by anticipation. Concretely: identify the specific query, confirm it is actually hot, confirm indexing and query structure are exhausted first, and only then duplicate data.

When you do, be explicit about how the copy stays correct. That is the whole cost of denormalization and it is where the bugs live. The options are a trigger, an application-level write path that updates both, a materialized view refreshed on a schedule, or an asynchronous projection built from a change stream. Each has a different staleness window, and you should be able to state it.

A useful middle ground is a covering index or a materialized view, which gives read performance without inventing a second source of truth the application must maintain.

The rule I would state in an interview: normalize until a measurement tells you not to, denormalize deliberately with a named consistency mechanism, and never denormalize by accident because a column was convenient to copy.

Curated: · Written: · Reviewed:

Q&A-26What index would you create for a query filtering on tenant_id and status and sorting by created_at, and why that column order?

A composite index on (tenant_id, status, created_at), in that order.

The rule for a B-tree composite index is equality columns first, then the range or sort column. Both tenant_id and status are equality predicates here, so they lead; created_at trails so the index also satisfies the ORDER BY, letting the planner skip a sort entirely.

Column order within the equality group matters for reuse rather than for this query: an index on (tenant_id, status, ...) can serve a query filtering on tenant_id alone, because a B-tree can seek on any leading prefix. The reverse is not true: filtering on status alone cannot use this index efficiently. So the more broadly useful column leads.

Selectivity is the tiebreaker people usually cite, and it is secondary to the prefix rule. Putting a highly selective column first narrows the scan fastest, but only if queries actually filter on it.

If the query also returns a small fixed set of columns, extending the index to cover them turns it into an index-only scan and avoids the heap lookup entirely: worth it for a genuinely hot query, at the cost of a larger index and slower writes.

Every index is paid for on every insert, update, and delete, so the honest answer includes checking whether an existing index already has this prefix. Three overlapping indexes on the same leading column is a common and expensive smell.

Curated: · Written: · Reviewed:

Q&A-57Explain what happens under the hood when you add an index, and why it is not free.

The 90-second answer

An index is a second copy of some of your data, kept sorted, plus a pointer back to each row. Adding one is a bulk build: the engine reads the whole table, sorts the key values, and writes out a new tree, then it must keep that tree correct on every subsequent write, forever. So the cost is threefold: the build, the per-write maintenance, and the space and cache the structure occupies.

What the structure actually is

A B-tree is a balanced tree of fixed-size pages. In PostgreSQL (8 KB pages, default B-tree fillfactor 90), a leaf entry for a bigint key is roughly 8 bytes of key plus a 6-byte row pointer, so a leaf page holds about 400 entries. That gives a height of 3 for 10 million rows: root, one internal level, leaves. A lookup is 3 page reads worst case, but the root and internal level are almost always resident, so a point lookup is typically one physical read.

Worked example (hypothetical, but the arithmetic is the point)

10M rows at 100 bytes per row → ~1 GB heap. An index on a bigint column: ~24,000 leaf pages ≈ 190 MB, about 19% of the table. The same table with an index on a 40-byte text column: ~68,000 leaf pages ≈ 540 MB. Over half the heap. That is the "indexes can approach table size" claim with numbers behind it.

What a write costs

Trace an INSERT into a table with three indexes:

  1. Heap: new tuple written, one WAL record.
  2. Index A: descend 3 levels, insert into the leaf, one WAL record.
  3. Index B: same.
  4. Index C: same.

One logical insert becomes four WAL records and four page modifications. Five indexes → six records. That is the amplification, and it is why write-heavy tables get slower with every index added.

Cache

Index pages compete with heap pages in the buffer pool. A rarely-read index still gets dirtied by every insert, so it consumes cache and generates checkpoint I/O without serving reads: it slows unrelated queries by evicting pages that were being used.

Failure modes worth naming

  • HOT updates. PostgreSQL can skip index maintenance when an UPDATE touches no indexed column and the new tuple fits on the same page. Set fillfactor to 100 and that room disappears: every update becomes a new entry in every index, and the indexes bloat until REINDEX.
  • CREATE INDEX CONCURRENTLY does not block writes, but it scans the table twice, can fail, and leaves an INVALID index you must drop and retry.
  • InnoDB secondary indexes store the primary key, not a row pointer. A CHAR(36) UUID primary key adds 36 bytes to every entry of every secondary index; a BIGINT primary key adds 8.
  • An unused index is still maintained on every write. It is not free just because no query reads it.

What to do

Index for the queries you actually run. One composite index on (a, b) serves both WHERE a = ? and WHERE a = ? AND b = ?; three single-column indexes serve neither as well and cost three times the write maintenance. Check usage stats (pg_stat_user_indexes.idx_scan, MySQL's sys.schema_unused_indexes) and drop what nothing reads.

Curated: · Written: · Reviewed:

Q&A-70What is a materialized view, and when does it beat a cache?

A materialized view is a table whose contents are the result of a query, stored in the same database and refreshed by the database. A cache is the same idea with the storage and the invalidation moved into your application. So the question isn't which is faster: it's who owns invalidation, and how many distinct keys the recompute has to cover.

The view wins when the result is expressible in SQL, the consumers already hold database credentials, and the recompute is a scan you would otherwise repeat once per key. The cache wins when the value isn't a query result at all (rendered HTML, a third-party response, a session blob), when consumers shouldn't reach the database, or when key cardinality is so high that the view would be as large as the base table.

Refresh semantics decide the rest. "Materialized view" means very different things per engine:

EngineRefreshIncremental?Blocks readers?
PostgreSQL 17REFRESH MATERIALIZED VIEW [CONCURRENTLY]NoPlain: yes, ACCESS EXCLUSIVE. CONCURRENTLY: no
OracleON COMMIT or ON DEMAND, FAST/COMPLETEYes, with materialized view logsFast: no. Complete: rebuilds the whole view
SQL ServerIndexed view, maintained on base-table DMLYes, synchronouslyNo, but every write pays

PostgreSQL's CONCURRENTLY requires a unique index on the view over plain columns (no expressions, no WHERE), requires the view to already be populated, and cannot run inside a transaction block. It is not cheaper, it re-runs the query and diffs against the old contents, it just doesn't take the exclusive lock.

Worked example (hypothetical figures). A 400M-row events table, 120 GB, 200 tenants. A dashboard shows per-tenant daily totals for the last 30 days.

  • Full aggregate across all tenants: 38 s.
  • Per-tenant aggregate, index on (tenant_id, ts): 0.35 s.
  • Materialized view over the full aggregate: 6,000 rows; dashboard read is 2 ms.
  • Refresh every 15 min: 38 s per 900 s window = 4.2% duty cycle, one query.
  • Cache with a 15-min TTL keyed per tenant: 200 misses × 0.35 s = 70 s of query time per window, and without TTL jitter all 200 land in the same second.

The view does less total work because one scan serves every consumer. The cache pays the scan once per key. That ratio, consumers per scan, is the real decision variable.

Failure modes. A refresh job that fails leaves the view serving old rows with no error at read time; PostgreSQL records no last-refresh timestamp, so you write one yourself (a metadata row updated in the same transaction as the refresh) and alert on its age. A plain refresh queues behind any long-running reader and then blocks every reader behind it for the full 38 s. On SQL Server and Oracle ON COMMIT, the cost moves to writes: a 10M-row bulk load into a table with three indexed views can take several times longer than into the bare table.

When the cache wins. Values that aren't SQL: rendered fragments, third-party API responses, per-user feeds with millions of keys. Consumers that shouldn't touch the database at all: edge workers, mobile clients. Read paths needing sub-millisecond p99 at tens of thousands of QPS, where even a 2 ms indexed read on the primary is too much. And anything where the recompute is cheap per key but the key space is huge, so a view would just be a copy of the base table.

Staleness. Both are stale by construction; state the bound the same way: "as of the last refresh" versus "up to TTL old". The difference is who is accountable when it's wrong: the database, or your code.

Curated: · Written: · Reviewed:

Q&A-12How do you choose between a relational database and a document store for a new service?

The 90-second answer. Three inputs decide it: what a single request reads or writes, what must be atomic, and what write throughput you need in the next 12 months. If a request touches one aggregate by key and you need writes beyond a single primary, take a document store. Otherwise take PostgreSQL. Data shape is not an input: jsonb stores nested documents fine, and MongoDB's 16 MB BSON cap makes deeply nested or unbounded documents a liability rather than a feature.

PostgreSQL 16MongoDB 7.0DynamoDB
Per-item size capno practical row limit (TOAST offloads large values)16 MB BSON400 KB
Multi-item atomicityany rows, one transaction4.0 replica sets, 4.2 sharded clusters; 60 s default transaction lifetimeTransactWriteItems, ≤100 items / 4 MB
Cross-entity queryjoins on any indexed column$lookup; not across shardsnone: denormalize or query a GSI
ConstraintsFK, UNIQUE, CHECK, NOT NULLunique index onlynone
Write scalingone primarysharding; resharding since 5.0per-partition 1,000 WCU / 3,000 RCU

What actually differs. Constraints and cross-entity queries are the real axis. A foreign key in PostgreSQL rejects a dangling reference at write time; in MongoDB nothing does, so the check lives in application code and a bug writes orphans that surface months later as nulls in a report. MongoDB's default write concern is w:1: the primary acknowledges before the write reaches a majority, so a failover can lose it; w:majority fixes that at the cost of a round trip to a majority of the set. PostgreSQL's default synchronous_commit=on waits for the local WAL flush; standby acknowledgement is opt-in via synchronous_standby_names.

Failure modes, and how you see them.

  • Document store chosen to dodge schema design. The schema moves into application code. Detect it by adding a $jsonSchema validator or a CHECK constraint and watching what it rejects on the first day.
  • Bad shard key. A monotonically increasing key puts every insert on one chunk. sh.status() from mongos shows the skew. Before 5.0 you could not change a shard key at all; resharding is now online but rewrites the collection.
  • Unbounded embedded arrays. Appending events to a document hits the 16 MB cap and writes start failing. Track max document size, not average.
  • Relational write ceiling. Watch replication lag, p99, and connection saturation against max_connections (default 100) in pg_stat_activity.

Do the arithmetic before you migrate. 5,000 writes/s at 1 KB each is 5,000 WCU, which needs at least 5 DynamoDB partitions at 1,000 WCU each. A single PostgreSQL 16 primary on 16 vCPUs typically sustains 10k–40k pgbench TPS depending on the read/write mix: hardware-dependent, but the order of magnitude means 5,000 writes/s is not by itself a reason to leave Postgres. The asymmetry that decides the default: denormalizing a normalized schema later is a mechanical migration, while reconstructing relationships from documents that were never modeled as relationships is a rewrite.

Curated: · Written: · Reviewed:

Q&A-35When would you use a graph database rather than expressing relationships in SQL?

Short answer. Reach for a graph database when the queries that matter are variable-depth traversals: "is A connected to B within N hops", "shortest path", "everything reachable from this node", and traversal, not aggregation, is the dominant access pattern. Having relationships is not the criterion; SQL models relationships fine. A join is a one-hop traversal.

Why SQL degrades. A recursive CTE is a breadth-first search, and it is the right tool up to a point. PostgreSQL 14+ even has the SQL-standard CYCLE clause, so you don't hand-roll a path array to avoid looping forever. The cost is per hop: each frontier row does an index probe into the edge table (edges(src_id)) plus a heap fetch, and the planner has no statistics for the recursive term. It estimates from the anchor term alone and can pick a nested loop that is catastrophic at depth 4.

Illustrative arithmetic, not a benchmark: 50M accounts, ~40 edges each. A 6-hop reachability query whose component holds 2M accounts means the CTE materializes a frontier growing toward 2M vertices, each costing an index probe that returns ~40 edge rows: order 10^8 edge rows touched. A traversal that stops at the first hit at depth 3, or runs bidirectional from both ends, touches thousands.

What the graph store actually buys. Neo4j's index-free adjacency makes following a relationship a pointer dereference inside the store file rather than a B-tree probe: a constant-factor win per hop, not an asymptotic one. The larger wins are early termination, bidirectional search, and shortestPath() in Cypher or ANY SHORTEST in GQL (ISO/IEC 39075:2024) as first-class operators instead of hand-written Dijkstra in SQL.

Alternatives before you add a database. A recursive CTE plus an index on the edge table's source column. A materialized transitive-closure table if the graph is small and changes slowly. SQL/PGQ (ISO/IEC 9075-16:2023) has begun shipping in PostgreSQL, letting you write path patterns in SQL. That removes the language objection, not the per-hop cost.

Failure modes. Supernodes: one vertex with 10M edges blows up any traversal through it; check the degree distribution before committing. Distributed graph stores (Amazon Neptune, JanusGraph, Neo4j Fabric) pay a network round trip per partition crossing, so a 6-hop path spanning 6 partitions is 6 RTTs per candidate path. Aggregation over all edges (GROUP BY counts, rollups) is a columnar scan in a relational engine and a slow full traversal in a graph one. And you now operate a second database: backups, upgrades, a Cypher/GQL skill set, weaker BI tooling.

Decision test. Are several of the important queries variable-depth traversals? Have you measured the recursive CTE failing on real data? If either answer is no, keep the relational store.

Curated: · Written: · Reviewed:

Q&A-44What is the risk of putting business logic in a database trigger, and when is it still the right choice?

The risks are invisibility and testability. A trigger executes on a write that a developer reading application code cannot see, so behavior appears without a call site. It runs inside the writing transaction, so it extends lock duration and can deadlock in ways that are hard to attribute. It does not participate in code review the way application code does unless migrations are reviewed with equal care, and it is awkward to unit test.

Triggers also interact badly with bulk operations: a trigger written for single-row inserts can turn a million-row load into a million procedure calls, and a trigger that writes to another table can cause a bulk operation to deadlock against normal traffic.

They are still the right choice when the guarantee must hold regardless of the writer. Audit trails are the clearest example: if the requirement is that no row can change without an audit record, application code cannot provide it, because a migration, an admin console, or a manual fix bypasses the application. The same reasoning applies to enforcing a temporal invariant, or maintaining a denormalized counter that must never disagree with its source.

The deciding question is whether the rule is a property of the data or a policy of the application. Data properties (this must always be true of any row, however it got there) belong in the database, alongside constraints and foreign keys. Application policy belongs in application code where it is visible, testable, and versioned with the feature it serves.

Curated: · Written: · Reviewed:

Q&A-50What is connection pooling, and why does it become a problem specifically in serverless architectures?

The 90-second answer

A connection pool keeps a small set of already-open database connections and hands them out to requests, so you pay the TCP handshake, TLS negotiation, authentication and session setup once instead of per query. On a long-lived server that cost is fixed: one process, one pool, sized to the CPU count.

Serverless removes the "one process" part. Every Lambda execution environment is its own process with its own pool, and concurrency is decided by traffic, not by you. At 200 concurrent invocations with a pool of 5, that is 1,000 connections pointed at a database whose max_connections defaults to 100 on PostgreSQL (151 on MySQL). The excess is rejected with FATAL: sorry, too many clients already (SQLSTATE 53300), and rejected for everyone, including the long-running services sharing that instance.

The fix is to move the pool out of the function: an external pooler (PgBouncer, RDS Proxy, or your provider's pooled endpoint) holds a bounded set of server connections and multiplexes many client connections onto them. Failing that, use an HTTP data API so there is no persistent connection at all.

Why the arithmetic is so lopsided

Opening a Postgres connection costs roughly 20–50 ms (TCP, TLS, auth, backend fork); an indexed point query is 1–2 ms. That ratio is why pooling exists. It is also why the failure is invisible until it isn't: nothing in the function's configuration mentions the database's limit, and concurrency tracks traffic.

Idle connections are not free either. Lambda freezes an execution environment between invocations rather than closing it, so a pool of 5 sits idle holding 5 Postgres backends: each a few MB of baseline memory plus whatever work_mem allocations its queries made (work_mem defaults to 4 MB per sort/hash node). A thousand frozen environments is gigabytes of RAM on the database doing nothing.

Choosing a fix

OptionWhat it doesCost / caveat
External pooler in transaction modeBounded server pool, many clients multiplexedSession state breaks: SET, advisory locks, LISTEN, temp tables, cursors. Protocol-level prepared statements only from PgBouncer 1.21 (Oct 2023); before that, session mode or a driver-level workaround
HTTP data API (e.g. Aurora RDS Data API)No persistent connection at allPer-request overhead, no transaction spanning calls unless the API supports it, different SDK
Reserved concurrency + pool size 1Hard ceiling on instance countThrottles: excess invocations get TooManyRequestsException (429) and are retried or dropped. You traded database protection for availability

Pool size 1 is enough because Lambda runs one invocation at a time per execution environment; a larger pool buys nothing and multiplies the ceiling. Reserved concurrency is a ceiling, not a fix: set it to max_connections / pool_size minus headroom for everything else on the instance. With pool size 1 and 100 server connections, 80 reserved leaves 20 for migrations, admin work and the long-running services.

Detection

Alert on the database's connection count, not on function errors: by the time the function sees 53300, the instance is already refusing connections. On Postgres, SELECT count(*) FROM pg_stat_activity against max_connections; on RDS, the DatabaseConnections CloudWatch metric. Also watch the pooler's own queue depth: a saturated pooler converts a connection error into latency, which is better but still a p99 problem.

Curated: · Written: · Reviewed:

Q&A-6Your cache hit rate is 95% and the origin still falls over every time you deploy. What is happening?

A 95% hit rate means the origin is sized for 5% of traffic. A deploy that empties or invalidates the cache does not increase load by 5%: it multiplies origin load by twenty, instantly. That is a thundering herd, and the origin is being asked to serve a volume it was never provisioned for.

The related failure is a stampede on a single hot key: when a popular entry expires, every concurrent request misses simultaneously and all of them recompute the same value.

Three mitigations, in order of how much they help here. First, do not empty the cache on deploy: version cache keys so old and new entries coexist and the new namespace fills gradually, or keep the cache external to the deployment unit so a rollout does not touch it. Second, request coalescing: the first miss for a key computes the value while the rest wait on that single in-flight computation, turning a thousand origin requests into one. Third, serve-stale-while-revalidate: return the expired entry immediately and refresh in the background, so an expiry never becomes a miss on the request path.

Randomized TTL jitter matters too. Entries populated together expire together, so a uniform TTL reproduces the herd on a schedule.

The underlying point is that a high hit rate is not resilience, it is a dependency. The number to know is what the origin can serve at 0% hit rate, because that is your actual floor.

Curated: · Written: · Reviewed:

Q&A-42Your team wants to add a read-through cache in front of a database. What do you ask them to specify before agreeing?

Five things, and if they cannot answer them the cache will cause more incidents than it prevents.

What is the staleness budget? Every cache is a decision to serve old data. The answer is a duration, per data type, agreed with whoever owns the product consequence, not a TTL chosen because it sounded reasonable.

What is the invalidation mechanism? TTL-only expiry is fine for tolerant data. Anything a user edits and immediately re-reads needs explicit invalidation on write, and the write path has to be identified. The hardest cases are writes that happen outside the application: a batch job, an admin tool, a database migration.

What happens when the cache is empty or unavailable? If the origin cannot serve the full load, the cache is a single point of failure wearing a performance disguise. This should be a measured number, not an opinion.

What is the key design, and can it collide? Multi-tenant caches with a key that omits the tenant leak data across customers, and this is a real and recurring incident class.

What will you measure? Hit rate alone is not enough: you want hit rate, origin load, and staleness-related error reports, so you can tell a healthy cache from one that is quietly serving wrong data.

If the underlying problem is a missing index or an N+1 query, the cache is hiding a fixable cause behind a permanent operational commitment.

Curated: · Written: · Reviewed:

Q&A-62When is it worth introducing a CDN, and what does it not solve?

It is worth it as soon as you have geographically distributed users and any cacheable content, which is nearly every public product. The gain is not only bandwidth: terminating TLS close to the user removes handshake round trips, which is often a larger share of perceived latency than the response itself for small requests.

It solves: static asset delivery, image and video distribution, absorbing traffic spikes for cacheable content, and a meaningful share of volumetric attack traffic at the edge.

It does not solve dynamic, personalised responses, which is where the interesting latency usually lives. A per-user dashboard cannot be cached at the edge without careful design: cache keys that include the user, or edge computation, both of which are real work rather than a configuration toggle.

It does not fix a slow origin. Cache misses go to the origin, and a cold cache after an invalidation sends the full load there, so origin capacity still has to be sized for the miss case.

It does not fix write latency at all. Writes go to the origin regardless of edge presence.

And it introduces its own failure modes worth planning for: a bad cache key that mixes users' responses, an invalidation that propagates slowly across hundreds of locations, and a cached error page that outlives the error. Caching a 500 for an hour turns a one-minute incident into a long one, which is why negative caching needs a much shorter TTL.

Curated: · Written: · Reviewed:

Q&A-13What is the difference between a load balancer health check and a readiness check, and what goes wrong when a team conflates them?

A liveness check answers "is this process broken and in need of a restart?" A readiness check answers "should this instance receive traffic right now?" They have different consequences: failing liveness kills the process, failing readiness only removes it from rotation.

Conflating them produces two characteristic outages.

The first is the cascading restart. A readiness check that verifies the database connection is used as a liveness check. The database has a brief hiccup, every instance reports unhealthy, every instance is restarted, and the restart storm (cold caches, reconnection thundering herd) turns a five-second database blip into a twenty-minute outage. Liveness should test only the process itself.

The second is traffic to instances that cannot serve. A health check that returns 200 as soon as the HTTP server binds, before dependencies are connected and caches are warmed, means the load balancer sends traffic during startup and users get errors during every deploy. Readiness must reflect actual ability to serve.

The nuance worth mentioning is that readiness should usually not fail on a downstream dependency either. If every instance marks itself unready because a shared dependency is degraded, you have converted partial degradation into total unavailability, and removed the capacity that could have served the requests not touching that dependency. Better to serve degraded responses and let the dependency's own signals drive the alert.

Curated: · Written: · Reviewed:

Q&A-36What is the practical difference between a 502, a 503, and a 504 from a load balancer, and what does each tell you to check?

The 60-second answer. These three codes are not a taxonomy of causes. They are three proxy-level events (bad response from upstream, no upstream available, upstream too slow) that each product maps to codes differently. So the first question is not "what does a 502 mean" but "which component generated this response, and what does this balancer map to that code." Check the balancer's own reason telemetry (Envoy response flags, the nginx error log, ALB target-group metrics) before you touch the application.

Balancer502503504
nginxupstream connection refused/reset, or malformed response headerlimit_req/limit_conn exceeded, or explicit return 503proxy_read_timeout exceeded (default 60s)
HAProxyserver connection failure after retriesno server available: all marked downtimeout server exceeded
Envoymaps upstream connection failures to 503, not 502no healthy upstream, circuit-breaker overflow, upstream connection failureroute timeout exceeded (default 15s)
AWS ALBtarget closed the connection or sent a malformed responsetarget group has no healthy targetstarget didn't respond within the idle timeout (default 60s)

Where the code was generated decides the investigation. A 503 from the balancer means the request never reached your code: check target registration, health checks, and circuit breakers. A 503 from the application means it did: check shutdown hooks and overload shedding. Same code, opposite direction. Envoy's response flags (UH, UO, UF, UT) and nginx's error-log line naming the upstream address settle this in one lookup. On ALB, HTTPCode_ELB_5XX_Count versus HTTPCode_Target_5XX_Count splits balancer-generated from target-generated 5xx directly.

Two failure modes worth naming. First, keep-alive mismatch: if the backend's keep-alive timeout is shorter than the balancer's idle timeout (nginx keepalive_timeout defaults to 75s, ALB idle timeout to 60s) the balancer reuses a connection the backend has just closed and gets a 502. It shows up as a low, load-independent 502 rate with an nginx log line reading "upstream prematurely closed connection." Second, a 504 that is not application slowness: the timeout can fire during TCP connect or TLS handshake, before any request is sent. If the application logs show no matching request, the request never arrived: look at connect time and handshake failures, not at your handler.

The trade-off. Balancer-side reason codes are cheap, uniform, and describe the connection; they say nothing about what the application was doing. Application-side tracing describes the stack but not why the connection failed. Alert on the split, balancer-generated versus target-generated 5xx, rather than on aggregate 5xx, and join the two on a request ID. A single aggregate 5xx alert cannot separate a capacity event from a latency event from a connection-lifecycle bug, and those three have different runbooks.

Curated: · Written: · Reviewed:

Q&A-3Walk me through what happens when a request enters your VPC and reaches a private database. Which controls does it pass, and what breaks if one is misconfigured?

The request arrives at an internet-facing load balancer in a public subnet: public meaning its route table has a default route to an internet gateway. The load balancer's security group allows 443 from the internet.

The load balancer forwards to an application instance in a private subnet, whose route table has no internet gateway route; its outbound traffic goes through a NAT gateway instead. The application's security group allows the application port from the load balancer's security group, referencing the group rather than a CIDR range, so the rule stays correct as instances come and go.

The application connects to the database in a data subnet. The database security group allows the database port from the application's security group only. Network ACLs may sit at the subnet boundary as a coarse second layer; unlike security groups they are stateless, so return traffic needs an explicit rule.

The instructive misconfigurations: a private subnet with an internet gateway route is not private, and nothing in the console labels it wrong. A security group opening the database port to 0.0.0.0/0 is a direct path from anywhere the subnet is reachable. A stateless NACL that allows inbound but not ephemeral outbound ports produces connections that establish and then hang, which reads like an application bug for hours.

The invariant worth stating: a subnet is public or private because of its route table, and a security group is safe because of what it references, not because of what it is named.

Curated: · Written: · Reviewed:

Q&A-24What is the difference between authentication and authorization in an API, and where do teams usually get authorization wrong?

Authentication establishes who is calling; authorization decides what that caller may do. They fail differently. An authentication bug lets a stranger in. An authorization bug lets a legitimate user reach someone else's data: more common, and usually more damaging because it is silent.

The mistake teams make most often is object-level authorization. The endpoint verifies the token and that the caller holds the invoices:read scope, then loads the invoice by the ID in the path without checking that this invoice belongs to this caller. Any authenticated customer reads any invoice by changing a number. OWASP's API Security Top 10 lists this as API1:2023, Broken Object Level Authorization, for exactly this reason: tests written by people who only use their own data never hit it. The test that catches it is two tenants: authenticate as A, request B's object ID, assert 404. A 403 confirms the object exists and leaks that much.

Fixes, and what each costs. The structural fix is to make ownership part of the query rather than a separate check: select the invoice by ID and tenant ID, so a mismatch returns nothing and there is no code path that skips the check. That is the strongest default when the resource has one owner and the datastore can express the predicate. It stops working when ownership is not a column: a shared document with per-user grants, delegated access, admin impersonation, or a resource behind a search index or a third-party API you cannot add a WHERE clause to.

ApproachWhere the check livesFits whenCost
Query-scoped ownershipData access layerOne owner per row, relational storeRules that are not expressible as a predicate; duplicated across queries
Route middleware / guardPer endpoint, before the handlerCoarse rules (role, scope, tenant)Easy to forget on a new route; cannot see the object
Policy engine in-process (OPA, Cedar)Sidecar or library, decisions from a policy fileMany rules, auditability, rules that change oftenPolicy and data drift apart; the engine cannot see the row unless you feed it
Centralized authz service (Zanzibar-style, e.g. OpenFGA)Network call per checkCross-service, relationship-heavy, list filteringExtra hop and a new availability dependency; needs caching and a fail-closed default

Most production systems combine them: middleware for coarse gates, query scoping for the object, a policy engine or service for the rules that fit neither. The failure mode to design against is a check that lives in one place and is bypassed by a second code path: a batch job, an export, an admin tool.

Two more recurring errors. Trusting claims in a JWT without verifying the signature and the issuer: a token is a bearer of assertions, not evidence, until verified. And enforcing authorization only in the UI, so hidden buttons are the only thing stopping an action the API will happily perform.

Scopes limit what an application may attempt; they do not answer whether this user may touch this record. Both checks are required.

Curated: · Written: · Reviewed:

Q&A-2A team wants to give an application permanent access keys so it can read from an object store. Why do you push back, and what do you propose instead?

What I'd say first

The key's lifetime is the incident's lifetime. CreateAccessKey sets no expiry date; the only clock on an access key is access_key_1_last_rotated in the IAM credential report, and that moves only when a human rotates it. A leaked key stays live until someone notices and deletes it. Federated identity replaces it with a credential the platform mints per session: 1 hour by default for AssumeRoleWithWebIdentity, bounded by the role's MaxSessionDuration (12 hours maximum). Same read access, exposure measured in hours instead of quarters.

What I propose

Attach the identity to the compute, not to a config file:

  • EC2 / ECS: instance profile or task role.
  • EKS: IRSA. The pod's service account is annotated with the role ARN, and the projected token is exchanged through AssumeRoleWithWebIdentity.
  • GCP / Azure: workload identity federation or managed identity.
  • CI outside the cloud: OIDC federation. GitHub Actions mints a JWT when the job requests id-token: write; the trust policy checks aud = sts.amazonaws.com and sub = repo:org/repo:ref:refs/heads/main, and STS returns temporary credentials.
Long-lived access keyFederated session
LifetimeUntil deleted1 h default, ≤12 h
Where it livesConfig, CI variables, laptopsIn memory, per process
RevocationDelete key, redeploy every consumerStop issuing; expires on its own
Audit identityKey ID: identical for every callerSession ARN naming role and workload
Blast radiusEverywhere the key was copiedOne workload, one session

Narrow the permission too

"Read from an object store" is not a policy. Name the bucket and prefix:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::reports-prod/incoming/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::reports-prod",
      "Condition": {"StringLike": {"s3:prefix": "incoming/*"}}
    }
  ]
}

GetObject takes the object ARN; ListBucket takes the bucket ARN. A policy that puts ListBucket on .../incoming/* silently denies listing, and the usual fix is to widen to s3:* on *. Start narrow and widen on a real failure; that surfaces the actual requirement instead of guessing it.

Failure modes

  • Trust policy wildcard. sub = repo:org/* lets any repository in the org assume the role. Pin the repo, branch or environment.
  • Credential theft from the instance. GuardDuty's UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration fires when instance credentials are used from outside AWS: that signal exists only because the credential is short-lived and tied to a source.
  • Unused permissions. IAM Access Analyzer generates a policy from CloudTrail history; run it after a month rather than hand-guessing.

If a key is genuinely unavoidable

A SaaS with no federation support. Then: a named owner, rotation that a scheduled job actually performs, and alerting on use from an unexpected ASN or region. That is a compensating control, not a design.

Curated: · Written: · Reviewed:

Q&A-33How do you handle secrets for an application running across several environments?

What a strong candidate says first

The property to preserve is that a secret never exists in a form a human copies. That rules out checked-in config files, hand-set environment variables, and CI variables pasted by an engineer: all three put the value somewhere a person can read and paste elsewhere.

So: a managed store, one per environment, and the workload fetches at start-up using its own platform identity.

  • AWS: Secrets Manager or SSM Parameter Store (SecureString), read via an EC2 instance profile, an ECS task role, or EKS IRSA. A service account annotated with a role ARN, whose projected OIDC token is exchanged through sts:AssumeRoleWithWebIdentity.
  • GCP: Secret Manager with GKE Workload Identity.
  • Azure: Key Vault with managed identity.
  • Self-hosted: Vault, with Kubernetes auth or AppRole.

Separate AWS accounts (or GCP projects) per environment, not just separate paths. A path prefix is an authorization boundary only if the IAM policy actually scopes to it; an account boundary survives a misconfigured policy.

Rotation

Two tiers.

Dynamic credentials are the strongest. Vault's database secrets engine creates a real database user per workload and drops it when the lease is revoked. Pin the role's default_ttl low: 1 h with max_ttl 24 h is a common shape; Vault's system default lease TTL is 32 days (768 h), far too long to inherit by accident. A leaked credential expires on its own.

Static secrets need a dual-secret window. AWS Secrets Manager's rotation Lambda writes the new value under the AWSPENDING staging label, tests it, then promotes it to AWSCURRENT; the previous value stays reachable as AWSPREVIOUS. Consumers reading AWSCURRENT pick up the new value while in-flight connections still hold the old one. Without that window, rotation is a synchronized outage.

Cost, so you can defend the choice: Secrets Manager list price is $0.40 per secret per month plus $0.05 per 10,000 API calls. 50 secrets across 3 environments is 150 secrets ≈ $60/month: cheap enough that "we'll just use env vars" is not a budget argument.

Failure modes

  • Kubernetes Secrets are base64-encoded, not encrypted; etcd encryption at rest is opt-in. Mount through the Secrets Store CSI Driver or sync with External Secrets Operator so plaintext isn't sitting in etcd.
  • Environment variables are readable from /proc/<pid>/environ by anything running as the same user or root, and they land in crash dumps and child processes. Prefer a tmpfs file mount or an in-process fetch.
  • Log leakage is the biggest real exposure: a secret in a URL query string ends up in the access log, an exception tracker captures a config object. Redact at the logging layer, not by asking developers to be careful.
  • Audit every read (Vault audit devices, CloudTrail GetSecretValue) so you can answer which workload read which secret, and when.

Encryption at rest in the store is table stakes. Access control and rotation are what protect you.

Curated: · Written: · Reviewed:

Q&A-41What does least privilege actually look like for a CI pipeline that deploys infrastructure?

It looks like several narrow identities rather than one broad one, separated along the axes where mistakes are expensive.

Separate by stage: the plan step needs read access and nothing else, and it is the step that runs on untrusted pull-request code. The apply step needs write access and runs only from the protected main branch after approval. Giving the plan step write permissions means an attacker who can open a pull request can modify infrastructure.

Separate by environment: the pipeline identity for staging cannot touch production. Federate via OIDC with a trust policy that pins the repository and the branch or environment, so a workflow on a fork or a feature branch cannot assume the production role even with a valid token.

Scope the permissions themselves to the resource types the pipeline manages, with conditions on tags or naming prefixes where the provider supports it. Administrator access assigned because scoping was tedious is the normal state of affairs and the normal cause of incidents.

Add guardrails that do not depend on the pipeline being correct: service control policies or organisation-level rules that deny destructive actions on protected resources regardless of the caller, and deletion protection on stateful resources.

Finally, treat the pipeline as production infrastructure. It has credentials to your entire estate, so its own configuration deserves code review, its logs should be immutable, and third-party actions should be pinned to a commit hash rather than a moving tag.

Curated: · Written: · Reviewed:

Q&A-64What does idempotency mean for infrastructure code, and where does it commonly break?

It means applying the same configuration twice produces the same result as applying it once. Declarative tools give this by construction: the tool compares desired state to actual state and acts only on the difference.

It breaks in several recognisable places.

Provisioners and local scripts embedded in the configuration. A shell script that appends a line to a file runs again on the next apply and appends it twice. Anything imperative inside a declarative tool is where idempotency dies.

Resources with generated names. A resource named with a timestamp or a random value that is regenerated each run produces a new resource on every apply instead of matching the existing one.

State drift from out-of-band changes. Someone edits a resource in the console; the tool now sees a difference and reverts it, which is idempotent with respect to the configuration but surprising to whoever made the change, and destructive if their change was an emergency fix.

Resources the tool cannot fully read back. Where a provider cannot inspect an attribute, it either always shows a diff or never shows one, and both are wrong in different ways.

And ordering assumptions: implicit dependencies that happened to work in the order the tool chose last time and do not this time.

The practical defences are reading every plan rather than skimming it, treating a plan showing changes on an unchanged configuration as a bug to investigate, and keeping imperative steps in a pipeline stage where they can be made idempotent explicitly.

Curated: · Written: · Reviewed:

Q&A-10Your Terraform plan shows it will destroy and recreate a production database. What do you do?

Stop, and do not apply. A destroy-and-recreate on a stateful resource is data loss, and the plan is telling you before it happens, which is the entire value of a plan step.

First, find which attribute forced replacement. The plan output names it, marked as forces replacement. Common causes: a change to an immutable field like engine version, availability zone, or subnet group; a resource identifier that changed because a module was refactored; or drift where someone modified the resource in the console and the configuration now disagrees.

The remedy depends on the cause. If a module was renamed or moved, the resource is fine and the state is wrong: terraform state mv (or a moved block, which is reviewable and belongs in the code) fixes the address without touching infrastructure. If a genuinely immutable attribute must change, that is a migration with a data path, not a Terraform apply: create the new resource alongside, replicate, cut over, retire the old one.

Two guardrails should already be in place. prevent_destroy on stateful resources turns this from a scary plan into a hard error. And a CI pipeline that runs plan on every pull request, posts the output, and requires human approval before apply means nobody encounters this for the first time at the keyboard at 2am.

The wider lesson is that console changes create exactly this class of drift, which is why break-glass console access should be time-boxed and reconciled afterward.

Curated: · Written: · Reviewed:

Q&A-37How do you keep an infrastructure-as-code repository maintainable once it covers dozens of services?

The 90-second answer

Three levers, in payoff order: state boundaries, module scope, and making CI the only path to apply.

Split state by blast radius and change frequency. A single state file means every plan re-reads every resource it tracks (a 3,000-resource state spends minutes refreshing before you see a diff) and one bad apply can take out services that had nothing to do with the change. Networking, identity and the cluster get their own rarely-touched states; each service gets its own. Cross-boundary values come from terraform_remote_state data sources or, better, from a published SSM/parameter value, so no state file is a hard dependency of another's apply.

Modules should encode a decision, not wrap a resource. A module with fifty pass-through variables is indirection with no payoff. A module that encodes "how we run a service here" (logging, tags, encryption, network placement already correct) is the paved road. Version it (version = "~> 5.0" from the registry, a git tag for private modules) and let consumers upgrade on purpose.

Then make CI the only way to apply: plan on every PR with the output posted, apply only from main, and no human credentials that can apply directly.

State boundaries in practice

StateContentsChange frequencyApplied by
orgaccounts, OUs, SCPsquarterlyplatform
networkVPCs, subnets, transit gateway, DNSmonthlyplatform
identityIAM roles, OIDC providersmonthlyplatform
clusterEKS/GKE control plane, node groupsweeklyplatform
svc-<name>one service's infradailyservice team

Refactoring into these boundaries is not a destroy/recreate. moved blocks (Terraform 1.1+) let you relocate a resource in config and state in one apply:

moved {
  from = aws_instance.web
  to   = module.service.aws_instance.web
}

Locking, drift, policy

terraform {
  backend "s3" {
    bucket       = "acme-tfstate"
    key          = "svc-payments/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true   # Terraform >= 1.10; DynamoDB lock table before that
  }
}

Drift detection is a scheduled terraform plan -detailed-exitcode per state. Exit codes are documented: 0 no changes, 1 error, 2 changes present. Exit 2 on a state nobody touched is a finding, not a surprise during an unrelated apply.

Policy as code runs on the plan JSON, not on the HCL: terraform show -json plan.out piped into OPA/Conftest, or Sentinel if you are on Terraform Cloud/Enterprise. A rule like "no 0.0.0.0/0 ingress on a database security group" fails the PR before a human reads it.

What breaks anyway

  • Two services sharing one state because they were split "later": the later never comes. Split at creation.
  • Module versions pinned to main in a git ref. A module change then lands in every consumer's next plan with no review. Pin tags.
  • Drift detection that only alerts. If nobody reconciles, the console stays the source of truth and the repo becomes documentation.

Curated: · Written: · Reviewed:

Q&A-59Your architecture diagram and your production environment have diverged. Why does that matter, and what do you do about it?

The 90-second answer. The diagram is an input to decisions (incident triage, risk review, onboarding) so when it diverges, every one of those decisions is made from a fiction. The fix isn't to draw more carefully; it's to stop hand-maintaining the parts a machine can derive, and to make drift visible within one pipeline cycle instead of one audit cycle.

Split the diagram in two. The derived layers come from systems that already know the truth:

LayerDerived fromHow stale it gets
Service-to-service callsOpenTelemetry spans → Jaeger service graph or Datadog Service Mapas traffic flows; Jaeger's graph is built by a scheduled job you configure
Infra and network topologyTerraform state, refreshed by terraform planone pipeline run
Resources that exist but aren't in stateAWS Config configuration items, or a driftctl scanConfig records on change; driftctl per scan
Ownership and tierBackstage catalog-info.yaml per repoone catalog poll
Intent: boundaries, trust zones, what the redundancy protects againsthand-drawn, in the repoone PR

The intent layer is the only part worth drawing by hand, and it's the part no generator can produce.

Then close the loop mechanically. A nightly CI job:

terraform plan -refresh=true -detailed-exitcode -out=tfplan
# exit 0 = no changes, 2 = changes present, 1 = error

Exit code 2 means state and reality disagree: either drift or an unapplied change. Run it against the commit that was last applied so the two are distinguishable. Caveat: plan only sees what's in state; a resource created entirely outside Terraform is invisible to it, which is why the Config/driftctl layer exists.

For console changes, CloudTrail is the detector: it typically delivers events within about five minutes of the API call, so an alert on write calls from a principal that isn't the pipeline role catches hand-edits the same day. If the platform is Kubernetes and reconciled by Argo CD with selfHeal: true, manual edits are reverted on the next reconciliation (default 3 minutes) rather than merely reported.

Worked example, hypothetical: a service starts calling a new cache and the diagram isn't updated. An incident at 02:00 has the on-call chasing the old dependency chain; each wrong hypothesis costs a 10-minute deploy-and-check cycle, and three of them add 30 minutes to a 45-minute MTTR. The same change with a trace-derived map shows the new edge in the first five minutes of traffic.

Failure modes worth naming: a derived map that's empty because sampling dropped the edge (check the span sampling rate before trusting an absent edge); a drift alert that fires on every legitimate pipeline run because the check runs against the wrong commit; and a catalogue that ends up populated by hand after all, which is the original problem with extra steps.

The rule that keeps it honest: the architecture record changes in the same PR as the change. A diagram reviewed alongside the code stays current; a diagram in a wiki does not.

Curated: · Written: · Reviewed:

Q&A-66Explain the transactional outbox pattern and the problem it solves.

It solves the dual-write problem: a handler that must both commit a database change and publish a message has no way to make those two operations atomic, because they are different systems. Commit then publish loses the message if the process dies in between. Publish then commit emits an event for something that never happened.

The outbox removes the second system from the critical path. In the same transaction that writes the order, insert a row into an outbox table containing the message. Either both are committed or neither is: one transaction, one database.

A separate relay then reads unpublished outbox rows and publishes them, marking them sent afterward. If the relay crashes between publishing and marking, it republishes on restart, which is why consumers must be idempotent, but at-least-once is exactly the guarantee message brokers give anyway, so this adds no new burden.

The relay can be a polling loop or, better, a change data capture stream reading the database's replication log, which avoids polling load and reduces latency to milliseconds.

The costs are an extra table, a component to operate, and a small publication delay. What you get is that the database is the single source of truth for both state and the events derived from it, which means no event describes a state that does not exist and no state change silently fails to notify anyone.

The mirror-image pattern on the consumer side is the inbox: record processed message IDs transactionally to make consumption idempotent.

Curated: · Written: · Reviewed:

Q&A-32Explain what a saga is and when you would prefer it over a distributed transaction.

The 60-second answer. A saga is a business transaction split into a sequence of local transactions, one per service, each with a compensating action that semantically undoes it. There is no global lock and no global commit: intermediate states are visible to other readers, and consistency is restored by running compensations in reverse order rather than by rolling back. The pattern is Garcia-Molina and Salem's, SIGMOD 1987.

Prefer it over a distributed transaction whenever the participants are separate services with separate databases. Two-phase commit holds locks from prepare until the coordinator's decision, so lock hold time is the duration of the slowest participant plus the coordinator's log write, and a coordinator that dies after prepare leaves those locks held until a timeout. Across service boundaries that is an availability liability you cannot buy your way out of, and most cloud stores don't expose a transaction manager at all: PostgreSQL ships max_prepared_transactions = 0, so XA is off until an operator turns it on, and DynamoDB's TransactWriteItems caps at 100 items and 4 MB inside one account and region. It is not a cross-service coordinator.

What it costs. Compensation is not rollback. A refund is not the same as never charging: the customer saw the charge, and the receipt email already went out. So the domain has to tolerate visible intermediate states, and someone who understands the business consequences has to design each compensation.

Worked trace. Order 8842, four steps, failing at the last one:

#StepLocal txCompensationTime
1createOrderinsert PENDINGdelete order8 ms
2reserveInventorydecrement availableincrement available40 ms
3chargePaymentinsert charge, key=8842refund charge900 ms
4confirmOrderset CONFIRMED—fails, 120 ms

Step 4 fails, so the orchestrator runs 3, 2, 1 in reverse: refund, release, cancel. Total elapsed 1.07 s. Under 2PC the inventory row locked in step 2 would be held for that whole 1.07 s, and if the coordinator died after prepare, until the participant's lock timeout: MySQL's innodb_lock_wait_timeout defaults to 50 s, Oracle's DISTRIBUTED_LOCK_TIMEOUT to 60 s. That is the trade in one number: 1.07 s of visible intermediate state versus up to a minute of a locked row.

The step that commits the business decision, the charge here, is the pivot. Everything before it must be compensatable; everything after it must be retriable, because you can no longer undo the charge without a refund the customer sees.

Requirements. Every step and every compensation must be idempotent, because retries are certain. Key the charge on the order id and let the PSP dedupe: Stripe retains idempotency keys for 24 hours, so a retry after that window is a second charge. Persist saga state so an orchestrator crash can resume; Temporal keeps it in workflow history, which warns at 10,000 events / 10 MB and hard-fails at 50,000 events / 50 MB, so a long-running saga needs continue-as-new. And you need a plan for a compensation that itself fails: Temporal's default retry policy (1 s initial interval, 2.0 backoff, 100 s max interval, unlimited attempts) will retry forever, which is the wrong answer for a refund. Route it to an alert and a human after a bounded number of attempts.

Choreography vs orchestration. Choreography, services reacting to each other's events, drops the coordinator but makes the flow implicit: reconstructing "why is order 8842 stuck" means joining logs across four services. Orchestration keeps it in one ordered history. The threshold is not step count so much as whether the compensation order differs from the reverse of the forward order; once it does, or once you're past three or four services, the coordinator earns its coupling. Named implementations: Temporal, AWS Step Functions (25,000 history events, 1-year max for Standard workflows), Camunda/Zeebe, Axon, and Seata's AT/TCC/Saga modes.

Curated: · Written: · Reviewed:

Q&A-28Two services need the same data with different access patterns. Do you share a database, duplicate the data, or add an API between them?

Shared database is the option to argue against first, because it is the one that looks cheapest. Two services writing the same tables have no boundary: a schema change requires coordinating deploys, and neither team can reason about invariants alone. It is acceptable when the two services are really one system with an artificial split, and rarely otherwise.

An API between them is the default. One service owns the data and the invariants; the other asks. The cost is a runtime dependency: a synchronous call in the request path means the caller's availability is now the product of both, and latency that compounds if the call is in a loop.

Duplication via an event stream or change feed is right when the reader's access pattern is genuinely different: a search index, an analytics store, a denormalized read model. The reader keeps its own copy shaped for its queries, and accepts staleness measured in seconds. It removes the runtime dependency and adds an eventual-consistency contract you must state and monitor.

The deciding question is who owns the invariant. If both services must enforce the same rule, they should not both hold the data: one owns it and the other asks. If the second service only reads, and can tolerate a defined staleness window, duplication is usually the better architecture because it decouples availability.

Read replicas of another team's database are a shared database wearing a disguise; the schema coupling is identical.

Curated: · Written: · Reviewed:

Q&A-7When would you choose eventual consistency, and how do you explain that choice to a product owner who hears "sometimes wrong"?

Choose it when the cost of unavailability during a partition exceeds the cost of a brief disagreement between replicas, and when the domain has a sensible merge or last-writer rule.

Good fits: view counts, feed ranking, product recommendations, notification delivery, most analytics. Bad fits: account balances, inventory decrement at the point of sale, uniqueness constraints, and anything where two "correct" answers combine into a real-world loss.

The explanation for a product owner should be in their units, not CAP's. Not "we relax consistency" but "if the network between our regions breaks, would you rather the checkout page show a slightly stale item count, or return an error?" That is the actual trade, and it is a product decision, not an engineering one.

It also helps to name the window. Eventual consistency does not mean unbounded: replication typically converges in tens or hundreds of milliseconds, and the interesting question is what happens during the rare partition. Saying "readers may be up to two seconds behind, and during a regional partition up to a few minutes" is concrete enough to reason about.

Finally, be precise that CAP applies during a partition. When the network is healthy a system can be both consistent and available, and the real day-to-day trade is PACELC's second half: even without partitions, stronger consistency costs latency. That is usually the trade the product owner actually feels.

Curated: · Written: · Reviewed:

Q&A-54What would make you choose event-driven architecture over synchronous request/response between services?

Choose events when the producer does not need to know who consumes, when the consumer's availability should not affect the producer's, and when the work is genuinely asynchronous from the user's point of view.

Order placed is the canonical example: the order service publishes, and inventory, fulfilment, analytics, and notifications each react. Adding a fifth consumer requires no change to the producer, which is the real architectural benefit: the coupling removed is coupling to the list of consumers.

It also decouples availability. A synchronous call chain has availability equal to the product of every link; an event consumer that is down accumulates a backlog and catches up.

The costs are substantial and worth stating plainly. Debugging becomes harder: there is no stack trace across an event boundary, so distributed tracing with correlation IDs stops being optional. Ordering and duplicate delivery become the consumer's problem. The overall behavior of the system is not written down anywhere: you cannot read one file and know what happens when an order is placed. And eventual consistency becomes user-visible, so the UI has to represent in-progress states honestly.

I would keep synchronous calls where the caller needs the result to proceed and the user is waiting, and use events where the reaction is a side effect. Mixing them deliberately along that line gives most of the benefit; making everything an event because events are decoupled produces a system nobody can reason about.

Curated: · Written: · Reviewed:

Q&A-74When does adding a message queue make a system less reliable rather than more?

When it converts a fast, visible failure into a slow, invisible one, and when the team treats it as durable storage rather than as a buffer.

The clearest case is an unbounded queue in front of an under-provisioned consumer. Without the queue, overload produces immediate errors and the problem is obvious within a minute. With it, producers keep succeeding, the backlog grows for hours, and by the time anyone notices, the queue holds work that is no longer relevant, notifications about events users have long since seen, and draining it takes longer than the incident that caused it. The queue turned a capacity problem into a much longer degradation.

It also hurts when the work was genuinely synchronous. Moving a step out of the request path only helps if the user does not need the result. If the UI must poll for completion, you have added a component, an eventual-consistency problem, and a new failure mode in exchange for a faster response that the user cannot act on.

Other ways it degrades reliability: retries without a dead letter queue, so a poison message blocks a partition indefinitely; ordering assumptions that the broker does not guarantee; and a dual write between the database and the broker, which silently loses messages or invents them.

The mitigations are bounded queues with explicit rejection when full, alerting on backlog age rather than depth, and a documented answer to what happens when the queue is unavailable, because for many systems the correct answer is to fail the request rather than to lose the work.

Curated: · Written: · Reviewed:

Q&A-11Design the queue between an order service and a slow fulfilment system. What guarantees do you need, and what do you do with messages that fail repeatedly?

What I'd say in the first two minutes

Assumptions: the order service is an OLTP service on Postgres; fulfilment is a separate system with p99 around 2 s and a history of 10–20 minute outages. The order must be accepted even when fulfilment is down, so the queue is a durability boundary, not a latency trick. I'd use Kafka, or SQS if I want zero ops, with at-least-once delivery.

Three guarantees, in priority order:

  1. At-least-once, made effectively-once at the consumer. No broker gives end-to-end exactly-once. Fulfilment keys on order ID with a uniqueness constraint, so a redelivery is a no-op.
  2. Per-order ordering, not global. Global ordering forces one consumer per topic, capping throughput at a single consumer's rate. Partition by order ID (Kafka) or use a FIFO queue with a message group ID (SQS) so unrelated orders run in parallel.
  3. Durability before ack. Producer acks=all, replication factor 3, min.insync.replicas=2. The publish returns only once the message is on a quorum.

Retry budget

Bounded retries with exponential backoff and jitter, plus a hard cap on total time:

AttemptDelay before attemptCumulative
100
21 s1 s
32 s3 s
44 s7 s
58 s15 s
616 s31 s
732 s63 s
832 s (capped)95 s
→ DLQ

Jitter of ±20% on each delay, so a fulfilment outage doesn't produce a synchronised retry storm the moment it recovers. Eight attempts over ~95 s rides out a blip; anything longer belongs in the DLQ where a human can see it.

Don't sleep inside the consumer to implement that backoff. Kafka's max.poll.interval.ms defaults to 300000 ms (5 minutes); a consumer that blocks longer than that without polling is evicted from the group and the message is redelivered to another consumer, which is exactly how you get duplicate fulfilment. Use retry topics with increasing delay, or SQS with a visibility timeout.

DLQ

A DLQ nobody reads is a data-loss mechanism with extra steps. Alert on depth > 0 sustained for 5 minutes, and on DLQ rate above 0.1% of throughput. Each DLQ message carries the original payload, the failure reason, the attempt count, and the first and last failure timestamps: without those, replay is guesswork.

The dual write

The real bug is the order committed to Postgres and the publish failing, or the reverse. Transactional outbox:

BEGIN;
INSERT INTO orders (id, ...) VALUES (4471, ...);
INSERT INTO outbox (id, topic, payload, created_at)
  VALUES (gen_random_uuid(), 'fulfilment', '{"order_id":4471,...}', now());
COMMIT;

A relay polls outbox every 100 ms, publishes, and marks the row sent. If the relay dies mid-publish, the row is still there and gets republished: at-least-once again, which the consumer's idempotency key absorbs. Delete sent rows after 7 days.

Failure modes I'd watch

  • Consumer crashes after fulfilment succeeds but before ack → duplicate fulfilment. Detect by counting idempotency-key conflicts; a nonzero rate means the ack window is too wide.
  • One poison order blocks its partition. Bounded retries cap the damage at ~95 s per message.
  • RabbitMQ with prefetch=0 (unlimited) hands a consumer the whole queue and OOMs it. Set prefetch to 10–50.
  • DLQ depth grows during an outage and nobody notices until reconciliation. Alert, don't graph.

Curated: · Written: · Reviewed:

Q&A-46How do you prevent a retry storm from turning a partial outage into a total one?

Retries multiply load exactly when a system is least able to absorb it, and every layer that retries multiplies the layers below it. If the client, gateway, and service library each make three total attempts, one logical request can become twenty-seven dependency attempts; three simultaneous client requests can become eighty-one.

The controls, in order of importance.

Retry at one layer only. Decide where retries live and make every other layer fail fast. This single decision removes the multiplication.

Bound the attempts and use exponential backoff with full jitter. Backoff without jitter synchronises clients into waves that arrive together; jitter spreads them. The number of attempts should be small, two or three, because a request that has failed three times is unlikely to succeed on the fourth and is certainly adding load.

Budget retries as a fraction of total traffic, typically 10%. When the budget is exhausted, stop retrying entirely. This is what prevents a widespread failure from producing a self-inflicted denial of service, because a global failure means every request retries at once.

Circuit-break so that once a dependency is clearly down, calls fail immediately rather than waiting for a timeout and then retrying.

On the serving side, shed load explicitly: reject early with a 429 and a Retry-After header rather than queuing work you cannot complete. A fast rejection costs almost nothing; a slow timeout holds a connection for seconds.

Curated: · Written: · Reviewed:

Q&A-23Your service depends on a third-party API that is occasionally slow. How do you keep that from taking down your service?

Three mechanisms, layered.

A timeout on every call, always. A call without a timeout inherits the default, which is often minutes or infinite, and a slow dependency then consumes your threads or connections until you have none. The timeout should be derived from the dependency's real latency distribution, a little above p99, not chosen as a round number.

A circuit breaker. After a threshold of failures or timeouts, stop calling and fail immediately for a cooling-off period, then let a single probe test recovery. This does two things: it stops you from queuing work against something that cannot serve it, and it stops you from adding load to a dependency that is struggling. Without it, retries make an overloaded dependency worse.

A bulkhead. Give the dependency its own bounded connection pool or concurrency limit, so exhausting it cannot starve unrelated request paths. This is what converts "the recommendation API is slow" into a degraded recommendations panel rather than a site outage.

Then decide the fallback deliberately: cached previous result, a default, a degraded response, or a clear error. The fallback is a product decision and it should be made before the incident.

Retries need care in this context: retry only idempotent operations, bound the attempts, use exponential backoff with jitter, and budget retries as a fraction of traffic so they cannot multiply into a self-inflicted denial of service.

Curated: · Written: · Reviewed:

Q&A-77How do you design for graceful degradation rather than total failure?

The 90-second answer. For each user journey, list the dependencies it touches and mark each one critical or optional. Critical means the journey cannot be completed without it; optional means the page is still worth serving without it. Then, for every optional dependency, write down four things as design outputs: a timeout, a circuit breaker, a bulkhead, and a fallback. Anything you don't classify gets the default behaviour, which is to hang until the caller's own timeout fires and take the page down with it.

On a product detail page: catalogue and price are critical. Recommendations, reviews, "recently viewed", and the live inventory count are optional. That's a product decision, not an infrastructure one, and it's the decision that determines everything downstream.

Concrete numbers. Budget the page at 300 ms server-side. Catalogue p99 is 120 ms, so recommendations get a 50 ms timeout: set from the dependency's own latency distribution, not from a round number. If recommendations hang, the page returns at roughly 170 ms with that module omitted. A timeout of 500 ms "to be safe" is how one slow dependency becomes a slow page.

Circuit breaker: Resilience4j's defaults are a 100-call count-based window, a 50% failure-rate threshold, 60 s in the open state, and 10 permitted calls in half-open. The one that bites is minimumNumberOfCalls = 100: on a low-traffic endpoint the breaker never trips because it never sees 100 calls. Lower it to 10–20 for anything that isn't high-QPS. Hystrix is the name most people still reach for; it's been in maintenance mode since 2018, so Resilience4j (JVM), Polly (.NET), or Envoy/Istio outlier detection at the mesh layer.

Bulkhead: Resilience4j's default is 25 concurrent calls with zero wait. That's the point: the 26th caller fails fast instead of queueing behind a stalled dependency. Give each optional dependency its own thread pool or connection pool so its exhaustion can't consume the catalogue's.

DependencyClassTimeoutIsolationFallback
Cataloguecritical200 msshared poolnone: fail the page
Recommendationsoptional50 msown pool, 25 concurrentomit module
Reviewsoptional80 msown poolcached copy, stale-if-error
Inventory countoptional40 msown pool"check at checkout"

Fallbacks that actually work. Prefer a cached copy over a default. Cache-Control: max-age=60, stale-while-revalidate=30, stale-if-error=86400 (RFC 5861) lets a CDN serve day-old reviews rather than nothing. Where there's no cache, omit the section: an empty recommendations rail is better than a spinner that resolves into an error.

Read-only mode. Most read-heavy systems can serve their core value with the write path off. Postgres has default_transaction_read_only; MySQL has read_only and super_read_only. Turning that on converts a failed primary into a degraded period where browsing works and checkout returns a clear "try again in a few minutes". Build it as a flag you can flip, and test it, because the first time you use it should not be during the incident.

Feature flags are the manual version of the same control: OpenFeature (CNCF), LaunchDarkly, Unleash. A kill switch on every non-critical, resource-intensive feature (recommendations, personalisation, the ML ranking pass) is worth building before you need it.

Test it. Fallback paths are the code that never runs, so they're the code that's broken when you need it. Inject the failure: Toxiproxy to cut the connection, LitmusChaos or AWS Fault Injection Service in a staging cluster, or just kill the recommendations pod and assert the page still returns 200 with the module absent. Add a synthetic check that asserts the degraded response, not just the healthy one.

Failure modes to watch. Retries without a budget turn a blip into a retry storm: cap them at ~10% of traffic (Envoy's retry_budget). Envoy's default circuit-breaker thresholds are 1024 connections / pending / requests, which is far above what most services can absorb, so set them from measured capacity. And a fallback that returns a default silently can mask a dependency that's been down for a week: alert on the fallback rate, not just on errors.

Curated: · Written: · Reviewed:

Q&A-8How do you make a payment API safe to retry?

With an idempotency key supplied by the client and enforced by the server.

The client generates a unique key per logical operation, not per HTTP attempt, and sends it as a header. The server records the key and a request fingerprint under a unique constraint, then returns the stored result for a completed duplicate and rejects the same key with different parameters.

If the "charge" is only a mutation in the service's own database, record the key and the mutation in one transaction. If it calls an external payment processor, no local database transaction can make both systems atomic. Forward the same idempotency key to a processor that supports idempotent requests and persist an in-progress/succeeded/failed state locally; an outbox or recovery worker must reconcile the uncertain case where the remote call succeeds but the local result is not recorded.

Concurrency needs care: two simultaneous retries can both reach the server. The unique constraint makes one win; the loser must wait for the winner's result rather than returning "in progress" and inviting another retry. Storing a state column (in-progress, succeeded, failed) and having the loser poll or block briefly handles this.

The key must also be scoped to the request body. If a client reuses a key with different parameters, that is a client bug and the server should reject it with a 4xx rather than silently returning the wrong prior response.

Keys need a retention window, 24 hours is typical, long enough to cover any realistic retry and short enough to bound storage. And the response you replay should include the original status code, because a retry of a failed charge must not look like a success.

Curated: · Written: · Reviewed:

Q&A-55How do you handle schema changes for a service deployed continuously with no maintenance window?

The 60-second answer. Never ship a schema change and the code that depends on it in the same deploy. Split every change into expand, migrate, contract, and make each step compatible with the code version before and after it. During a rolling deploy two versions run against one schema: that constraint is what everything else follows from.

Expand. Make the schema accept both shapes. Add the new column nullable, add the new table, build the new index concurrently. Old code is untouched and keeps working.

ALTER TABLE orders ADD COLUMN currency text;   -- nullable, metadata-only on PG 11+
CREATE INDEX CONCURRENTLY idx_orders_currency ON orders (currency);

Deploy code that writes both, reads old. This is the reversible point: rolling back the app is safe because the old shape is still being maintained.

# psycopg

def insert_order(conn, o):
    conn.execute(
        "INSERT INTO orders (id, amount, currency) VALUES (%s, %s, %s)",
        (o.id, o.amount, o.currency))

def currency_of(row):
    return row["currency"] if row["currency"] is not None else "USD"

Migrate. Backfill in batches, then flip reads to the new shape while still writing both.

UPDATE orders SET currency = 'USD'
WHERE id IN (SELECT id FROM orders WHERE currency IS NULL LIMIT 5000);

40M rows at 5k per batch is 8,000 batches; 30 ms of work plus a 100 ms sleep between them is roughly 17 minutes of wall clock, and the sleep is what keeps autovacuum and replication from falling behind. Watch pg_stat_replication for lag, if a replica is 30 s behind, lengthen the sleep.

Then tighten the constraint without a rewrite:

ALTER TABLE orders ADD CONSTRAINT orders_currency_nn
  CHECK (currency IS NOT NULL) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_currency_nn;  -- no write lock
ALTER TABLE orders ALTER COLUMN currency SET NOT NULL;      -- PG 12+ skips the scan

Contract. Stop writing the old shape, then drop it: after at least one full deploy cycle, so a rollback would not need it. Dropping a column the previous release still writes to turns a routine rollback into an incident.

Locks, precisely. ACCESS EXCLUSIVE blocks reads and writes; the trick is holding it only for metadata-only operations, which is milliseconds.

StatementLockBlocks writes
ADD COLUMN (nullable, or DEFAULT, PG 11+)ACCESS EXCLUSIVE, metadata-onlybriefly
CREATE INDEX CONCURRENTLYShareUpdateExclusiveno
VALIDATE CONSTRAINTShareUpdateExclusiveno
SET NOT NULL (PG 12+, validated CHECK)ACCESS EXCLUSIVE, metadata-onlybriefly
DROP COLUMNACCESS EXCLUSIVE, metadata-onlybriefly

Failure modes. Set SET lock_timeout = '3s' on every migration. A DDL statement waiting behind a long-running query queues every subsequent query behind itself: a 5 ms ALTER TABLE becomes a full outage. Fail, retry, don't wait. CREATE INDEX CONCURRENTLY can fail and leave an INVALID index behind; check pg_index.indisvalid and drop-and-retry rather than assuming it worked. And never rename a column in place: old code selects the old name; add the new column and dual-write instead.

When not to. If you can take a window, a plain ALTER TABLE is simpler. On MySQL, gh-ost or pt-online-schema-change do the copy-and-swap for you; on Postgres, pg_repack covers bloat. The discipline above is what you do when neither is available.

Curated: · Written: · Reviewed:

Q&A-14You need to add a NOT NULL column to a 500-million-row table in production. How?

Not in one statement. Adding a NOT NULL column with a default historically rewrites the table and holds an exclusive lock for the duration, which on 500 million rows means an outage. Modern PostgreSQL avoids the rewrite for a constant default, but the approach below is what survives across engines and versions.

Break it into steps that each hold a short lock:

Add the column as nullable with no default. This is a metadata-only change and is effectively instant.

Deploy application code that writes the new column on every insert and update, while still tolerating nulls on read. Now the set of null rows is closed and can only shrink.

Backfill in batches: tens of thousands of rows per batch, committing between them, with a pause to let replication catch up. A single UPDATE over 500 million rows produces an enormous transaction, bloats the table, and can push replicas hours behind.

Verify no nulls remain, then add the NOT NULL constraint. In PostgreSQL, add it as a NOT VALID check constraint first and validate it separately, which takes a weaker lock than a direct table alteration.

Finally, remove the null-tolerance from the application.

The general principle is expand-migrate-contract: make the schema accept both shapes, move the data, then narrow it. It is more steps and it is the only version that is safe to abandon halfway.

Curated: · Written: · Reviewed:

Q&A-53A batch job holds locks that block user traffic every night. How do you fix it?

What a strong candidate says first (60–120 s)

The lock isn't the problem; its scope and hold time are. Moving the job an hour later just relocates the collision. I'd identify which lock is blocking: a long UPDATE transaction holding row locks, an ALTER TABLE taking an exclusive table lock, or a long read pinning a snapshot, then shrink each one.

For the common case, a batch UPDATE over millions of rows: chunk it by primary key, commit between chunks, and cap how long any single statement waits for a lock. That turns one multi-minute transaction into thousands of sub-100 ms ones, so user traffic interleaves instead of queuing.

Which lock, which fix

SymptomLockFix
Writers blocked on specific rowsRow locks held for the whole transactionChunk by PK, commit per chunk
All writers blockedALTER TABLE exclusive lockCREATE INDEX CONCURRENTLY (PostgreSQL); ALGORITHM=INPLACE or instant DDL (MySQL 8.0.12+); gh-ost / pt-online-schema-change
Table bloat, vacuum laggingLong read holding a snapshotMove analytics to a replica
Deadlocks, not waitsDifferent acquisition orderSame order in both paths (PK order)

Chunking, concretely

PostgreSQL 15:

SET lock_timeout = '2s';   -- default is 0 (wait forever)

UPDATE orders
   SET status = 'archived'
 WHERE id IN (
   SELECT id FROM orders
    WHERE status = 'stale'
    ORDER BY id
    LIMIT 5000
 );
-- COMMIT, sleep 50 ms, repeat until 0 rows affected

MySQL 8.0:

SET SESSION innodb_lock_wait_timeout = 5;  -- default 50 s

UPDATE orders SET status = 'archived'
 WHERE status = 'stale'
 ORDER BY id
 LIMIT 5000;

Chunk size: 1,000–10,000 rows. Rationale: 5,000 single-row updates is on the order of 50 ms of work, so no row lock is held longer than roughly 50–80 ms. 5M rows / 5,000 = 1,000 chunks; at 50 ms work plus 50 ms sleep, the job finishes in about 100 s wall-clock instead of holding locks for the full multi-minute transaction. The sleep matters: without it, 1,000 back-to-back transactions still saturate CPU and starve user queries.

Chunk by id > :last_id ORDER BY id LIMIT n, not OFFSET, which re-scans from the start each time: O(n²) over the run.

Lock timeout as a yield valve

lock_timeout = '2s' (PostgreSQL) or innodb_lock_wait_timeout = 5 (MySQL) makes the batch abort and retry rather than block user traffic. The job takes longer; users don't notice. Set it on the batch session only, never globally.

Ordering

If the batch updates rows in a different order from the application, you get deadlocks rather than waits. Both paths should acquire in primary key order.

Analytics

Read-heavy work belongs on a replica or an analytics store. That removes contention rather than managing it.

Monitoring

Alert on p99 lock wait > 500 ms, or any session waiting > 2 s. PostgreSQL: pg_locks joined to pg_stat_activity where wait_event_type = 'Lock'. MySQL 8.0: performance_schema.data_lock_waits. The reason this was found by user complaints is that nobody was watching that signal.

Curated: · Written: · Reviewed:

Q&A-25How would you approach migrating a monolith's database to a separate service without downtime?

As a sequence of reversible steps, each safe to stop at, with CDC as the bridge and the old tables kept until a full business cycle has passed.

Assume Postgres 15, a few hundred GB across the tables in scope, and a hard requirement that no committed write is lost and no read goes stale for more than a second.

0. One owner in code. The tables get a single owning module; nothing else queries them. Enforce with a lint rule or a CODEOWNERS entry, not a wiki page. This is code archaeology and it is the long pole: budget more time here than for the infrastructure.

1. Interface. The module sits behind an interface the rest of the monolith calls. Still one process, one database, but the call graph now matches the boundary you intend.

2. Stream changes out. Run Debezium against the monolith's Postgres WAL (wal_level = logical) or MySQL binlog (binlog_row_image = FULL), publishing to Kafka; the new service consumes and applies to its own store. The monolith still writes only the old tables: this is CDC, not application-level dual write, and the distinction matters: dual write fails partially (old write commits, new write doesn't) and needs a repair path, whereas CDC has one source of truth and only lag to manage. A reconciliation job compares per-PK-range row counts and checksums every 5 minutes and alerts above a 0.1% mismatch in any range. This is where you find out what the boundary actually is.

3. Backfill. Chunk by primary-key range, 10k rows per batch, throttled so replication lag stays under 5s. Worked example: a 400 GB table at 10k rows/s is roughly 11 hours; run it during low traffic and pause on lag.

4. Shift reads. Shadow first: send each read to both paths, return the old answer, log divergence. Then move traffic 1% → 5% → 25% → 50% → 100%, gated on p99 and error rate, with a per-tenant routing table so the largest tenants can be moved individually. Rollback is setting the percentage to 0.

5. Stop writing the old tables. Keep them read-only for one full business cycle, 30 days for a monthly-close system, then drop. Keep a logical dump for 90 days.

PhaseWritesReadsRollback
0–1monolithmonolithrevert code
2monolithmonolithstop connector
3monolithmonolithtruncate new store
4monolithsplit by %set % to 0
5new servicenew serviceflip flag; old tables still present
6new servicenew servicerestore from dump

What people underestimate. Foreign keys crossing the boundary become application-level checks or are dropped. Transactions that spanned the boundary become sagas with compensating actions, or are redesigned so they don't span it. Publish events through a transactional outbox: write an outbox row in the same transaction as the state change, relay it to Kafka, and have consumers dedupe on (aggregate_id, version): otherwise you get lost or duplicated events at exactly the cutover. If neither sagas nor redesign is acceptable for a given pair of tables, the boundary is in the wrong place; better to learn that at step 2 than after the split.

Failure modes to instrument. A Postgres logical replication slot retains WAL while the Debezium consumer is down: alert on retained WAL bytes and slot lag, or the primary's disk fills. A lagging new store during read shift produces stale reads: gate the shift on replication lag under 1s. Saga compensation loops need a retry cap and a dead-letter queue.

Curated: · Written: · Reviewed:

Q&A-71How do you approach a legacy system that must keep running but cannot be modified safely?

The 90-second answer. Don't rewrite it, don't freeze it: put a seam in front of it and move behaviour out one capability at a time. Concretely: stand up a facade (nginx, Envoy, or the API gateway you already run) so every caller addresses the facade rather than the legacy host. Then pick one capability, route a copy of its traffic to a new implementation, compare outputs in production, and only then flip the read path. Reads before writes, because reads don't need a consistency story. Where writes must move, dual-write and reconcile until drift is zero for a sustained window. Characterisation tests protect the parts you must touch. Two constraints hold it together: no new features land in the legacy system, and every extraction step ships value on its own.

Choosing the first capability. Rank by read/write and blast radius:

CapabilityAccessShare of trafficWhy this position
Product lookupread~40% of requestspure read, cacheable, trivially diffable
Price calculationread~15%deterministic given inputs, high learning
Order creationwrite~5%needs dual-write plus reconciliation
Billingwrite~2%money; do it last

(Figures illustrative for a mid-size commerce system.)

The seam, concretely. nginx's mirror module (1.13.4+) copies requests to a second upstream without affecting the client:

location /api/product/ {
    mirror /mirror;
    proxy_pass http://legacy;
}
location = /mirror {
    internal;
    proxy_pass http://new-product-service$request_uri;
}

Mirroring all traffic doubles load on the new service; sample with split_clients when that matters. Envoy does the same job with request_mirror_policies on the route. The mirrored response is discarded: you compare it out of band, never in the request path.

The comparison harness. Log both responses keyed by request ID, then diff. Start on a 1% sample, alert when the mismatch rate crosses your threshold (0.1% is a reasonable starting point), and read the first hundred diffs by hand: they tell you more than the dashboard does.

Writes: dual-write and reconcile. Legacy commits first, new path second; if the new path fails you have divergence. Don't paper over it: run a reconciler. Illustrative: 1.2M order rows/day, reconciler every 15 minutes over the trailing 2 hours, comparing row counts and a per-shard checksum. Cut reads to the new path when drift is zero for 7 consecutive days and the reconciler has caught at least one real divergence you then fixed.

Characterisation tests. Feathers' technique: pin current behaviour, including behaviour that is arguably wrong, as tests before you touch anything. Approval or golden-master tests over recorded inputs and outputs are the cheap version. They protect against unintended change, not correctness: a test asserting the buggy rounding is doing its job.

Failure modes. The facade accretes business logic and becomes the second legacy system; keep it to routing, auth and observability. Dual-write divergence during partial failure, detected by the reconciler rather than by user complaints. Characterisation tests rot the moment someone ships a feature into the legacy system, which is why the freeze is a rule, not a preference.

Curated: · Written: · Reviewed:

Q&A-29How do you decide the right granularity for a service boundary?

Follow the data and the rate of change, not the org chart or an aesthetic sense of size.

The strongest signal is transactional integrity: things that must change atomically belong together. If splitting two entities forces a distributed transaction or a saga with compensating actions, the split is probably wrong: you have taken a database guarantee and rebuilt it in application code, badly.

The second signal is the deploy coupling. If two services are always released together, they are one service with extra network hops, latency, and failure modes. Count how often a change touches both.

The third is different scaling or availability requirements. A component that needs ten times the capacity of its neighbours, or one that must stay up while the rest is down, has a genuine reason to be separate.

Team ownership matters, but as a constraint rather than the criterion: a boundary no single team can own will erode.

The practical advice is to start coarser than the target. Splitting a module that has clean internal boundaries is straightforward; merging two services whose data has diverged is not. Modules within one deployable, with enforced import rules, give most of the design benefit and none of the distributed-systems cost, and they make the eventual split cheap because the seams are already there.

The failure I would call out is splitting by technical layer: a service for the API, one for business logic, one for data access. Every feature then touches all three.

Curated: · Written: · Reviewed:

Q&A-40How do you design an API so it can evolve without breaking existing clients?

Decide first that additive changes are free and everything else is a version. Adding an optional field or a new endpoint breaks nobody, provided clients ignore unknown fields, which is a contract you should state explicitly in the documentation, because clients that validate strictly will break on your first addition.

Breaking changes are: removing or renaming a field, changing a type, tightening validation, changing the meaning of a value, and adding a required request field. Each of these needs a new version.

For versioning, URL path versioning is the most operationally honest choice (it is visible in logs, cacheable, and trivially routable) even though header-based versioning is more theoretically pure. What matters more than the mechanism is that you can measure usage per version, because you cannot retire a version you cannot see.

Deprecation needs a process, not an announcement: a stated support window, a deprecation header on responses, usage metrics per client, and direct contact with the remaining callers before removal. Most failed API migrations fail because nobody knew who was still calling.

Two design choices that buy the most future room: return objects rather than bare values, so fields can be added; and avoid exposing internal enumerations directly, since a new internal state should not force a client change.

Idempotency, consistent error shapes, and pagination from day one also count as evolvability: retrofitting pagination onto an endpoint that returned an array is a breaking change.

Curated: · Written: · Reviewed:

Q&A-93How would you design an API that returns a large result set?

With cursor-based pagination, a bounded default page size, and no ability for a caller to request everything.

Cursor pagination (an opaque token encoding the position, typically the last seen sort key) is preferable to offset pagination for two reasons. It is stable under concurrent inserts and deletes, so a caller paging through results does not see duplicates or skip items when the underlying data changes. And it performs consistently: OFFSET 100000 requires the database to scan and discard 100,000 rows, so deep pages get progressively slower, whereas a cursor is an indexed seek.

The sort must be deterministic and must include a tiebreaker, usually the primary key, or the cursor is ambiguous for rows sharing a sort value.

Set a default page size and a maximum, and enforce the maximum server-side. A caller requesting a million items is either a mistake or an attack, and either way the server should not attempt it.

Return the next cursor in the response, and prefer not to return a total count. Counting the full result set is often as expensive as the query itself, and callers rarely need it, where they do, offer it as an explicit, separately-priced option.

For genuinely large exports, pagination is the wrong shape entirely. Offer an asynchronous job: the caller requests an export, receives a job ID, polls or receives a webhook, and downloads a file from object storage. That keeps a multi-gigabyte result out of the request path where it would tie up connections and time out.

Curated: · Written: · Reviewed:

Q&A-100What is the difference between an API gateway and a load balancer, and when do you need both?

Short answer. A load balancer picks a backend; an API gateway decides whether the request is allowed and what it means. The LB's input is a connection or an HTTP request line. The gateway's input is the caller's identity, the API key, the body. You need both when the public contract changes on a different schedule from the backend fleet: auth, quotas and versioning owned by a platform team, capacity and health owned by service teams. You need one when they don't.

ConcernL4 LB (AWS NLB, HAProxy mode tcp)L7 LB (AWS ALB, NGINX, Envoy)API gateway (Kong, Apigee, AWS API Gateway)
Routing key5-tuplehost, path, headerpath + method + consumer/API key
TLSpass-through or terminateterminateterminate, mTLS to consumers
Authnoneoptional (ALB OIDC)required, per-consumer
Rate limitnoneconnection-levelper-consumer quota
Bodyopaqueopaqueschema validation, transformation
Metricsconnections, bytesRPS, latency, 5xxper-consumer usage, quota burn

What the extra hop costs. Illustrative, not measured: a mobile client calls GET /v2/orders/42. ALB terminates TLS and routes by host+path to the gateway target group, ~1 ms p50. The gateway verifies the JWT (RS256 against a cached JWKS, ~0.2 ms), checks the quota in Redis (one in-AZ round trip, ~0.5 ms), validates the request schema (~0.2 ms). It forwards to the service through an internal NLB, ~0.5 ms. Service time is 20 ms. The two hops add ~2–3 ms p50, roughly 10–15%: invisible next to a 20 ms service, decisive if the service is a 1 ms cache read. Measure it before arguing about it.

When you need both. External consumers, more than one service, per-consumer policy. The gateway owns the public contract; each service's own LB owns its fleet. Note that managed gateways ship their own HA front end, AWS API Gateway and Apigee are already load balanced, so "both" often means one gateway product plus an internal LB, not two products you operate.

When you don't. One internal consumer, no per-consumer policy, no public exposure. A gateway there is a hop and a component with nothing to enforce.

Failure modes worth naming.

  • A gateway on every path is a single point of failure. If its rate-limit store (Redis) is unreachable, you choose fail-open (quota leaks) or fail-closed (all traffic stops). Kong's rate-limiting plugins expose this as fault_tolerant; decide it deliberately rather than inheriting the default.
  • ALB's idle timeout defaults to 60 s. If a backend's keepalive timeout is shorter, the backend closes first and you get sporadic 502s under low traffic. Keep backend keepalive above the LB's idle timeout.
  • Envoy's default circuit breakers (max_connections 1024, max_pending_requests 1024, max_requests 1024, max_retries 3) shed load with a 503 UO before your service is actually saturated. Raise them per cluster or you will debug a capacity problem that isn't one.
  • AWS API Gateway's default account throttle is 10,000 rps with a 5,000 burst per region. That is your ceiling until you raise it, whatever the backends can take.

The design caution. A gateway is a policy enforcement point, not a service. Routing rules that encode domain decisions, "orders over $10k go to the manual review queue", belong in the service, where they can be unit tested. Keep the gateway to cross-cutting policy: auth, quota, schema, contract version.

Curated: · Written: · Reviewed:

Q&A-58How do you design a rate limiter for a public API, and where do you put it?

The 90-second answer

Put it at the edge (in the gateway or CDN, before the request reaches anything that costs money) and back it with a shared counter. Per-instance limits multiply: 20 pods each allowing 100 req/min gives a round-robining client 2000 req/min.

Use a token bucket, not a fixed window. Capacity B is the burst you tolerate; refill rate r is the sustained rate. B=10, r=2/s is 120 req/min sustained with a 10-request burst: five seconds of steady-state budget. A fixed window lets a client send two full windows across a boundary; a sliding-window log is exact but stores a timestamp per request.

Limit per API key for authenticated traffic, per IP for unauthenticated. Return 429 with Retry-After plus RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset (the IETF ratelimit-headers draft; X-RateLimit-* is the older de facto convention) so clients pace themselves instead of polling.

The counter

One Redis key per bucket, one Lua script, one round trip:

-- KEYS[1] bucket key; ARGV[1] capacity B; ARGV[2] refill r tokens/sec; ARGV[3] tokens wanted
local B, r, want = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])
local t = redis.call('TIME')
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)

local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens, ts = tonumber(b[1]), tonumber(b[2])
if tokens == nil then tokens, ts = B, now end

tokens = math.min(B, tokens + math.max(0, now - ts) / 1000 * r)

local allowed, retry_ms = 0, 0
if tokens >= want then
  tokens = tokens - want
  allowed = 1
else
  retry_ms = math.ceil((want - tokens) / r * 1000)
end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(B / r * 1000) + 1000)
return {allowed, math.floor(tokens), retry_ms}

TIME inside the script means every instance reads Redis's clock, so there is no skew between pods; that is safe because Redis 5+ replicates script effects rather than the script body. Multi-field HSET needs Redis 4.0+. The TTL is B/r seconds plus a second of slack: after that long idle the bucket is full anyway, so the key is disposable.

Trace

B=10, r=2/s, fresh bucket, client fires 12 requests at t=0.

reqt (s)tokens beforedecisiontokens afterRetry-After
10.00010.0allow9.0—
100.0001.0allow0.0—
110.0000.0deny0.01
120.0000.0deny0.01
130.5001.0allow0.0—

Request 11 computes a 500 ms deficit, so Retry-After: 1 (seconds, rounded up). At t=0.5 s the bucket has refilled exactly one token.

Where, precisely

Two tiers. Coarse per-IP at the CDN edge, say 1000 req/min per IP, absorbs volumetric floods before they reach your origin. Per-key at the API gateway or service middleware, where the key is already parsed and you can vary the limit by plan or endpoint cost. Login and password-reset get their own stricter buckets: the risk there is credential stuffing, not capacity.

If a Redis round trip per request is too much (typically ~0.5 ms p50, low single-digit ms p99 same-AZ), give each instance a local bucket at r/N and reconcile with Redis once a second. You trade exactness for one Redis call per second per instance: a client pinned to a single instance gets only 1/N of the burst.

Failure modes

  • Redis down. Decide deliberately: fail open (availability) with a local fallback limiter, or fail closed (protection). Auth endpoints should fail closed.
  • Hot key. One key at 10k req/s makes one shard hot. Shard the counter across N keys and sum, or fall back to local buckets.
  • Retry storm. Every denied client retries at the same instant. Jitter Retry-After.
  • Detection: 429 rate per key, Redis p99, and the ratio of denied requests followed by an immediate retry.

Rate limiting bounds volume, not intent. It is not abuse prevention on its own.

Curated: · Written: · Reviewed:

Q&A-73How do you prevent one tenant from affecting another in a multi-tenant system?

Isolate along the dimensions where one tenant's behavior can consume a shared resource: request capacity, data, storage, and the database's working set.

Request capacity: per-tenant rate limits and concurrency limits, enforced at the edge. Without them, one tenant's runaway integration consumes the whole fleet's capacity, and the incident looks like a general outage.

Data: tenant identity must be part of every query, ideally enforced structurally rather than by convention. Row-level security in the database, or a data access layer where the tenant filter cannot be omitted, is much stronger than reviewing every query. Cross-tenant data leakage is the failure with the worst consequences and it usually comes from one forgotten WHERE clause or one cache key missing the tenant.

Work: background jobs should be scheduled fairly rather than first-in-first-out, or one tenant's bulk import starves everyone else's jobs for hours. Per-tenant queues with round-robin consumption is the usual answer.

Database resources: a single tenant with a hundred times the data skews query plans, fills the buffer cache, and makes shared indexes less effective. Beyond a threshold, very large tenants are better served on dedicated infrastructure, which is why most mature multi-tenant products end up with a tiered model: pooled for the many, silo for the few.

And instrument per tenant. Aggregate metrics conceal exactly the situation you are trying to detect, so error rate and latency should be sliceable by tenant before an incident, not after.

Curated: · Written: · Reviewed:

Q&A-90How do you approach securing service-to-service communication inside a private network?

In 90 seconds: Assume Kubernetes, dozens of services, no mesh yet. Give every workload a cryptographic identity (a SPIFFE ID), enforce mTLS with certificates that expire in hours rather than years, authorize every call against that identity with default-deny, and carry the end-user's identity as a separate token. NetworkPolicy is the second layer, not the first. The network boundary is not the authorization boundary: one compromised pod otherwise reaches everything.

Identity. SPIFFE gives a URI-shaped identity: spiffe://prod.example.com/ns/payments/sa/checkout. SPIRE issues X.509 SVIDs with a default TTL of 1 hour and rotates at half-life, so a stolen key is useless within about 30 minutes. Istio's istiod issues workload certificates with a default 24-hour lifetime in trust domain cluster.local; Linkerd's identity controller also defaults to 24 hours. Shorter TTLs cost more CA traffic and more rotation failures; 1h is a reasonable floor for most fleets.

mTLS. TLS 1.3 is a 1-RTT handshake (0-RTT on resumption, RFC 8446), so per-connection cost is small; the real cost is the CA and the rotation path. Envoy's SDS pushes rotated certificates into the proxy without a restart. In Istio, PeerAuthentication with mtls.mode: STRICT rejects plaintext, which closes the "pod talks straight to the port, bypassing the sidecar" hole.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata: { name: default, namespace: payments }
spec:
  mtls: { mode: STRICT }

Authorization. Identity is authentication. Map it to policy: an Istio AuthorizationPolicy is default-deny for any workload it selects. Once a policy exists in the namespace, unmatched requests are rejected. For richer rules (method, path, JWT claims) put OPA behind Envoy ext_authz, or use Cilium's identity-aware policy. Default-deny means a new service cannot call everything by accident.

User context. Keep service identity and user identity in separate tokens. Use RFC 8693 token exchange to mint a downstream token whose sub is the user and whose act claim names the calling service. Conflating them is how an internal endpoint ends up doing privileged work for any caller holding a service certificate.

Network policy. Kubernetes NetworkPolicy is default-allow; write default-deny ingress and egress per namespace and open only what is needed. Cilium and Calico enforce it, and Cilium can bind policy to SPIFFE identities. This catches a different failure than mTLS: a misconfigured or bypassed proxy.

Failure modes.

  • Rotation outage: if the SPIRE agent cannot reach the server, SVIDs expire and every call fails mTLS. Alert on time-to-expiry, not just on rotation errors.
  • Clock skew: certificates are validated against wall clocks, so a node minutes off fails validation. Run NTP and alert on skew.
  • Trust-domain mismatch: a service in cluster.local calling one in prod.example.com fails unless the domains are federated. Federation is explicit in SPIRE.
  • Policy drift: an allow added for a debug session and never removed. Review policies as code.

Choosing. Istio: most features, 24h certs, heaviest. Linkerd: simpler, 24h certs, fewer knobs. Cilium: identity-aware policy without a sidecar, with SPIFFE integration. SPIRE standalone: identity for non-Kubernetes workloads (VMs, bare metal) as well.

Curated: · Written: · Reviewed:

Q&A-27Describe how you would design the network for a workload that must not have any route to the internet.

What a strong candidate says first. Don't block egress: delete it. In AWS that means: no internet gateway attached to the VPC, no NAT gateway, no egress-only internet gateway, and no route in any route table with destination 0.0.0.0/0 or ::/0. A route that doesn't exist can't be created by a permissive security group, a wide-open NACL, or an app that ignores its proxy setting. The security group is a filter; the route table is the boundary.

Then rebuild the paths the workload genuinely needs, because an isolated workload still pulls images, reads secrets, writes logs and reaches object storage.

  • S3 and DynamoDB: gateway endpoints. No hourly charge, installed as a route to the service's managed prefix list (pl-…) in the subnet's route table.
  • Everything else: interface endpoints (PrivateLink). ECR needs ecr.api and ecr.dkr plus the S3 gateway endpoint for layer blobs; Session Manager needs ssm, ssmmessages, ec2messages; add secretsmanager and logs. These are ENIs with private IPs in your subnets, so security groups and endpoint policies apply to them. With private DNS enabled the public hostname resolves to the endpoint IP and SDKs work unchanged.
  • Metadata: 169.254.169.254 is link-local, never routed off the host, so it keeps working. Require IMDSv2: PUT /latest/api/token with X-aws-ec2-metadata-token-ttl-seconds returns a token that every later call must carry. Without it, one SSRF bug reads instance credentials with a plain GET.

Cost, list price us-east-1: interface endpoints are about $0.01/hour per AZ (~$7.30/month) plus $0.01/GB processed. The NAT gateway you removed was $0.045/hour ($33/month) plus $0.045/GB. Gateway endpoints are free.

Keep it that way. SCPs denying ec2:CreateInternetGateway, ec2:AttachInternetGateway, ec2:CreateNatGateway and ec2:CreateRoute with a gateway target. Caveat: SCPs don't apply to the management account, so the workload shouldn't live in the org root. Endpoint policies and bucket policies do the rest: "aws:SourceVpce": "vpce-…" in a bucket policy means only traffic arriving through that endpoint can read the bucket, which closes the exfil path where the workload writes to an attacker-controlled bucket in another account.

Failure modes.

  • Peering and Transit Gateway are transitive: a peered VPC, or a TGW attachment with a default route, reopens egress even though your VPC has no IGW. Network Access Analyzer finds those paths across the account.
  • IPv6: a subnet with an IPv6 CIDR is globally routable by default, and an egress-only IGW permits outbound v6. For zero egress, attach neither and keep ::/0 out of every route table.
  • DNS: Route 53 Resolver DNS Firewall in block mode with an allowlist. DNS is the classic low-and-slow exfil channel once everything else is shut.
  • Software distribution: mirror the package repo inside, or bake images in a connected build account and promote them inward. ECR pull-through cache is real but needs egress, so it belongs on the build side.
  • Operator access: Session Manager, not an inbound SSH port. Workload security groups carry no inbound rules at all.

Verify, don't assume. A scheduled check that fails if any route table has a default route:

aws ec2 describe-route-tables --output json | jq -r '
  .RouteTables[] | .RouteTableId as $rt
  | .Routes[] | select(.DestinationCidrBlock=="0.0.0.0/0" or .DestinationIpv6CidrBlock=="::/0")
  | "\($rt) \(.GatewayId // .NatGatewayId // .TransitGatewayId // "unknown")"'

Empty output is the pass condition; run it per region and per account. Pair it with VPC Flow Logs and an alert on any accepted flow whose destination is outside the VPC CIDR and the endpoint ENI addresses. Isolation drifts the moment someone attaches a TGW for a "temporary" migration.

Curated: · Written: · Reviewed:

Q&A-76What does defense in depth mean concretely for a public-facing web application in the cloud?

It means no single control failing results in compromise, and it is best described as the layers a request passes and the layers an attacker would have to defeat after that.

At the edge: DDoS protection and a web application firewall, TLS terminated with modern ciphers, and rate limiting. This layer stops volume and known patterns, not a determined attacker.

At the network: the application in private subnets with no inbound route from the internet, security groups referencing each other rather than CIDR ranges, and egress restricted so a compromised instance cannot freely reach the internet. Egress filtering is the layer most often skipped and it is what limits exfiltration and command-and-control.

At the identity layer: short-lived workload identities, narrowly scoped policies, MFA for humans, and no long-lived keys. This is what limits what a compromised process can do.

At the application: input validation, parameterised queries, output encoding, object-level authorization checks, and dependencies scanned and patched.

At the data: encryption at rest with keys the application cannot export, encryption in transit internally as well as externally, and separate credentials per service so one leak does not reach everything.

And crucially, detection across all of them: audit logs, flow logs, configuration change alerts, and anomaly detection, because the assumption behind defense in depth is that some layer will fail, and the value of the remaining layers depends on someone noticing.

The test is to name a layer and ask what still protects you if it fails entirely.

Curated: · Written: · Reviewed:

Q&A-99If you could enforce only three technical controls across an entire cloud estate, which would you choose and why?

Assumption: one AWS organization, many accounts. The pattern maps to GCP and Azure with different names.

1. No long-lived credentials: OIDC federation into STS, IMDSv2 on every instance, MFA for humans.

Every CI job gets a GitHub-issued OIDC token (minutes of life) and exchanges it with AssumeRoleWithWebIdentity for an STS session: one hour by default, twelve at most. The role's trust policy pins the subject claim, e.g. repo:acme/api:ref:refs/heads/main, so a token minted for a fork PR cannot assume it. Instances get credentials from IMDSv2 only: HttpTokens: required, token TTL 21600 s, hop limit 1, which closes the SSRF-to-IMDS credential theft path. Humans go through IAM Identity Center with MFA; sessions default to one hour, twelve maximum.

Why first: a static access key in a repo secret or a .env is valid until someone rotates it, and nobody rotates it. Federation makes the credential's lifetime the job's lifetime. It also forces a coherent identity model, because OIDC trust policies do not work without one.

2. All change through a reviewed pipeline; no standing human write access to production.

Terraform in a repo, plan posted on the pull request, apply on merge by a role the pipeline assumes via OIDC. Humans hold read-only plus a break-glass role that is time-boxed, MFA-gated, and pages the security channel when assumed. SCPs at the org root deny iam:CreateAccessKey and deny writes outside the pipeline role.

Three properties from one control: peer review before the change, an author and an approver in the record, and the environment matching its code. Drift detection is terraform plan -detailed-exitcode on a schedule: exit code 2 means someone changed something by hand.

3. Tamper-evident audit logging, in an account the logged accounts cannot write to.

An organization trail in CloudTrail covering every account, delivering into a dedicated log-archive account, with log file validation on (SHA-256 digest files) and S3 Object Lock in compliance mode on the destination bucket. Management events are on by default; S3 data events and Lambda Invoke are not, and are usually the ones you need: enable them for the buckets and functions that matter.

The number that catches people: the CloudTrail console's Event history is 90 days. Anything older exists only in the trail's S3 bucket, so retention is whatever you set on that bucket, not what the console shows.

ControlStopsDoes not stop
OIDC + IMDSv2 + MFAStolen static keys, SSRF credential theft, replay of a leaked token after the session endsA compromised workload using its own valid identity
Reviewed pipelineUnreviewed change, hand-edited prod, most audit findingsA malicious change that passes review
Org trail + Object LockEvidence destruction, "we can't tell what happened"The incident itself

Why these three: the first removes the most common entry path, the second removes the most common self-inflicted damage and produces evidence as a byproduct, the third reconstructs whatever the first two missed. Encryption and network segmentation protect narrower classes and are usually already handled by the platform; they are not what decides whether an incident is survivable.

Curated: · Written: · Reviewed:

Q&A-61How would you detect and respond to a compromised set of credentials in a cloud account?

First 60–120 seconds

Clarify two things: which cloud, and what kind of credential. A long-lived access key, or a session STS issued from a role. That decides the containment step. Assume AWS unless told otherwise.

Then: assume compromise, contain the session, preserve the log range, scope the blast radius, rotate everything reachable, and only then remove the credential class.

Detection

Most compromises surface as behavior, not as an alert on the credential itself.

SignalWhere it shows up
Calls from a new ASN or country for that identityGuardDuty UnauthorizedAccess:IAMUser/*; CloudTrail sourceIPAddress
Enumeration bursts: ListBuckets, DescribeInstances, ListRoles, GetCallerIdentityCloudTrail management events
Logging tampering: StopLogging, DeleteTrail, UpdateTrailGuardDuty Stealth:IAMUser/CloudTrailLoggingDisabled
New persistence: CreateUser, CreateAccessKey, CreateLoginProfile, AttachUserPolicy, UpdateAssumeRolePolicyCloudTrail
Egress: GetObject volume, or CreateSnapshot followed by ModifySnapshotAttribute to publicGuardDuty S3 protection Exfiltration:S3/AnomalousBehavior; CloudTrail data events

GuardDuty's *:IAMUser/AnomalousBehavior family (CredentialAccess, Discovery, Persistence, Exfiltration) is the baseline: it's trained on your account's own history, so it catches the identity-specific anomalies you'd otherwise hand-roll.

Two gaps worth knowing. CloudTrail logs management events by default, but S3 GetObject and Lambda Invoke are data events: off unless you enable them, and billed per 100,000 events. And GuardDuty's foundational sources are CloudTrail management events, VPC flow logs and DNS: it sees nothing inside a service that emits none of those. So "unusual egress" is only visible if you turned on data events or are watching flow logs.

Response, in order

  1. Contain the session, not just the key. Deleting an access key does not invalidate a session token already issued from it: STS signed it and it stays valid until expiry. AssumeRole defaults to 3600 s and caps at 43200 s; GetSessionToken defaults to 43200 s and reaches 129600 s with MFA. A key deleted at minute 5 can leave a live session for up to 12 hours. Attach the AWSRevokeOlderSessions inline policy (denies when aws:TokenIssueTime predates the attach) or an explicit deny on the identity.
  2. Preserve before remediating. Snapshot the affected EBS volumes, export the CloudTrail range to S3. The console event history only reaches back 90 days; without a trail, that is your entire window.
  3. Scope it. Every action the identity took, every resource it touched, every credential it could have read. Assume anything readable was read: SSM Parameter Store values, secrets in instance user data, config files on instances it could reach.
  4. Rotate everything in reach, not just the compromised key.

Trade-offs

Revoke-all vs scoped deny. Revoking every session for the role kills legitimate automation mid-flight. If the identity backs production automation, prefer a scoped deny (named actions, named source IPs) plus rotation; if it's a human identity, revoke everything. Criterion: does something else depend on this identity staying live?

Federation vs long-lived keys. Federation removes the standing secret but adds an IdP dependency, IdP down means nothing can assume the role, plus an STS call per assumption. Scoped long-lived keys on a 90-day rotation are still defensible where the consumer can't present an OIDC token: third-party SaaS, on-prem appliances, break-glass. Criterion: can the consumer present a verifiable identity token, and can you tolerate the IdP dependency?

Failure modes

  • Declaring victory after deleting the key while the session runs to expiry.
  • Rotating the key but not the secrets the identity could read.
  • Missing the second credential the attacker created: alert on CreateAccessKey and CreateUser as a standing rule.

Curated: · Written: · Reviewed:

Q&A-34Your object storage bucket was found to be publicly readable. Walk me through your response and the follow-up.

Assume AWS S3. The core answer: contain without destroying evidence, scope what was actually readable, start the notification clock, then fix the system that allowed it.

Containment. Turn on all four account-level Block Public Access settings. Account-level settings apply to every bucket in the account, including ones you have not audited, and override less restrictive bucket-level settings. Crucially, RestrictPublicBuckets and IgnorePublicAcls make an existing public bucket policy and public ACL grants ineffective without deleting them, so the policy document and the ACL stay intact as evidence.

aws s3control put-public-access-block --account-id 111122223333 \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

aws s3api get-bucket-policy-status --bucket my-bucket
# {"PolicyStatus": {"IsPublic": false}}

Then snapshot get-bucket-policy, get-bucket-acl, get-bucket-policy-status and the relevant CloudTrail events into a case folder before anyone "cleans up."

Scope: two different questions. What could be read is the object inventory plus the policy as it stood. What was read needs object-level logs: CloudTrail S3 data events are off by default and billed per event, and S3 server access logs are best-effort with delivery delays. If neither was enabled, you cannot enumerate reads: say that plainly rather than implying a clean bill of health.

Exposure window runs from the configuration change, not from discovery. CloudTrail management events (PutBucketPolicy, PutBucketAcl, DeletePublicAccessBlock) are logged by default; AWS Config's get-resource-config-history gives the timeline.

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutBucketPolicy \
  --start-time 2024-03-01T00:00:00Z

Illustrative timeline:

Time (UTC)Event
2024-03-04 14:02PutBucketPolicy by deploy-bot grants s3:GetObject to Principal: "*"
2024-03-11 09:15GuardDuty finding Policy:S3/BucketPublicAccessGranted
2024-03-11 09:22account-level BPA enabled. Public access cut
2024-03-11 09:40policy, ACL and CloudTrail events snapshotted

Exposure window: 6 days 19 hours 13 minutes, not the 7 minutes between detection and containment.

Notification. GDPR Art. 33 gives 72 hours from awareness to notify the supervisory authority; Art. 34 requires notifying data subjects without undue delay where there is high risk. Awareness is when you have reasonable certainty a breach occurred, so the clock starts at 09:15, not when scoping finishes. Run a Macie sensitive data discovery job to classify what was in the bucket: that is the input legal needs.

Follow-up is structural.

ControlLayerStopsLatency
Account-level BPA (all four)S3 accountpublic ACLs/policies taking effectimmediate
SCP denying s3:PutBucketAcl, s3:PutBucketPolicy, s3:DeletePublicAccessBlockOrganizationsthe change itself, every account in the OUat API call
Object Ownership = BucketOwnerEnforcedbucketACL-based grants entirelyimmediate
Config rules s3-bucket-public-read-prohibited, s3-account-level-public-access-blocks-periodicAWS Configdrift from console/CLI editson config change
GuardDuty S3 ProtectionGuardDutyPolicy:S3/BucketPublicAccessGrantedminutes
CloudTrail S3 data eventsCloudTrailobject-level read visibilitynear-real-time, per-event cost
Checkov/tfsec in CI + a module defaulting to private, versioned, ACLs disabledIaCthe change at PR reviewat review

New buckets already default to ACLs disabled (April 2023) and SSE-S3 encryption (January 2023), so the module's real job is versioning, BPA and refusing public policy documents.

The postmortem question is not who made the bucket public: it is why the account allowed it silently for six days.

Curated: · Written: · Reviewed:

Q&A-84How would you architect a system that must prove which user performed every action, for a regulator?

The 90-second answer. Make the audit record a byproduct of the write, not a call the application remembers to make, and make it verifiable after the fact. Identity comes from a token the user cannot forge: an OIDC sub claim, or an RFC 8693 token-exchange act claim when a service acts on their behalf. The record is written in the same transaction as the change. The store is append-only with a SHA-256 hash chain anchored outside the system. Retention is set to the longest regime you are subject to. And the retrieval path is load-tested as hard as the write path, because a trail that takes a week to search fails the requirement in practice.

Identity. No shared accounts, no service acting for a user without carrying the original identity, no human path into the database on a shared credential. Workloads get SPIFFE IDs or mTLS client certs; humans get short-lived OIDC tokens. Where an operator acts on a user's behalf, the record carries both: sub = the user, act.sub = the operator.

Capture. Two mechanisms catch changes regardless of which code path made them: a database trigger writing to an audit table inside the same transaction, and CDC reading the WAL. In Postgres, an AFTER trigger means a rolled-back transaction leaves no phantom audit row. But a superuser can run ALTER TABLE ... DISABLE TRIGGER, so pair it with logical decoding (Debezium on a Postgres replication slot, or wal2json) which reads the WAL and cannot be bypassed by the application. That slot is also the failure mode: an inactive slot retains WAL indefinitely and will fill the disk. Alert on pg_replication_slots retained bytes.

Tamper-evidence. Append-only storage plus a hash chain:

# Python 3.12
import hashlib, json

def record_hash(prev_hash: str, event: dict) -> str:
    body = json.dumps(event, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(bytes.fromhex(prev_hash) + body).hexdigest()

prev_hash is 64 hex chars, so bytes.fromhex yields 32 bytes; the genesis record uses "00" * 32. Verification recomputes the chain and fails at the first mismatch, which localises a deletion to the gap between two records. Chaining alone cannot detect truncation of the tail, so anchor the head hash daily: an RFC 3161 timestamp from a TSA, or a write into a separate AWS account's S3 bucket under Object Lock in compliance mode. Azure Immutable Blob Storage and GCS Bucket Lock give the same WORM guarantee. No identity should hold both write access to the data and delete access to the audit store.

Retention. Set it to the longest regime that applies, then tier with the guarantees intact.

RegimeRetentionImmediately retrievable
SEC 17a-4(f) / FINRA 45116 yearsfirst 2 years
MiFID II RTS 115 years (extendable to 7)—
HIPAA §164.316(b)(2)(i)6 years—
PCI DSS 10.5.112 months3 months
CFTC 1.315 yearsfirst 2 years

Sizing, illustrative: 200 events/s at 500 bytes is 100 KB/s, about 8.6 GB/day and 3.1 TB/year. Roughly 19 TB over six years before replication. That is cheap on S3 Glacier Instant Retrieval or Azure Archive, provided the Object Lock retention is set at write time and cannot be shortened.

Retrieval. Partition by month and index on (subject_id, occurred_at); Postgres declarative partitioning handles this, or ship to ClickHouse or OpenSearch if the query mix is exploratory. Regulators expect production within days, not weeks, so rehearse the extraction against a real request shape.

Failure modes to name. Clock skew breaks ordering: use the database's clock_timestamp(), not the application's clock. A restore from backup can fork the chain, so record the chain head alongside each backup. And a synchronous audit write costs a round trip: same-AZ roughly 0.5–1 ms, cross-region 30–70 ms, which is why the audit store is usually in-region and replicated asynchronously.

Curated: · Written: · Reviewed:

Q&A-88How do you handle personal data in logs and analytics without breaking the product?

Decide what identifiers the downstream consumer actually needs, and supply the weakest one that works.

For most operational logging, a stable pseudonymous ID is enough. Debugging needs to correlate a user's requests, not to know their email address, so log a user ID and resolve it to a person only through a separate, access-controlled lookup. That single change removes personal data from the highest-volume, widest-access data store you have.

For fields that must not appear at all (tokens, card numbers, health data, message contents) redact at the logging layer rather than at the call sites. A middleware that strips known-sensitive keys and pattern-matches obvious secrets protects against the case that actually causes incidents, which is a new field added by someone who did not know the rule.

For analytics, aggregate early and retain the raw event for a short window only. Most analytical questions are answered by counts and distributions, and keeping raw per-user events indefinitely creates a deletion obligation and a breach exposure for value that has usually already been extracted.

The deletion path is the part teams underestimate. A user's deletion request must reach every copy: primary store, replicas, backups, search index, cache, analytics warehouse, and the third-party tools. Designing this at the start: a documented inventory of where identifiers propagate, and a deletion job that covers them: is far cheaper than reconstructing it under a regulatory deadline.

Backups are the honest exception, and the standard answer is documented retention windows with deletion applied on restore.

Curated: · Written: · Reviewed:

Q&A-65How do you decide what to log, given that logging everything is expensive and logging nothing is worse?

Decide by asking what question each log line will answer during an incident. A line that answers nothing is cost.

The tiers I would set. Always log, at low volume: every request with a correlation ID, method, path, status, duration, and the authenticated identity. That single structured line answers most operational questions and its volume is predictable. Log every error with enough context to reproduce (inputs, identifiers, the failing dependency) not just a stack trace.

Log every security-relevant event unconditionally and retain it longest: authentication outcomes, authorization denials, permission changes, and access to sensitive data. These are the ones an audit or an investigation needs, and they cannot be reconstructed later.

Sample the high-volume, low-information categories: successful health checks, per-item debug lines in a batch loop, verbose third-party client output. Head sampling for traces with tail-based retention of everything that errored or was slow gives you the interesting tail without the volume.

Make level changeable at runtime, so debug logging can be enabled for one service during an incident without a deploy.

Two hard rules regardless of volume. No secrets, tokens, or personal data in logs: enforced by redaction in the logging layer, since developer discipline fails eventually. And structured output, because grep across terabytes is not an incident response strategy; fields you can filter and aggregate are.

Then set retention by category rather than globally: 7-30 days hot for operational logs, a year or more in cheap storage for audit.

Curated: · Written: · Reviewed:

Q&A-30What signals would you put on a dashboard for a service you have never operated before, and what would you alert on?

Assume I can get three things before I touch the dashboard: the service's SLO, its dependency list, and its deploy pipeline. If there's no SLO, that's the first fix: every alert threshold below is derived from it, and without it you're guessing at what "bad" means.

Dashboard. Four golden signals for the service, and the same four for each dependency, because most incidents arrive from below.

SignalWhat I plotWhy it earns a panel on an unfamiliar service
Latencyp50/p95/p99 per endpoint, as percentile lines or a heatmapA mean hides the tail; the tail is what users feel
Trafficrequests/s by endpoint and status classThe baseline that makes every other panel readable
Errorsrate split by cause (4xx, 5xx, timeout) not one blended numberA 4xx spike is a client or a bad deploy; a 5xx spike is you
Saturationthe scarcest resource: connection pool, thread pool, queue depth, CPU throttlingWhich resource runs out first is service-specific; the dependency list tells you where to look

Overlay deploys and config changes. A large share of incidents correlate with a change, and the overlay makes that visible without a second dashboard.

Alerts. Alert on symptoms users feel, not causes. High CPU is a cause; it may or may not matter. Elevated 5xx on a user-facing endpoint is a symptom. The cleanest symptom formulation is error budget burn rate, because it scales urgency to damage.

Worked example. SLO 99.9% over 30 days, 2M requests/day. Budget = 0.1% of 60M requests = 60,000 errors.

  • Burn rate 14.4 sustained 1h → 1.44% error rate → ~1,200 errors → 2% of budget. Page.
  • Burn rate 6 sustained 6h → 0.6% → ~3,000 errors → 5% of budget. Page.
  • Burn rate 1 sustained 3 days → 0.1% → ~6,000 errors → 10% of budget. Ticket.

Each alert pairs a long window with a short one and fires only when both exceed the threshold. The long window catches slow burns; the short window clears the alert within minutes of recovery instead of leaving it hot for an hour.

The trade-off is detection time against noise. Cause-based alerts fire earlier but page on events that never reach users; symptom-based alerts are precise but only fire once damage is happening. Burn rate sits between them: it fires on real damage, at a rate that lets you act before the budget is gone.

The exceptions I page on that aren't yet symptoms are saturation signals with a lead time: disk fill rate, certificate expiry, a queue whose backlog grows faster than it drains, replica lag approaching the RPO. Each is an outage with a countdown, and each is cheap to predict.

What breaks. A mean latency panel hides p99 going 40 ms → 180 ms. Alerting on CPU pages you for a nightly batch job no user touches. Per-user or per-request-ID labels blow up the time series database: keep labels bounded. And an alert with no owner and no runbook is a graph, not an alert: if nobody can say what to do when it fires, it shouldn't page anyone.

For a service I've never run, the first week is discovery: instrument, watch the baseline, then set thresholds from what I observe rather than copying another service's numbers.

Curated: · Written: · Reviewed:

Q&A-38A customer reports intermittent errors that you cannot reproduce and that do not appear in your dashboards. How do you proceed?

Assume the dashboards are aggregating the signal away, because that is usually what has happened. An error rate of 0.2% is invisible next to a 99.8% success line, and a p99 graph hides a p99.9 that one customer experiences constantly.

So first, break down rather than drill in: split error rate and latency by customer or tenant, by region, by client version, by instance, and by availability zone. Intermittent problems that are invisible in aggregate are usually concentrated in one dimension: a single bad host, one zone with a network issue, one client version with a shorter timeout.

Second, get identifiers from the customer. A request ID or a precise timestamp with a time zone turns an unreproducible report into a trace. If requests are not carrying a correlation ID end to end, that is the finding: the gap is in observability, and fixing it is more valuable than this one incident.

Third, look at what aggregation cannot show: distributed traces for slow requests, and structured logs filtered to that tenant. Tail-based sampling that keeps every erroring trace is what makes this possible after the fact.

Common causes for this shape of report: retries hiding a real failure rate, one unhealthy instance passing a shallow health check, connection pool exhaustion at a specific hour, DNS or TLS handshake failures that never reach application logs, and clients timing out earlier than your latency budget assumes.

Curated: · Written: · Reviewed:

Q&A-9A query that ran in 20ms last month now takes 8 seconds. The code has not changed. Walk me through your diagnosis.

Assumptions first: PostgreSQL 15/16, one query, same SQL text, same hardware. "Code has not changed" narrows the cause to four things: the data, the statistics, the schema/indexes, or the plan the planner picked. A 20 ms → 8 s jump is 400×, which is not gradual degradation: it is a plan change or a threshold crossing.

First move: get the real plan, not a theory.

EXPLAIN (ANALYZE, BUFFERS, VERBOSE) <query>;

Read it top-down and find the first node where rows= (estimate) diverges from actual rows= by more than ~10×. Everything above that node is downstream of a bad estimate; everything below it is fine. That one comparison splits the diagnosis in two: bad estimate → statistics or plan-cache problem; good estimate → the work genuinely grew.

Worked trace. A plan that explains the 8 seconds exactly (illustrative numbers):

Nested Loop  (cost=0.86..1842033.11 rows=50 width=64) (actual time=0.06..8123.44 rows=4213880 loops=1)
  ->  Seq Scan on orders  (cost=0.00..1204.00 rows=50 width=16) (actual rows=4213880 loops=1)
        Filter: (status = 'pending'::text)
        Rows Removed by Filter: 1204
  ->  Index Scan using customers_pkey on customers  (actual rows=1 loops=4213880)
        Index Cond: (id = orders.customer_id)

The planner estimated 50 rows out of the outer scan; it actually got 4.2 M. It chose a nested loop because 50 inner lookups are cheap. At 4.2 M lookups, each ~1.5 µs of index descent plus buffer access, that is 4.2e6 × 1.5 µs ≈ 6.3 s, plus the outer scan and tuple overhead → the 8 s floor. Nothing about the query changed; the estimate did.

Fix: ANALYZE orders; then re-run EXPLAIN. If the estimate corrects, the planner switches to a hash join and you are back near 20 ms. Then stop it recurring: check pg_stat_user_tables.last_analyze, if it is weeks old, autovacuum's analyze threshold (50 + 0.1 × rows) was not crossed fast enough after a bulk load; lower autovacuum_analyze_scale_factor on that table.

The other causes, with the signal that identifies each:

CauseSignal in the plan / catalogFix
Stale statsestimate vs actual >10× divergence; last_analyze oldANALYZE, tune autovacuum
Generic cached plansame query fast with a literal, slow with a bind paramSET plan_cache_mode = force_custom_plan
Index unusableSeq Scan where Index Scan used to be; filter shows a cast or function on the columnmatch types, or add an expression index
Bloatpg_relation_size ≫ live rows; pgstattuple dead-tuple ratio highVACUUM FULL / pg_repack
Data growthplan unchanged, row counts up 100×new index, or partition

What I would rule out, and why. Lock contention, a saturated buffer cache, and noisy neighbours produce variable latency: p50 fine, p99 terrible. An 8-second floor on every run is deterministic, and deterministic means the plan. I would still check pg_stat_activity for a blocker once, because a lock wait looks like a floor too, but I would expect the plan to be the answer.

Before trusting "the code has not changed": ORMs and query builders change the SQL without a commit to your file. A new default filter, a changed bind type, a lazy-load that now fires per row. Diff the actual SQL from pg_stat_statements, not the source.

Curated: · Written: · Reviewed:

Q&A-4What actually breaks first when traffic to a service grows ten times, and how do you find out before your users do?

The 60-second version. Assume a stateless service behind a load balancer, one primary Postgres, autoscaling on CPU. Ten times traffic almost never breaks the code first: it breaks the first fixed limit in the request path. In rough order of how often I've seen it: the database connection pool, a single-writer primary, a downstream rate limit or quota, then a queue whose consumers can't keep up. The pool is first because it's the one that scales against you: instances autoscale, connections per instance are fixed, and the database's max_connections is fixed, so capacity grows linearly into a wall.

Why the pool, with numbers. Postgres ships with max_connections = 100. Say the service runs 40 instances with a pool of 20 each: 800 potential connections against 100 slots. At baseline only a fraction are busy, so nothing complains. At ten times traffic the pool fills, the 101st connection is refused, and the failure is binary: existing queries keep working while every new request fails to get a connection. That reads as a total outage, not degradation, which is why it surprises people.

The size you actually need is Little's Law: in-flight = arrival rate × hold time. Baseline 200 rps at 25 ms mean query time → 5 concurrent queries. Ten times traffic → 2,000 rps × 25 ms = 50 concurrent. Fine. But once the pool saturates, wait time is added to hold time: at 100 ms effective, in-flight = 200. The pool is now the queue, and the queue creates the latency that fills it. That loop is what turns a busy hour into an outage.

What to watch. Saturation, not absolutes. "400 connections" tells you nothing; "400 of 500" tells you how many weeks you have.

SignalWhat it measuresAlert shape
Pool in-use / pool maxheadroom per instance> 70% for 5 min
DB connections / max_connectionsheadroom at the shared limit> 80%
Queue depth and d(depth)/dtarrival minus drain ratederivative > 0 for 10 min
Replica lagread-path staleness> 5 s
Burstable CPU credit balancehidden throttlebalance < 30%
Remaining downstream quotafixed external limit< 20% of daily budget

Latency and error rate are lagging indicators: by the time p99 moves, users are already in it.

Finding out first. Load test at 10x against production-shaped data: same row counts, same index selectivity, same hot rows. Empty tables hide the problem: a query that's 1 ms on 1k rows is 400 ms on 50M. Ramp rather than spike, and watch the saturation signals above; the first one to cross 80% is your answer. Then test the overload path deliberately: exhaust the pool, add 200 ms to the downstream, and confirm the service sheds instead of queueing.

Overload behaviour. At ten times traffic something will exceed capacity eventually; the design question is whether it degrades or collapses. Queue arithmetic makes it concrete: arrival 2,000/s, drain 1,500/s → backlog grows 500/s, so a 60,000-message queue is full in 120 s and then every producer blocks. A bounded queue plus fast rejection is the difference between a slow minute and a cascading failure.

# Python 3.12, single-threaded event loop, so the counter check is atomic
inflight = 0
LIMIT = 50

async def handle(req):
    global inflight
    if inflight >= LIMIT:
        return Response(503, "shed")   # reject fast, don't queue
    inflight += 1
    try:
        return await process(req)
    finally:
        inflight -= 1

So: what breaks first is the fixed limit nobody is measuring, and you find it by ramping to 10x against real data while watching saturation rather than symptoms.

Curated: · Written: · Reviewed:

Q&A-15How do you size and structure an autoscaling policy so it actually helps during a traffic spike?

What I'd say first

Autoscaling follows capacity; it does not absorb spikes. Before choosing a metric I need three numbers: how fast load rises (rps per second), how long a new instance takes from launch to serving traffic, and what one instance actually carries at the SLO. If the rise time is shorter than the provisioning time, no policy saves you: you pre-provision or you absorb the spike somewhere else (a queue, load shedding, a cache).

Sizing

maxReplicas = peak rps / per-instance rps at SLO, rounded up, plus one for failure. Measure per-instance capacity under load; don't infer it from a dashboard average.

Worked example: hypothetical numbers, but this is the shape of the arithmetic:

  • One instance serves 150 rps at p99 = 200 ms, measured under load.
  • Steady state is 600 rps → 4 instances.
  • A campaign email lands at 12:00:00. Traffic climbs 600 → 4,500 rps over 90 s, then holds for 10 minutes.
  • Target tracking at 70% of capacity = 105 rps/instance → 4,500 / 105 = 43 instances at peak.
  • Launch plus health check = 100 s.

At t=0 the policy sees 150 rps/instance, above target, and starts a scale-out. But the metric is a 1-minute datapoint, the alarm needs an evaluation period, and the default cooldown is 300 s. In a 90-second ramp you complete roughly one launch wave. You reach maybe 8–10 instances while demand sits at 4,500 rps: 450–560 rps per instance, p99 well past SLO, and the excess queues or 503s.

The headroom version of the same arithmetic: covering a provisioning window reactively costs spare capacity equal to growth rate × warm-up. 50 rps/s × 100 s = 5,000 rps of idle headroom, i.e. 33 idle instances. That is the price of reactive-only, and it is why scheduled scaling exists.

Structure

WorkloadLeading signalWhy not CPU
Request/response behind an ALBALBRequestCountPerTarget, or concurrent connectionsCPU lags; a service blocked on a downstream is idle-CPU and saturated
Queue consumerApproximateAgeOfOldestMessage (backlog age)Message count hides per-message cost variance; age is the actual SLA
CPU-bound computeCPU utilizationFine here, but verify the correlation under load before trusting it

Asymmetry: scale out fast, scale in slow. Kubernetes HPA defaults: scaleUp.stabilizationWindowSeconds: 0, scaleDown.stabilizationWindowSeconds: 300 (default since 1.18), tolerance 0.1, sync period 15 s. AWS target tracking ships a 300-second cooldown; drop scale-out to 60 s and leave scale-in long.

Failure modes worth naming:

  • Oscillation. Scale-in raises per-instance load above target, which immediately triggers scale-out. Detect it as a sawtooth in replica count in the HPA or ASG event log. Fix with the scale-down stabilization window and the tolerance band.
  • Downstream herd. 43 instances × 20 pooled connections = 860 connections against a database max_connections of 200. The failure surfaces as connection errors at the pool, not as latency in your service. Cap maxReplicas at the downstream limit or put a pooler in front.
  • Metric staleness. 1-minute metrics plus a 60 s evaluation means you react to a two-minute-old world. Push a custom high-resolution metric from the app when the ramp is faster than that.
  • A ceiling you can't reach. Instance quota, subnet IP exhaustion (a /24 gives ~250 usable addresses), ENI limits. Test that the group actually reaches maxReplicas.

Test the scaled state

Load test to maxReplicas, not to steady state, and confirm the downstream survives it. A group that can reach 40 instances but shares a connection limit sized for 10 has moved the failure, not removed it.

Curated: · Written: · Reviewed:

Q&A-43How would you approach capacity planning for a launch you have never seen traffic for?

Convert unknown demand into a tested capability, then buy margin.

Start with what you can bound: the marketing reach, the size of the user base being notified, historical conversion for similar launches, and the shape of arrival. Peak matters more than total: a campaign email sent at once produces a spike far above the daily average, often 10-20x, in the first ten minutes.

Then load test to find where the system actually breaks, not to confirm it survives an estimate. Ramp until something fails, note which resource saturated first, fix or provision it, repeat. A test that passes at the expected load teaches you nothing about the margin you have. Use representative data volumes; a load test against an empty database measures the wrong system.

Pre-provision rather than relying on autoscaling for the spike, because provisioning lag is longer than the spike's rise time. Warm caches and connection pools before opening the gate.

Then design the overload behavior, because capacity planning that assumes the estimate is right is a wish. Rate limiting at the edge, a queue with admission control, and a graceful degradation path (read-only mode, a static waiting page) turn an underestimate into a slow experience rather than an outage.

Finally, agree the rollback: a feature flag or a staged regional rollout means you can reduce demand, which is the only lever that works when capacity cannot be added fast enough.

Curated: · Written: · Reviewed:

Q&A-94What would you look at first if a system is slow only for users in one geographic region?

Establish whether it is the network path or the serving path, because they lead to entirely different fixes.

Network path first, since regional-only slowness is most often distance or routing. Measure round-trip time from that region to your edge and to your origin. If the region is served from a distant point of presence, or if traffic is being routed to a far region because of a DNS or anycast configuration, latency is structural and no amount of application optimisation will help. Traceroutes from the affected region, and real user monitoring broken down by geography, answer this quickly.

Then check whether the edge is actually serving that region. A CDN with a low hit rate in one region, because content is not being cached there, or because a cache key includes something region-varying: means those users are paying the full origin round trip while everyone else is not.

If the network path is normal, look at the serving path for anything region-correlated: a regional replica with high lag, a dependency deployed in fewer regions, or a database read that crosses regions for those users specifically.

Then the non-technical causes, which are common: peak hours in that region coinciding with a batch job, a regional ISP issue, or regulatory inspection adding latency in some jurisdictions.

The measurement that settles it is a synthetic probe from inside the affected region against both the edge and the origin. Aggregate metrics from your own infrastructure cannot see a problem that occurs before the request arrives.

Curated: · Written: · Reviewed:

Q&A-89What is your process for evaluating whether an architecture will meet a latency requirement before it is built?

Clarify three things before doing any math: which percentile (p95 and p99 are different designs), which path (the critical render path, not the whole service), and at what load. A latency target without a concurrency figure is unanswerable: 200ms at 10 rps and 200ms at 2,000 rps are different architectures. Then build a budget and check it against measured components, not guesses.

Decompose the target into a budget. Take a 500ms p95 page render and write the numbers down:

SegmentBudgetSource
Client → edge, TLS 1.3 new conn (1 RTT)40msmeasured from a real client region
Edge → origin, cross-region RTT70mspublished inter-region latency
App processing20msprototype
3 sequential downstream calls @ 15ms45msprototype against a representative dataset
Primary DB query, indexed, warm5msEXPLAIN + timing on prod-shaped data
Render + transfer40msmeasured
Total220ms

Two things fall out immediately: sequential calls sum, and the budget is half spent before any interesting work happens. Same-region hops are ~0.5–1ms RTT; cross-region is ~70–80ms. That gap decides where you put caches and replicas.

Replace assumptions with measurements. Inter-AZ and inter-region RTTs are knowable. Query latency for a representative dataset is measurable with a prototype in an afternoon: representative meaning row counts and cardinality, not a 100-row fixture. Third-party latency is published or observable. Anything you can't measure yet is a spike, not a number.

Fan-out is where budgets die. A chain of five sequential 20ms calls floors at 100ms. A fan-out to 20 parallel calls costs the slowest of the 20, and that is not the per-call p99. If each call's p99 is 50ms and they're independent, the probability all 20 land under 50ms is 0.99²⁰ ≈ 0.82, so the fan-out's p99 sits near the 99.95th percentile of a single call. If the tail is fat, that's far worse than 50ms. If the calls share a database or connection pool, they aren't independent and the tails correlate, which is worse still.

Design against the budget: parallelise independent calls, cache stable data, move non-blocking work off the request path. Each has a cost: caching buys staleness, parallelism buys connection pressure and partial failures, async buys eventual consistency. Say which you're accepting.

Failure modes to check for, and how you'd see them:

  • Connection pool saturation: p99 goes from 40ms to 180ms once the pool hits its 20-connection cap. Detect via pool wait-time, not request latency.
  • Queueing: at 70% utilisation an M/M/1 queue is ~3.3× service time; at 90% it's ~10×. Detect via in-flight request count and queue depth.
  • Retry amplification: retries at three layers multiply offered load up to 27×. Detect via retry counters per layer.
  • Cold start: first requests after deploy hit cold caches and cold JIT. Detect by segmenting latency by process age.

Validate under load. Run at 1.5× peak concurrency with production-shaped data, long enough to cross a GC cycle and a cache expiry, and read p99 plus pool wait, not the mean. Latency at one request per second tells you nothing about latency at capacity.

Curated: · Written: · Reviewed:

Q&A-31A stakeholder asks for 99.99% availability. What questions do you ask before agreeing?

Four questions, in order.

Availability of what, measured how? 99.99% is 4.4 minutes per month, and it means nothing until you define the indicator: which endpoints, measured from where, counting which responses as failures. Measured at the load balancer, a client-side network failure is not your outage. Measured from synthetic probes in three regions, it might be. Teams routinely agree to a number and then discover they disagree about the denominator.

What is the cost of the gap between 99.9% and 99.99%? The difference is 43 minutes versus 4 minutes per month, and it is usually the difference between a single-region deployment with good practices and a multi-region active-active architecture with all the consistency complexity that implies. Often several times the infrastructure spend, plus a permanent tax on delivery speed. Stated that way, many stakeholders discover 99.9% was what they meant.

What are the dependencies' SLAs? Your availability cannot exceed the product of everything in the critical path. Three dependencies at 99.9% each put a ceiling near 99.7% before your own code fails at all. If the number is not achievable, that is a fact to establish now.

What happens at the boundary? An SLO without a consequence is a wish. What does the team stop doing when the error budget is exhausted?

Then write it down as an SLO with an error budget, and treat the budget as the shared decision-making currency between reliability and feature work.

Curated: · Written: · Reviewed:

Q&A-19A service you designed must keep serving if an entire cloud region goes down. How do you design for that, and what do RTO and RPO change about your answer?

RTO picks the topology, RPO picks the replication. Everything else is downstream of those two numbers. Assume an order-processing API: 5,000 req/s peak, 200 writes/s, RTO 5 minutes, RPO 60 seconds, budget ceiling around 2x baseline run cost.

TargetTopologyReplicationSteady-state cost
RTO 5 min / RPO 1 minactive-active serving, single-writer data per cellasync cross-region~2x
RTO 30-60 min / RPO 5 minwarm standby at 25% sizeasync~1.3x
RTO 2-4 h / RPO 15 minpilot lightsnapshot + log shipping~1.1x
RPO 0multi-region consensussynchronous quorum commit~2.5x plus 60-120 ms per write

For these numbers I go active-active for the stateless tier and single-writer-per-cell for data: clients hit an anycast address (Global Accelerator), which routes to region A or B at roughly 3,250 req/s each; a cell router hashes tenant to cell, and each cell is an Aurora Global Database with the writer in A and a secondary in B.

Failover arithmetic: health checks at 30-second intervals with three consecutive failures is about 90 seconds to detect. I would verify that interval in a test rather than trust the default. Aurora Global Database cross-region promotion is documented at around a minute and is not automatic, so it is a runbook step I automate to overlap detection. Add 10-20 seconds for clients to re-establish connections and I land near 2.5 minutes, with the rest of the RTO for cache warm-up and autoscaling. DNS alone cannot make 5 minutes: a 60-second TTL is honoured inconsistently, and I have watched subsets of clients sit on a dead IP for 10-15 minutes. Anycast removes DNS from the critical path; Route 53 stays as a second layer.

On conflicts: single-writer per cell means there is no conflict resolution to get wrong. Multi-writer stores (DynamoDB global tables, Cassandra) resolve concurrent writes to one key last-writer-wins, which silently discards one of them during a partition: acceptable for a cart, not for order state. The price is that reads outside the writer region either cross the backbone or read slightly stale.

RPO honesty: with async replication, RPO is the replication lag at the moment of failure, so I alert per cell at 10 seconds of lag. The 60-second RPO is a budget being spent. RPO 0 forces synchronous cross-region commit onto every write and roughly doubles the data-tier bill. That is a negotiation with the business, not a config flag.

Capacity: surviving a lost region means each region carries peak, so two regions at 65% cost about 1.3x and the survivor jumps to 1.55x its normal load. It either autoscales inside the RTO or sheds non-critical traffic. Cells bound blast radius: losing one cell is 5% of tenants, and I can fail one cell over without touching the region.

Shared responsibility: the provider compensates failures inside its service boundaries, not your decision to run in one region. The multi-region wiring is mine: replication configured and lag monitored, KMS keys (regional, so an encrypted snapshot does not decrypt in the other region; you re-encrypt with a target-region key), ACM certificates, WAF rules, service quotas and vCPU limits in region B, secrets, and the promotion runbook with a named authorized operator.

Proof: a quarterly game day in business hours that blackholes region A at the network edge rather than stopping the app, so health checks and failover are exercised as one path. Two numbers measured from outside: time from declared failure to p99 error rate back under SLO, and acknowledged writes lost, counted from a synthetic client writing a numbered record every second. Include failback, most DR incidents happen there, and run a single cell first.

Failure modes I design for: split brain if promotion happens before the old writer is fenced; health checks green while writes fail, which is why the check must be a synthetic write; and the single-region dependency nobody inventoried, such as one Kinesis stream or one Lambda layer. The game day is what surfaces those.

Curated: · Written: · Reviewed:

Q&A-1Design a multi-region deployment for a service with a 15-minute RTO and a 5-minute RPO. Which topology do you choose, and what does that choice cost you?

What I'd say in the first two minutes. Assume the 5-minute RPO is a hard ceiling on committed-data loss, not a target, and that this is ordinary OLTP where writes are the scarce resource. Then the choice is asynchronous streaming replication from one writer region to a warm standby in a second region, with replica lag as a first-class SLI. Synchronous cross-region replication is the only thing that reaches zero loss, and it charges you a cross-region RTT on every commit plus a write-availability cliff when the standby is unreachable. If the business actually means "we cannot lose a committed order," that trade flips and you pay the latency.

Why not the alternatives.

TopologyRPORTOWrite latencyRelative cost
Cross-region backup/restoreSnapshot interval (hours)Restore time, scales with data size: 2 TB at 200 MB/s ≈ 2.8 hnonelowest
Pilot light (data replicated, compute off)= lag30–60 min (provision + warm)nonelow
Warm standby, async (chosen)= lag; target ≤ 60 s, alert at 180 s5–15 minnone~1.3–1.5×
Hot standby, sync01–5 min+RTT (~60–80 ms cross-continent)~2×
Active-active multi-region0secondslocal writes, but conflict resolution2–3×

Backup/restore fails on RPO before it fails on RTO. Sync fails on latency and availability. Active-active buys an RTO you don't need at a cost you can't justify for a 15-minute objective.

The design. One writer region. The standby runs the same stack at ~30% of primary capacity, continuously applying WAL, and is promoted on failure. No capacity is acquired during the incident: that is the whole point of "warm."

Artefact 1: measuring the thing your RPO depends on.

-- PostgreSQL 14, run on the standby
SELECT
  pg_last_wal_receive_lsn() AS received,
  pg_last_wal_replay_lsn()  AS replayed,
  pg_wal_lsn_diff(pg_last_wal_receive_lsn(),
                  pg_last_wal_replay_lsn()) AS replay_lag_bytes,
  now() - pg_last_xact_replay_timestamp() AS replay_lag_time;

Caveat that bites people: pg_last_xact_replay_timestamp() measures time since the last replayed commit, so on an idle primary it climbs even though nothing is behind. Alert on byte lag and on the primary's send position versus the standby's replay position, not on that timestamp alone.

Artefact 2: a failover trace (illustrative).

  • t=0 primary region loses power.
  • t=0–90 s monitoring fires after 3 consecutive failed 30 s health checks.
  • t=90 s on-call declares; runbook starts.
  • t=120 s lag verified at 12 s: inside RPO.
  • t=150 s promote standby; accepting writes by t=210 s.
  • t=210–270 s repoint DNS/global load balancer (TTL 60 s).
  • t=330 s clients reconnect, pools refill.
  • t=450 s caches warm, autoscaling reaches full capacity. Steady state.

Total ≈ 7.5 min against a 15-minute RTO, with ~7 min of margin. Data lost: 12 s of transactions.

Failure modes.

  • Lag silently exceeds RPO. Alert at 60% of budget (180 s), not at 300 s: by the time you alert at the limit, you are already out of compliance.
  • Failover to a lagging replica. You lose exactly the lag. A 4 min 50 s lag at the moment of failure puts you at the RPO edge with no room.
  • Split brain. The old primary returns and accepts writes. Fence it before repointing, and remember DNS TTL means some clients keep hitting it for up to the TTL.
  • Warm standby can't absorb full load. 30% capacity promoted to 100% traffic depends on autoscaling working in a region you've never scaled in. Pre-provision the floor and test scale-out in a drill.
  • If you do choose sync: synchronous_standby_names with FIRST 1 blocks writes when the named standby is unreachable; ANY 1 (a,b) survives one loss but needs a third copy to survive two.

What it costs. Duplicate compute at ~30%, if the primary fleet is $12k/month, the standby is ~$3.6k/month, illustrative. Duplicate storage plus cross-region transfer billed per GB. And engineering: quarterly failover drills and runbook upkeep. The drill is the only thing that turns "15-minute RTO" from a claim into a measurement.

Curated: · Written: · Reviewed:

Q&A-69Describe how you would test a disaster recovery plan.

By executing it, on a schedule, with the results measured against the stated objectives. A plan that has never been run has an unknown RTO, and the gap between the documented procedure and reality is discovered only under load.

Build up in stages. Start with a tabletop walkthrough: the team talks through the steps, which finds missing documentation and unclear ownership cheaply. Then a component test: fail over the database in a non-production environment and time it. Then a full regional failover in production, or in an environment that is a faithful copy, with the clock running.

Measure the actual RTO and RPO rather than the intended ones, and record where the time went. The delays are rarely in the technology: they are in deciding to declare a disaster, in finding the person with the necessary access, in a runbook step that references a system that was renamed, and in DNS TTLs that are longer than anyone remembered.

Test the restore, not the backup. A backup that has never been restored is an assumption. Restore to a fresh environment and verify the data, including that the application actually starts against it.

Test the failure of your recovery mechanism too: what happens if the standby region is where the incident started, or if the credentials needed to fail over are stored in the system that is down.

Then fix the findings and schedule the next one. Quarterly is a reasonable cadence for a serious objective, and the trend in measured RTO is the real indicator of readiness.

Curated: · Written: · Reviewed:

Q&A-80How would you migrate 50TB of data to the cloud with minimal downtime?

Separate the bulk from the delta. Fifty terabytes over a 1Gbps link is roughly five days at full utilisation, which is neither achievable nor acceptable as downtime, so the bulk moves ahead of time and only the delta moves during the cutover.

For the bulk, compare network transfer against a physical transfer appliance. The crossover point depends on available bandwidth: at 10Gbps, 50TB is around twelve hours and network transfer is fine; at 100Mbps it is weeks and a physical device is faster and cheaper. Whichever you choose, the bulk copy happens while the source stays live.

For the delta, use change data capture or incremental replication so the target continuously catches up with changes made since the bulk copy. The cutover window then shrinks to the time needed to drain the last few seconds of changes.

The cutover: stop writes at the source, wait for replication to reach zero lag, verify, repoint the application, resume. That is minutes, not days.

The parts that decide success. Verification must be by checksum or row-count reconciliation per table, not by eyeballing, and it must run before cutover, not after. A rollback path must exist: keep the source intact and writable until you have run on the target long enough to be confident. And bandwidth for the delta must be reserved, because a migration that saturates the link degrades the production system you are migrating.

Test the whole sequence on a subset first; the surprises are in the schema, character encodings, and data that violates constraints nobody knew existed.

Curated: · Written: · Reviewed:

Q&A-83Walk me through migrating an on-premises workload to the cloud.

The four decisions come in order: what the workload actually depends on, what to do with each piece, how the data moves, and when we cut over and how we come back. Jumping straight to the third is how migrations end with a Sunday-night rollback.

Assessment. The unit of migration is a dependency-consistent group, not a server. Discovery means flow data and config inventory cross-checked against what the owning team believes runs there, and I treat that output as a hypothesis until a dry run proves it. What discovery reliably misses is what breaks cutover night: the 3am batch job, a vendor VPN terminating on a firewall appliance, the licence server, the reporting database someone built in 2019 with a direct link to the primary. Every dependency gets a named owner and an explicit treatment before any date is agreed.

Data gravity decides the sequence and sometimes the strategy. 50 TB over a 100 Mb/s link is 46 days at line rate: 400 Tb ÷ 0.1 Gb/s: before retransmits, change freezes, or the time it takes to notice a bad checksum. So the realistic options are a dedicated link or a shipped device, moving the consumers along with the data, or leaving the data behind until last and streaming it behind continuous replication. If the consumers stay on-prem, every call now crosses a WAN, and I would move the application before the data rather than the reverse.

Strategy per group:

MoveChoose it whenWhat you accept
Retire / retainNothing consumes it, or it cannot legally leaveThe cheapest migration is the one you don't do
RehostCommodity tier, deadline-driven, no scaling or delivery problemKeeps old failure modes, often worse latency and worse licensing economics
ReplatformManaged equivalent exists: OS, database, queueBehavioural differences in failover, backups, extensions, collation, connection limits
RefactorThe code is the constraint: release cadence, scaling, cost per requestMost money and time; the only option that fixes the actual problem

Most estates end up mostly rehost/replatform with two or three refactors. That is a reasonable outcome; the failure is refactoring everything, or rehosting something whose architecture is the reason it needed to move.

Why a rehost fails quietly. Latency arithmetic: 200 sequential database round trips at 0.3 ms LAN latency is 60 ms of network. At 1.5 ms across an availability zone boundary it is 300 ms. Nothing in a low-load lift-and-shift test shows this; it appears under concurrency, as a queueing collapse in the app pool. That is why I measure the call pattern (round trips per request, not just bandwidth) before choosing rehost.

Cutover. Something like:

  • T-2w: target built as code, bulk copy started, smoke and restore tests green against copied data
  • T-2d: continuous replication running, lag under 60 s, reconciliation job proving it
  • T-0: freeze writes or force read-only, final delta, then reconcile. Row counts plus a business invariant (total ledger balance, open orders), not just "it's up"
  • T+30m: internal read-only traffic, then 10% of production
  • T+24h: rollback deadline, once reverse replication is running and verified

Rollback is a design decision, not a fallback. The moment the target accepts a write the source doesn't have, "just go back" becomes a data-reconciliation project. So the plan names the exact point after which the recovery path is forward-only, and before that point rollback is a DNS or traffic-weight revert plus reverse sync, tested before cutover, not improvised on the night.

De-risking. Pilot with a low-risk stateless workload first and pay for the runbook that gets. Test backup restore in the target: SAN snapshot DR has no automatic equivalent, and the first place anyone finds out is the first restore. Re-baseline cost with real figures: committed capacity for steady state, cross-AZ traffic, and licensing, where per-socket or per-core entitlements can cost more on cloud vCPUs than the server they replace. Re-check the security baseline, because network paths, key custody and log destinations all change. Finally, set abort thresholds as numbers the go/no-go owner can act on: error rate above 2× baseline for 10 minutes, p99 above 1.5×, replication lag above 60 s, and stop the wave rather than push through it.

Curated: · Written: · Reviewed:

Q&A-102You need to connect an on-premises data centre to a cloud VPC. Walk me through how you choose between a site-to-site VPN and a dedicated interconnect, and how you make the link survive a failure.

I'd start by pinning down four things: sustained and burst bandwidth in each direction, latency and jitter sensitivity, monthly data volume, and whether traffic must be encrypted on the wire. Those decide it.

A site-to-site VPN is IPsec over the public internet. It's up in hours, costs almost nothing, and encrypts by default. Its ceiling is real: AWS caps a single VPN tunnel at 1.25 Gbps, and latency follows internet weather. P99 jitter of 10–30 ms is normal, which breaks synchronous database replication or voice. A dedicated interconnect (Direct Connect, ExpressRoute) is a private circuit from your cage to the provider's edge. You get predictable sub-10 ms latency, 1/10/100 Gbps ports, and cheaper egress, but it takes weeks to provision, costs port hours plus a carrier cross-connect, and is not encrypted unless you add MACsec or run IPsec over it.

Site-to-site VPNDedicated interconnect
Bandwidth per link1.25 Gbps per tunnel1–100 Gbps per port
LatencyVariable, internet-dependentPredictable, often <5 ms
EncryptionIPsec by defaultNone; MACsec or IPsec over it
ProvisioningHoursWeeks
Cost modelTunnel hours + internet egressPort hours + cross-connect + carrier + cheaper egress
Best for<1 Gbps, backup path, fast start>1 Gbps sustained, latency-sensitive, large egress

The usual production answer is both: dedicated as primary, VPN as backup. Run BGP over each, prefer the dedicated path with local preference, and use BFD for sub-second failure detection. When the circuit dies, BGP withdraws the route and traffic shifts to the VPN in a few seconds. A single dedicated circuit is a single point of failure: two circuits in different provider locations with diverse last-mile paths is the only way to claim resilience.

Bandwidth arithmetic matters more than people expect. Four 1.25 Gbps VPN tunnels give 5 Gbps aggregate, but ECMP hashes per flow, so one TCP connection still tops out at 1.25 Gbps. If your workload is a single large replication stream, aggregation buys nothing. Same on a 10 Gbps DX port: a single flow may be limited by the on-prem firewall or the cloud-side virtual interface.

Failure modes to design for: BGP session flap from a carrier maintenance window, asymmetric routing when one path fails, and MTU blackholes. IPsec overhead drops usable MTU to ~1400; without MSS clamping or a correct tunnel MTU, large packets vanish silently. Detect with BFD, flow logs, and synthetic probes that measure both paths.

Cost example (list price, us-east-1, verify current): a 10 Gbps DX port is roughly $1.50/hour (~$1,100/month) plus cross-connect and carrier; internet egress is about $0.09/GB versus $0.02/GB over DX. At 100 TB/month, that's $9,000 versus $2,000 in egress alone: the circuit pays for itself, but only if you actually move that volume.

Curated: · Written: · Reviewed: —

Q&A-101We're moving 40 teams onto AWS. Walk me through the account structure you'd design, the guardrails you'd put on it, and how a new team gets a working account on day one.

Short answer. The account is the only containment boundary in a public cloud that a compromised principal cannot argue its way out of. IAM policies, service quotas, most resource policies and the bill are all scoped to it. So the landing zone is four things and nothing else: the OU tree, the guardrails attached to each OU, the pipeline that vends accounts, and the break-glass path for when the pipeline is wrong. Pick those four deliberately and the rest is detail.

The tree. Keep it shallow: depth buys nothing and makes SCP inheritance hard to reason about. Five OUs, not one per team.

OUAccountsPreventive guardrailWho can change it
Securitylog-archive, auditDeny organizations:LeaveOrganization; deny writes to the log bucket except the org trailPlatform, two-person rule
Infrastructurenetwork, shared-services, CIDeny regions outside the approved list; deny IAM user creation with long-lived keysPlatform
Workloads/Prodone per serviceDeny kms:ScheduleKeyDeletion; deny public S3 ACLsService team, via pipeline
Workloads/NonProdone per serviceSame, plus deny rds:DeleteDBInstance unless a snapshot tag is setService team
Sandboxone per engineer, 30-day TTLDeny VPC peering into prod; hard budget capThe engineer

Azure maps cleanly: management groups instead of OUs, subscriptions instead of accounts, Azure Policy assignments instead of SCPs, and the same six-level depth limit applies to the management group hierarchy.

Guardrails are two layers and teams conflate them. Preventive controls (SCPs, Azure Policy deny) filter what an identity can do; they never grant anything. A team whose role creation is blocked by an SCP still needs the IAM policy that allows it: the SCP only subtracts permission. Detective controls (Config rules, GuardDuty, Security Hub) find what got through anyway. You need both, because an SCP that is too tight stops work and one that is too loose stops nothing.

{"Version":"2012-10-17","Statement":[
 {"Sid":"NoLeave","Effect":"Deny",
  "Action":["organizations:LeaveOrganization","account:CloseAccount"],"Resource":"*"},
 {"Sid":"RegionLock","Effect":"Deny",
  "NotAction":["iam:*","sts:*","cloudfront:*","support:*","budgets:*"],
  "Resource":"*",
  "Condition":{"StringNotEquals":
    {"aws:RequestedRegion":["eu-west-1","eu-central-1"]}}}]}

The NotAction list matters: IAM, STS and CloudFront are global services, and a naive region lock breaks them.

Onboarding. A team opens a ticket with a service name, an owner and a data classification. The vending pipeline (Control Tower Account Factory for Terraform, or a Terraform module over aws_organizations_account) then creates the account into the right OU, applies the baseline (CloudTrail into the log archive, Config recorder, GuardDuty, default VPC deleted, budget alarm at 80% of forecast), creates a team role trusted from the corporate IdP, and registers the account ID in the service catalogue. Target is under an hour with no human in the loop after the ticket. The team's first commit is a Terraform stack in their own repo, not a console session.

What breaks. Three things, in order of frequency. Someone attaches an SCP at the root and locks out a service nobody remembered was there. The organization's default account quota is 10, so a 40-team rollout needs a quota increase before day one: 40 teams × 3 accounts plus 6 shared is 126. And the log archive account gets write access handed to a team "just for now", which destroys the one account whose entire value is that nobody can edit it.

Curated: · Written: · Reviewed: —

Q&A-103How would you design a data lake/lakehouse on a cloud provider to serve analytics at scale?

Core answer (first 90 seconds)

Three things decide the design: who queries it (BI, data science, or both), the freshness SLA (nightly, 5 minutes, seconds), and the governance regime (PII, retention, residency). Everything else follows.

My default is a lakehouse on object storage with an open table format, Iceberg on S3/GCS/ADLS, because it gives ACID, schema evolution and time travel without locking me to one engine. Three zones:

  • Raw/bronze: append-only, exactly as received, partitioned by ingest date. Never mutated. This is the replay source.
  • Curated/silver: deduplicated, conformed, typed. Iceberg tables with MERGE for upserts.
  • Consumption/gold: modelled marts, aggregated, often materialized views or clustered copies for BI.

Ingestion: batch for sources that only expose nightly extracts; CDC (Debezium → Kafka → Iceberg sink) for operational databases needing minutes; streaming for events. All three land in bronze first. A streaming path never writes straight to gold: that couples the consumer to the producer's schema.

Storage layout and file sizing

The dominant operational failure in lakehouses is the small-file problem. Every file is at least one task in a query engine. Target 128MB–1GB Parquet files with Zstd. That means:

  • Partition on low-cardinality, always-filtered columns (usually event_date), never on user_id.
  • Use Iceberg hidden partitioning or Delta generated columns so consumers don't need to know the layout.
  • Run scheduled compaction (rewrite_data_files, OPTIMIZE) as a real budgeted job.

Arithmetic: a 10TB table at 256MB per file is ~40k files. At 2MB per file it's 5M. The second will not scan in reasonable time regardless of compute, because the bottleneck is metadata and task overhead, not bytes.

Governance and catalog

One catalog, one source of truth, Glue Data Catalog or Unity Catalog, with table-level lineage captured. Column-level access control via Lake Formation LF-Tags or Unity Catalog grants, not by copying data into per-team buckets. PII columns get tagged at ingestion so a new consumer inherits the restriction instead of negotiating it.

Separation of storage and compute

Storage is S3/ADLS/GCS; compute is ephemeral. Serverless SQL (Athena, BigQuery, Snowflake) for interactive BI; managed Spark (EMR, Databricks, Glue) for heavy transforms. Both read the same Iceberg tables: that's the point. A BI team on Athena and an ML team on Spark don't need a copy.

Warehouse vs lakehouse vs hybrid

Managed warehouseLakehouseHybrid
Best whenSQL-only, high concurrency, <100TBMixed SQL+ML, open formats, petabyte scaleExisting warehouse plus new ML
Cost modelPer-slot or per-creditStorage + compute you manageBoth
GovernanceMature, built-inNeeds catalog disciplineTwo systems to reconcile
Lock-inHighLowMedium

Honest answer: if the workload is 90% SQL BI under 100TB, a managed warehouse beats a self-managed Spark lake on total cost and operational burden. The lakehouse wins when you have ML, unstructured data, or a genuine multi-engine requirement.

Cost control

Partition and column pruning are the biggest levers: scanning 10TB instead of 200GB is a 50x difference. Then Zstd over Snappy, spot/preemptible Spark executors with a small on-demand driver pool, S3 Intelligent-Tiering or lifecycle to Glacier IR for cold partitions, and per-team chargeback via query tags. Teams that can't see their spend won't optimize it.

Follow-ups an interviewer will ask

  1. Late-arriving data. Tests watermarks, idempotent MERGE, and reprocessing cost. Bronze keeps everything; silver MERGEs on a business key with a version column; a lateness window (say 48h) sends older records to quarantine rather than silently corrupting aggregates.
  2. Evolving the partition scheme without downtime. Tests Iceberg partition evolution (metadata-only; old data keeps its layout) versus Delta/Hudi where a rewrite may be needed, and whether you'd coordinate consumers, since a partition change that breaks a downstream job is an outage.
  3. GDPR deletion on an immutable log store. Tests whether you separate PII from immutable facts. Crypto-shredding (per-subject key, delete the key) or Iceberg row-level deletes plus compaction to physically remove. Also the difference between deleted at the query layer and deleted from storage.
  4. When does a managed warehouse beat a self-managed Spark lake? Tests judgement and cost honesty. The answer is more often than lakehouse advocates admit: concurrency, SQL workload shape, and the fully-loaded cost of a platform team.

Curated: · Written: · Reviewed: —

Q&A-67What questions do you ask before agreeing to a multi-cloud architecture?

What problem is it solving, specifically? The stated reasons are usually availability, negotiating leverage, or avoiding lock-in, and each deserves a different response.

If the reason is availability: is a provider-wide outage actually your risk? Most large outages are regional or service-specific, and multi-region within one provider addresses those at a fraction of the complexity. Multi-cloud active-active means your data layer must be consistent across providers, which is the hardest problem in the design and the one that most such projects fail on.

If the reason is negotiating leverage: is the discount larger than the engineering cost? Running two providers means two sets of expertise, two security models, two IAM systems, two monitoring stacks, and a permanent constraint to the intersection of both feature sets, which is roughly the capability of neither.

If the reason is avoiding lock-in: what is the actual exit scenario, and what would it cost to migrate if you did not prepare? Often the honest answer is that a focused six-month migration is cheaper than a permanent abstraction tax, and that the strongest lock-in is not compute anyway, it is data gravity and managed data services.

What I would usually recommend instead: portability where it is cheap (containers, standard protocols, infrastructure code with a clean module boundary) and provider-specific services where they earn their keep. Then, if a specific regulatory or contractual requirement demands a second provider, scope it to the workload that requires it rather than to everything.

Curated: · Written: · Reviewed:

Q&A-63How do you approach right-sizing a fleet without risking performance?

What a strong candidate says first. "Right-sized" is not "low utilization": it is the smallest fleet that still meets its SLO at the peak with one failure domain removed. So I measure at the peak, find the constraint that actually binds, change one dimension at a time in a canary, and keep the rollback cheap enough to use at 3 a.m.

1. Measure at the peak, on the dimension that binds.

Averages hide the answer. A fleet at 15% average CPU with an 85% p99 is correctly sized; the resize candidates are the ones where the peak is also low. Pull p50/p95/p99 over at least one full weekly cycle, per dimension:

Binding constraintMetric that decidesWhat shrinking it does
CPUp99 CPU at peakFine if peak is low; otherwise throttling
MemoryRSS high-water mark, not averageOOMKill under load
Connectionsconcurrent conns, open FDs per instancepool exhaustion, timeouts
Diskqueue depth, burst balancelatency cliff, not gradual
Burstable creditscredit balance trendthrottle to baseline

Burstable families are the trap. T3/T4g earn credits at a baseline fraction of vCPU (t3.medium: 20% of 2 vCPU) and cap the balance at 24 hours of baseline earnings. In standard mode, once the balance reaches zero the instance is held at baseline: throughput collapses while the CPU graph looks calm. The metric that matters is the credit balance trending down over days, not CPU over minutes.

2. Change one dimension, in a canary, through a full cycle.

Size, count, or generation: one at a time, or you cannot attribute the regression. Canary 5–10% of the fleet and hold it through a full traffic cycle including the weekly peak; a comfortable Tuesday says nothing about Monday morning. Keep the previous launch template or autoscaling group config intact so rollback is a config revert plus a scale-up, not a rebuild.

3. Headroom arithmetic.

Headroom = one AZ's capacity + autoscaling provisioning lag + application warmup. Hypothetical: 30 instances across 3 AZs at 60% peak CPU. Lose an AZ and the survivors sit at 90%: no margin left. A new instance takes roughly 3–5 minutes to boot and warm, so a 2-minute spike is absorbed by headroom, never by scaling. Right-sizing to the peak removes exactly the margin that failure needs.

4. Failure modes and how you detect them.

  • Memory-bound service shrunk on CPU: OOMKill count and container restart count climb before latency moves.
  • Connection-bound: pool wait time and "too many open files" in logs.
  • Cold caches after scale-in: p99 spikes immediately after a scale-in event. Plot p99 against fleet size.
  • Burstable: credit balance hits zero, then latency steps up and stays there.

5. Cheapest wins first.

Non-production schedules (nights and weekends off) and old-generation families usually beat instance-size tuning on savings per unit of risk. Do those before buying commitments: a savings plan locks in the size you picked.

Curated: · Written: · Reviewed:

Q&A-17Your monthly cloud bill doubled and nobody knows why. How do you investigate, and how do you prevent a recurrence?

First 90 seconds

Don't open the console. Open Cost Explorer, set granularity to Daily, group by Service, and compare the last 30 days against the 30 before. A doubling is almost never diffuse: it's one service, usually one usage type inside that service. Re-group by Usage Type, then by Linked Account or by your cost-allocation tag. Two questions decide the rest: is this a step change (a resource appeared, a config changed) or a ramp (something grows per request or per GB)? And is it one account or all of them?

Step change → find the resource. Cost Explorer won't give you resource IDs; the Cost and Usage Report will. CUR delivers hourly line items with resource IDs to S3, queried with Athena. Take the ID to CloudTrail event history (90 days) and find who created it and when.

Ramp → it's a meter, not a resource. Go to the usage type.

The meters that actually double bills

Rates are us-east-1 on-demand list, which is the number to quote before you know the region.

MeterRateWhy it hides
NAT Gateway data processing$0.045/GB + $0.045/hrNo instance owns it; absent from EC2 dashboards
Cross-AZ transfer$0.01/GB each direction ($0.02 round trip)Billed as a usage type, not a resource
Internet egress$0.09/GB for the first 10 TB/monthSpread across every service that talks out
CloudWatch Logs ingestion$0.50/GBA debug line left on at 10k rps
EBS snapshots$0.05/GB-monthNever expire by default
Public IPv4$0.005/hr per addressCharged since Feb 2024 whether attached or not
DynamoDB on-demand writes$1.25 per million WRUA hot-key retry loop

Worked example: a private-subnet service pulling 40 TB/month through a NAT gateway. 40 × 1024 = 40,960 GB × $0.045 = $1,843/month in data processing, plus 720 h × $0.045 = $32.40. If that traffic is S3, a Gateway VPC endpoint is free and takes the data-processing line to $0. That single line is a plausible doubling, and nothing in the EC2 console shows it.

Prevention

  1. Tag at provision time, enforced. Cost allocation tags must be activated in the Billing console before they appear in Cost Explorer, and take up to 24 hours to show. Enforce with an SCP that denies ec2:RunInstances unless aws:RequestTag/Project is present. Tags added after the fact are added by hand, and hand-added tags are incomplete.

  2. Budgets on forecasted spend. AWS Budgets: the first two are free, $0.02/day each after. Alert on forecasted, not actual: actual fires after the money is gone. Forecast needs history to project, so a brand-new account's first spike may not trigger it.

  3. Cost Anomaly Detection. Free, ML-based, per service, linked account, cost category or tag, alerts over SNS. It catches the ramp case that budgets miss.

Failure modes

  • Cost Explorer lags roughly 24 hours and carries no resource IDs: you see the spike a day late and still don't know what it is.
  • Unblended vs amortized: an expiring Reserved Instance or Savings Plan flips instances to on-demand and reads as a step change in unblended cost that isn't new usage. Check amortized before hunting.
  • Budget alerts on actual spend are post-mortem by construction.
  • A bill nobody owns is a bill nobody reduces. Cost visibility per team, and cost as a design-review criterion, where changing topology is still cheap.

Curated: · Written: · Reviewed:

Q&A-95How do you keep infrastructure costs visible to the teams that create them?

The 90-second answer

Make cost a property of the resource at creation, not a report assembled afterwards. Enforce a team tag at provisioning so every resource is attributable; where resources are genuinely shared, allocate by a published proxy and say plainly that it is an allocation, not a measurement. Convert the bill into the unit the team already reasons about (cost per million requests, per active tenant) and deliver it where the decision happens: a cost diff on the pull request, a weekly line in the team's channel. Then give them a lever, because visibility without the ability to change the number is just blame. This assumes teams own their services end to end; if they don't, the whole exercise is theatre.

Attribution: enforce at creation, not after

Tagging campaigns after the fact never finish. The version that works denies the create call when the tag is missing. In AWS that is an SCP:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyUntaggedRunInstances",
    "Effect": "Deny",
    "Action": "ec2:RunInstances",
    "Resource": "arn:aws:ec2:*:*:instance/*",
    "Condition": { "Null": { "aws:RequestTag/team": "true" } }
  }]
}

Two things people miss. AWS tag policies only report non-compliance: they do not block anything, so they are a detector, not an enforcement mechanism. And a cost allocation tag must be activated in Billing before it appears in Cost Explorer, with up to 24 hours of lag; a tag that exists on the resource but was never activated shows up as unallocated spend. SCPs also do not apply to the management account, so the root account needs its own guardrails.

Shared infrastructure: allocate, and show the idle

A 12-node EKS cluster of m5.2xlarge at $0.384/hr on-demand is 12 × 0.384 × 730 ≈ $3,364/month (illustrative, us-east-1 list price). That is 96 vCPU allocatable. Allocate by CPU requests, not usage:

NamespaceCPU requestedShareAllocated
team-a30 vCPU31.25%$1,051
team-b18 vCPU18.75%$631
team-c12 vCPU12.50%$420
unrequested36 vCPU37.50%$1,261 → platform

Allocating by requests rather than usage is deliberate: it charges teams for what they reserve, so over-requesting has a price, and the 37% nobody requested lands visibly on the platform team instead of being smeared across everyone. Allocate by actual usage and a team that reserves 30 vCPU and uses 3 looks cheap, while the cluster stays oversized forever.

Report in the team's units

$1,051/month is not actionable. team-a serves 420M requests/month, so it is $2.50 per million requests. That number moves when they change something and is comparable month to month. Trend beats absolute value.

Deliver where the decision happens

A monthly finance report is read by finance. Infracost posting a cost diff on the pull request reaches the engineer at the moment they pick an instance size. A weekly figure in the team's own channel keeps it alive between changes. Budget alerts are a backstop, not a control: they fire after the money is spent.

Failure modes and how you would catch them

FailureDetection
Unallocated bucket growsAlert when untagged spend exceeds 5% of total; it means a new resource type or an operator-created resource escaped the policy
Tag driftNightly job diffs live resources against the tag policy; autoscalers and operators recreate resources without tags
Allocation disputesPublish the formula and its inputs; a team that can see the denominator argues about the number instead of the method
Cost-per-request gamingBatching requests to shrink the denominator; pair the cost metric with a latency SLO
Under-provisioningCost reviewed alone drives incidents; review it next to the error budget, never as a standalone target

The lever matters as much as the number. If the team cannot change instance types, retention, or replica counts, they will ignore the report. Paved-road modules that make the efficient choice the default do more than any dashboard.

Curated: · Written: · Reviewed:

Q&A-49How do you decide whether to use a managed service or run something yourself?

The 90-second answer

Default to managed for stateful infrastructure you have no differentiated reason to run: Postgres, Redis, Kafka, queues, search. Self-host when the managed version can't give you a capability you need, when the price premium at your scale exceeds the loaded cost of the people who'd run it, or when regulation pins the data somewhere the provider won't go.

The decision is rarely about the compute bill. It's about labour and failure modes.

Do the arithmetic on labour, not instances

Worked example with assumed list prices: substitute current ones. A Postgres primary plus standby, 8 vCPU / 64 GiB, 1 TiB gp3, Multi-AZ:

Managed (RDS Multi-AZ)Self-hosted (2× EC2 + EBS)
Compute~$590/mo~$590/mo
Storage~$80–160/mo~$160/mo
Backups, PITR, patching, failoverincludedyour team
Steady-state ops~0~0.25 FTE

At a $200k loaded SRE, 0.25 FTE is ~$4,200/mo. The managed premium here is a few hundred dollars a month. Managed wins by roughly an order of magnitude, until the premium scales with node count and the labour doesn't. At 20 nodes the premium is $6–9k/mo while the 0.25 FTE is still 0.25 FTE. That's where self-hosting starts to pay, and only if you already have the expertise.

What managed actually buys, concretely

  • Tested restore. RDS point-in-time recovery to any second inside the retention window. A self-hosted pgBackRest plus WAL archive that you have actually restored from is a project, not a checkbox.
  • Failover. AWS documents RDS Multi-AZ failover as typically 60–120 s and Aurora as typically under 30 s. Your Patroni or repmgr cluster will do it too: after you've tuned it and drilled it.
  • Version currency. RDS forces major-version upgrades once a version reaches end of standard support; Aurora PostgreSQL charges extended support on old majors. That's a schedule you don't control.

What it costs you

  • Capability ceilings. No superuser on RDS or Aurora; max_connections is a function of instance class; extensions come from a curated list (pgvector is there, your obscure one may not be). DynamoDB gives you no query planner to argue with and no JOIN.
  • Debuggability. You get Performance Insights and pg_stat_statements, not perf on the host or the kernel's view of the I/O stack.
  • Exit cost that grows with adoption. Aurora → self-managed Postgres is pg_dump/pglogical plus a cutover; DynamoDB → Postgres is a rewrite of the access patterns. Egress at ~$0.09/GB is the visible part; the schema and IAM coupling is the expensive part.

Where self-hosting genuinely wins

A version or extension the provider won't ship; a workload whose shape (huge sequential scans, custom storage engines) the managed tier prices badly; data residency the provider's regions don't cover; or a team that already runs that system at scale and treats it as a core competency.

The tiebreaker

If it breaks at 3am, who is qualified to fix it, and do we want that to be us? If the answer is "nobody, and no", the managed premium is cheap insurance. If it's "we have three people who've done it and it's our differentiator", self-host.

Curated: · Written: · Reviewed:

Q&A-75How do you evaluate whether to adopt a new managed service that your team has no experience with?

Evaluate it as an operational commitment, not as a feature list, and time-box a spike that tests the parts that are expensive to discover late.

The questions I would want answered before committing. What are its failure modes, and what does the provider's status history actually show? What are the hard limits (throughput, size, connections, quotas) and where do they sit relative to our projected use in two years? What is the upgrade and maintenance model, and can we control when it happens? What does it cost at ten times our current scale, since the pricing curve is often the thing that changes the decision? What is the exit path, and how much data would have to move?

Then a spike that exercises the awkward paths rather than the tutorial: a failure injection, a restore from backup, a schema or configuration change under load, and the observability story. Can we see what it is doing when it misbehaves, or is it opaque?

Adoption should be staged. First a non-critical workload, in production, long enough to experience a real incident. That is where you learn whether support is responsive and whether the failure modes are survivable.

The decision criterion I would state: adopt when it removes work that is undifferentiated for us and is a core competency for them, and when the failure modes are ones we can detect and survive. Decline when the appeal is novelty, or when the exit cost grows faster than the benefit.

Curated: · Written: · Reviewed:

Q&A-68How do you keep a shared Kubernetes cluster safe for multiple teams?

What I'd say in the first two minutes

Assume one cluster, several teams that don't trust each other, and I own the platform. A namespace is a naming convention, not a boundary: the kernel is shared, so every control below has to be enforced at admission rather than written in a wiki. Four layers: identity, resources, network, admission. Node isolation and audit sit underneath them.

Identity. Each team gets namespaces and a RoleBinding scoped to them; nobody gets cluster-admin. Pods get cloud permissions through workload identity (EKS IRSA, GKE Workload Identity, AKS federated credentials) bound to a specific service account, so a pod's IAM role is as narrow as its RBAC. The thing people miss: anyone who can create a pod in a namespace can read every Secret in that namespace, so "create pods" is not a low-privilege grant.

Resources. Quota caps the namespace; LimitRange fills defaults so a pod with no requests doesn't land as BestEffort and get evicted first.

apiVersion: v1
kind: Namespace
metadata:
  name: team-payments
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.29
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: v1
kind: ResourceQuota
metadata: {name: quota, namespace: team-payments}
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    pods: "200"
    count/services.loadbalancers: "2"
---
apiVersion: v1
kind: LimitRange
metadata: {name: defaults, namespace: team-payments}
spec:
  limits:
  - type: Container
    defaultRequest: {cpu: 100m, memory: 128Mi}
    default: {cpu: 500m, memory: 512Mi}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny, namespace: team-payments}
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Network. Default-deny ingress and egress, then explicit allows. Without a policy every pod reaches every other pod, which turns one compromised container into a lateral-movement platform.

Admission. Pod Security Admission at restricted (stable since 1.25) blocks privileged containers, hostPath, hostNetwork, root users and writable root filesystems. PSA does not check image registries, so a Kyverno or Gatekeeper policy handles the registry allowlist and image signing. Admission runs in order: LimitRange fills defaults, ResourceQuota rejects over-budget pods, PSA enforces the level, then validating webhooks check what PSA can't.

LayerStopsDoesn't stop
IdentityCross-team API reads; over-broad pod → cloud accessPod reading Secrets in its own namespace
ResourcesNoisy-neighbour CPU/memory, unbounded podsA team legitimately using its whole quota
NetworkPod-to-pod lateral movementAnything, if the CNI ignores policy (Flannel alone)
Admissionprivileged, hostPath, hostNetwork, rootRegistry allowlists, image signing

Failure modes and how I'd catch them. Default-deny egress without a DNS allow rule breaks every lookup: allow egress to kube-system on port 53 first. Quota exhaustion doesn't leave pods Pending; the ReplicaSet gets a FailedCreate event, so alert on kubectl get events --field-selector reason=FailedCreate. Flipping enforce: restricted onto a live namespace doesn't evict running pods, only blocks new ones: run audit and warn for a week first and read the pod-security.kubernetes.io/audit-violations annotations in the audit log. And a CNI without NetworkPolicy support accepts the objects and enforces nothing, so verify with a test pod that should be blocked.

Underneath. For genuinely untrusted workloads, separate node pools with taints, or a sandboxed runtime (gVisor, Kata): container-to-container kernel isolation is weaker than VM-to-VM. Keep the audit log at Metadata level for most resources (RequestResponse on Secrets would write secret contents to disk) and ship it off-cluster, because "who changed this" comes up weekly.

Curated: · Written: · Reviewed:

Q&A-21How do you decide what belongs in a shared platform team versus in each product team?

Put something in the platform when it is undifferentiated, when getting it wrong is expensive, and when the cost of divergence exceeds the cost of coordination. Leave it with product teams when it is close to the domain or when teams genuinely need different answers.

Clear platform candidates: identity and access patterns, network topology, secret management, the CI/CD path to production, observability plumbing, and the base images and infrastructure modules. Nobody wants six answers to how a service gets credentials, and six answers means five of them are unreviewed.

Clear product-team territory: the service's data model, its API contract, its scaling behavior, its runbook. A platform team that owns these becomes a bottleneck and, worse, becomes accountable for decisions it lacks the context to make.

The design rule that keeps this healthy is the paved road: the platform's offering must be the easiest path, not the mandatory one. If the golden path is genuinely easier than doing it yourself, adoption is voluntary and the platform gets honest feedback. If it is mandatory but bad, teams route around it and you get shadow infrastructure that nobody reviews.

The failure mode to watch for is a platform team measured by tickets closed. That measures the platform as a service desk. The useful measures are adoption of the paved road, lead time from commit to production, and how many teams can deploy without asking anyone.

Curated: · Written: · Reviewed:

Q&A-45Describe how you would run a design review when two senior engineers disagree about the architecture.

Move the discussion from preference to evidence, and make the decision explicit rather than letting it dissolve.

First, establish what is actually being decided. Disagreements at this level are usually about different questions: one person optimizing for delivery speed, the other for operational cost in year two, and once that is visible, much of the heat goes out of it.

Second, write down the constraints that are not negotiable: the latency budget, the compliance requirement, the team's operational capacity, the deadline. Options that violate a constraint leave the discussion; this is often most of the disagreement.

Third, compare the remaining options on the same criteria, and require each side to state what evidence would change their mind. If neither can answer that, the disagreement is about values, not facts, and needs a decision rather than more discussion.

Fourth, where the disagreement is factual and cheap to resolve, resolve it: a prototype, a load test, a spike with a time box. Two days of measurement beats two weeks of argument.

Then decide, name the decider, and record it: the options considered, the criteria, the choice, and the conditions that would cause a revisit. An architecture decision record does most of this work, and the reversal condition matters most, because it converts a contested decision into a testable one.

The failure mode to avoid is deciding by seniority or by attrition. Both leave the losing side uncommitted, and the design gets undermined in implementation.

Curated: · Written: · Reviewed:

Q&A-98How do you introduce a significant architectural change to a team that is resistant to it?

Find out what the resistance is actually about, because it is usually not the technical merits.

The common reasons, each needing a different response: the team has been through a migration that consumed a year and delivered nothing, and expects a repeat; the change makes someone's expertise less valuable; the operational burden lands on people who were not in the decision; or the team can see a problem with the proposal that the proposer has not addressed. That last one is the most important, and it is the reason to listen first rather than to persuade.

Then reduce the size of the commitment. A proposal to re-architect is a bet nobody can evaluate. A proposal to move one low-risk service, with a defined success measure and a stated point at which you would abandon the approach, is a decision the team can agree to on Tuesday. Evidence from a real migration changes minds in a way that a document does not.

Give the burden to the proposer first. Whoever proposes the change should operate the first instance of it, including being on call for it. This is both fair and clarifying: proposals that will not survive contact with operations tend to be revised at this point.

Make the reversal condition explicit and honour it. A team that has seen an initiative abandoned when the evidence went against it will engage seriously with the next one.

And be honest that some resistance is correct. The right outcome of a design review is sometimes not proceeding.

Curated: · Written: · Reviewed:

Q&A-82How do you think about the trade-off between developer velocity and operational safety?

As a false dichotomy in the long run and a real one in any given week, which is why it needs a mechanism rather than a philosophy.

The long-run case: the practices that make deployment safe (small changes, automated tests, fast rollback, good observability) are the same practices that make it fast. Teams that deploy many times a day generally have fewer failures and recover faster than teams that deploy monthly, because small changes are easier to verify and easier to reverse. So most apparent trade-offs are actually a missing capability.

The short-run case is real: adding a required review, a soak period, or a manual gate does slow a specific change and does reduce a specific risk. The mistake is deciding that case by argument.

The mechanism I would use is the error budget. Agree an SLO, and let the remaining budget decide. Budget healthy means ship: the evidence says the current pace is not hurting users. Budget exhausted means reliability work takes priority until it recovers. Both sides agreed the number in advance, so the weekly negotiation disappears.

Alongside it, put safety into the path rather than into approval steps. Automated checks in the pipeline, progressive delivery with automated rollback on error-rate regression, and feature flags decoupling deploy from release make a change safe without making it slow. Manual gates are the tool of last resort: they are slow, and they degrade to rubber-stamping under time pressure, giving you the cost without the protection.

Curated: · Written: · Reviewed:

Q&A-96Describe a time-bounded plan for reducing risk in a system with known but unquantified problems.

The 60-second version. You have a list of known problems and no numbers. Spend a quarter turning that list into a ranked, evidence-backed one, and start fixing the top of it before the quarter ends. The plan fails in four predictable ways, each with a cheap detector.

Assumptions: the list exists, you have roughly a quarter, and the same people own both the risk work and the feature work.

WeeksActionOutput
1Inventory and rankEach problem gets a rough probability of causing an incident this quarter and a rough impact if it does
2–4Buy information on the top itemsRestore test, failover drill, load test: each converts a guess into a measured number
5–10Fix in expected-loss order, detection and recovery firstMeasured RTO, alert latency, capacity headroom
11–13Re-rankEvery item fixed, scheduled, or consciously accepted

Ranking first matters because without it the list gets ordered by whoever is loudest. Rough categories are enough: the goal is separating the three items carrying most of the expected loss from the twenty that don't. Include the ones nobody wants to write down: single points of failure, recovery paths never tested.

Weeks 2–4 buy information cheaply. A restore test takes a day and tells you whether backups work. A failover drill gives you real RTO. A load test finds where capacity ends. Hypothetical: a team assumed a one-hour restore; the test took six, because the largest table had never been restored at full size. That number moved the item from rank 9 to rank 1.

Fixing detection before prevention: cutting alert latency from 30 minutes to 1 reduces impact across every failure mode at once. Preventing one specific failure helps once.

How the plan itself fails.

Estimates get gamed. Whoever owns a component has an incentive to lowball its incident probability, because a high number means work. Detect it by comparing predicted incident counts against actuals each quarter: a team predicting 0.2 incidents and logging 3 is either unlucky or optimistic. Take the estimate in writing before the fix is scoped, so the estimate isn't set by the cost of the fix.

Drills get skipped or scoped down. The failover drill quietly becomes a tabletop. Detect it by requiring a number: a drill that produces no measured duration didn't happen. If the calendar shows the drill moved twice, that's the signal.

Backlog items die. Risk work loses every sprint to feature work. Detect it by tracking the age of each risk item; anything open six weeks with no status change gets escalated to the person who owns both sides of the trade.

The re-rank never happens. The list is written once and never revisited. Put the re-rank on the calendar with a date. If the top three are unchanged after a quarter of evidence, either the evidence was never gathered or the list was right: say which.

The deliverable is a list where every item is fixed, scheduled, or explicitly accepted, and a team that can name which risks it accepted.

Curated: · Written: · Reviewed:

Q&A-86How do you decide whether an incident warrants a postmortem, and what makes one worth reading?

Write one whenever the incident consumed a meaningful share of the error budget, affected customers visibly, required an unplanned rollback, or surprised the team: that last criterion matters most, because surprise means the mental model was wrong, which is worth understanding regardless of impact.

What makes one worth reading is that it explains the mechanism, not the sequence. A timeline of who did what at which minute is necessary and is not the value. The value is the answer to why the system permitted this, and why it took as long as it did to detect and to resolve.

Concretely, a good one contains: the customer impact in the customer's terms, the mechanism of the failure explained well enough that a reader could reproduce it, why existing safeguards did not catch it, what made detection slow, what made mitigation slow, and the action items with owners and dates.

Two things separate the useful ones from the ritual ones. It is blameless in a real sense: the question is what about the system made this action reasonable at the time, not who made the mistake. An engineer who fears the document will describe events carefully and reason vaguely, which destroys the value.

And the action items are prioritised honestly. Ten items nobody will do is worse than two that will, because it converts the process into theatre. The measure of the practice is whether the same failure recurs, not whether documents were produced.

Curated: · Written: · Reviewed:

Q&A-81What makes an on-call rotation sustainable?

The 90-second answer

A rotation is sustainable when three things hold at once: page volume stays inside a stated budget, the person on call has the access and authority to resolve what pages them, and repeat pages get converted into fixes. The budget is the part teams skip, and it is the part that decides everything else. Google's SRE book publishes the target as no more than two incidents per 12-hour on-call shift, and operational work (pages, tickets and manual toil together) at no more than 25% of an engineer's time, with 50% as the ceiling for the team. Argue about those two numbers in the design review; the rest follows.

TargetFigureSource / basis
Incidents per 12-hour shift≤ 2Google SRE book, "Being On-Call"
Operational work share≤ 25% per engineer, ≤ 50% per teamGoogle SRE book, "Eliminating Toil"
Overnight pagesZero for anything that can wait until morningPolicy, not a measurement

Sizing the rotation

24/7 primary coverage is 168 hours a week. If a primary shift is one week and nobody takes more than one week in four, that is four engineers for primary alone; add a secondary for escalation and you need eight trained people before anyone takes a holiday. Follow-the-sun across three regions covering eight hours each removes most overnight pages but still needs three primaries plus three secondaries, and each region needs a spare. A four-person rotation with no secondary is the configuration that breaks first: one illness puts someone on back-to-back weeks.

Alert hygiene, with the arithmetic

Take a hypothetical team logging 40 pages in a week. The weekly review classifies them: 26 non-actionable (a CPU > 80% alert with no user-visible symptom), 9 actionable but duplicated across two services, 5 real. Delete the 26, dedupe the 9, and the week drops to 5 pages: about 0.7 per 12-hour shift, inside budget, without hiring. That is the usual shape: most volume is alert-set debt, not workload.

Failure modes and how you detect them

  • Cause-based alerts. CPU, memory and queue-depth thresholds page constantly and nobody acts. Detect it in the weekly review: if fewer than half the pages led to an action, the alert set is the problem, not the rotation.
  • Rotation too small. Detect by tracking consecutive on-call weeks per person; two in a row means the roster is understaffed, not that someone was unlucky.
  • Missing access. If on-call cannot restart the service or read the logs, every page becomes a two-person page. Detect by counting pages that needed an escalation for permissions.
  • Alerts deleted without fixing the cause. The symptom returns in a month. Detect with a recurring-cause list carrying an owner and a date.

Fairness

Volume is only half of it. The rotation needs explicit compensation or time off in lieu for out-of-hours work, a documented escalation path so nobody sits alone with a problem beyond them, and a rule that a responder woken at 3am does not work the next morning. The on-call engineer also needs shift time to fix what paged them, not just to acknowledge it: otherwise the same page arrives next week.

Curated: · Written: · Reviewed:

Q&A-48You inherit a system with no documentation and an on-call rotation that pages nightly. What do you do in the first month?

Stabilise the humans first, then the system, then the design.

Week one: instrument the pages themselves. Every alert gets categorised: did it require action, was it a real user-facing problem, what did the responder do. Most nightly paging is a small number of alerts firing repeatedly, and you cannot fix what you have not counted.

Week two: eliminate the noise. Alerts that never required action get deleted or converted to tickets, not tuned. Alerts on causes rather than symptoms get replaced. Duplicate alerts for one condition get merged. This is usually a large reduction and it buys the team the capacity to do everything else.

Weeks two and three, in parallel: write the runbook as you go. The responder documents what they did while they are doing it. This is the only version of documentation that reliably gets written, and it is the most useful kind.

Weeks three and four: address the top recurring cause. There is almost always one (a disk filling, a leak requiring restarts, a dependency timing out) and fixing it removes a large share of remaining pages.

Only then look at architecture. Draw the actual dependency graph from traces and network flow, not from a diagram someone drew two years ago, and identify the single points of failure.

Resist the urge to rewrite. The system is loud, not necessarily wrong, and you do not yet know which parts encode requirements you have not discovered.

Curated: · Written: · Reviewed:

Q&A-87When would you accept a single point of failure in a design?

When the cost of removing it exceeds the expected cost of the outage it causes, and when that trade has been stated explicitly rather than overlooked.

The calculation needs three numbers: how often the component is likely to fail, how long recovery would take, and what an hour of that outage costs the business. A component with a mean time between failures measured in years, a fifteen-minute recovery, and an internal-only impact is frequently not worth eliminating.

Legitimate acceptances I would defend: a single control-plane component whose failure does not affect the data path, so running systems continue while it is down. A single instance of an internal tool with a documented, rehearsed rebuild. A single-region deployment for a product whose customers are in one region and whose contractual availability is 99.9%.

What must accompany the acceptance: monitoring that detects the failure quickly, a tested recovery procedure with a known duration, and a documented decision record so that a future engineer does not assume it was an oversight, or, worse, assume it was deliberate when it was not.

What I would not accept: a single point of failure in the data path that nobody has identified, which is the usual situation. Undocumented single points of failure are found during incidents, and the recovery time is then unbounded because nobody has practised it.

The useful exercise is to walk the request path and name what happens if each component disappears. The surprises in that exercise are the real risk, not the ones already on the diagram.

Curated: · Written: · Reviewed:

Q&A-91What is the value of chaos engineering, and how would you introduce it without causing an incident?

Its value is that it tests the failure paths, which are the only paths that are never exercised by normal traffic and are therefore the most likely to be broken. Every system has a documented behavior for a dependency timing out; a large share of them behave differently in reality, and the difference is discovered during an incident unless you go looking.

Introducing it safely is a matter of sequence and blast radius.

Start with a hypothesis, not an outage: state what you expect to happen when a specific dependency fails, and what signal would show it. An experiment with no prediction is just breaking things.

Start in a non-production environment to validate the tooling and the observation, then move to production, because non-production rarely reproduces the interesting behavior.

In production, start with the smallest blast radius that can teach you something: one instance, one availability zone, a small percentage of traffic, during working hours, with the team watching and an abort mechanism that is tested first. Working hours matter: the point is to learn with everyone present, not to simulate the 3am experience.

Have a stop condition defined in advance, tied to a customer-facing metric, and honour it.

Then broaden gradually as confidence builds. The organisational prerequisite is that the team already has good observability and a functioning incident process: chaos experiments on a system nobody can observe produce outages rather than knowledge, and that is how the practice gets a bad name.

Curated: · Written: · Reviewed:

Q&A-92How do you evaluate a vendor's claim that their service is highly available?

Read the SLA rather than the marketing page, and then read the exclusions.

The number itself is less informative than its definition. What counts as downtime: full unavailability, or degraded performance? Measured over what window: a month, so a four-hour outage may be under the threshold, or a shorter window that reflects real impact? Measured by whom, using what probes? Many SLAs exclude anything not detected by the vendor's own monitoring.

Then the exclusions, which are where the real terms live: scheduled maintenance, which in some contracts is unlimited; issues attributed to customer configuration; regional events; and dependencies the vendor itself relies on.

Then the remedy. Almost every SLA remedy is a service credit proportional to the fee, which is negligible compared to the business impact of an outage. So the SLA is not risk transfer: it is a statement of intent with a small penalty, and you should architect as though the remedy did not exist.

Evidence beats terms. A public status history with real incident detail, and postmortems that are specific rather than apologetic, tell you more than any number. So does asking directly about their largest incident in the last two years and how it was handled.

And architect for their failure regardless: what does your system do when this vendor is down for an hour? If the answer is that you are down too, then their availability is now your ceiling, and that fact belongs in your own SLO calculation.

Curated: · Written: · Reviewed:

Q&A-52How do you evaluate whether a system is ready for a compliance audit?

Audits ask for evidence, not for assurances, so readiness means the evidence exists as a byproduct of how you operate rather than being assembled the week before.

Four categories cover most frameworks. Access: who can reach what, how access was granted, how it is reviewed, and how it is removed when someone leaves. The hard part is usually the removal path and the periodic review, not the granting.

Change: every production change traceable to an approved, reviewed request. A pipeline that requires a pull request and records the approver produces this automatically; a process where engineers can apply from a laptop does not, at any level of discipline.

Data: what personal or regulated data exists, where it lives, how it is encrypted at rest and in transit, how long it is retained, and how deletion is performed and proven. Data mapping is the most common gap, because nobody maintains it unless it is enforced at the point where a new store is created.

Monitoring: log retention that meets the required window, logs that are tamper-evident, and evidence that alerts were acted on.

The readiness test I would apply is a dry run: pick a change from three months ago and produce the approval, the diff, who deployed it, and the access list for the system at that time. If that takes days, the controls exist on paper only.

Curated: · Written: · Reviewed:

Q&A-20What does a well-designed blue-green deployment give you that a rolling deployment does not, and what does it cost?

It gives you an instant, complete rollback and a fully-formed environment to test before any user reaches it. With a rolling deploy, rollback means another rolling operation over the same minutes, and during the roll two versions serve traffic simultaneously whether you wanted that or not.

Blue-green stands up the new version as a separate fleet, verifies it, then shifts traffic at the router. If the new version misbehaves, the shift is reversed in seconds because the old fleet is still running and warm.

The costs are real. You pay for double capacity during the transition, which for a large fleet is significant. More importantly, the shared database does not get a blue and a green: both versions read and write the same schema, so every schema change must be backward compatible with the version you might roll back to. That constraint (expand, migrate, contract) is the actual discipline; the traffic switch is the easy part.

Stateful connections complicate it too. Long-lived WebSockets or in-flight background jobs do not move at the router, so you need drain behavior and a defined maximum drain time.

A canary sits between the two: shift 1% of traffic, watch error rate and latency against the control, then proceed. It costs less capacity than blue-green and gives real production evidence that a staging test cannot, at the price of a slower rollout and the need for automated analysis to make the go/no-go call.

Curated: · Written: · Reviewed:

Q&A-79What is your approach to choosing partition keys for a sharded data store?

First 90 seconds

Three questions decide it: which query carries the most traffic, what the peak write rate is, and whether the key can ever change. The key should be the attribute that appears as an equality predicate in that dominant query, has enough distinct values that no single value exceeds one partition's capacity, and is never updated. When one attribute can't satisfy all three, use a composite key and pay for the fan-out.

Even distribution is a capacity question, not a vibe

Every sharded store caps what one partition absorbs. DynamoDB: 3,000 RCU and 1,000 WCU per partition, 10 GB per partition. Cassandra's widely cited guideline is to keep partitions under ~100 MB and ~100k rows. Redis Cluster spreads keys over 16,384 hash slots. MongoDB splits at 128 MB chunks by default. Those are the numbers that decide whether a key is safe.

Arithmetic for a multi-tenant SaaS: 40,000 writes/s at 1 KB average item. DynamoDB gives 1,000 WCU per partition at 1 KB per write, so throughput alone needs at least 40 partitions. If the largest tenant is 25% of writes, that tenant alone is 10,000 writes/s: ten partitions' worth of capacity behind a single partition key value, which DynamoDB will not give you. The same tenant at 1 KB per item and 50M items is 50 GB, five times the 10 GB per-partition limit. So the key becomes TENANT#<hash(user_id) % 16>: the tenant spreads across 16 partitions, and whole-tenant reads fan out over those 16.

Monotonic keys depend on the store

Range-partitioned stores sort by key, so a timestamp or auto-increment ID sends every new write to the last tablet or region: Bigtable, HBase, and MongoDB with a range shard key all behave this way. Fix it with a reversed or hashed prefix. Hash-partitioned stores don't have this problem: DynamoDB hashes the partition key internally, and Cassandra's default Murmur3Partitioner (256 vnodes per node by default) scatters sequential keys. There the failure is a low-cardinality key or one hot value, not monotonicity.

Query alignment

Any query without the partition key becomes a scatter-gather across every shard. That's acceptable for an admin report and fatal on the hot path. If two access patterns both need to be fast and share no attribute, denormalize into a second table keyed differently rather than compromising the primary key.

Failure modes and how you see them

  • Hot partition: CloudWatch Contributor Insights for DynamoDB reports the most-accessed partition keys; Cassandra's nodetool toppartitions and nodetool tablehistograms show the same thing. Watch for one key taking a disproportionate share of traffic. DynamoDB adaptive capacity isolates frequently accessed items, but a single partition key value still cannot exceed the per-partition ceiling.
  • Scatter-gather creep: p99 rises with shard count because every such query touches all of them. Track queries-per-request against shard count.
  • Cross-partition transactions: largely unavailable. Entities that must change atomically belong in the same partition, and that constraint often overrides the distribution argument.
  • Key change: repartitioning means a full rewrite. Load-test the distribution before committing, not after.

Curated: · Written: · Reviewed: