Skip to content
Tech Interview Prep

Top 100 Solutions Architect Interview Questions and Answers

The questions most likely to come up in your Solutions Architect interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.

Curated: · Written: · Reviewed:

Reviewed 100Review pending 2

102 questions

Q&A-23What questions do you ask on the first call with a new customer before proposing anything?

The 90-second version

I ask four things in order (problem, constraints, decision, success) and I don't ask anything their website, their 10-K, or their job postings already answer. The order matters: problem first earns the right to ask about budget. Budget first ends the call.

QuestionWhat it actually revealsFollow-up that decides it
"What's happening today that shouldn't be?"The pain, in their words"What have you tried, and why did it stop?"
"Why now?"The forcing function"What date does that land on?"
"What's the budget range?"Whether this is funded at all"Is that capex, opex, or already allocated?"
"Who signs?"The economic buyer"Who else has to agree, and what do they care about?"
"Has this been proposed before?"Org history"What killed it?"
"How will you judge this in a year?"Success criteria"What number moves, and by how much?"

That last-but-one question is the most informative on the call. A previous attempt that failed tells you about the organisation, and the reason it failed is usually still present.

Structured or open: pick by who's in the room

MEDDIC (Metrics, Economic buyer, Decision criteria, Decision process, Identify pain, Champion) and BANT (Budget, Authority, Need, Timeline) are the two frameworks people reach for. They're checklists, and a checklist run at the wrong person produces rehearsed answers.

  • Peer engineer on the call, buyer absent. Open conversation first. Ask what they escalated last month, what broke in the last release. Structure comes second, once you know who to point it at.
  • Pre-Series A, no budget line. BANT's budget question is noise. Ask the forcing function instead: "what has to be true by the end of the quarter?"
  • Existing customer, expansion. Skip "why now." You have the history; ask what changed since the last renewal.
  • Enterprise, procurement involved. Structure wins outright. If you can't name the economic buyer by the end of the call, you've had a friendly chat, not a discovery.

A worked trace (hypothetical)

Payments company, ~400 engineers, on-prem Postgres inside PCI DSS scope. They open with "we want a managed database."

  • Why now → an audit finding lands in March. Today is January: eight weeks.
  • Budget → "around $150k/yr opex." Funded, and opex, not capex, so no procurement cycle.
  • Who signs → VP Engineering signs; security has veto.
  • Previous attempt → a 2023 migration to a cloud provider stalled at the PCI scope re-draw and died after nine months.
  • Success → p99 write latency from 40 ms to under 15 ms, and DBA on-call pages halved.

The arithmetic kills the obvious proposal. Eight weeks does not fit a scope re-draw plus a migration, and the last attempt proves it. The first proposal is two phases: re-draw scope now, migrate after the audit.

Failure modes

Asking budget in the first ten minutes gets you an anchored-low number or a refusal. "What keeps you up at night?" gets a rehearsed answer: ask what they escalated last month instead. A champion who isn't the economic buyer leaves you with a fan and no signature. And a vague answer to the one-year question is a dispute waiting at the end; fix it before you propose anything.

I also ask what they're worried about. People describe risks more candidly than requirements, and that answer reshapes the proposal more than anything in the formal brief.

Curated: · Written: · Reviewed:

Q&A-1A customer says they need a system that is 'fast, secure, and scalable'. How do you turn that into an architecture?

Those three words are not requirements, they are the absence of requirements, and accepting them is how a project ends up rebuilt in year two. The job is to convert each into something with a number and a consequence.

Fast becomes a latency budget attached to a named user action: the search results page renders in under 400ms at p95 for users in their primary market. That is testable, and it immediately raises the questions that matter: measured from where, at what concurrency, with how much data.

Secure becomes a threat model and a compliance boundary: who are we defending against, what data classes exist, which regulations apply, what does an auditor need to see. A customer in payments and a customer running an internal tool both say secure and mean entirely different budgets.

Scalable becomes a growth curve with a horizon: 2,000 orders a day now, 20,000 expected within eighteen months, with a seasonal peak of four times the daily average in one week of the year. That last figure usually changes the design more than the average does.

The technique that gets these out of people is to ask what would make this a failure rather than what they want. Customers describe aspirations vaguely and disasters precisely.

Then write the numbers down and get them signed off, because every one of them is a constraint you will be held to, and the ones you inferred silently are the ones that will be disputed.

Curated: · Written: · Reviewed:

Q&A-7A customer's stated requirement conflicts with what you believe they actually need. How do you handle it?

Assume first that they know something you do not. Stated requirements that look wrong are often carrying a constraint nobody mentioned: a regulator's expectation, a commitment made to a client, a political reality inside the organisation. Ask why the requirement exists before arguing with it.

If, having asked, the requirement still looks like it will not serve them, the useful move is to separate the requirement from the outcome. They asked for a real-time dashboard; what they need is to know within the working day whether a batch failed. Those have very different costs, and once the outcome is on the table the conversation is about how to achieve it rather than about who is right.

Where the conflict is genuine, put both options in front of them with the consequences priced. Not an argument, a comparison: this is what you asked for, here is what it costs and what it constrains later; here is the alternative, cheaper by this much, with this limitation. Then let them decide, because it is their system and their budget.

Record the decision either way, including the case you made. If they choose the option you advised against, that record protects the relationship later: it is a documented decision rather than a surprise.

The one place I would not simply defer is a requirement that creates a security, privacy, or data-integrity problem. Those get raised explicitly, in writing, because the consequence lands on their customers rather than only on them.

Curated: · Written: · Reviewed:

Q&A-74How do you specify non-functional requirements so they can actually be tested at acceptance?

A testable non-functional requirement is a sentence with five parts: the metric, the threshold, the conditions, the measurement point, and who runs it. Drop any one and acceptance becomes an argument about interpretation rather than a pass/fail.

  • Metric: what is counted: p95 latency, error rate, recovery time. Not "responsiveness".
  • Threshold: the number and the comparison: ≤ 800 ms, ≥ 99.95%.
  • Conditions: load, data volume, duration, and system state (cold cache, warm, mid-failover).
  • Measurement point: load balancer, application log, client, synthetic probe. These differ by tens of milliseconds.
  • Owner and environment, who executes, where, against which dataset.
NFRUntestable formTestable form
Latency"search is fast"p95 ≤ 800 ms at the load balancer, 200 concurrent users, 50M-row orders table, 30-min steady state
Availability"highly available"≥ 99.95% of 5-minute probes succeed per calendar month; planned maintenance excluded and capped at 4 h/quarter
Throughput"handles peak load"sustains 250 req/s for 60 min with p99 ≤ 1.5 s; 3× headroom to 750 req/s without redesign
Recovery"RTO/RPO met"restore from backup in ≤ 30 min with ≤ 5 min data loss, demonstrated in a quarterly drill, clock started at incident declaration
Security"secure"TLS 1.3 in transit, AES-256 at rest, quarterly access review, pen test with all high findings remediated within 30 days

Worked example: the concurrency figure is where most specs break. "200 concurrent users at 800 ms" implies 200 / 0.8 = 250 req/s by Little's law (L = λW). If the business case needs 500 req/s, the spec is already wrong before anyone writes code. Fix the number in the requirement, not at acceptance.

Availability arithmetic. 99.9% of a 30-day month is 43 min 12 s of allowed downtime; 99.95% is 21 min 36 s; 99.99% is 4 min 19 s. Writing "99.9%" without the window and the probe interval leaves the measurement undefined: a 5-minute probe interval cannot detect a 2-minute outage, so the target is unfalsifiable at that granularity.

Failure modes worth settling in the statement of work:

  • Environment scale. Queueing is non-linear: for a single-server queue, mean wait scales as ρ/(1−ρ), so moving from 50% to 90% utilization multiplies waiting time by 9. A test environment at a tenth of production capacity does not validate a latency threshold; it validates a different system.
  • Percentile aggregation. A p95 computed per minute and then averaged is not the p95 of the run. Pin the aggregation rule in the spec.
  • Data volume and shape. Latency against an empty table proves nothing; specify row counts and index selectivity.
  • Assertion vs demonstration. RTO/RPO and failover requirements are accepted from a drill with timestamps, not from a design document.
  • Drift. Pin the build and configuration under test; a threshold met on last quarter's build is not evidence for this one.

Close with the acceptance clause: who runs the test, in which environment, with which dataset, and what happens when the number is missed. Retest window, remediation, or a documented waiver. That clause is what makes the requirement enforceable.

Curated: · Written: · Reviewed:

Q&A-6You are presenting an architecture to a room containing a CTO, a finance director, and two engineers. How do you structure it?

Lead with the decision you want, not with the design. Everyone in that room is there to approve, fund, or object, and burying the ask behind twenty slides of topology means the finance director disengages before you reach it.

Structure it as: here is the problem in business terms, here is what we propose, here is what it costs, here is what it buys, here are the two risks and what we would do about them. Then the technical detail, for the people who want it.

Address each audience's actual question. The CTO wants to know whether this is a sensible bet and what it commits them to. The finance director wants the number, its confidence, and when it is spent. The engineers want to know whether it will work and whether you have thought about the parts that are hard.

The failure that loses credibility fastest is answering a business question with technical detail. If someone asks whether it will be ready by March, the answer is a date and a confidence, not a description of the pipeline.

Diagrams should be layered: one that fits on a slide and shows five boxes, and detailed ones in the appendix for the people who will read them.

And be honest about uncertainty. Stating that the cost estimate is plus or minus thirty percent until the integration is scoped builds more trust than a precise number that turns out wrong, and it is the engineers in the room who will notice which one you did.

Curated: · Written: · Reviewed:

Q&A-27What makes an architecture diagram useful to a customer rather than decorative?

The 90-second answer

A diagram is useful when it changes what the customer does next: approves the design, clears the security review, funds the next phase, or asks one specific question. Decorative diagrams show the system; useful ones answer a single decision the reader owns, at the abstraction level they can act on, and are honest about what is still open.

So name two things before drawing: the reader and the decision. "The customer's CISO deciding whether to approve cross-border data flow" and "the customer's finance director deciding whether the run cost fits the budget" need different pictures. If you cannot name the decision, you are drawing decoration.

One diagram per decision, not per system

ViewReaderDecision it supportsSizeMust show
ContextCustomer exec, financeApprove scope and cost envelope5–8 boxesSystem, users, external systems, trust boundary
ContainerCustomer's architects, securityApprove integration and data flow8–15 boxesDeployables, protocols, sync vs async, data stores
DetailOne named reviewerApprove a single mechanism (auth, key rotation)one flowCall sequence, failure path

The C4 model (Simon Brown) names these levels; you don't need the tooling to use the discipline.

Worked example (hypothetical)

A customer's security team has to approve an integration with your SaaS. First draft: 40 boxes, every internal microservice, unlabelled arrows. The review stalls: the security lead asks "where does our data leave our VPC?" and nobody can point at it.

Redraw as one context view: the customer's IdP, the customer's VPC, your API gateway, your region, plus a dashed box for the undecided "EU-only processing" option. Six boxes. Arrows labelled "OIDC token exchange, synchronous, TLS 1.3" and "nightly batch export to S3, customer-managed key". The trust boundary is a labelled line where the VPC ends. The review clears in one meeting because the question they arrived with is answerable on the page.

Failure modes and how you catch them

  • The everything diagram. Detection: the reader asks "what am I looking at?" or squints at a projector. Fix: split by decision.
  • Unlabelled arrows. Detection: two engineers in the room disagree about whether a flow is synchronous. Arrow direction alone hides the property that drives reliability and cost.
  • A confident box for an undecided component. Detection: it turns up in an SOW or a contract. Draw it dashed, labelled "open: decision needed by <date>".
  • The undated diagram. Detection: someone quotes a two-year-old picture as a commitment. Put version, date and owner in the title block.

Trade-offs

Multiple views cost maintenance. Keep one source: diagram-as-code (Structurizr DSL, PlantUML, Mermaid) checked into the repo beside the code, so a PR that changes a component forces the diagram to change. Slides in a shared drive drift within a quarter. If the customer only ever needs the context view, don't build the rest. And be deliberate about showing internals: a detailed view of your own components can read as a commitment about how they will stay, so show detail only where the customer has a decision to make about it.

Curated: · Written: · Reviewed:

Q&A-51How do you make sure a design review with the customer produces decisions rather than discussion?

Send the material in advance, state the decisions being asked for, and make sure the people who can make them are in the room.

The most common failure is a review where the design is presented for the first time. The audience spends the session absorbing it, asks clarifying questions, and leaves with an action to think about it, which means another session and a two-week delay. Circulating a document beforehand, with the specific decisions listed at the top, converts the session into a decision meeting.

Name the decisions explicitly: we are asking you to confirm the availability target, approve the integration approach with the billing system, and choose between these two options for reporting. Three named decisions with their consequences priced is a productive hour. "Here is the architecture, thoughts?" is not.

Check attendance against the decisions. If the security team must approve the trust boundaries and they are not there, that decision will not be made regardless of how well it is presented, and it is better to know that beforehand.

Time-box the discussion of each and be willing to park what cannot be resolved, with an owner and a date. A single unresolvable point can consume a session otherwise.

Record decisions in the room and circulate them the same day, including what was decided against and why. Undocumented decisions get relitigated, usually by whoever was absent, and the reasoning is what stops that from being a fresh argument each time.

Curated: · Written: · Reviewed:

Q&A-95What makes a good architecture decision record for a customer engagement specifically?

The same structure as any decision record, plus two things that matter more when the reader will be someone else's team in two years.

The standard content: the context and the forces in play, the options considered including the rejected ones, the decision, and its consequences stated honestly including what became harder. The rejected options are the part that is impossible to reconstruct later and the part most often omitted.

The first addition for a customer engagement is the constraint's source. "We chose this because the customer's security policy prohibits X" or "because their contract with vendor Y runs until 2028" is essential, because those constraints expire. A future engineer finding a decision with no stated source will either preserve a constraint that no longer exists or overturn one that still does.

The second is the reversal condition, written specifically: what would have to change for this to be worth revisiting. That converts the record from a historical note into something actionable, and it is what makes a decision defensible when a new team questions it.

Practically: keep them in the customer's repository rather than in your own systems, because they must outlive the engagement. Write them at the time rather than reconstructing them at handover, when the reasoning has already faded. Keep them to a page. And make them immutable, superseded by a new record rather than edited, so the history of the reasoning survives.

The test is whether a new engineer's "why is it like this?" has an answer that is not a person who has left.

Curated: · Written: · Reviewed:

Q&A-21A stakeholder challenges your design in front of the room with an objection you had not considered. What do you do?

Say that you had not considered it, and think about it out loud. The instinct is to defend, and defending an objection you have not evaluated is how a design review turns into a contest that the design loses either way.

Then establish what kind of objection it is. If it is factual (a constraint you did not know about, a system you were unaware of) the design may need to change and the useful next step is to capture it and say when you will come back with an answer. If it is a difference in judgement about a trade-off, put both sides on the table and identify what evidence would settle it.

What not to do: accept it immediately to avoid conflict, which is as damaging as defending badly. A design that changes with whoever spoke last is not a design, and the room notices.

Take it offline when the answer needs work, but be specific: what you will look at, and when you will report back. "Let me think about it" without a commitment reads as dismissal.

Being visibly comfortable with not knowing is what builds credibility in this role. The customer is deciding whether to trust your judgement on the hundred decisions they will not see, and someone who defends everything is someone whose confidence carries no information.

Afterward, ask why it was not raised earlier. A significant objection surfacing at the review usually means the wrong people were in the earlier conversations.

Curated: · Written: · Reviewed:

Q&A-28How do you approach a customer whose real constraint is political rather than technical?

Recognise it explicitly rather than designing around it silently, and work with the person who owns the constraint rather than over them.

The signals are familiar: a requirement with no technical justification that will not move, a stakeholder who blocks without engaging on the merits, or two groups whose designs are both defensible and mutually exclusive. Continuing to make technical arguments in that situation is the most common way this role wastes months.

What usually helps is finding the underlying interest. A team insisting on owning a component may be protecting headcount, or may have been burned by a dependency before. A director rejecting a platform may have committed publicly to a different one. Those are addressable: often by changing how the work is framed, who presents it, or how ownership is distributed, while the stated objection is not.

Give people a way to change position without losing face. New information is the usual mechanism: a load test, a cost model, a pilot result. It lets someone move because the facts changed rather than because they were wrong.

Escalate carefully and rarely. Going over someone converts a disagreement into an enemy, and this role depends on relationships that outlast the project.

And be honest with your own sponsor about what is happening. A schedule slipping because a decision is blocked internally is information they can act on; a schedule slipping for reasons nobody names is one they cannot.

Curated: · Written: · Reviewed:

Q&A-52A customer wants to know what happens if you get hit by a bus. How do you answer?

What I say in the room

"The answer isn't a promise, it's a demonstration, and you can check it this afternoon."

Then I show four things, in this order, because each is verifiable in under five minutes:

ClaimWhere they check it
The environment rebuilds without meterraform plan from a clean clone against remote state in their S3 bucket; CI runs it on every PR
Decisions are recorded, not remembereddocs/adr/: numbered ADRs, each with status and the alternatives rejected
Ops knowledge is transferablerunbooks in the repo next to the code, each headed Last executed by / date
Two people per areaCODEOWNERS: every path maps to at least two handles, one from each side

A continuity clause in the contract is weaker than this, because it's a promise about a future state. The repo is evidence about the present.

The drill that makes it real

Week 4, I block a week in the calendar and stop answering Slack. Their engineer picks a P2 off the backlog:

  • Monday: reads ADR-014, runs runbooks/rotate-db-credentials.md, hits a permission error on step 3.
  • Tuesday: escalates to my second, who fixes the runbook and commits the fix.
  • Wednesday: the change ships through the pipeline; no local credentials, no SSH key from anyone's laptop.
  • Friday: I read the merged PR from home.

If instead they'd had to call me, that's the finding, and it costs a week to fix in month 4 versus a quarter in month 9.

What "no" actually looks like

When the answer is really no, it's usually one of these, and each has a cheap detector:

  • terraform.tfstate sitting on someone's laptop → a fresh clone can't plan. Detect by running the pipeline in a clean container.
  • A runbook with no execution record → have the counterpart run it in staging, timed, and watch where they stall.
  • CODEOWNERS entries with a single owner → a script that lists paths with one handle.
  • Root or break-glass credentials in one person's password manager → pull the IAM credential report; move them into the customer's own account with two holders.

The reciprocal question

It's worth taking the bus question at face value as key-person risk generally, including on their side. I ask who the only person is who understands the nightly settlement batch job. If the answer is one name, the handover plan should cover that too: customers recognise the pattern the moment it's named.

And if the honest answer is that the project does depend on me, I say so. That's a finding to act on, not something to reassure away: it means knowledge transfer hasn't been happening, and the fix is a calendar block and a drill, not a paragraph in a contract.

Curated: · Written: · Reviewed:

Q&A-5Two systems need to exchange data. How do you choose between a synchronous API, asynchronous messaging, batch, and webhooks?

Decide from three properties: whether the caller needs the result to continue, how fresh the data must be, and how much coupling in availability is acceptable.

A synchronous API fits when the caller cannot proceed without the answer: a credit check during checkout, an address validation. The cost is that the caller's availability becomes the product of both systems, and its latency includes the callee's. Anything in a user-facing path needs a timeout and a defined behavior when the call fails.

Asynchronous messaging fits when the second system needs to know something happened but the first does not need its response. It decouples availability: the consumer can be down and catch up. The cost is eventual consistency the user may see, and the operational burden of a broker.

Batch fits when the freshness requirement is measured in hours, when volumes are large, or when the other side is a legacy system that offers a nightly file and nothing else. It is unfashionable and frequently correct: a nightly reconciliation is simpler and more debuggable than a stream nobody needed.

Webhooks fit when the other party owns the event and you want push rather than poll. Assume at-least-once delivery, verify signatures, persist the event, then acknowledge quickly so the sender is not left waiting on your downstream work.

For integration with a customer's existing estate, the deciding constraint is usually what the other system can actually do, not what you would prefer. Design to their real capability, and be explicit when it forces a compromise.

Curated: · Written: · Reviewed:

Q&A-98How do you assess whether a customer's proposed integration will scale to their projected volume?

By finding the constraint at the far end, because the bottleneck in an integration is almost always the other system rather than yours.

The measurements to get: what rate does their system actually sustain, what are its documented and undocumented limits, and how does its latency behave as concurrency rises. Call it and find out rather than reading the documentation, because published limits and observed behavior diverge often, particularly with older enterprise systems.

Then do the arithmetic against the projection, and specifically against the peak rather than the average. A daily volume divided by 86,400 seconds is a meaningless number if the traffic arrives in a two-hour window; the peak rate is what has to fit.

Then check for the shapes that break at volume even when the rate fits: an integration that makes one call per record where a batch endpoint exists, a synchronous call in a loop, or pagination that gets slower with depth. Each of these works fine at pilot volume and fails at production volume, which is why pilots are poor evidence for this question.

If the constraint binds, the options are to negotiate a higher limit, which is often possible and worth asking; to reshape the integration: batch instead of per-record, or a bulk file for the initial load; or to decouple with a queue so the peak is smoothed to a rate the far end can absorb.

And state the ceiling explicitly in the design, so the customer knows at what volume this needs revisiting rather than discovering it during a busy period.

Curated: · Written: · Reviewed:

Q&A-83How do you decide whether a customer needs an API gateway?

By whether there are cross-cutting concerns that would otherwise be implemented in every service, and by whether the consumers are external.

For a single internal service with one known consumer, a gateway is a hop and a component for no benefit, and I would not recommend one.

For external or partner consumers, it earns its place quickly. Authentication, per-consumer rate limiting, request validation, usage metering, and a stable public contract that does not expose internal service boundaries are all things that otherwise get reimplemented per service: inconsistently, and the inconsistencies are where the gaps appear. Centralising them also means a policy change happens once.

The other case is a customer with several services and a public surface, where the gateway provides routing and versioning independent of how the services are arranged internally, so the internal structure can change without breaking consumers.

The cautions I would state. It is on every request path, so its availability becomes theirs, and it needs the same care as any production dependency. And it accumulates logic: teams put transformation and business rules there because it is convenient, and it becomes an untested tier that nobody owns. The rule I would set is cross-cutting policy only: anything domain-specific belongs in a service where it can be tested.

Also worth checking what they already have. A load balancer with authentication and rate limiting, or a CDN with edge rules, may cover the requirement without another component to operate.

Curated: · Written: · Reviewed:

Q&A-46A customer wants an API for their partners. What do you specify beyond the endpoints?

What I'd specify, in the order it decides whether partners ship.

Assume external partners, multi-tenant, who can't read your code or your Slack, and a contract that has to survive you changing the implementation. The endpoints are the smallest part of it.

1. Auth and onboarding. OAuth 2.0 client credentials (RFC 6749 §4.4) with a client ID per partner, mTLS or a signed JWT for the confidential case (RFC 8705, RFC 9068), and a revocation path (RFC 7009). The onboarding SLA matters more than the protocol: if issuing credentials takes two weeks of manual steps, that is the partner programme's throughput ceiling, not a technical detail.

2. Rate limits, stated and enforced. Publish the limit, return it on every response, and return 429 with Retry-After when exceeded. The IETF draft draft-ietf-httpapi-ratelimit-headers standardises RateLimit-Limit / RateLimit-Remaining / RateLimit-Reset; X-RateLimit-* is the de facto alternative. Use per-partner buckets, not one global bucket, or one noisy partner starves the rest.

3. Errors. One shape, application/problem+json per RFC 9457 (which obsoletes RFC 7807), stable machine-readable type codes, and no prose the partner has to interpret. Every 4xx should tell the partner what to change.

4. Idempotency. Any POST that mutates takes an Idempotency-Key header (IETF draft; Stripe's convention is the common reference). Retries are normal on flaky networks; without this, a timeout becomes a duplicate charge.

5. Versioning and deprecation. Pick URL-path (/v1) or media-type versioning and write down the support window. Announce with Deprecation and Sunset headers (RFC 8594 for Sunset; Deprecation is still an IETF draft). You cannot retire a version whose usage you cannot count, so per-version request metrics are part of the contract.

6. Sandbox, docs, ops. A sandbox with realistic data and a reset, an OpenAPI 3.1 description as the source of truth, a status page, an SLA, and a named owner for change communication.

AreaConcrete decisionCost of omitting
AuthOAuth 2.0 client credentials, per-partner client, revocation endpointShared keys; can't revoke one partner
LimitsPer-partner bucket, RateLimit-* headers, 429 + Retry-AfterPartners discover limits by failing
ErrorsRFC 9457 problem+json, stable type URIsA support ticket per error
IdempotencyIdempotency-Key on all mutating POSTsDuplicate writes on retry
Versioning/v1 path, 12-month support window, Sunset headerVersions accumulate, none retire
Ops99.9% SLA, status page, support pathOutages become escalations

Arithmetic: 99.9% over a 30-day month is 43 min 12 s of allowed unavailability; 99.95% is 21 min 36 s. Decide explicitly whether scheduled maintenance counts: that one sentence changes the number by an order of magnitude in practice.

Rate-limit sizing: 600 requests/min per partner is 10 rps. A partner running 50 workers at 200 ms p99 generates 250 rps and will hit it, so the headers and Retry-After are what let them back off instead of opening a ticket.

# pseudo-code, not a library API
def create_payment(idem_key, body):
    cached = store.get(idem_key)        # (status, response) or None
    if cached:
        return cached                   # replay, no side effect
    if not store.claim(idem_key):       # SETNX, 24h TTL
        return 409, problem("idempotency_in_progress")
    result = charge(body)
    store.put(idem_key, (201, result), ttl=86400)
    return 201, result

Trace: first call with key K → get misses → claim succeeds → charge runs once → stored → 201. Second call with K → get hits → returns the stored 201 and body, charge not called. A concurrent second call while the first is in flight → claim fails → 409 problem+json with Retry-After: 1.

Failure modes and how they show up:

  • No idempotency: duplicate charges after client timeouts; detect by a reconciliation count mismatch between your ledger and the partner's.
  • Global rate limit: one partner's batch job 429s everyone; detect by per-key 429 rate.
  • No per-version metrics: v1 can't be retired; detect by counting requests per version at the gateway.
  • Unsigned webhooks: spoofed events; detect by the share of deliveries failing signature verification.

Trade-offs: path versioning is visible in access logs and trivially routable, header versioning keeps URLs stable but is invisible in logs and easy to forget. Webhooks are cheaper than polling but need signatures, retries and a replay endpoint. Per-partner quotas cost a counter per key and buy fairness.

Curated: · Written: · Reviewed:

Q&A-84The customer's team asks why you are proposing message queues when direct calls are simpler. How do you justify it?

By naming the specific failure that direct calls produce in their case, and by conceding the simplicity point, because they are right that it is simpler.

The justification has to be concrete. If their order service calls the fulfilment system synchronously and fulfilment is down for an hour, orders fail: the customer cannot buy, and the business loses revenue for a failure in a system that is not on the critical path of taking money. A queue changes that outcome to a backlog that drains when fulfilment returns. That is the whole argument, and it is a business argument rather than an architectural preference.

The second justification, where it applies, is load smoothing: a spike that would overwhelm a downstream system becomes a queue that drains at whatever rate that system can sustain.

Then be honest about what it costs, because a team that only hears the benefits will resent the consequences. Eventual consistency the user may notice, so the interface has to represent in-progress states. A broker to operate. Duplicate delivery, so consumers must be idempotent. Harder debugging, since there is no stack trace across the boundary and correlation IDs become necessary rather than optional. And a dead letter queue that someone has to watch, or failures accumulate invisibly.

Given that, I would use queues only where the decoupling earns those costs: typically at boundaries where a downstream failure should not fail the user's action, and keep direct calls everywhere the caller genuinely needs the result.

Curated: · Written: · Reviewed:

Q&A-39How do you explain the consistency trade-off to a business stakeholder who is deciding between two designs?

In their scenario, with their consequence, and without the theory.

Not "eventual consistency" but: "if the link between our two sites breaks, we can either keep taking orders and risk two customers buying the last item, or stop taking orders until it is fixed. Which is worse for you?" That is the actual decision, it is theirs to make, and stated that way most stakeholders answer immediately and correctly for their business.

The answer usually varies by operation within the same system, which is the more useful insight to give them. Showing an approximate stock level on a product page can be stale; decrementing inventory at checkout cannot. Framing it per-operation avoids a single system-wide choice that is wrong for half the cases.

Quantify the window so it is not abstract. "Under normal conditions the two sites agree within a second; the case we are discussing is the rare failure where that could stretch to minutes" turns an alarming-sounding property into a bounded risk.

Then name the cost of the stronger option in terms they feel: not milliseconds of coordination latency, but that the checkout takes noticeably longer for every customer, every day, to protect against a failure that happens twice a year.

And write the decision down with the reasoning. This is precisely the choice that gets revisited in an incident, and having a record that the trade was made deliberately, and by whom, is what keeps that conversation productive.

Curated: · Written: · Reviewed:

Q&A-60How do you decide whether a customer's workload should use a relational database or something else?

What I say first (60–120 seconds)

Three questions, in this order.

  1. What is the unit of access? If the application reads and writes a whole aggregate, an order with its lines, and rarely queries across aggregates, a document store is a candidate. If it joins across entities, relational.
  2. What must the database enforce? Foreign keys, uniqueness, check constraints and multi-row atomicity are things a relational engine enforces for you. Move to a store without them and you reimplement them in application code, where a second writer or a bug bypasses them silently.
  3. What is the read/write shape and volume? Point lookups by key, append-only time series, and relevance-ranked text each have a store that beats a B-tree at that one shape.

Default: relational. Not because it wins at any single shape, but because the second year of requirements is unspecified, and a new query is a new SELECT rather than a migration. Postgres 16 covers three of the "something else" cases in the same engine: jsonb with a GIN index for document-shaped data, tsvector with a GIN index for full-text ranking, and TimescaleDB for time series, so "we need a document store" often means "we need a jsonb column."

Workload shapeStoreThe specific thing that decides it
Aggregate read/written whole, shape variesDocument (MongoDB, Postgres jsonb)MongoDB caps a document at 16 MB; if your aggregate exceeds that, the aggregate is wrong
Point lookups by key, no cross-key queriesKey-value (Redis, DynamoDB)DynamoDB caps an item at 400 KB and a partition at 1000 WCU / 3000 RCU
Relevance-ranked textSearch index (Elasticsearch/OpenSearch)Postgres tsvector + GIN gives ranking and phrase queries; it does not give you sharding or per-field BM25 tuning
Append-only metrics, time-window queriesTime-series (TimescaleDB, InfluxDB)Retention and downsampling are first-class; a B-tree on (series, ts) works until the table outgrows one node
Variable-depth traversalGraph (Neo4j, Neptune)A recursive CTE handles fixed depth; variable depth with cycles needs cycle detection and blows up

Where the relational default stops holding

Postgres 16 defaults max_connections to 100; MySQL 8.0 defaults to 151. Each Postgres connection is a backend process, so the ceiling is roughly connections ÷ query time: 100 connections at a 5 ms p50 is a theoretical 20,000 queries/s, and you never reach it because the tail queues behind the head. The signal to move a workload off the primary is not a QPS number: it is p99 crossing the SLO while CPU sits under 60%, which means you are waiting on locks or I/O, not compute. PgBouncer in transaction mode raises the connection ceiling before a second database is warranted.

Failure modes

  • Dual writes. Writing to Postgres and Elasticsearch in the same request handler loses updates when one succeeds and the other fails. Fix with a transactional outbox plus CDC (Debezium) or logical replication. Detect by reconciling row counts against document counts on a schedule.
  • Schema drift in a document store. Nothing rejects a document missing a required field. Detect with a query for documents where the field is absent, run as a monitor, not a one-off.
  • Choosing for anticipated scale. The second technology's operational cost starts on day one: on-call, backups, upgrades, a second consistency model. A well-indexed relational primary handles more than most customers estimate.

Two things I push back on

Choosing a document store to avoid designing a schema: that moves the schema into application code where nothing enforces it. Postgres keeps it in the database:

-- Postgres 16
CREATE TABLE events (
  id      bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  payload jsonb NOT NULL,
  CONSTRAINT payload_has_type CHECK (payload ? 'type')
);
CREATE INDEX events_payload_gin ON events USING gin (payload jsonb_path_ops);

jsonb_path_ops indexes containment (@>) only; it is smaller and faster than the default jsonb_ops when that is all you query.

And choosing for anticipated scale. Most of these stores are additions, not replacements: the usual right answer is a relational system of record with one specialised store alongside it for the workload that needs it, fed asynchronously.

Curated: · Written: · Reviewed:

Q&A-56A customer asks whether they should put a cache in front of their database. What do you establish before answering?

What problem it is meant to solve, because a cache is a permanent operational commitment being proposed as a performance fix, and half the time the underlying problem is cheaper to fix directly.

Establish first whether the slowness is a small number of queries. If it is, indexing or a rewrite removes the need entirely, for days of work rather than a new component with a staleness contract and an invalidation problem.

If a cache is warranted, the questions the customer has to answer are business questions rather than technical ones. How stale may this data be, per data type, and who accepts that? Which writes must invalidate it, including the ones that happen outside the application: a batch job, an admin tool, a support person editing a record directly? What should users see if the cache is unavailable?

That last one is the question that most often changes the design. If the database cannot serve full traffic without the cache, the cache is a single point of failure and its own availability now sets theirs. The number to establish is what the origin can serve at a zero hit rate, and it should be measured rather than assumed.

Then the multi-tenant trap, which is worth raising explicitly for any customer with distinct client organisations: a cache key that omits the tenant serves one client's data to another, silently, for as long as the entry lives.

If they cannot answer the staleness and invalidation questions, the cache is premature and I would say so.

Curated: · Written: · Reviewed:

Q&A-57How do you explain to a customer why their reporting queries should not run against the production database?

In terms of the risk to the thing they care about, with a concrete mechanism rather than a principle.

The mechanism: reporting queries are large, unpredictable, and written by people optimising for the answer rather than for the cost. One of them scanning a large table consumes the same memory and IO the transactional workload needs, and the effect is not that the report is slow: it is that checkout is slow, during business hours, for reasons that are hard to attribute because the report looks unrelated.

It also creates a coupling nobody intends. A query someone wrote for a monthly board pack becomes load-bearing, and then the schema cannot change without breaking a report that no engineer knows exists. That is a real constraint on their ability to evolve the system, and it accrues quietly.

The remedy is proportionate to their situation. For most customers a read replica is enough: cheap, straightforward, and it removes the contention entirely. Where the reporting is heavy or the shapes are analytical, a separate store fed on a schedule suits the queries better and costs less to run than making the transactional database do both jobs.

One caution worth stating if they already use a replica for recovery: heavy reporting raises its replication lag, which quietly degrades the recovery objective they think they have. If the same replica is both the reporting target and the failover target, those two purposes are in conflict and they should know it.

Curated: · Written: · Reviewed:

Q&A-91The customer wants a single system of record but has three departments with conflicting data definitions. What do you do?

Treat it as a governance problem with a technical component, and resolve the definitions before building anything, because the conflict does not disappear when it is encoded.

The pattern is familiar: sales counts a customer at the point of signature, finance at first invoice, and support per contract, so three departments report three different customer counts and each is correct within its own definition. A single system with one definition makes two departments wrong.

The first step is to make the conflict explicit and specific: write the three definitions down side by side with worked examples. That alone frequently resolves it, because people have been using one word for three things without noticing.

Then the options. Agree a single canonical definition, which requires a decision-maker senior to all three and is the cleanest outcome when achievable. Or model the underlying reality more precisely, a party with several relationships, and derive each department's view from it, which is usually the better technical answer because each department keeps its correct number and they reconcile to the same base. Or accept separate systems with a documented mapping, which is the honest choice when the definitions are genuinely different business concepts rather than a disagreement.

What I would avoid is choosing a definition by default because a schema had to be written. That is how a system becomes untrusted: two departments see numbers that do not match their own, stop believing it, and keep their spreadsheets, which is the outcome the project was meant to eliminate.

Curated: · Written: · Reviewed:

Q&A-55How do you handle a situation where the customer's data quality is worse than anyone assumed?

The 90-second answer

Don't absorb it. Quantify it, split it into what the delivery team can fix and what only the business can decide, then re-baseline the plan with the customer in writing. Data quality is one of the most common causes of migration overrun, and the failure mode is rarely the bad data itself: it's discovering it at cutover with no paper trail and no decision-maker on record.

Quantify against the real data, not the demo dataset

One profiling query per table turns a vague concern into a table the customer can act on:

-- PostgreSQL 16
SELECT
  count(*)                                                AS rows,
  count(*) FILTER (WHERE customer_id IS NULL)             AS missing_customer,
  count(*) FILTER (WHERE email !~ '^[^@]+@[^@]+\.[^@]+$') AS bad_email,
  count(*) FILTER (WHERE updated_at > now())              AS future_timestamps
FROM staging.orders;

Figures below are from a representative migration, hypothetical:

ConstraintViolation rateOwnerFix
customer_id null3.4%delivery teamquarantine, backfill via legacy join
duplicate (customer_id, order_ref)6.1%businesspick a survivorship rule
country not ISO-316611%delivery teamnormalisation map
order references a deleted customer0.8%businessdecide the orphan policy

Compute is cheap; decisions are expensive

4.2M order rows at a 6.1% duplicate rate is ~256k rows. Deduplicating at 20k rows/s is about 13 seconds of compute. The expensive part is the survivorship rule, which of the two rows wins, and who signs off. That is a business decision with a named owner and a date, not a ticket the delivery team quietly closes. Escalate it early; a stalled decision blocks the migration just as hard as a missing feature.

Three options, with costs

  • Cleanse before migrating. Most work, best outcome, and it blocks on every business decision above.
  • Migrate as-is into a tolerant system. Defers the problem and usually makes it worse: the tolerance becomes permanent and the new system's reports inherit the garbage.
  • Migrate the clean subset, handle the remainder separately. Frequently the pragmatic answer: it unblocks the bulk of the migration while the messy tail gets its own process and owner.

Failure modes and how you detect them

  • Silent absorption. Detected at cutover, typically as a three-week slip with no explanation the customer will accept. Detect earlier with per-table row-count reconciliation on every batch.
  • Over-cleansing. Normalising USA/US/United States into one value destroys variation a downstream system depends on. Detect by diffing distinct-value counts before and after normalisation.
  • Tolerance becoming permanent. Detect by tracking the violation rate per constraint over time; if it isn't falling, the tolerant path became the destination.

Keep it visible

Put the checks into the pipeline: a quarantine table with reason codes, and a dashboard showing violation rate per constraint per batch. The state stays visible continuously instead of being rediscovered at cutover, and the customer sees the same numbers you do.

Curated: · Written: · Reviewed:

Q&A-40The customer's existing database is the bottleneck, but replacing it is out of scope. What options do you present?

First 60 seconds. "Bottleneck" is not a diagnosis. Before proposing anything I want the top queries by total time (pg_stat_statements on PostgreSQL, performance_schema/sys on MySQL) and whether the primary is pinned on CPU, write IOPS, lock waits, or connection count. The answer changes completely between those. Then I present options cheapest-first and say where each stops working.

1. Fix the queries. A small number of statements usually dominate. An index or a rewrite is days of work, not a project. Failure mode: an unused index costs write throughput and disk. Detect with pg_stat_user_indexes where idx_scan = 0 after a full workload cycle. This is always step one, because everything below is expensive.

2. Move reads off the primary. A read replica for reporting and analytics. Failure mode: replication lag. Detect with pg_stat_replication (replay_lag) or MySQL SHOW REPLICA STATUS (Seconds_Behind_Source). Two traps: routing read-your-writes traffic to the replica gives stale answers, and on PostgreSQL a long query on the replica can hold back replay. max_standby_streaming_delay defaults to 30s before that query is cancelled. Cost: a second instance plus cross-AZ traffic.

3. Cache the hot, tolerably stale reads. Redis or Memcached. Failure modes: a hot key expiring causes a stampede on the origin. Detect as a p99 spike plus an origin QPS spike at the TTL boundary; mitigate with jittered TTLs and single-flight. And Redis's default maxmemory-policy is noeviction, so once memory fills, writes fail with OOM rather than evicting. Set maxmemory and a policy deliberately.

4. Reduce the work reaching the primary. Batch writes, put write-heavy paths behind a queue (Kafka, SQS), archive cold rows out of hot tables. Failure mode: the queue becomes the backlog. Detect with consumer-group lag (kafka-consumer-groups --describe) or SQS ApproximateNumberOfMessagesVisible. If the queue is on the write path, the customer sees "saved" before it is durable; that is a product decision, not a side effect. Archiving breaks any query still scanning the old range: detect as a spike in errors on date-ranged queries.

5. Pool connections. PgBouncer's default default_pool_size is 20; when it saturates, clients queue. Detect with SHOW POOLS and cl_waiting > 0. Cheap, and often the real fix when the primary is fine but connection count is the ceiling.

6. Scale vertically. A configuration change, no schema work. On RDS, db.r6g.large (2 vCPU, 16 GiB) to db.r6g.4xlarge (16 vCPU, 128 GiB) is a real multiple. Failure mode: you hit the instance-class ceiling, and the change itself costs a failover. AWS documents Multi-AZ failover as typically 60–120 seconds.

7. Offload one workload at the boundary of scope. Move search to OpenSearch, or metrics to TimescaleDB, while the primary keeps the transactional core. This is the last option before "replace the database", which is out of scope.

Where it ends. Don't quote a made-up multiple. Give the customer a measurable horizon: "when the primary's CPU or write IOPS is pinned at the instance-class ceiling and replica lag exceeds your staleness budget under normal load, vertical scaling is done and the database itself has to change." That sentence makes the constraint their informed choice rather than a surprise.

Curated: · Written: · Reviewed:

Q&A-50The customer wants real-time reporting on operational data. What do you propose?

First, establish what real-time means to them, because the range of answers spans three orders of magnitude in cost. In my experience the requirement usually resolves to within a few minutes, and often to within the working day, once the underlying decision it supports is named.

Ask what action the report triggers and how quickly. A dashboard someone looks at each morning does not need streaming. An alert that dispatches an engineer does, but that is an alert rather than a report. A trading decision genuinely needs seconds. Same words, entirely different architectures.

For most cases the honest answer is a read replica plus scheduled aggregation, refreshed every few minutes. It is cheap, it is operable by an ordinary team, it does not touch the transactional path, and it covers a large share of stated real-time requirements.

Where the freshness requirement is genuinely seconds, propose change data capture into a store shaped for the queries, a stream into a columnar or search store, and be explicit about what that adds: a pipeline to operate, a consistency window to explain, and a materially higher run cost and skill requirement.

The mistake to avoid either way is reporting queries against the transactional primary. It is the cheapest thing to build and the most common cause of an outage six months later, when a report someone wrote grows past what the primary can absorb during business hours.

Whatever is chosen, agree the freshness as a number and show it on the dashboard, so users can see how current the data is rather than assuming.

Curated: · Written: · Reviewed:

Q&A-65A customer's peak traffic is 40 times their average, one day a year. How do you design for that?

By separating what must scale from what can be shed, and by not sizing the whole system for one day.

First establish the shape precisely: is it 40 times for an hour or for a minute, and which operations does it affect? Frequently the spike hits a narrow path (a product page, a queue join, a payment) while most of the system sees normal load. Designing the whole estate for the peak when a fifth of it needs to scale is how these projects become unaffordable.

Then pre-provision rather than relying on autoscaling for the event itself. Autoscaling responds in minutes and the spike arrives in seconds; on a known date, the capacity should be in place beforehand and the caches warm. Autoscaling remains useful for the recovery and for the unknown days.

Then design the overload behavior, because a 40 times estimate will be wrong in one direction or the other. A queue with admission control, a waiting room for the entry path, rate limiting at the edge, and a defined degraded mode (read-only, or a static page for the parts that cannot serve) turn an underestimate into a slow experience rather than an outage. Customers usually accept queuing far better than errors, and that is a product conversation worth having in advance.

Then load test to failure against representative data, not to confirm the estimate, and rehearse the day: who is watching, what the thresholds are, and who can pull a feature flag.

And check the dependencies. Their payment provider and their own suppliers need to know the date too.

Curated: · Written: · Reviewed:

Q&A-30How do you validate that a proposed design will actually meet the customer's latency requirement?

Turn the requirement into a budget with units, then replace every number in it with a measurement. A requirement you cannot decompose into hops you can measure is not yet a requirement. Then check the two shapes (sequential hops sum, fan-out takes the max) and load-test at the arrival rate the customer actually expects, not at whatever rate your generator happens to achieve.

Decompose into measurable hops

400ms p95 for a page, from a user on a 4G phone 800km from the nearest edge:

HopHow you get the number
Network RTT to edgeRIPE Atlas or ThousandEyes probes from the customer's cities; or great-circle distance × 1.5 ÷ 200,000 km/s
TCP + TLS1 RTT for TCP, 1 more for TLS 1.3 (2 for TLS 1.2)
Edge → originsame probe method, region to region
App processingserver span from OpenTelemetry
Each downstream callclient span, p95 not mean
QueryEXPLAIN ANALYZE (Postgres) against a generated dataset at production row counts

NYC→London is ~70ms RTT in practice. A new TLS 1.3 connection costs 2 RTTs, so ~140ms is gone before your code runs. That line kills budgets in the first meeting.

The two shapes

Sequential: means add exactly. Tails do not: the p95 of a five-hop chain is not the sum of the five p95s, so measure the chain rather than adding the parts.

Fan-out: the aggregate takes the max. For N independent parallel calls, P(all under the per-call p99) = 0.99^N. At N=10 that is 0.904, so 9.6% of requests blow the per-call p99 and the fan-out's p99 sits near a single call's p99.9. Width is expensive in the tail.

Load-test at the right arrival rate

Little's Law sizes it: 500 rps at 200ms means 100 requests in flight. Use an open-model generator, k6's constant-arrival-rate executor or wrk2, not a fixed-VU closed loop.

Failure modes: passing in the lab, failing in production

  • Coordinated omission. A closed-loop generator waits for each response, so when the server stalls it sends fewer requests and the stall never shows up in the latency numbers. You pass at p95 200ms; production at the real arrival rate shows p99 3s. Detect it by comparing achieved throughput to the target, if throughput sags mid-run, the percentiles are fiction.
  • Warm connections. Connection reuse hides the handshake. Segment RUM by first visit vs repeat; first visits pay 2 extra RTTs.
  • Server-side only. A 200ms server p95 can be a 2.5s LCP on a mid-tier Android over 4G. Lighthouse and WebPageTest are lab tools and cannot see your users; field data can: CrUX, or your own PerformanceObserver beacon.
  • Toy data. A query over 10k rows says nothing about 10M. Check EXPLAIN ANALYZE estimated vs actual rows and the buffer cache hit ratio.
  • Bimodal latency. A 100ms mean can hide 5% of requests at 4s from a GC pause or a lock. Read the histogram, not the mean.

What it costs

A design that passes in the lab and fails in production is found at launch, when the fix is a cache tier or moving compute to the edge: weeks, not a config change. If the budget does not close on paper, raise it before the contract, not during acceptance testing. Requirements are frequently negotiable once the cost of meeting them is concrete.

Curated: · Written: · Reviewed:

Q&A-58A customer's checkout occasionally double-charges. They want to know how the architecture prevents it. What do you tell them?

That preventing it is a property of the payment path's design rather than of care taken by developers, and then show them where the property comes from.

The mechanism is an idempotency key. The client generates one identifier for the logical purchase, not per attempt, and sends it with the request. The server keeps that identifier under a uniqueness constraint so a retry is recognised as the same purchase rather than a new one.

What I would not tell them is that the key and the charge simply go in one database transaction. That is true only when "charge" means a row they own. In checkout the money is taken by a processor they do not own, and a local commit cannot stand in for that processor's own record. Leaving a transaction open while waiting on that call is the wrong shape as well: they pay the processor's latency on a held connection, and a crash after the processor has accepted the payment still has no local row that prevents a second attempt.

So the path I would show them is: persist the key as soon as the request arrives, pass that same key through to the processor, then record what the processor returned. If the process dies in the gap, the retry presents the same key and the processor returns the original payment instead of taking the card again. If two clicks arrive together, one of them must own the key and the other must wait for its result: a uniqueness error sent back as a failure is how the shopper tries again under a new key and is charged twice.

Then the honest framing of why retries happen at all: the client cannot tell a lost request from a lost response, so it must retry, and the payment provider does the same to you. At-least-once behavior is not a bug to be eliminated but a property to be designed for.

I would also ask what their current symptom actually is, because double-charging sometimes has a simpler surface: a button with no client-side guard, or a reconciliation job re-submitting settled transactions. Client-side debounce helps the shopper; it does not replace the server-side key. Reconciliation needs the same identifier, or a uniqueness check on the settlement, or it will create the duplicates the architecture was meant to prevent.

Curated: · Written: · Reviewed:

Q&A-82A customer asks for a system that never loses data. How do you respond?

Take the requirement seriously, then make it precise, because as stated it is unachievable and the useful conversation is about which loss modes matter.

The precise version has three parts. Durability: once we confirm receipt, the record survives hardware failure. That is achievable with synchronous replication and tested backups. Recovery point: after a disaster, how much recently-written data may be lost, which is the RPO and is a cost decision. And integrity: the record cannot be silently altered or lost through a software fault, which is a different problem entirely and is where most real data loss comes from.

That third one is worth dwelling on with the customer, because replication faithfully copies corruption. The losses I have seen were not hardware: a migration that dropped a column, a bug that overwrote records, a deletion that cascaded further than intended, an integration that silently discarded messages. Against those, the protections are point-in-time recovery, immutable audit records, validation at boundaries, and backups retained long enough to predate a slow-moving corruption.

Then the acknowledgement contract, which is the part systems get wrong: never confirm to a user or a partner until the data is durably committed. Most data loss visible to customers is a system that said yes before it had persisted anything.

I would propose specific numbers (an RPO, a retention period, a tested restore) rather than an absolute, and demonstrate them with a drill. A tested restore is more reassuring than any promise, and the customer can see it.

Curated: · Written: · Reviewed:

Q&A-64How do you handle a customer requirement for data deletion on request?

The first 90 seconds

When a customer asks for "deletion on request", they are usually promising their own users a right they have not built yet. Pin down two things before designing anything: which regime they are promising under, and which legal entity is the controller. The clock differs. GDPR Art. 17 gives you one month, extendable by two more for complex requests (Art. 12(3)). CCPA/CPRA §1798.105 gives 45 days, extendable once to 90 (§1798.130(a)(2)).

Then say the thing that decides the design: deletion is a data-flow problem, not a DELETE statement. The primary row is the easy part. The request fails at the copies nobody wrote down.

Inventory first: it is the deliverable

SystemWhat holds the identifierMechanismPropagation
Postgres primaryusers.id, 6 child tablesDELETE + cascadeimmediate
Streaming replicassame rowsfollows primary<1 s lag
Nightly logical backupsfull dumpnot editableexpires at 35 days
Elasticsearchuser_id in 4 indicesdelete_by_queryseconds
Redissession keys, cached profileDEL by key patternimmediate
Snowflakeevents.user_idDELETE + re-run derived tablesnext dbt run
Datadog logsemail inside log linesno selective deleteindex retention, 15 days
SendGridcontact recordAPI deleteminutes
SalesforceContactdelete, then Recycle Bin15 days

Building this table usually reveals two or three systems nobody in the room knew were receiving personal data.

A worked trace

Subject 8842, verified 2025-03-03, GDPR.

  • 09:12. Postgres: DELETE FROM users WHERE id=8842 cascades to 41 rows across 6 tables. Replicas follow in under a second.
  • 09:12. Elasticsearch: delete_by_query across 4 indices, 1,203 docs, ~4 s.
  • 09:13. Redis: 3 session keys and 1 cached profile removed by key pattern.
  • 09:14. SendGrid contact deleted via API; Salesforce Contact deleted, sits in Recycle Bin for 15 days.
  • 09:20. Snowflake: DELETE FROM events WHERE user_id=8842 removes 88,410 rows from the live table, but Time Travel keeps them queryable for 1 day (default) and Fail-safe for 7. The bytes are gone on day 8, not at 09:20.
  • Datadog: no selective delete. The last log line containing the email expires 2025-03-18.
  • Backups: 35-day retention. The oldest backup holding the row was taken 2025-03-03 and expires 2025-04-07. Until then the row exists in backups and is deleted on restore.

So the honest completion date is 2025-04-07, not 2025-03-03, and the ledger says so.

The hard cases need decisions, not defaults

Backups cannot be selectively edited. The position most DPAs accept in practice: a documented retention window, deletion applied on restore, and backups used only for disaster recovery. Write it down and get it agreed.

Legal hold: Art. 17(3)(b) and (e) let you keep what another obligation requires. Delete the profile, keep the invoice. German commercial law (HGB §257) requires 10 years for books; most EU member states require 6–10 years for VAT invoices. That ledger row reads retained_legal with the basis, not deleted.

Anonymisation: hashing an email with unsalted SHA-256 is pseudonymisation, not anonymisation. The input space is enumerable, so Recital 26 does not apply and it is still personal data. Aggregate to cohort counts and drop the join key; if a cohort can be narrowed to one person, it is still personal data.

Soft versus hard delete: a soft delete the application can still read is not erasure. If you need a tombstone, keep the ID and the fact of deletion, not the payload.

The request path

CCPA regulations (§7020–7022) require you to verify the requester; GDPR Art. 12(6) lets you ask for more information when you have reasonable doubts. Log the verification, not the ID document.

CREATE TABLE deletion_ledger (
  request_id   uuid,
  subject_id   text,
  system       text,
  status       text CHECK (status IN ('pending','deleted','retained_legal','expired')),
  legal_basis  text,
  completed_at timestamptz,
  PRIMARY KEY (request_id, system)
);

The ledger is what you hand the auditor. "We ran a script" is not evidence.

Curated: · Written: · Reviewed:

Q&A-59The customer wants an audit trail for a compliance requirement. What do you design?

First 90 seconds. Ask which regime, because it sets the numbers: SOX 7 years, HIPAA 6 years, PCI DSS 1 year with 3 months immediately available, SEC Rule 17a-4 6 years with the first 2 readily accessible. Then ask what the auditor actually queries, usually "everything that happened to record X between date A and date B", and what the response SLA is. Then design: capture at the database, store append-only, propagate identity, tier by age, and test the read path.

Capture point. A trail written by the application misses the migration, the admin console, and the manual fix applied during an incident: exactly what an auditor asks about.

Capture pointSees migrations/admin/manual fixesEnd-user identityTamper-evident
ORM hooks / app codeNoYesNo
DB triggersYesOnly if propagatedOnly with WORM + chain
Log-based CDC (Debezium on Postgres logical replication → Kafka)YesNo: sees the DB principalDepends on sink
Control plane (CloudTrail, S3 Object Lock)Yes, for API/object opsNoYes, compliance mode

Use triggers for the row-level trail and CDC for the "who touched the database at all" layer. They answer different questions.

Identity. No shared accounts. The app sets the actor per transaction:

BEGIN;
SELECT set_config('app.actor', 'u_4821', true);        -- true = transaction-local
SELECT set_config('app.on_behalf_of', 'agent_77', true);
UPDATE accounts SET tier = 'gold' WHERE id = 991;
COMMIT;

true matters: with PgBouncer in transaction pooling mode a session-level SET leaks the previous user's identity onto the next borrower's connection. A support agent acting for a customer puts both identities in the record.

The trigger (Postgres 16; sha256() is built in since 11):

CREATE OR REPLACE FUNCTION audit.capture() RETURNS trigger
LANGUAGE plpgsql AS $$
DECLARE v_prev bytea; v_subject text; v_before jsonb; v_after jsonb; v_payload jsonb;
BEGIN
  SELECT row_hash INTO v_prev FROM audit.event ORDER BY id DESC LIMIT 1;
  v_prev := coalesce(v_prev, '\x00'::bytea);

  IF TG_OP = 'DELETE' THEN
    v_subject := OLD.id::text; v_before := to_jsonb(OLD);
  ELSE
    v_subject := NEW.id::text; v_after := to_jsonb(NEW);
    v_before := CASE WHEN TG_OP = 'UPDATE' THEN to_jsonb(OLD) END;
  END IF;

  v_payload := jsonb_build_object('actor', current_setting('app.actor', true),
                                  'action', TG_OP, 'subject', v_subject,
                                  'before', v_before, 'after', v_after);

  INSERT INTO audit.event(actor, on_behalf_of, db_user, action, subject,
                          before, after, prev_hash, row_hash)
  VALUES (coalesce(current_setting('app.actor', true), session_user),
          current_setting('app.on_behalf_of', true), session_user, TG_OP,
          v_subject, v_before, v_after, v_prev,
          sha256(v_prev || convert_to(v_payload::text, 'UTF8')));
  RETURN NULL;
END $$;

CREATE TRIGGER accounts_audit AFTER INSERT OR UPDATE OR DELETE
  ON accounts FOR EACH ROW EXECUTE FUNCTION audit.capture();

The TG_OP = 'DELETE' branch is not decoration: in a DELETE trigger NEW is unassigned and touching NEW.id raises record "new" is not assigned yet. The trigger runs in the same transaction as the change, so a rolled-back update leaves no audit row: atomicity you want, but it also means the app can suppress its own trail. CDC from the WAL cannot be suppressed that way.

Storage and volume. 500 writes/s is 43.2M rows/day, 15.8B/year, ~110B over 7 years; at ~500 B/row that is ~55 TB. Keep 90 days hot in Postgres partitioned by month with an index on (subject, occurred_at), ship to S3 with Object Lock in compliance mode (governance mode is bypassable with s3:BypassGovernanceRetention), 7-year retention, transition to Glacier Instant Retrieval at 1 year. Object Lock requires versioning and is enabled at bucket creation.

Failure modes. A stalled Debezium consumer leaves the replication slot holding WAL: alert on pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) above 5 GB or the primary fills its disk. A hash chain with no external anchor is rewritable by anyone with table write access; publish the head hash daily to a separate account, or sign it with a KMS key the DB role cannot use. And run a real auditor query against the read path before the auditor does.

Curated: · Written: · Reviewed:

Q&A-79A customer wants to keep seven years of transaction history queryable. How do you design for that?

The 90-second answer. Split by access pattern, not by storage cost. Keep a hot window in the transactional store, sized to what the application actually reads: 90 days is a common default, a year if the app does year-over-year comparisons. Move everything older to columnar files on object storage, partitioned by time so the move is a metadata operation. Then find out what the historical access actually is, because a regulator pulling one record and a monthly report scanning seven years need different query paths.

Why not just keep it all in Postgres. Storage is cheap; the working set is not. Indexes grow with the table, so a B-tree that fit in shared_buffers at 50M rows doesn't at 350M, and plans flip from index scans to sequential scans. Backups and restores grow linearly, so the restore half of your RTO quietly doubles every couple of years for data nobody queries. Partitioning by created_at doesn't fix the working set, but it makes the exit cheap: DETACH PARTITION is a catalog update, not a delete. On Postgres 14+ use DETACH PARTITION ... CONCURRENTLY to avoid the ACCESS EXCLUSIVE lock; pg_partman automates the rolling window.

The tiers. Figures below are AWS list prices at the time of writing: verify before quoting them to a customer.

TierAgeStoreFormatCostQuery
Hot0–90dPostgres 16, range-partitioned on created_atrowgp3 EBS ~$80/TB-moms
Warm90d–2yS3 Standard, Iceberg tablesParquet~$23/TB-moseconds, Athena
Cold2–7yS3 Glacier Instant RetrievalParquet~$4/TB-moseconds first byte

Athena bills $5 per TB scanned, and that number is what decides the query path.

Worked example (illustrative). 50M transactions/year × 7 years = 350M rows at ~400 bytes = ~140 GB heap plus ~160 GB of indexes in Postgres. The same data as Parquet with column pruning and dictionary encoding lands around 40 GB. A full-history scan in Athena is 40 GB = $0.20 and ~10 s. A single-record lookup with no date predicate costs the same $0.20 and 10 s; with a predicate on the partition column it touches one day ≈ 200 MB = $0.001 and ~1 s. That gap is why the partition column has to be in every query, and why you enforce it rather than document it.

Failure modes, and how you'd see them.

  • Archive query cost surprise. A dashboard drops its date filter and scans all 40 GB on every refresh. At 10k refreshes/day that's $2,000/day. Detect it with the Athena workgroup DataScannedInBytes metric and per-query stats; cap it with a per-query data limit on the workgroup.
  • Schema drift. The source adds a column or widens amount to numeric(18,4); the Parquet files written last year don't have it, and readers either fail or silently return null. Detect it with a contract test in the export job that compares the source DDL hash against the archive schema, plus a nightly per-day row-count and checksum reconciliation between hot and archive. Iceberg schema evolution keeps old files readable under a new schema; plain Parquet does not.
  • Boundary bug. A row is archived twice, or dropped from both sides. The same reconciliation query catches it: alert when hot_count + archive_count != ledger_count for any day.
  • Deleting instead of detaching. A DELETE FROM transactions WHERE created_at < ... on 350M rows writes roughly the row volume again in WAL, holds an ACCESS EXCLUSIVE lock, and leaves dead tuples until vacuum catches up. Detect it with replication lag, WAL generation rate, and a pgstattuple bloat estimate.
  • Cold retrieval latency. Glacier Flexible Retrieval is 3–5 hours for standard retrieval. If a regulator asks for one record and that's your SLA, you've failed. Use Glacier Instant Retrieval (millisecond first byte) when single-record retrieval is in the contract, and track retrieval requests.
  • Restore time. If the archive lives in the same cluster, backups still grow and the restore drill still takes hours. Watch backup size trend and time an actual restore quarterly.

Two things to confirm before building. Whether the seven years is a legal retention obligation: that decides whether it can ever be deleted and whether it must be immutable. If it must be, use S3 Object Lock in compliance mode (WORM) rather than a copy someone can edit, and revoke UPDATE/DELETE on the archive tables. And whether the history must be byte-identical to what was recorded; if so, the export is an append-only log with a checksum per batch, not a nightly overwrite.

Curated: · Written: · Reviewed:

Q&A-38A customer wants their data to never leave their country. What does that actually constrain?

More than the region selector, and the gaps are usually in the supporting services rather than the primary datastore.

The obvious part: compute and storage in an in-country region, which may not exist for their provider, and which may lack some managed services or have only one availability zone. A material availability constraint they need to know about before committing.

The parts that are missed. Backups and their replication targets. Log aggregation, monitoring, and error tracking, which frequently default to a vendor's home region and carry payload data including personal information. Support tooling and any screen-sharing or debugging path where an engineer abroad views production data: that is often a transfer in the regulatory sense. CDN and edge caching, where content is copied by design. Email, SMS, and notification providers. Any third-party SaaS in the data path, including analytics.

Then the question of who can access it, which is separate from where it sits: some regimes care about foreign access to in-country data, so an engineering team abroad with production access may be a problem even though the bytes never move.

The practical approach is a data flow map naming every system that touches each data class, with its residency and its access. That is the artifact a regulator or auditor asks for, it is what makes the gaps visible, and it is almost never in place before someone asks for it.

And confirm the requirement's source (law, contract, or preference) because the three admit very different solutions.

Curated: · Written: · Reviewed:

Q&A-78How do you explain database indexing to a non-technical stakeholder who is being asked to fund performance work?

The 90-second version

Lead with the outcome, not the mechanism. "Checkout is slow because one report reads every order we've ever taken, about 40 million rows, to find the 200 it needs. An index makes it read those 200 directly. The report goes from eleven seconds to under a second, and the load it was putting on the checkout database during business hours goes away."

Then the analogy, one sentence, only if they ask how: "It's the index at the back of a book. Without it you read every page to find one name; with it you go straight to the page. The book gets a bit thicker and takes longer to print."

Then the trade, one sentence: "That's the cost. Every index makes writes slower and takes disk, so we add them for the questions the system actually asks, not everywhere. That's why this is scoped engineering, not a switch we flip."

Then stop. Stakeholders fund outcomes.

The trade, with something concrete behind it

If they push on "why not just index everything", have a real example ready, because "slower writes" is not a number.

Every insert and delete writes an entry into every index on the table; an update writes entries for the indexes whose columns changed. A nightly import of 2 million rows into a table with six indexes writes six extra B-tree entries per row, plus the write-ahead log for each. Illustrative but realistic: that job went from 4 minutes to 22 minutes after three indexes were added for a reporting feature, and it now runs into the 06:00 window where the finance extract starts. You detect it the first night, from the job's own duration metric: no investigation needed.

The other failure mode is an index that buys nothing. An index on a column with four distinct values, like a status flag, is usually ignored by the query planner, because reading the index and then fetching the rows costs more than scanning the table. It still takes disk and still slows every write. Detection is a quarterly check of which indexes have never been used across a full business cycle, month-end batch included: in Postgres that's pg_stat_user_indexes.idx_scan sitting at 0. To the stakeholder: "we check every quarter which indexes nothing uses and drop them."

Two more worth naming if the conversation goes long:

  • On a primary with replicas, extra index maintenance increases WAL volume, so replicas fall behind. Detected by replica lag; the cost is stale reads on the reporting replica.
  • Building an index on a live table takes a lock. In Postgres, plain CREATE INDEX blocks writes for the duration; CREATE INDEX CONCURRENTLY doesn't, but takes longer and can leave an invalid index if it fails. That's the risk window you're asking them to accept.

"Why wasn't this done at the start?"

Usually the real question. The honest answer: indexes are chosen for the query patterns and data volumes that exist at the time. Both changed: the report is new, or the table grew from 2 million rows to 40 million and the plan that was fine at 2 million is not fine at 40. That's normal system evolution, not an earlier mistake, and saying it plainly protects the team without sounding defensive.

Set the expectation

Say up front that this is periodic, not one-off: as data grows and queries change, some indexes stop being used and new ones are needed. Give them the shape of the ongoing cost (a few days a quarter, not a project) so the next request isn't a surprise. And commit to the check that keeps it honest: the quarterly unused-index review and the slow-query log, so you're removing as well as adding.

Curated: · Written: · Reviewed:

Q&A-26How do you design an integration that must survive the other system being unavailable for hours?

Make the boundary asynchronous and durable, so their downtime becomes your backlog rather than your outage.

Writes to them go through a durable queue that you own. Your side accepts the work, persists the intent, and returns; a worker delivers when they are reachable. Hours of unavailability then cost you a growing backlog and nothing else, provided the queue is bounded and monitored on age rather than only depth.

Reads are harder because a caller may be waiting. The options are a local cache of their data with an explicit staleness window, a degraded response that omits what they provide, or failing the specific operation with a clear message. Which one is right is a business decision, and it should be made with the customer before it is needed rather than by whoever writes the catch block.

Everything delivered must be idempotent, because retry after an outage means duplicates are certain. An operation key that they honour, or one you agree with them, is what makes the recovery safe.

Recovery behavior deserves as much design as the outage. A backlog of four hours released at full rate will knock them over again the moment they return, so the drain should be rate-limited to something they can absorb, and messages that are no longer relevant, a status update superseded by a later one, should be collapsed rather than replayed.

And agree the expectations with their team in advance, including how you will tell each other that something is wrong.

Curated: · Written: · Reviewed:

Q&A-9The customer wants 99.99% availability. What do you do before agreeing to it in a proposal?

Find out what they mean, what it is worth, and what it will cost them: in that order, before any number goes in a document.

What they mean: 99.99% is 4.4 minutes a month, and it is meaningless until you define which service, which endpoints, measured from where, and what counts as down. Degraded performance? A single failing feature? Planned maintenance? Customers frequently mean "we do not want the outage we had last year", which is a different and more tractable requirement.

What it is worth: ask what an hour of downtime costs them. If the answer is a large number, the architecture is justified. If they cannot answer, that is itself informative: the requirement is probably inherited from a template rather than derived from the business.

What it costs: the gap between 99.9% and 99.99% is usually the gap between a well-run single-region deployment and a multi-region active-active one, with the consistency complexity that implies. Often several times the infrastructure spend and a permanent tax on delivery speed. Presented as a priced option, many customers choose 99.9%.

Also check it is achievable: your availability is capped by the product of every dependency in the path, including their systems and their network. Promising four nines on top of a third-party integration with three nines is a commitment you cannot keep.

Then write it as an SLO with a defined indicator, and make sure the SLA in the contract is looser than the SLO you engineer to.

Curated: · Written: · Reviewed:

Q&A-37How do you decide whether a customer needs multi-region, given the cost?

From their tolerance for a regional outage, expressed in money and in regulatory obligation, rather than from a preference for robustness.

The question to put to them: if this system were unavailable for four hours, what happens? For some customers that is a bad afternoon. For others it is contractual penalties, a regulatory notification, or trading that cannot happen. Those are different architectures and the customer is the only one who knows which they are.

Then be precise that multi-region addresses one specific failure, a whole region becoming unavailable, which is rarer than the failures most outages come from. Their own deployments, a bad configuration, a dependency failure, and single-zone events cause far more downtime in practice, and multi-zone plus good change control addresses those at a fraction of the cost. Recommending multi-region to a customer whose last three outages were self-inflicted is spending their money on the wrong risk.

If it is warranted, distinguish the tiers honestly, because the cost steps up sharply: restore from backups, a running standby whose data loss is however far replication has lagged, and active-active. Active-active still leaves a product decision about conflicts, two sites selling the last item, and a global database offering moves the replication, not that choice.

And whatever tier is chosen, price the drills. An untested failover has an unknown recovery time, so the ongoing cost of rehearsing it is part of the decision, not an optional extra, and it is the line customers most often cut, which is how they end up paying for a capability that does not work.

Curated: · Written: · Reviewed:

Q&A-62How do you scope disaster recovery for a customer who has never tested a restore?

Start by testing one, before designing anything, because the current state is unknown and frequently worse than assumed.

A restore drill on their existing backups answers questions no design conversation can. Do the backups complete? Can they be restored into a usable system? How long does it take? Does the application start against the restored data? Who has the access required, and is that person available? In my experience this exercise finds a problem more often than not: a backup that has been failing silently, a restore that takes far longer than anyone believed, or credentials nobody can locate.

That result becomes the baseline for the conversation, and it is far more persuasive than a diagram. A customer who has just watched a restore take eleven hours engages with recovery objectives quite differently.

Then set the objectives from the business consequence: how much data can they afford to lose, and how long can they be down, expressed with the cost of each. Those two numbers select the tier (restore from backups, a running standby, or active-active) and the cost steps up sharply, so the numbers deserve scrutiny rather than aspiration. Nightly backups cannot meet a minutes-level loss budget; a standby's loss is the lag at the moment of failure.

Then design to the objectives, and price the drills as an ongoing commitment rather than a one-off. An untested plan has an unknown recovery time, so the rehearsal is not optional extra work; it is the thing that makes the objective real.

And include the recovery of the recovery path: what if the credentials or the runbook live in the system that is down?

Curated: · Written: · Reviewed:

Q&A-76A customer's workload has to survive the loss of an entire cloud region. How do you choose between backup-restore, pilot light, warm standby and active-active, and how do you tie the choice to RTO/RPO and cost?

Pin the failure model and the numbers first, because the strategy is an output of them, not a menu item to pick from. Two figures come before any design: how long the business can be down (RTO) and how much data it can afford to lose (RPO), and I ask for them per workload tier, not as one number for the estate. Checkout and the reporting cluster almost never deserve the same treatment, and the cheapest way to cut DR spend is to stop paying warm-standby money for components that can wait.

Then the four options, with the planning ranges I'd put in front of a customer. These describe what the design delivers, not what a provider's SLA promises, and they are only real once the customer has measured them:

StrategyRTORPOExtra steady-state costWhat has to be true
Backup and restore4–24 hBackup interval, often 12–24 h~5–15% of primary run costBackups copied out of region, a restore target that exists (account, network, IAM), and a measured restore time
Pilot light30 min – 3 hNear zero if the data tier replicates continuously~20–40%Data services running in the second region; everything else is code, images and no capacity
Warm standby5–30 minSeconds~50–100%A scaled-down but complete environment that can take load, plus tested scale-up
Active-activeEffectively zeroEffectively zero~100%, plus cross-region data transferWrite conflicts resolved, state localised, and both regions genuinely serving traffic before the event

The tie to money is two different conversations. RPO is a data decision: asynchronous replication is cheap but has lag, synchronous or consensus replication buys near-zero RPO at the cost of write latency and engineering effort. RTO is mostly operational: detection, the decision to fail over, the traffic shift, and scale-up. A ten-minute technical failover behind a two-hour escalation path is a 2h10 RTO, and the proposal should commit to the number the business will actually experience.

Worked example, hypothetical figures. Primary runs $25k/month, so $300k/year. Pilot light adds roughly 25%, warm standby roughly 75%: a $150k/year gap. Warm standby moves RTO from about 3 hours to about 30 minutes. Assume a region-loss event every 18 months; that assumption is the softest number in the whole model, and I say so. Warm standby then buys back about 1.7 hours of downtime per year, and it pays for itself against pilot light once an hour of downtime costs more than roughly $90k. Above that, warm standby. Below it, pilot light plus a rehearsed runbook is the honest recommendation, and the money saved should fund the drills that make pilot light's RTO real. Where the loss is regulatory or reputational rather than lost revenue, name that separately: expected-value arithmetic understates it and the board knows it.

Matching strategy to funding usually means tiering the estate, showing the breakeven so the customer chooses their own number, and phasing: backup and restore built and tested this quarter beats warm standby that exists only in a diagram.

The failure modes I raise in the same conversation:

  • The failover has never been run. The paper RTO and the measured RTO diverge badly, and the usual gap is restore throughput and scale-up, not the switchover itself. Detected only by scheduled drills with a stopwatch and by timing a real restore from real backups.
  • Failback is a second migration. After failing over, the old region has stale or diverged data; without reverse replication and a conflict plan, coming home loses writes. Design it before the event, not during it.
  • The dependencies nobody modelled: identity provider, DNS registrar, certificate authority, the deployment pipeline, third-party allowlists, and service quotas in the DR region. If the pipeline runs in the region you just lost, you cannot deploy anywhere.
  • Active-active specific: write conflicts across regions, cache and session locality, cross-region data transfer becoming a visible line item, and traffic splitting that misbehaves when one region returns bad health signals.

And the region choice itself, second region of the same provider versus a second provider, is a blast-radius question against duplicated operations and expertise. It is a bigger, separate decision than the strategy tier, and I would not let it hide inside the DR discussion.

Curated: · Written: · Reviewed:

Q&A-86A customer asks what happens to their system when a cloud region has an outage. How do you answer honestly?

The first 90 seconds

Ask one question before answering: "Is everything in one region?" If it is, say plainly that a full regional outage takes them down for its duration, and that duration isn't theirs to control. Multi-AZ covers a data-centre failure, not a regional one: AWS documents Multi-AZ RDS failover at typically 60–120 seconds, and that's the failure class it's built for.

Then split "region outage" into the three things it actually means, because they have different answers:

ClassWhat it looks likeReal examples
Full regionalControl plane and data plane both impairedus-east-1, 25 Nov 2020 (Kinesis degraded ~19h for some customers); 7 Dec 2021 (~7h)
Control plane onlyConsole, EC2/IAM/CloudWatch APIs impaired; running instances keep servingSame events, control-plane phase
Single serviceOne managed service degraded in one regionMinutes to hours

The middle row decides most designs. If the workload doesn't need to launch instances, mint IAM credentials, or deploy during the event, a control-plane outage is survivable. If it autoscales on a schedule or calls IAM per request, it isn't.

Then the four questions that turn this into a plan

  1. Where are the backups? If the only copy is in the same region, raise it in the meeting: it's the cheapest fix on the list. Cross-region S3 replication with Replication Time Control is SLA-backed at 99.99% of objects within 15 minutes; plain CRR has no time SLA, just "usually minutes."
  2. What's the RPO? Nightly snapshots mean up to 24h of loss. Continuous backup/PITR means minutes. Aurora Global Database documents RPO ~1s and RTO under 1 minute for a managed cross-region failover: that's the top of the range, not the default.
  3. What's the RTO? Restore-from-backup is hours to days at real data volumes. Warm standby is tens of minutes. Active-active is seconds.
  4. Has a restore been tested? An untested restore is a plan, not a capability.

Options, with rough costs

  • Cross-region backups. Cheapest. Storage plus per-GB cross-region egress. RTO hours, RPO = backup interval.
  • Warm standby. Roughly doubles steady-state compute for the standby footprint, plus replication traffic. RTO tens of minutes. Check per-region service quotas first: a region you've never scaled into may not have the EC2 capacity or quota headroom you need.
  • Active-active. 2–3x steady-state compute plus cross-region egress on every write, plus a conflict-resolution story. DynamoDB global tables resolve last-writer-wins; that may not be what the domain wants. RTO seconds.

What I'd avoid

Don't imply more protection than exists. Route 53 failover isn't instant: health checks default to 30-second intervals and need three consecutive failures, so with DNS TTL you're at 1–3 minutes before traffic moves. Assuming the standby is healthy. And IAM is global, so if it's impaired, new credentials fail no matter which region the compute sits in.

This question usually comes from a board asking it. An optimistic answer surfaces at the worst moment.

Curated: · Written: · Reviewed:

Q&A-75The customer wants to know the security posture of the design in one page. What goes on it?

One page, four blocks and a gap list. The page is a map from data → boundary → control → evidence, not a feature list. A reviewer reads it top to bottom and can answer: what are we protecting, where does it cross a trust line, what stops it there, and how would we know if it didn't. If a line on the page can't be tied to a mechanism or an artefact, it comes off.

1. Data classification and flow. Name the classes and their stores, because everything downstream is scoped by this. Typical four tiers: Public, Internal, Confidential, Restricted. Then map the regulated subsets explicitly: cardholder data (PAN) under PCI DSS 4.0, personal data under GDPR Art. 4, PHI under HIPAA: to the systems that hold them, so the compliance scope is a list of named stores, not a claim. One diagram: classes on the left, stores and flows on the right.

2. Trust boundaries. Every place data crosses a zone: internet → edge, edge → app, app → data, app → third party. Each crossing is where a control belongs, and each gets a row.

3. Controls at each boundary, stated as mechanisms. This is the table that carries the page:

CrossingDataControl (mechanism)Evidence
Internet → edgeallTLS 1.3 at the ALB, AWS WAF managed + custom rules, Shield AdvancedWAF logs, ACM cert
Edge → apppersonal, financialmTLS with SPIFFE SVIDs; OIDC tokens, 15-min TTLSPIRE attestation logs
App → datafinancialIAM auth to RDS, TLS in transit, KMS CMK envelope encryption, AES-256-GCM at restCloudTrail data events
App → third partypersonalOAuth2 client-credentials, scoped tokens, egress allowlistvendor SOC 2 report, DPA

Key management gets its own line: KMS customer-managed keys, automatic annual rotation (AWS default is 365 days), separate CMK per data class. Secrets in Secrets Manager with 30-day rotation for RDS credentials. "Short-lived federated workload identity" is a fact; "we use least privilege" is a claim and doesn't go on the page.

4. Detection and response. What is logged, where it lands, how long it's kept, what pages someone. Concretely: CloudTrail management events to a dedicated log-archive account (S3 with Object Lock), VPC flow logs, GuardDuty and Security Hub findings into the SIEM. Retention: PCI DSS 10.5.1 requires 12 months of audit logs with 3 months immediately available; CloudTrail Event history alone is only 90 days, so the trail-to-S3 path is what satisfies it. Alerts: root account use, IAM policy changes, GuardDuty high-severity, failed-auth spikes.

5. Compliance mapping. If a regime applies, a column mapping each control to its requirement: PCI DSS 4.0 requirement number, ISO 27001:2022 Annex A control, SOC 2 TSC. That mapping is the artefact the auditor asks for.

6. Gaps, deliberately. What isn't in place yet, what's accepted risk with an owner and a date, what depends on the customer. A one-page posture with no gaps reads as marketing; naming three real ones is what makes the other five blocks credible.

Failure modes to watch. A page that lists controls without boundaries can't show where a control is missing. A page with no retention numbers can't be checked against PCI. A page with no gaps gets discounted wholesale. And a page that names properties instead of mechanisms ("encrypted", "least privilege") gives the reviewer nothing to verify.

Curated: · Written: · Reviewed:

Q&A-42A customer asks for a single sign-on integration with their corporate identity provider. What do you need to establish first?

Which protocol, who owns which part of the user lifecycle, and what happens to people who are not in their directory.

Protocol: SAML and OIDC are both common in enterprise estates, and the answer is usually dictated by what their identity team supports rather than by preference. Establish it early because it affects the library choices and the effort meaningfully.

The lifecycle question is the one that gets missed. Authentication tells you who someone is; it does not create their account, assign their permissions, or remove them when they leave. Provisioning may be just-in-time on first login, or via SCIM, or manual, and deprovisioning is the part with the compliance consequence. A leaver who retains access because the integration only covered login is a finding waiting to happen.

Then authorization: does their directory carry group membership that maps to your roles, and who maintains that mapping? Groups drifting from intended access is a common source of over-permission, so someone has to own the review.

Then the edge cases, which need answers before build: contractors and partners who are not in the corporate directory, break-glass access when the identity provider is down, which makes their availability your availability, and any service accounts that cannot use interactive login.

Finally, get their identity team into the conversation directly and early. They are frequently a separate group with their own change process and lead times, and that queue is a schedule dependency that surprises projects far more often than the implementation does.

Curated: · Written: · Reviewed:

Q&A-34The customer asks for a feature that would compromise the security of their own users. How do you respond?

Raise it clearly and in writing, propose the alternative that meets the underlying need, and do not implement it quietly if they insist.

First understand what they are actually after, because the request is usually a means rather than an end. A request for support staff to see customer passwords is really a request to help users who are locked out, and that is solvable with impersonation tokens, audited access, and a proper reset flow. Most of these requests dissolve once the underlying need is separated from the proposed mechanism.

Where the conflict is real, be specific about the consequence and who bears it. Not "that is insecure" but "this stores card data in a system that is not in scope for your PCI assessment, which puts your compliance status at risk and exposes your customers' data if that system is breached." The consequence landing on their customers and their regulator, rather than on an abstract principle, is what makes the point land.

Put it in writing. This is partly about the customer's record and partly about yours, and a documented recommendation is what protects both when it is reviewed later.

If they still insist and it is lawful, it is their system and their risk, but the decision should be recorded as theirs, made with the consequence stated, and approved by someone with the authority to accept it. If it is not lawful, or it would harm their users in a way I would not want attached to my name, I would decline that piece of the work and say so plainly.

Curated: · Written: · Reviewed:

Q&A-19The customer's security team rejects your design late in the process. How do you recover?

First 60–120 seconds

Ask one question before anything else: is this a blocking gate or a recommendation? Then get the objection in writing, mapped to a control ID, with the evidence that would clear it. "Security says no" is not actionable. "PCI DSS v4.0 req 4.2.1: PAN crosses the public network without TLS 1.3, and there's no key-management evidence" is, because it names the control, the gap and the artefact.

Then classify it. Three classes, three different responses:

ObjectionWhat it actually isResponseTurnaround
Missing controlDesign gapPrice it, take to sponsor as a scope decisiondays–weeks of build
Policy violationA rule with a named ownerAsk for the documented exception pathdays, if the risk owner signs
Evidence gapDesign is fine, artefacts missingProduce the artefacthours–days

The third is the most common late-stage rejection and the cheapest to fix. A DFD showing where each data class crosses a trust boundary, a STRIDE threat model, an IAM policy export, a data-retention statement: these are documents, not code.

Worked example (hypothetical numbers)

Six weeks to launch. Security blocks on two things: no customer-managed keys for the PII store, and no evidence of who can read it.

  • Control: envelope encryption with a KMS-held CMK. 3 engineer-weeks, plus 1 week for the key-rotation runbook and access review. Slips launch 3 weeks.
  • Evidence: DFD + IAM policy export + data-event logging on the bucket. 2 days, zero code change.

Split it. Ship the evidence now, and take the key-management change to the sponsor as a decision: slip 3 weeks, or accept the risk in writing with a compensating control (field-level encryption at the app tier) and a dated remediation item. That converts an open-ended block into a choice with a price on it.

What not to do

Don't argue the control is unnecessary: the objection is usually correct, and being right about the argument costs more than the control. Don't accept a verbal exception: an exception without a named risk owner, an expiry date and compensating controls is an undocumented risk, not an exception. And don't let the objection block the whole release. Establish precisely which components are in scope; typically the payment path is blocked and the admin console is not.

Fix the process cause

A late rejection means security was engaged after the trust boundaries were drawn. Bring them into the design review at the DFD stage: one hour, and their constraints are cheap to satisfy while the design is still fluid. Two habits prevent the repeat: a standing security review at design sign-off, and a written list of the artefacts each gate expects (DFD, threat model, data classification, pen-test remediation plan) so nobody discovers them at the end. If the objection turns out to be a patch SLA rather than a design flaw: PCI DSS v4.0 requires critical and high patches within one month of release: the answer is a remediation schedule, not a redesign.

Curated: · Written: · Reviewed:

Q&A-90How do you evaluate whether a customer's proposed SaaS product will meet their compliance obligations?

Pin down two things before reading any document: which regimes actually attach, and which data classes are in scope. Regimes follow the data subjects and the data type, not the vendor's headquarters: EU personal data pulls in GDPR, PHI pulls in HIPAA, cardholder data pulls in PCI DSS, a US federal agency customer pulls in FedRAMP. The customer is almost always the controller and stays accountable; the vendor's paperwork reduces the customer's work, it does not transfer the obligation. So the evaluation is an evidence review against a control mapping, and the deliverable is a gap list, not a verdict.

Demand artefacts, not assurances, and read the scope section first:

RegimeArtefactWhat to read in it
GDPRArt. 28 DPA; SCCs (Commission Implementing Decision (EU) 2021/914) if data leaves the EEAAnnexes: sub-processor list, security measures, transfer module
SOC 2Type II reportAudit period, scope, the bridge letter for the gap since period end, exceptions in section 4
ISO/IEC 27001Certificate plus Statement of ApplicabilityCertification body accreditation, scope statement, excluded Annex A controls
HIPAABAA under 45 CFR 164.504(e)Breach terms, safeguards, subcontractor flow-down
PCI DSSAttestation of Compliance plus SAQ typeWhich SAQ, the assessed environment, the assessment date
FedRAMPAuthorization package or marketplace listingAuthorization boundary, impact level, date of last assessment

A SOC 2 covering the corporate entity but not the production environment, or an ISO certificate whose scope names a different product line, is the most common gap and it is invisible unless you read the scope paragraph. Same for the sub-processor list: data routinely reaches a fourth party nobody in the room has heard of, and Art. 28(2) requires the customer to have authorised it.

Then the questions the paperwork does not answer. Where do support engineers connect from? If that country has no adequacy decision, it is a transfer needing SCCs plus a transfer impact assessment: the Schrems II (C-311/18) analysis, not a checkbox. What is the deletion process, does it reach backups and derived data, over what period, and how is it evidenced? What is the exit format and window?

Breach notification is where the arithmetic bites. GDPR Art. 33 gives the customer 72 hours from awareness to notify the supervisory authority, and Art. 34 requires notifying data subjects without undue delay. A vendor contract promising notification "within 72 hours" consumes the entire budget before the customer has assessed anything. Push for 24 hours, and check whether the clock starts at the vendor's awareness or at confirmation.

Finally, the customer's own side of the shared responsibility model, where most failures actually occur: configuration, access management, and what they put into the product. A compliant product configured carelessly is not a compliant system. Concrete failure modes to name: certification scope mismatch (detected by reading the scope statement), undisclosed sub-processor (detected by diffing the list against the DPA annex), support access from a non-adequate country (detected by asking for the access path), deletion that skips backups (detected by asking for a deletion runbook and a sample evidence record), and a breach window longer than the regulator's deadline (detected by reading the clause against Art. 33).

Bring the customer's compliance or legal function in early. This is their decision; a technical recommendation that pre-empts a legal judgement is one they cannot rely on.

Curated: · Written: · Reviewed:

Q&A-66How do you decide what to monitor for a customer who currently monitors nothing?

What a strong candidate says first

Before picking a tool, ask two questions: which user action earns or costs money, and what would stop it? Then monitor the smallest set that answers "are users affected, and roughly where", page only on user-visible symptoms, and let the first few incidents decide what to add. The decision rule for every candidate signal is: would this change what I do at 3am? If not, it is a dashboard nobody opens.

State your assumptions out loud: no metrics, logs or traces worth the name; one or two user-facing services; nobody on call yet. If they already have usable logs, the plan changes: you would alert on log-derived symptoms before adding new instrumentation.

The starting set

SignalQuestion it answersWhy it is first
Synthetic check on the main journey, from outside their networkAre users affected now?Sees DNS, TLS, load balancer and third-party failures that internal metrics cannot
Request rate, error rate, latency distribution, per user-facing endpointHow bad, and where?These three map directly to user pain
Saturation of the scarcest resource: usually DB connections and diskWhat is about to break?Cheap, leading, and the usual first cause
Deploy and config-change annotationsWhat changed?Free, and correlates with a large share of incidents

That is enough to know something is wrong and roughly where. It is a step change from nothing.

Alerting: symptoms only, and few

A team new to this will alert on everything and have an ignored channel within a month. Two or three rules that are always real beat thirty that are usually noise. Page on the user-visible symptom, not the cause:

# error ratio over 5m, user-facing API
sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="api"}[5m])) > 0.02

# p99 latency
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m]))) > 1

# synthetic probe failing for 2m
probe_success{job="blackbox",target="checkout"} == 0

Do not label by user ID or request ID: that is how a metrics backend falls over. Do not page on CPU; page on the latency it causes, if it causes any.

Leading indicators with a deadline

Each of these is an outage with a countdown, and none shows up as a symptom until too late: certificate expiry (warn at 30 days), disk filling (predict_linear(node_filesystem_avail_bytes[6h], 7*24*3600) < 0), backup failure (any failure pages), and quota or licence exhaustion.

Then let incidents drive it

Every incident ends with one question: what signal would have shown this sooner? The answer becomes the next thing instrumented. That produces a setup shaped by their failures rather than a vendor checklist, and it stays small enough to read.

Failure modes to name

  • The synthetic check covers the homepage but not login, so the outage that matters is invisible. Walk the journey with the customer and script each step.
  • Alerting on cause rather than symptom produces pages with no user impact, and the channel dies.
  • Monitoring only what they own. A payment provider's outage is their outage; probe it.
  • No owner or runbook per alert. An alert nobody knows how to act on is noise with extra steps.

Worked example (illustrative figures)

A three-service SaaS on one Postgres. Pool of 25 connections, average query 20 ms, so the ceiling is 25 / 0.020 = 1,250 queries/s. At 1,300 queries/s of arrivals the wait queue grows without bound: p99 latency goes from 45 ms to 210 ms and the error rate climbs as clients time out. Nothing in CPU or memory shows it. The pool-saturation metric and the p99 alert both fire; the deploy annotation from 20 minutes earlier explains it. That is the whole argument for measuring saturation and a latency distribution rather than averages.

Curated: · Written: · Reviewed:

Q&A-81What do you look for when a customer says their system is slow but has no metrics?

The 90-second answer. Don't theorise. Get one measurement from the user's vantage point, then bisect. "Slow" is a report about experience, and the cause is usually not where the reporter thinks it is. Three moves: measure the real journey from outside, split the problem along dimensions that each eliminate whole categories, then mine the telemetry that already exists in almost every system even when nobody calls it monitoring.

Measure from outside first. Run the user's journey with curl and read the phase breakdown:

curl -sS -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' https://app.example.com/orders

Hypothetical result: dns=0.004 connect=0.021 tls=0.048 ttfb=3.180 total=4.910. A TTFB of 3.18 s says the server, or something in front of it, is the problem. Total minus TTFB is 1.73 s, so the page then spends longer fetching assets than generating HTML. If instead ttfb were 0.09 and total 4.9, you'd be looking at front-end assets or the client's network, and no amount of server profiling would help. Run it from the customer's region, not your laptop: a corporate VPN or a CDN edge that doesn't serve them changes the answer.

Bisect by dimension. Each question kills a category:

QuestionAnswerEliminates
Everyone or one tenant?one tenantglobal infra, CDN, deploy
Everywhere or one region?one regionapplication code
All day or a window?09:00–09:30steady-state capacity; points at a batch job or cron
All endpoints or some?one endpointnetwork, DNS, TLS
Always, or since a deploy?since Tuesdayeverything except the change

Mine what already exists. Uninstrumented rarely means nothing: nginx and Apache access logs carry $request_time and $upstream_response_time; PostgreSQL has pg_stat_statements and log_min_duration_statement (default -1, i.e. off: turn it on); MySQL's slow query log has long_query_time defaulting to 10 s, which is why it looks empty; the OS has iostat, vmstat and ss. That is usually enough to name the dominant cause.

The usual suspects, in order. A missing index on a table that grew past the point where a scan was cheap; an ORM N+1; an unoptimised front end; a synchronous third-party call in the request path; connection pool exhaustion at peak.

Worked trace of the N+1, hypothetical: an orders page renders 50 rows. Each row lazily loads its customer, one query per row at 12 ms round trip, so 600 ms of pure query time plus 50 round trips of connection overhead. EXPLAIN ANALYZE on the parent query shows 3 ms: the 600 ms is invisible to it. Fix: one join, or a batched IN fetch, taking 50 queries down to 2.

Failure modes and how you catch them. Measuring from your own network gives a false negative: always measure from the user's path. Sampling at 11am misses a 9am peak: pull logs for the reported window. Cache-warm tests hide cold-start cost: clear the cache or test a rarely-visited page. And "slow" sometimes means throughput, not latency: requests queue behind a saturated pool, so p50 looks fine while p99 explodes. Compare p50 to p99 before believing a single number.

Instrument as you go. Add request-time logging and a slow-query threshold while you're in there, because the same question arrives again in six months and shouldn't require this exercise twice.

Curated: · Written: · Reviewed:

Q&A-41How do you scope the observability a customer needs, given they are cost-sensitive?

Start from the questions they will need answered during an incident, and buy the cheapest thing that answers each. Observability spend gets out of hand when it is specified as a product tier rather than as a set of questions.

The irreducible set: is the system up and are users affected, which is a small number of synthetic checks and a request-level metric; what changed, which is deploy and configuration change annotations and costs almost nothing; and what is failing, which is structured request logging with a correlation ID and errors captured with context.

That covers most incidents at modest cost. The expensive tiers: full distributed tracing at high sampling, long metric retention at high cardinality, every log line kept hot for months: should be bought where they answer a question the cheap tier cannot, not by default.

The specific economies that work: sample traces heavily but keep every trace that errored or was slow, which preserves the interesting tail at a fraction of the volume; keep operational logs hot for a week or two and tier the rest to cheap storage; and be careful with metric cardinality, since a label with unbounded values is the usual cause of a surprising bill.

Retention should be set by category rather than globally: security and audit records kept long because they cannot be reconstructed, verbose debug output kept briefly.

And point out that the cost of not having it is measured in incident duration. A customer with a four-hour mean recovery time is paying for the gap already, just not on that invoice.

Curated: · Written: · Reviewed:

Q&A-68How do you present a risk register to a customer without it becoming a formality nobody reads?

Keep it short, make each entry concrete, and bring it to a decision each time rather than reporting it.

Short means the risks that could actually change the outcome: usually five to eight. A register of forty entries is a document that has been optimised for coverage rather than for use, and the effect is that nobody reads any of them, including the three that matter.

Concrete means each risk names a specific event, its consequence in the customer's terms, and what would be done. Not "integration complexity" but "the billing system's API has undocumented rate limits; if they are lower than our peak, the nightly run will not complete in the window, and we would need to negotiate a limit increase or move to a batch file, which adds about three weeks."

Each entry needs an owner and a next action, and several will be owned by the customer rather than by you: decisions pending, access not granted, a dependency on another of their programmes. Making those visible is often the register's main value, because they are the risks the delivery team cannot address alone.

Review it at the regular checkpoint by exception: what changed, what closed, what is newly urgent. Reading the whole list aloud is what trains people to stop listening.

And close risks explicitly when they are retired. A register that only grows reads as a list of complaints; one that visibly closes items demonstrates that raising a risk leads somewhere, which is what keeps people raising them.

Curated: · Written: · Reviewed:

Q&A-33What do you include in a handover so the customer's team can actually run the system?

The 90-second answer. Handover is done when their engineers have run the system without you, not when the documents are delivered. So I'd agree exit criteria up front, a named list of operations their team must have performed unaided, and treat the documents as inputs to that, not the deliverable.

Assumptions: a production system with on-call, a receiving team of at least two engineers, and a departure date you control.

The package. Four things, each with a definition of done:

ArtifactDone means
Architecture recordDecisions and the alternatives rejected, with the constraint that drove each. Not a diagram of the current shape.
RunbooksOne file per alert, in the service repo next to the code, Markdown. Each: what the alert means, first three commands, how to confirm recovery, who to escalate to.
Infrastructure as codeThe pipeline that applies it, and no path to production that bypasses it: verified by trying to bypass it.
Access inventoryEvery credential, its owner, its rotation interval, and the transfer completed, not scheduled.

Credentials are the step most often left half-done: the new team has console access but not the break-glass account, or the CI deploy key still belongs to a departing engineer's personal account.

Shadowing is the part that decides it. Their engineers do the work while yours watch, not the reverse. Concretely, over the last two weeks: their engineer runs three deploys, one rollback, one credential rotation, and one restore, each unaided, with you reachable but not driving. If the system has on-call, they take one full rotation with you as secondary.

A restore drill is worth more than any document because it exercises the whole chain at once: access, procedure, and the assumption that the backups are actually restorable. Hypothetical example: a 400 GB Postgres cluster, nightly snapshots, a stated RTO of one hour. The drill restores the latest snapshot into a scratch environment; it takes 3 h 40 m, because the snapshot has to be downloaded and replayed and nobody had measured that. That single number is the most valuable output of the handover, and it only appears if someone runs it.

Failure knowledge. A known-issues register: what has broken during the build, the weak points you'd fix first with more time, and the workarounds currently in place. Uncomfortable to write, most useful thing in the package. Include the things that are load-bearing but fragile: the cron job nobody owns, the queue with no dead-letter handling.

The boundary after. A hypercare window with an end date, I'd propose 30 days, a named contact on each side, and an explicit list of what is no longer covered once it closes. Write the boundary down before the incident, not during it.

What goes wrong. Runbooks written by their author and never executed by anyone else. Access transferred on paper. A support arrangement with no end date, which becomes permanent and unpaid. Exit criteria agreed verbally.

Curated: · Written: · Reviewed:

Q&A-13What do you do when the customer's team will not be able to operate what you are proposing?

Treat it as a design constraint of the same weight as latency or cost, because a system that cannot be operated will degrade to whatever the team can actually run: usually within a year, and usually unsafely.

Assess it honestly and early: what do they run today, what is their on-call maturity, how many people, what is their appetite for learning something new, and what happens when the one person who understands it leaves. Ask what broke last year and how it was fixed; the answer tells you more than a capability matrix.

Then adapt the design rather than the training plan. Managed services over self-hosted, even at higher unit cost, because the operational burden is the constraint. Fewer moving parts, even if a more sophisticated design would be cheaper to run in expert hands. Boring, well-documented technology over the better tool their team has never seen.

Where sophistication is genuinely necessary, price the capability as part of the project: runbooks, a handover period with their engineers doing the work while you watch, and a defined support arrangement with an end date and a plan for what happens after.

And say it plainly to the sponsor. "This design assumes two engineers who can operate Kubernetes; you have none, so the options are to hire, to buy managed support, or to choose the simpler design" is a conversation that is uncomfortable once and cheap. Discovering it after go-live is expensive and lands on the customer.

Curated: · Written: · Reviewed:

Q&A-80How do you handle a customer whose developers want a technology the operations team cannot support?

Both sides are usually right about their own constraint, so the job is to convert a technical argument into a funding decision the sponsor can make.

In the first minute: name the two constraints, refuse to adjudicate on technical merit, and put priced options in front of whoever owns both budgets. The developers are right that the technology fits the problem. Operations is right that they cannot run it to the standard the business expects without something changing. Neither of those is a claim about the other team's competence, and treating it as one is how the conversation goes wrong.

Worked example: figures illustrative, for a customer with 50 GB/day of events, four engineers on call, Postgres and one Kubernetes cluster already in production:

OptionIllustrative costTime to first production topicWho is paged at 3am
Managed Kafka (Confluent Cloud, Amazon MSK)~$2–4k/month at this volume. MSK still leaves you broker version upgrades and partition rebalancing; Confluent Cloud removes more of that at a higher per-GB rate2–3 weeksVendor for the broker; your team for consumer lag, schema changes, ACLs
Self-managed Kafka on the existing clusterNo new line item, ~0.5 FTE of ops time ongoing6–10 weeksOps team, on a system none of them has run in production
Don't adopt: outbox table in the Postgres you already run~2 weeks of dev work, no new ops surface2 weeksNobody new

The third row is not a consolation prize and should not be presented as one. It is the right answer until you need replay or multiple independent consumer groups, at which point the outbox poller becomes the thing that wakes people up.

Ask the sponsor for two numbers: how much the dev team's delivery slips if they use the outbox, and how many on-call hours per month operations is being asked to add. If the ops number is 0.5 FTE and the dev number is one sprint, the managed service at $3k/month is usually the cheapest of the three, and it is the only one that removes the disagreement instead of assigning it.

The failure mode to name out loud is adopting it without funding the capability. That produces a system nobody is confident operating, which degrades quietly and fails at the worst moment. Detection is a question, not a load test: ask who is paged when the broker is unhealthy, and check that person is on the rota this quarter. If the answer is "the developer who introduced it" and they are not on the rota, you have already chosen this option by accident.

Development owning operations is a legitimate model, not a punishment. It needs a named rota, the same alerting and runbook standard as everything else, and, if it is meant to be temporary, a handover date written down. Without those three it is the unfunded option with better PR.

Curated: · Written: · Reviewed:

Q&A-48What do you do when a customer's team disagrees with your recommendation and they will be operating the system?

Take it seriously, because they have information you do not and they are the ones who live with the outcome.

Their objection usually comes from one of three places: they know something about their environment that has not been shared, they have been burned by this approach before, or they have a preference formed by familiarity. The first two are substantive and should change your recommendation. The third deserves examination: familiarity is a legitimate factor, since a team operating something they understand will run it better than something they do not, and that is a real engineering consideration rather than a soft one.

The way to find out which it is: ask what specifically concerns them and what would have to be true for the recommendation to be right. If they can name a concrete failure mode, you have learned something. If the answer is general discomfort, the conversation is about familiarity and can be addressed with evidence or with training, or by changing the recommendation, if the gap is large enough.

Where the disagreement persists and it is their system to run, I would weight their preference heavily. A design they believe in and can operate beats a marginally better design they resent and will erode. That is not capitulation; the operability of a system is part of its quality.

Where I would hold the line is on a decision that creates a security, data-integrity, or availability risk they are not seeing. Then it goes to the sponsor as an explicit trade-off with both positions recorded, rather than being settled by whoever is more persistent.

Curated: · Written: · Reviewed:

Q&A-72How do you decide the right level of automation for a customer's deployment process?

From their deploy frequency and their tolerance for a bad release, and by automating the steps that are error-prone before the steps that are merely tedious.

For a customer deploying monthly with a change window, a fully automated continuous deployment pipeline is more machinery than the situation warrants, and it will not be trusted or maintained. What they need is the build, the tests, and the deployment itself automated and repeatable, with a human deciding when it runs.

The steps worth automating first are the ones where a human error is likely and consequential: database migrations, configuration differences between environments, and the deployment sequence itself. Manual steps in those places are where releases go wrong, and each one is also an undocumented dependency on the person who knows it.

Rollback should be automated before anything else, because it is the control that makes everything else safe, and it is the step most often left manual, which means it is slowest exactly when it is needed.

Then let frequency drive further investment. A team moving to weekly deploys benefits from automated verification and progressive rollout; a team deploying quarterly does not yet.

The trap to avoid is copying a mature practice wholesale. A customer with no tests who builds a continuous deployment pipeline has automated the delivery of untested changes, which is worse than what they had. The sequence is tests, then repeatable deployment, then rollback, then frequency, then progressive delivery: in that order, because each depends on the one before it.

Curated: · Written: · Reviewed:

Q&A-63The customer asks why they need infrastructure as code when their environment rarely changes. How do you make the case?

What I'd say in the first two minutes

One clarifying question first: what's the blast radius if this environment disappears tonight. Customer-facing, and is there a compliance obligation attached to it? That changes the emphasis, not the argument.

Then I'd concede the premise. Change frequency is a bad reason to adopt infrastructure as code, and if that's the only case being made, the customer is right to push back. The case is recovery, review, and the fact that the environment is currently undocumented.

Recovery is the strongest argument. If the environment were lost (a region event, a mistaken deletion, an account compromise) can they rebuild it? Without code, someone reconstructs it from memory and a diagram, under pressure, and the result differs from the original in ways nobody notices until later. With code, the rebuild is a pipeline run against a clean account.

Review is the second. A console change is made by one person with no record of what it was, who approved it, or why. The same change as code is reviewed before it happens and recorded afterward, which is what most compliance regimes are actually asking for when they ask about change management.

The third is documentation. Infrastructure as code is the only description of an environment that cannot go stale, because it is the thing that produced it.

QuestionConsole-onlyUnder IaC
Rebuild after lossFrom memory, days, silently divergentapply against a clean account, minutes
Who approved this changeNobody; CloudTrail shows the API call, not the intentPR review, then commit history
Where is the current configA wiki page of unknown ageThe repo, at a known commit
Can you diff today vs last monthNogit diff

The drift trace. This is the argument that lands with engineers, because it's a thing they've all lived through. Someone opens port 22 to 0.0.0.0/0 in the console at 02:00 during an incident. A scheduled CI job runs terraform plan -detailed-exitcode; exit 0 means no drift, 2 means drift, 1 means error. The next run exits 2 and prints the diff: cidr_blocks 10.0.0.0/8 → 0.0.0.0/0, from_port 443 → 22. A human then decides: codify it as a reviewed PR, or revert it. Without IaC, that change is invisible forever.

Failure modes, and how you'd catch them.

  • Drift fights the hotfix. The next apply reverts an undocumented console change and breaks whatever it was patching. Detected by the exit-2 CI job above; the fix is a policy that console changes get codified within a day, not that drift detection is turned off.
  • State is a single point of failure. State on a laptop means the mapping from config to real resources is gone with the laptop. Detected the first time two engineers apply at once. Fix: a remote backend with locking. S3 plus a DynamoDB lock table, or S3 native locking in Terraform 1.10 and later.
  • Secrets land in state. Database passwords and generated keys are stored in plaintext in the state file. Detected by anyone with read access to the bucket. Fix: encrypted, tightly-scoped state bucket, and secrets sourced from a secret manager rather than written into config.

The rejoinder to the premise. Environments that rarely change are exactly the ones where nobody remembers how they were built. The knowledge is at its most fragile where the change rate is lowest, because there's been no recent rehearsal.

The honest cost. Importing existing resources is tedious, and it requires a discipline that console changes stop. Illustratively, a 60-resource environment rebuilt by hand at 20 minutes per resource including verification is 20 hours of one engineer's time, plus the parts nobody remembers. I'd propose starting with the next new component rather than a big-bang import: Terraform 1.5 and later can generate config for imported resources, which shortens the tedious part, so the practice establishes itself without becoming a project.

Curated: · Written: · Reviewed:

Q&A-69The customer asks whether microservices are right for them. How do you answer?

By asking what problem they expect it to solve, because the honest answer for most customers is not yet, and the reasons they usually give point to cheaper solutions.

The problems microservices genuinely address: independent deployment when multiple teams are blocked on a shared release train, independent scaling when one component needs far more capacity than the rest, and fault isolation when one component's failure must not take the others down.

If they have one team, none of those apply. The coordination cost that microservices remove does not exist, and they will pay the distributed-systems cost: network failures between components, no cross-component transactions, distributed debugging, and an operational surface several times larger: in exchange for nothing.

The reasons customers usually cite are worth examining individually. Deploying is slow: usually a pipeline and test-suite problem, cheaper to fix directly. The codebase is hard to work in: usually a modularity problem, and modules with enforced boundaries inside one deployable give most of the benefit at none of the cost. Scaling: usually one component, which can often be extracted alone.

So my typical recommendation is a well-modularised single deployable, with boundaries enforced by tooling, and extraction of the specific components that have a real independent scaling or deployment need. That keeps the option open, because clean internal boundaries make a later split straightforward, while an unnecessary split is expensive to reverse.

If they have several teams blocking each other, that is the case where I would agree.

Curated: · Written: · Reviewed:

Q&A-67A customer wants to expose their internal API to a mobile app. What changes?

The 90-second answer

The trust boundary moves. An internal API is protected mostly by being unreachable; a mobile client ships to devices you don't control, so every assumption that rested on "the caller is our own service" has to be re-established server-side. I'd treat this as designing a public API with exactly one known consumer: the same threat model as public, but you can still coordinate releases with the app team.

Four things change. Authentication becomes a public-client problem. Authorization has to be enforced per object on every call. The API surface gets reshaped for high-latency networks. And you need a versioning and deprecation policy, because client and server no longer deploy together.

Trust

The app binary is public. Anything shipped inside it (an API key, a client secret, a signing key) is extractable in minutes with strings or a decompiler. So the app is an OAuth public client: authorization code flow with PKCE and no client secret (RFC 8252). Access tokens short-lived, 15 minutes is a common choice; refresh tokens in the platform keystore (Keychain, Android Keystore), rotated on every use with reuse detection so a stolen refresh token invalidates the whole family.

Authorization is what internal APIs get wrong most often. Internal services tend to check "is this caller allowed to hit this endpoint" and trust the IDs in the request. A mobile client will send any ID it likes. Every handler needs an object-level check against the authenticated subject.

# Python 3.12 / FastAPI
@app.post("/v1/orders/{order_id}/cancel")
def cancel(order_id: str, idem: str = Header(alias="Idempotency-Key"),
           user = Depends(current_user)):
    order = db.get_order(order_id)
    if order is None or order.owner_id != user.id:
        raise HTTPException(404)   # 404, not 403 — don't confirm the ID exists
    cached = redis.get(f"idem:{user.id}:{idem}")
    if cached:
        return JSONResponse(json.loads(cached), status_code=200)
    result = orders.cancel(order)
    redis.set(f"idem:{user.id}:{idem}", json.dumps(result), ex=86400)
    return result

Trace a replay: the app sends the cancel, the response is lost when the phone enters a tunnel, the app retries with the same Idempotency-Key. The second call finds the cached body and returns it without touching orders.cancel. Without the key, that second call either double-cancels or 409s depending on state: exactly the ambiguity a retrying mobile client can't resolve on its own.

Network shape

Chatty internal APIs assume sub-millisecond RPC. On LTE each request costs a radio wake-up plus a round trip; 150–300 ms per request is realistic (illustrative). A screen that fires 12 calls pays roughly 2 s of pure network time before any server work. Reshape so one screen is one request, trim payloads, and paginate with cursors rather than offsets so a list doesn't shift under a client that was offline.

Versioning

You can't deploy the client. Assume a tail of installs that never update: if 5% of your base sits more than two versions behind (hypothetical), every breaking change either strands them or forces you to run both shapes. So: additive changes only, version in the path, per-version usage metrics from day one, and a min_supported_version check that returns 426 Upgrade Required with a store link.

Failure modes and how you'd see them

  • Replayed or modified request → idempotency keys plus short token TTL; alert when the same key arrives with a different body.
  • Old client hitting a removed field → a 400 spike concentrated in one app version. You only see it if every request carries an X-App-Version header and you dashboard error rate per version.
  • Stolen refresh token → rotation with reuse detection; the reuse event itself is the alarm.
  • Abuse of unauthenticated endpoints (login, signup) → per-IP and per-device rate limits, since there's no user identity to limit on yet.

First three things I'd do: put X-App-Version on every request, add object-level ownership checks to every handler that takes an ID, and stand up per-version error dashboards before the first release.

Curated: · Written: · Reviewed:

Q&A-96How do you decide whether to recommend a customer build on a platform their team already knows but that is a worse technical fit?

The 90-second answer

Decide by whether the mismatch is a wall or a slope, then price the slope over the life of the system, not the length of the project. If the platform the team already knows clears every hard requirement with headroom, recommend it and eat the inelegance. If it misses one, the recommendation is the better fit, and it has to arrive with funded capability, because recommending a system the customer cannot operate is not a recommendation.

State the assumptions out loud: the hard requirements (throughput, retention, latency, compliance boundary), the team's size and hiring market, and the horizon you are pricing over. "Worse technical fit" means nothing until you name the requirement it misses.

Wall or slope

A slope is ongoing cost: more code, more nodes, a rollup you maintain by hand. A wall is a requirement the platform cannot meet at any reasonable cost. Slopes are weighable against the learning curve. Walls are not.

The test for a wall is a load test at 2× projected peak against a production-shaped dataset, running the real query pattern, not a vendor benchmark. Pass at 2× and it is a slope.

Worked example (hypothetical figures)

Team of 8 backend engineers, all strong in MySQL and Python. Workload: 40k events/sec sustained, 120k peak, 90-day retention, ~30 TB, per-second windowed aggregation.

MySQL (known)Kafka + ClickHouse (unfamiliar)
Nodes at 120k peak20 shards (measured 6k inserts/sec/node with batched writes)6 brokers + 4 ClickHouse nodes
Cross-shard aggregationapp-side merge, ~3× query latencynative
Ramp cost03 engineers × 6 weeks = 18 eng-weeks ≈ 0.35 FTE-years
Ongoing ops~1.5 FTE (rebalancing, shard skew, app-side merges)~0.5 FTE

Break-even: 0.35 FTE-years of ramp ÷ 1.0 FTE-year of extra ops saved per year ≈ 4 months. Over a 3-year life: MySQL 4.5 FTE-years of ops; Kafka + ClickHouse 1.5 + 0.35 = 1.85. Recommend the unfamiliar stack, funded with the 18 eng-weeks of ramp and a managed ClickHouse offering for the first year.

Now flip one number: at 5k events/sec and 2 TB, MySQL is one partitioned node, the mismatch is a slope (app-side rollups instead of native materialized views, a bit more code) and the 18 eng-weeks buys nothing. Recommend MySQL.

Failure modes

  • The wall you did not test: you benchmarked at 1× and the system falls over at 1.4×. Detect with the 2× load test before the recommendation, not after.
  • The learning curve you under-priced: ramp is not 6 weeks of training, it is 6 weeks plus the first production incident, which is where the real cost lands. Budget the on-call shadowing.
  • The fit argument that is yours: if you are the one who would operate it, you are biased toward elegance. Ask who carries the pager.

Funding the capability

Training, hiring, a managed offering, or a support contract with an end date. Without one of those, the recommendation is a system the customer cannot run, and the known platform, slope and all, is the honest answer.

Curated: · Written: · Reviewed:

Q&A-11A customer asks why they should not just use the vendor's reference architecture unchanged.

The 90-second answer

A reference architecture is a proof that the vendor's products compose into something that works: under a workload shape and a set of constraints it usually doesn't state. It's a starting point and a checklist, not a design. Use it, then walk it against the customer's own numbers and record a reason for every divergence.

What it actually encodes

Four things, and only one of them is on the diagram:

  1. Workload shape: read/write ratio, working-set size, peak-to-average ratio, latency budget.
  2. Failure model: how many AZs, what quorum, what RPO/RTO.
  3. Trust boundaries, where TLS terminates, what sits in the VPC, what is public.
  4. Cost envelope and team capability: the vendor assumes you run their managed services and that someone on staff can operate them.

Named artefacts that do state some of this: the AWS Well-Architected Framework's six pillars, the Azure Well-Architected Framework's five, and the Google Cloud Architecture Framework. The reference architectures themselves live in the AWS Solutions Library and the Azure Architecture Center. Read the "considerations" section, not the diagram: that's where the assumptions leak out.

It is also, reasonably, a showcase for the vendor's managed services. That isn't malice, but it means the cost-optimal option at the customer's volume may simply not appear in the diagram.

Worked example

Vendor reference architecture: ALB → ECS Fargate across three AZs → Aurora PostgreSQL → ElastiCache Redis → S3 + CloudFront. Customer: 40 GB Postgres database, 120 rps peak, 15 rps average, single region, six-person platform team, no Redis in production today.

Reference saysCustomer's numbersDivergence and reason
ElastiCache RedisHot set ~3 GB; a db.r6g.xlarge buffer pool is 32 GiBDrop the cache tier. The buffer pool holds the hot set, so the cache is buying latency you already have, and it removes a stateful tier from the on-call rotation.
Three AZs for the stateless tierALB health-checks, no local stateTwo AZs. Three only buys something for quorum-based stateful systems: etcd, Cassandra RF=3, Kafka with min.insync.replicas=2. Aurora's storage is replicated across three AZs regardless of where your tasks run.
NAT Gateway per AZ10 TB/month egress to S3Three NAT gateways at $0.045/hr ≈ $98/month, plus 10,240 GB × $0.045 ≈ $461/month in data processing. An S3 gateway endpoint is free and takes that traffic off the NAT bill entirely.
Managed queueTeam already runs Kafka on-premKeep Kafka. The cost of learning a new queue exceeds the licence saving.

(NAT figures are AWS's published us-east-1 list rates; check current pricing before quoting them to a customer.)

Where it goes wrong

Two failure modes. Adopting it unchanged: the design is correct on paper and fails on the customer's constraints. Usually cost at their volume, or a component nobody can operate. Diverging without recording why: six months later nobody knows whether the missing cache tier was a decision or an oversight, and the next architect re-adds it.

The constraint most often skipped is the team's operational capability, and it's the most expensive to get wrong. A design nobody on the customer's side can run degrades the week after you leave.

Curated: · Written: · Reviewed:

Q&A-54What is the role of a reference architecture inside a consulting practice, and how do you keep it from becoming dogma?

A reference architecture is the practice's compressed memory: the decisions that survived ten engagements, written down so the eleventh doesn't re-derive them. What it buys is measurable. If a design phase costs roughly three weeks of a senior architect's time and you run twelve engagements a year, the reference returns about 36 weeks: 0.7 FTE: before you count the consistency benefit, which is that quality stops tracking whoever happened to be staffed.

The thing that decides whether it helps or hurts is what you write down next to each decision. A decision without its assumption is a rule; a decision with its assumption is a hypothesis you can test on the next customer. The table below is the shape I'd want: the numbers are illustrative, the structure is not.

DecisionAssumption it encodesThreshold that invalidates it
Kafka, 3 brokers, RF=3, acks=allsustained produce ≥ 20k msg/s, p99 produce budget < 5 msbelow ~2k msg/s the operational cost isn't repaid
Multi-region active-passiveRTO ≤ 15 min, RPO ≤ 1 min, data-residency constraintcustomer accepts RTO ≥ 4 h → single region
Kubernetes + service mesh≥ 8 services, ≥ 3 teams owning them independentlyfewer than 5 services → managed containers

Dogma is what happens when the left column travels and the middle column doesn't. The recognisable failure: a customer at 2M writes/day gets a cluster sized for 50M. Twelve nodes instead of three, at an illustrative $1.00 per node-hour, is $8,760/month against $2,190: $6,570/month, $78,840/year, which is more than the design time the reference saved. Nobody can point at the requirement that justified the extra nine nodes, because there wasn't one.

Four mechanisms keep it honest, and each has a number attached.

Divergence is logged, not approved. When an engagement departs from the reference, the architect writes a one-page record within five business days: what they changed, which assumption failed, what it cost. No sign-off gate: a gate turns divergence into a negotiation and people stop recording it. These records are the only honest input to the next revision.

Revision is triggered by clustering, not by calendar. Review quarterly, but the trigger is that ≥30% of engagements diverged at the same node. Three of twelve teams dropping the second region is a signal; one is a customer quirk. When the trigger fires, the node changes or gets deleted.

It's modular at the seams teams actually adopt. A team should be able to take the identity and tenancy pattern (the OIDC flow, the tenant-isolation model, the audit-log shape) without inheriting the streaming stack. In practice that means the reference ships as versioned Terraform modules and a Backstage software template, each independently consumable, not as one repository you fork wholesale.

It's explainable from first principles by the person applying it. If a junior engineer asks why the mesh is there and the answer is "it's in the reference," that's a defect in the reference, not a gap in them. Run the test deliberately: at engagement kickoff, the architect on the account walks the customer's own numbers against each row of the table and marks the ones that don't hold. Anything unmarked is inherited by default, and default inheritance is where dogma lives.

Two things this is not. It is not a review lens: the AWS Well-Architected Framework and Azure Landing Zones are checklists you run against a design, and treating either as the design is a category error. And it is not a pre-sales artefact: if the reference is committed in the proposal before discovery, you've sold the assumption instead of the fit, and the divergence record becomes an argument about the contract rather than an input to the next revision.

The failure mode to watch for is silent: engagements quietly build something else, nobody writes it down, and the reference keeps being cited in proposals while the delivery teams route around it. Detect it by diffing what shipped against what the reference prescribed at each engagement close, if the diff is large and the divergence log is empty, the problem isn't the architecture, it's that recording divergence has become expensive.

Curated: · Written: · Reviewed:

Q&A-3A customer wants to build a component in-house that a vendor already sells. How do you evaluate that decision?

The 60-second answer

Default to buy. Build only when the component is the thing the customer's business is actually good at, when their requirements are unusual enough that every vendor is a poor fit, or when the vendor relationship puts unacceptable risk on their critical path. Everything else is a five-year total-cost comparison, and the number that decides it is the maintenance tail, not the licence fee.

The three tests

Differentiation: would a competitor notice this component existing? If it is SSO, no. If it is the pricing engine or the matching algorithm, yes.

Fit: how many of the vendor's defaults would the customer have to fight? A handful of config flags is normal. A fork of the vendor's core is a signal to build.

Risk: what does the vendor's roadmap, pricing or acquisition do to a component on the customer's critical path? A rate limiter behind an API gateway is low risk. The identity provider every request depends on is not.

Worked comparison (hypothetical figures)

Customer: 40-engineer B2B SaaS, 2,000 end users, deciding whether to build SSO plus SCIM provisioning or buy it. Assume a fully loaded engineer-year of $200k.

Line itemBuildBuy
Initial delivery2 engineers × 6 months = 1.0 engineer-year ≈ $200kintegration, 0.3 engineer-year ≈ $60k
Subscription$02,000 users × $3/user/month × 60 months = $360k
Maintenance and edge cases0.25 FTE/yr × 5 = 1.25 engineer-years ≈ $250kvendor absorbs
Security patching and on-call0.1 FTE/yr × 5 ≈ $100kvendor absorbs
Five-year total≈ $550k≈ $420k

The two cost curves move differently, and that is the real argument. The vendor's price scales with the customer's user count; the build's cost scales with the customer's edge cases and headcount. A customer growing 3× a year should expect the subscription line to grow with it, and a customer with genuinely unusual requirements should expect the maintenance line to grow instead. Build wins when the second curve is flat and the first is steep, and when the component is the differentiator, in which case neither number matters much.

Why the build estimate is wrong in a predictable direction

It prices the happy path. The vendor's product already contains the edge cases their thousand customers found: the IdP that returns a malformed SAML assertion, the tenant that deprovisions users in the wrong order. The customer's users will find the same class of bugs, one at a time, in production, on the auth path, at 2am. That work is not in the estimate and it never ends.

Failure modes, and how to detect them

  • Bus factor of one. Detect with git shortlog -sn -- path/to/component and a CODEOWNERS file with a single entry. If one person owns most of the commits, the component is a resignation letter away from unmaintained.
  • Patching lag. Track mean time from CVE publication to deployed patch. On an auth-path component, anything over 30 days is a finding worth escalating.
  • Silent scope creep. The vendor ships quarterly; the in-house component ships when someone complains. Compare open versus closed issues per quarter: a rising backlog means the maintenance estimate was fiction.
  • Integration lock-in. Count the call sites importing the vendor SDK. Four hundred call sites is a quarter of work to exit; one adapter module is a week.

What I would ask the customer

Would a competitor notice you having this? What happens if the person who built it leaves? What is the five-year cost of each option including staff time? And what does exiting the vendor cost if it goes wrong?

That last question is where I would push back on a pure buy answer: buy, but keep the integration behind an interface you own. One adapter module, contract tests against the vendor's API, and a flag on the swap. That converts an unbounded exit cost into a bounded one, which is usually worth more than the licence-fee difference.

Curated: · Written: · Reviewed:

Q&A-18How do you evaluate two vendors whose feature lists both cover your customer's requirements?

First 90 seconds

Both vendors cover the requirements, so the feature list has done its job and stops being the decision. I'd move to three things it can't show: the price at the customer's projected volume, the cost of leaving, and evidence of how each behaves when something breaks. I'd agree the weighted criteria with the customer before the next demo, so the scoring can't be reverse-engineered toward whoever presented better.

Score it, with the weights written down first

Scores are 1–5; each contribution is weight × score, so the total is checkable line by line.

CriterionWeightA scoreA contribB scoreB contrib
3-year TCO at projected volume30%41.2030.90
Exit cost (export format, egress, re-integration)20%20.4051.00
Reliability evidence vs SLA20%51.0040.80
Support model in the customer's timezone15%40.6030.45
Security artefacts (SOC 2 Type II, pen-test summary, subprocessor list)10%50.5040.40
Roadmap alignment and vendor stability5%30.1530.15
Total100%3.853.70

A wins by 0.15, which is inside the noise of the weights, so the honest move is a sensitivity check rather than a verdict. Shift 10 points from TCO to exit cost: TCO 20%, exit 30%, reliability 20%, support 15%, security 10%, roadmap 5%:

  • A = 4×0.20 + 2×0.30 + 5×0.20 + 4×0.15 + 5×0.10 + 3×0.05 = 0.80 + 0.60 + 1.00 + 0.60 + 0.50 + 0.15 = 3.65
  • B = 3×0.20 + 5×0.30 + 4×0.20 + 3×0.15 + 4×0.10 + 3×0.05 = 0.60 + 1.50 + 0.80 + 0.45 + 0.40 + 0.15 = 3.90

The ranking flips. The decision hinges entirely on how much the customer values being able to leave, so argue that explicitly instead of letting the total pretend to decide.

Make the exit cost a number

"Proprietary format" is not a figure. If the dataset is 40 TB and the vendor passes through S3 egress at list ($0.09/GB for the first 10 TB/month, after the first 100 GB free), a full export is 40,000 GB × $0.09 = $3,600 per pull, plus the engineering weeks to re-ingest into the alternative. Vendor B exporting Parquet to S3-compatible storage makes that number near zero. Same for the SLA: 99.9% monthly availability permits 43 minutes of downtime (0.1% of 43,200 minutes), 99.99% permits 4.3. Read the credit schedule too: a 10% credit on a $10k/month contract returns $1,000 for a six-hour outage, so the SLA is not the risk control. The vendor's 12-month status-page history is.

Get evidence, not claims

A four-week trial against the customer's own data and integration, with exit criteria written before it starts: restore from backup into a clean environment, run a schema migration while under load, kill a node and measure failover, and instrument p99 on the customer's real query mix. A vendor that won't let you test failure paths has answered the question.

Three reference calls at similar size and industry, scripted around one question: walk me through your worst incident in the last 12 months. Time to first response, time to resolution, and whether you renewed. Ask for a churned customer if one exists.

Failure modes

Both feature lists are true in a demo tenant with 10,000 rows; the gap appears at 10^8. A POC that only exercises the happy path proves nothing. Reference lists are curated to happy customers. Weighted scoring done after the demos gets rationalised toward the preferred vendor: the fix is agreeing weights first and writing them down. Support nine timezones away means a P1 waits overnight.

What would change the recommendation

A 10x volume assumption, an acquisition on the horizon (exit cost dominates), or a roadmap that treats the customer's use case as a side bet. If the two genuinely tie after the sensitivity check, take the one that is cheaper to leave.

Curated: · Written: · Reviewed:

Q&A-31A customer asks you to compare cloud providers for their workload. How do you approach it without a religious answer?

The 90-second answer

Ask what the workload actually is before naming a provider. Three questions decide most of these: what the dominant cost line is (compute, storage, or egress), what compliance regime and data-residency rules apply, and what the team already runs. Then score the providers against criteria you weighted with the customer, and show the scoring.

For mainstream workloads (a containerised API, a Postgres-backed app, a batch pipeline) the technical gap between AWS, Azure and GCP is small. The decision is usually commercial and operational: existing commitments, existing skills, and a handful of specific services.

Where the money actually goes

Egress is the classic trap. List prices to the internet, us-east-1 / East US / us-central1, early 2025:

ProviderFirst 10 TB/moTiering above that
AWS (S3)$0.09/GB after 100 GB free$0.085 at 10 TB, $0.07 at 50 TB, $0.05 at 150 TB
Azure (zone 1)$0.087/GB after 100 GB free$0.083 at 10 TB, $0.07 at 50 TB, $0.05 at 150 TB
GCP premium tier$0.12/GB$0.11 at 10 TB, $0.08 at 150 TB
GCP standard tier$0.085/GB$0.065 at 10 TB, $0.05 at 150 TB

Worked example, 100 TB/month out of object storage to the internet (1 TB = 1024 GB, as the tiers are billed):

  • AWS: 100 GB free, then 10,240 GB × $0.09 = $921.60, 40,960 GB × $0.085 = $3,481.60, remaining 51,100 GB × $0.07 = $3,577.00 → $7,980.20/month.
  • GCP premium: 10,240 GB × $0.12 = $1,228.80, remaining 92,160 GB × $0.11 = $10,137.60 → $11,366.40/month.

That is $3,386/month, about 42% more, on the same workload, and it never appears in an instance-price comparison.

The other hidden lines: cross-AZ traffic at $0.01/GB in each direction in AWS and Azure (so $0.02/GB round trip), and AWS NAT Gateway at $0.045/hour plus $0.045/GB processed. A chatty service mesh can outspend its compute on those two.

Commitments

Vendor-published maxima: EC2 Reserved Instances up to 72%, Compute Savings Plans up to 66%, Azure Reserved VM Instances up to 72%, GCP committed use discounts up to 57% (70% for memory-optimised). Enterprise agreements (AWS EDP, Azure MACC) trade a multi-year spend commitment for a negotiated discount that is not published. Hypothetically, 15% off a $5M/year bill is $750k/year: larger than any technical difference you will find between the three.

Egress lock-in is also weaker than it was: AWS has waived data-transfer-out charges for customers moving off since March 2024, and the EU Data Act (in force January 2024, applying from September 2025) pushes the same way.

Compliance and regions

Check this first, because it eliminates options. FedRAMP High, DoD IL4/IL5, HIPAA BAA, EU residency and sector rules are authorised service-by-service, not provider-wide: AWS GovCloud, Azure Government and Google's Assured Workloads each cover a different subset. Sovereign offerings (AWS European Sovereign Cloud, Azure EU Data Boundary, Google Sovereign Cloud) differ in what is actually operated by in-region staff. Region counts move every quarter; check the current list rather than quoting a number.

Specific services

This is where a real technical difference can exist, and it is narrow. BigQuery's on-demand $6.25/TB scanned (first 1 TB/month free) versus Athena's $5/TB changes the cost curve for a warehouse. Spanner's external consistency versus DynamoDB global tables' last-writer-wins versus Cosmos DB's five consistency levels changes what you can build. If the workload touches none of these, do not manufacture a differentiator.

Running the exercise

Weight the criteria with the customer before scoring, so the result is not reverse-engineered from a preference. Price their real workload (same instance family, same storage tier, same egress volume) rather than comparing list prices for "equivalent" instances. Present the scoring, not just the conclusion. And say plainly where the difference is small: a recommendation that claims a large technical gap where none exists gets discounted entirely, along with the rest of the analysis.

Curated: · Written: · Reviewed:

Q&A-61A customer is worried about vendor lock-in. How do you address it proportionately?

By identifying where the lock-in actually is, since the concern is usually attached to the wrong thing.

Compute is rarely the problem. Containers and virtual machines move without much difficulty, and the effort is measured in weeks. What genuinely locks a customer in is data gravity (the cost and risk of moving large volumes, and everything integrated with it) along with proprietary managed services with no equivalent elsewhere, and the skills their team has built.

So the proportionate response is to be deliberate where the cost is high and relaxed where it is low. Keep data in portable formats and standard engines where that is reasonable. Put genuinely proprietary services behind an interface you own, so a replacement is a module rather than a rewrite. And use provider-specific services freely where they earn their keep and the exit cost is modest.

What I would argue against is a full abstraction layer designed for portability. It is a permanent tax: more code, the intersection of both providers' capabilities, and a layer that has to be maintained: paid every day against a migration that usually never happens. Teams that build it typically end up with the worst of both: the abstraction's cost, and provider-specific behavior leaking through it anyway.

The reframing I would offer the customer: the useful question is not how to avoid lock-in but what an exit would cost and whether that is acceptable. Estimate it honestly for the two or three largest commitments. Usually the answer is that a focused migration is cheaper than the insurance policy.

Curated: · Written: · Reviewed:

Q&A-20How do you decide the boundary between what you build and what the customer's team builds?

By where the knowledge needs to end up, not by what is convenient to hand over.

The components the customer will own and change most often should be built by their team, with you alongside. If they build it, they understand it; if you build it and hand it over, the knowledge transfer is a document that nobody reads until something breaks. Domain logic in particular belongs with them, because it changes with their business and they are the ones who know what it should do.

What is sensible for you to build: the parts that need expertise they do not have and are not trying to acquire, the parts that are genuinely one-off, and the scaffolding that establishes patterns (the first service, the pipeline, the infrastructure modules) which then serve as the worked example their team extends.

The arrangement that works best is pairing on the boundary rather than splitting it cleanly. Their engineers building with yours produces both a system and a team that can run it, which is the actual deliverable for most engagements.

Two constraints shape it in practice. Their capacity is usually the binding one: a team already fully committed cannot absorb this regardless of the plan, and pretending otherwise produces a schedule that fails. And a hard deadline pushes work toward whoever is fastest, which is a real trade against knowledge transfer that should be made explicitly rather than by default.

Whatever the split, name the owner of every component before go-live. Unowned components are how systems decay.

Curated: · Written: · Reviewed:

Q&A-35How do you keep a design coherent when several teams are building parts of it in parallel?

The 90-second answer. You don't keep a design coherent by reviewing everything. You keep it coherent by freezing the seams and letting everything behind them move. Three things get frozen before parallel work starts: the interfaces between teams, who owns each piece of data, and the cross-cutting conventions. Identity, error shape, logging, correlation-ID propagation. Everything behind a team's own interface is that team's business, and changing it needs nobody's permission.

Then the frozen parts have to be executable, not prose. A schema plus a recorded example plus a contract test each side runs in its own pipeline turns an agreement into a build failure.

What's frozen, what's free

Frozen before parallel workFree per team
Request/response schema and its semanticsInternal data model
Entity ownership: one writer per entityStorage engine, indexes
Error taxonomy, identity, correlation-ID propagationLanguage, framework, deploy cadence
Versioning and deprecation policyInternal service decomposition

Make the contract fail loudly. Billing's pipeline, Python 3.11:

import json, jsonschema

SCHEMA = json.load(open("contracts/orders_v2.json"))  # owned by Orders, versioned in git

def test_billing_can_consume_order_created():
    payload = json.load(open("fixtures/order_created.json"))  # Billing's recorded example
    jsonschema.validate(payload, SCHEMA)  # raises ValidationError on drift
    assert payload["currency"] in {"USD", "EUR", "GBP"}

Trace what happens when Orders renames total_cents to amount_cents and updates the schema: Billing's next pipeline run fails inside jsonschema.validate with the failing JSON pointer, before anything deploys. The rename PR can't land until Billing's fixture and code move with it. That is the point: the mismatch surfaces in a pull request, not in an integration environment three weeks later.

One writer per entity. Before code, name the single team that writes each entity; everyone else reads through their interface. Two services writing the same table is the coupling that costs a quarter to unpick, and if you don't decide it explicitly you decide it by omission.

Integrate continuously. Get a thin end-to-end path working in week two, stubs and all. Integration as a phase at the end of parallel workstreams is where schedules die; integration as a daily habit is just the build.

Someone holds the whole picture. Usually this role. It works through short cross-team design reviews with the standing to say no, not through a document nobody reads.

Expect drift anyway. The built system diverges from the intended boundaries even when everyone agrees. Detect it mechanically: import-linter or ArchUnit rules in CI for forbidden dependencies, schema-level grants so only the owner's service account can write the table, and periodic call-graph extraction from traces to see which service actually calls which. A boundary that exists only in a diagram is already gone.

Failure modes worth naming. Contract tests only the provider runs, so consumer drift stays invisible; a shared database adopted "temporarily" and never removed; an architect who reviews but can't block, which is advice rather than arbitration; breaking changes shipped with no deprecation window, leaving consumers pinned to an old version forever.

Curated: · Written: · Reviewed:

Q&A-17Question (locked): A customer wants to move an on-premises monolithic application to the cloud with near-zero downtime and no big-bang risk. How do you structure the migration?

Assumptions: one monolith, one primary database, a leased link to the data centre, and a business that can take minutes of read-only but not an outage. Those two constraints rule out moving the whole thing in one window. The shape is discovery, per-component decisions, then waves: each with its own cutover and its own tested rollback.

Start with evidence, not interviews. Flow logs, firewall rules, DNS zones, database session lists and the batch scheduler show what actually talks to what; interviews tell you who owns it. Then establish data gravity: which datastore is the system of record, its size, its change rate, and who reads its tables directly. In a monolith the database is the real integration hub, and the reports reading its tables never appear on the diagram. Those hidden readers are what turn a two-week move into a two-quarter one.

Then a strategy per component, with the reason recorded:

StrategyWhenHere
Rehostnothing needs to changefile/print server, an FTP endpoint
Replatformruntime change, code intactthe bulk: app servers to containers, database to a managed service
Refactorsomething blocks the movea Windows-only COM dependency, 2 components of 12
Replacecommodity functionality, liability codeCRM, monitoring, three internal tools
Retirea year of access logs showing no useusually 10–15% of an estate
Retainregulatory or gravity-boundadjacent batch system, deferred

Wave zero is the landing zone (network, identity, observability, keys) not a workload, but it lands first. The pilot wave should be small, self-contained and visible, carrying real users on real data: an internal admin tool or the read-only reporting path. Two to three weeks of parallel running there proves the run model and the reconciliation tooling before anything that matters moves. Then order waves by the dependency graph: leaves first, the gravity centre last, so the datastore with the most consumers moves when everything around it is proven.

Bulk load plus continuous replication: DMS or Debezium. Do the arithmetic first: 1.2 TB over a 500 Mbps VPN is about 5.3 hours at line rate, and a source changing 1 GB an hour needs only 280 KB/s of delta to hold the gap. Bandwidth is rarely the failure mode; a nightly job that rewrites 200 GB is. Keep the source as system of record through the wave and replicate forward; at cutover flip the source of truth and replicate back for a 72-hour window, so rollback is a routing change rather than a restore. Two-way active-active I avoid: split-brain writes and schema drift are how these projects lose data.

Each wave gets a runbook with timings in minutes, a go/no-go at a named point, and rollback triggers with numbers attached: error rate above 1% for five minutes, replication lag over 60 seconds at the freeze, any reconciliation mismatch. Freeze writes for ten minutes, apply the final delta, have the business owner, not the project team, run the smoke checks, then release. Rehearse the rollback too: a rollback nobody has executed is a document, not a rollback.

During transition, run both paths and compare: nightly reconciliation of row counts and financial totals to the cent, synthetic transactions against both sides, and a routing layer moving traffic 5%, 25%, 100% with the old path still hot. Observability must span both environments in one pane, or a regression after cutover cannot be attributed to either side of the cut.

With a hard deadline, trade scope and blast radius, never the rollback. Widen the waves, rehost more and refactor less, cut dual-running to days, shrink the rollback window from a week to 24 hours. Put the two consequences in writing (more rework later, more exposure per wave) and let the sponsor choose. If the deadline is a lease ending, the fallback is rehosting the stragglers and modernising in the cloud afterwards.

Compliance changes sequence more than design. Where the target must be assessed before regulated data lands, the landing zone and its evidence become a gate, and that data moves last or stays. Keep the audit trail unbroken across the cut; a gap an auditor finds nine months later is not a saving. Customer-managed keys, access logging, retention, and a rollback path that preserves the chain of evidence. If an assessor signs off each wave, build the waves around their calendar and start that conversation in week one.

Curated: · Written: · Reviewed:

Q&A-70How do you handle a customer who wants to lift and shift to the cloud with no changes?

What I'd say in the room. Ask what the driver is before answering. If it's a deadline (a lease expiring, hardware out of support, an acquisition to absorb) lift and shift is the right call and I'd back it, because it decouples the move from the modernisation, and trying to do both under one deadline is how both fail. If the driver is cost, lift and shift does not deliver it, and I'd say that before the contract rather than after the first invoice. Either way the plan is lift and shift plus a named next phase with its own business case, agreed in writing at the same time as the migration scope.

The arithmetic. Illustrative rates, us-east-1-shaped, 2024: re-price for the customer's region and commitment. 200 VMs at 8 vCPU / 32 GB, average CPU utilisation 12%, peak 35%. On-prem: 8 VMs per 2-socket host (64 cores, 512 GB), 25 hosts at $28k over 5 years, 600 W each at PUE 1.6 and $0.12/kWh, plus $60k/yr storage/network/space and 1.5 loaded ops FTE at $150k.

MonthlyAnnual
On-prem, fully loaded$37.5k$450k
Cloud, same shape, on-demand, 24×7$58.4k$701k
Right-sized to 4 vCPU / 16 GB$29.2k$350k
Right-sized + 60 non-prod VMs on 12×5$23.6k$283k

Host amortisation is $140k/yr, power $25k/yr, storage/network/space $60k/yr, ops $225k/yr. Cloud at $0.40/hr per 8 vCPU instance is $292/month per VM, so the like-for-like move is 56% more expensive to run: that is the number to put in front of the customer, not "the cloud is cheaper". Right-sizing to $0.20/hr halves it; running the 60 non-prod VMs 60 hours a week instead of 168 cuts their cost by 64% and takes the total to $283k, 37% below the data centre. The savings come from elasticity and from work you stop doing, and a straight lift captures neither.

Managed databases are the honest exception. The 12 database VMs at 16 vCPU / 128 GB are roughly $7k/month of the lift-and-shift bill; a Multi-AZ managed equivalent is about 2× the raw instance price, so ~$14k/month. That is close to a wash against the ~0.5 FTE of DBA patching, backup and failover work it removes: sell it on reliability and headcount, not on the invoice.

Two things to fix during the move, because retrofitting them across 200 hand-built VMs is a six-month project: the network topology (non-overlapping CIDRs, hub-and-spoke or peering decided up front, egress path chosen) and the identity model (federated SSO from day one, no local accounts). Land everything in infrastructure as code as it arrives, even if the code is a thin wrapper over an image.

Failure modes. The month-three invoice shock, which usually kills the funding for phase two: detect it by tracking actual run-rate against the business case monthly from the first billing cycle. And lift-and-shift-as-steady-state: the VMs become someone's permanent estate and the modernisation case never gets written. Set a date for the right-sizing review before the migration starts; if it hasn't begun by the end of the second quarter, escalate it as a delivery risk, not a finance one.

Curated: · Written: · Reviewed:

Q&A-71A customer's system works but nobody can change it safely. What do you recommend?

The 90-second answer. Don't change it yet. Spend the first few weeks buying the ability to change it, in this order: pin current behaviour with tests, make deploys reversible, make the result observable. Then change something small to prove the loop works. Every improvement attempted before that ability exists is a bet placed on a system nobody can reason about.

Concretely: characterisation tests in the sense Michael Feathers uses in Working Effectively with Legacy Code. Tests that assert what the code does today, including the parts that look wrong. A deploy path where rollback is one command and has actually been executed. Three signals (error rate, p99 latency, and the one business metric the system exists to move) on a dashboard within minutes of a deploy.

StepArtefactExit criterion (targets I'd set, not industry constants)
1. Pin behaviourCharacterisation tests over the top call pathsSuite runs in CI in under 5 minutes; covers the paths carrying most traffic
2. Reversible deploysBlue-green or canary plus one-command rollbackRollback rehearsed in staging and timed at under 5 minutes
3. ObservabilityError rate, p99, business metric on one dashboardAll three visible within 2 minutes of a deploy
4. First changeA low-risk change behind a feature flagShipped, observed, and deliberately rolled back once

What a characterisation test looks like.

# tests/test_characterisation_pricing.py
# Python 3.12, pytest 8.2
from pricing import total

def test_gold_discount_applied_after_tax():
    # Pins today's behaviour. The discount landing after tax is almost
    # certainly a bug; this test detects change, it does not bless it.
    items = [{"price": 100, "qty": 2}]
    assert total(items, {"tier": "gold"}) == 198.0

Trace: subtotal is 2 × 100 = 200; tax at 10% takes it to 220; the gold discount is applied to the taxed total, 220 × 0.9 = 198.0. That is why the assertion is 198.0 and not 180.0. If someone later reorders the calculation, the test fails and the failure is the conversation, which is the point. The test asserts nothing about correctness.

Failure modes, and how you'd catch them.

  • Snapshot suites that get regenerated wholesale. jest -u or approvals verify -r turns the safety net into a rubber stamp. Guard: snapshot updates land as their own reviewed commit, never inside a feature change.
  • Tests pinned to internals. Characterisation tests that reach into private methods break on every refactor and get deleted. Pin at the boundary (HTTP response body, function return value) not at the seam you intend to move.
  • A rollback path that has never run. The rollback is itself untested code. Rehearse it in staging on a schedule and record the wall-clock time; if it takes 40 minutes, you don't have a rollback, you have a second outage.
  • Flags that never expire. Each one is a permanent branch. Give every flag an owner and an expiry date at creation.
  • Observability watching the host, not the business. CPU green while orders fail. The business metric is the one that decides.

The organisational half. A team punished for outages will not take risks regardless of tooling, so the first incident under the new process decides whether the process survives: blameless postmortem, and the change that caused it stays in. An SLO gives that a number: 99.9% availability over a 30-day month is 43,200 × 0.001 ≈ 43 minutes of budget to spend on change. DORA's Accelerate State of DevOps reports put elite performers at on-demand deploys, a change failure rate around 5%, and recovery from a failed deploy in under an hour; that is the shape to aim at, not a target to hit in week one.

I'd also expect some of the fear to be well-founded: a component with no tests and real consequences. Name those explicitly, keep them behind the flag longest, and treat them differently rather than claiming change is now safe everywhere. For the eventual rewrite, Fowler's strangler fig is the pattern: route a slice of traffic to the new implementation, keep the old one serving the rest until the slice is 100%.

Curated: · Written: · Reviewed:

Q&A-43How do you decide how much of a customer's estate to modernise versus leave alone?

By rate of change and by pain, not by age. A twenty-year-old system that works, nobody modifies, and costs little to run is not a problem to be solved; modernising it spends money and introduces risk to produce the same outcome.

The candidates worth changing are where the business needs to move and the system will not let it: components that block a capability the customer needs, that change frequently and are expensive to change, that carry unacceptable risk. Unsupported platforms, unpatchable dependencies, a single person who understands them, or whose run cost is genuinely large.

Map their estate on two axes: how often it changes, and how much pain it causes. High on both is where the return is. Low on both is where the correct action is to leave it, document it, and revisit if the situation changes, which is a recommendation customers rarely receive and often need.

Where change is warranted, prefer containment over replacement. A facade in front of the legacy system, with capabilities extracted one at a time, delivers value incrementally and can be stopped at any point. A full rewrite delivers nothing until the end and is the pattern with the worst track record in this industry.

The honest framing for the customer is that modernisation is an investment with a return, not hygiene. For each component, what does changing it enable, and what does it cost? Components that fail that test should be named as deliberate exclusions in the proposal, so leaving them is a documented decision rather than an omission.

Curated: · Written: · Reviewed:

Q&A-8How do you estimate a migration that involves a legacy system nobody in the room fully understands?

The 90-second answer

I'd give a range, not a number, and I'd make the first line item inside that range a paid discovery phase. The uncertainty here isn't in how long it takes to move a table: that's estimable. It's in how many tables, how many consumers, and how much logic lives outside the application. You can't estimate a count you haven't counted, so the estimate's job is to buy the counting.

Assumptions I'd state out loud before any number: the legacy system keeps running during migration, so no big-bang cutover; we can read production data and access logs; and whoever knows the system is available a few hours a week. If any of those is false, the range moves before anything else does.

Estimate the count first, the unit second

Two to four weeks of discovery, two engineers, producing an interface inventory, a data-volume profile, and a list of scheduled jobs. That's 4–8 person-weeks. Against a migration that will run 6–12 months with five engineers, it's under 5% of the budget to collapse the top of the range.

Arithmetic: pre-discovery range 4–11 months, post-discovery 5–8. You spent 6 person-weeks to remove a 7-month tail. Had you committed to 4 and it was 11, you'd have burned 7 months × 5 engineers = 35 person-months of overrun. The discovery phase is cheap insurance against exactly that.

Probes that pay for themselves in a day

ProbeWhat it measuresCostWhat it narrows
Distinct callers in the access logundocumented consumers1 hourintegration count
Row counts and growth per tabledata volume, cutover window2 hourscutover strategy
Procedure/trigger catalog (pg_proc, sys.sql_modules)logic outside the app1 dayrewrite scope
Peak TPS from existing monitoringdual-write feasibility1 hourwhether both systems can run at once

For the first row, on a combined-format access log:

# distinct clients hitting legacy endpoints in the last 30 days
awk '$7 ~ /^\/(legacy|v1)\// {print $1}' access.log | sort -u | wc -l

Field 7 is the request path in that format; field 1 is the client. If that count is 40 and the architecture diagram shows 6, you've just found the estimate's real driver.

What discovery actually finds

Four classes, and you budget for the class rather than being surprised by the instance: undocumented consumers (detected by the log probe above, or by shadow traffic with response diffing); business logic in triggers and stored procedures (detected by the catalog query plus a search for dynamic SQL); data violating constraints the schema never enforced (detected by running validation queries against a read replica before migration); and a scheduled job on someone's workstation (detected by egress monitoring or a cron inventory. It only shows up when that machine is off).

Commercial structure

Fixed price on a system nobody understands transfers risk to whoever is most optimistic, which serves neither side. I'd price discovery as a fixed, small engagement, then re-estimate at its end with a gate the customer can accept or walk from. After that, time-and-materials with a cap, or milestone-based with a re-estimate trigger if the interface count moves more than, say, 30%.

One more input worth having: your own last three migrations, actual versus estimated. If they ran 1.6× over, apply that factor to the range and say so: it's more defensible than any bottom-up number built on a system nobody in the room understands.

Curated: · Written: · Reviewed:

Q&A-12How do you handle a customer integration where the other side's API is poorly documented and unreliable?

Design so their unreliability is contained, and get the facts you need empirically since the documentation will not give them to you.

Empirically first: call it, record real behavior, and build a small conformance suite. What are the actual response times and their distribution, what error shapes appear, is it idempotent, what are the undocumented rate limits, does it return 200 with an error in the body. That last pattern is common in older enterprise APIs and breaks naive clients silently.

Then contain it. An anti-corruption layer: one module that owns every call to them and translates into your own domain model: means their oddities do not leak into the rest of the system, and a change on their side has one blast radius. Timeouts derived from measured latency, a circuit breaker, bounded retries with backoff, and idempotency on anything that mutates.

Then decide the degraded behavior with the customer, because it is a business decision: queue and retry, fall back to a cached or default value, or fail the operation visibly. For a partner integration, queueing with a visible backlog is often right: the work is not lost and someone can see it is stuck.

Finally, instrument it separately and share the data. When their API is the cause of a problem, having a graph of their error rate and latency turns a dispute into a conversation, and it is frequently the thing that gets the other side to fix something.

And put the observed behavior in writing as an assumption in the design, since it is not in their documentation.

Curated: · Written: · Reviewed:

Q&A-15A customer has an existing vendor contract that constrains your design. How do you work with that?

First 90 seconds. Ask for the contract, not a summary of it. The person who told you "we're locked into vendor X" is usually repeating an assumption; the order form and the MSA say something narrower. Five things decide the design: end date, what the obligation actually names (a spend, a product, a workload), minimum volume, exit terms, and who inside the customer owns the relationship.

Then answer directly. If the obligation is a committed spend, it is already sunk and the only design question is where this vendor fits well. If it names a product for a specific workload, that workload uses the product and everything else is free. A genuine technical mandate, the product must carry this traffic, is rare, and when it exists you escalate with evidence rather than route around it quietly.

Worked example (illustrative figures). Three-year contract, 14 months remaining, $150k/yr minimum spend on a managed event-streaming product, exit fee equal to the remaining commitment, auto-renewal unless notice is given 90 days out.

Remaining obligation: 14/12 × $150k ≈ $175k, owed whether or not you use the product.

Routing around it: a self-managed equivalent at roughly $10k/month of compute plus half an FTE of ops (~$90k loaded) ≈ $210k/yr. So the "avoid the vendor" option costs $210k/yr on top of the $175k you still owe. That is the number the sponsor needs, not "it's expensive."

Contract termWhat it actually obligesDesign move
$150k/yr minimum, 14 months leftMoney, not a product choiceUse the vendor where it fits; the spend is sunk either way
Names the product for event streamingThat workload onlyAdapter interface; other workloads are unconstrained
90-day notice before auto-renewalA calendar datePut the renewal date in the design doc; decide 6 months out
Exit fee = remaining commitmentSunk costDo not let it drive the interface design

Failure modes.

  1. Designing against an assumed constraint. Detection: ask for the clause. If nobody can produce the sentence within an hour, it isn't a constraint. Cost: a parallel stack you pay for twice, and a migration nobody needed.

  2. Vendor types leaking past the adapter. Detection: a CI check. grep -rn "vendor_sdk" src/ --exclude-dir=adapters fails the build. Cost: exit becomes a rewrite of every call site instead of swapping one module.

  3. Absorbing the cost silently. Detection: the vendor line item in the run-cost report grows and no design doc mentions it. Cost: the sponsor renegotiates without your input, or the number lands at renewal as a surprise.

  4. Missing the notice window. Detection: renewal date in the design doc and the team calendar, with a decision point six months out. Cost: another full term on the same terms.

Design moves. Use the contracted product where it fits, and use it properly: half-hearted adoption gives you the cost without the benefit. Put the parts that are a poor fit behind an interface so the cutover is a swap rather than a project. Treat the end date as a design input: if it is 12 months out, a replaceable component is worth real effort and the cutover gets planned instead of forced.

Curated: · Written: · Reviewed:

Q&A-25The customer wants to reuse a system they already paid for, and it is a poor fit. How do you make that case?

Not as a technical argument, and not by dismissing the investment. Sunk cost is a real emotional and political fact inside their organisation, and treating it as a fallacy to be pointed out is how a good recommendation gets rejected.

Make the case forward-looking: the question is not what was spent, but which option costs less from today. Price both: adapting the existing system, including the ongoing cost of the compromises it forces, against the alternative including migration. Often the existing system loses on the run cost rather than the build cost, and the five-year view makes that visible where the first-year view does not.

Be specific about the misfit rather than general. "It is not designed for this" is an opinion. "It handles 200 transactions a minute and you need 2,000, and the constraint is a single-threaded component we cannot change" is a fact, and it is much harder to argue with.

Look genuinely for the middle option, because there usually is one: keep the system for the workload it does well, and put the new requirement alongside it behind a common interface. That preserves the investment, limits the change, and is frequently the right answer on the merits rather than merely the diplomatic one.

And find out whether someone's credibility is attached to that system. If so, the conversation needs to happen with them privately before it happens in a room, or the technical merits will not decide it.

Curated: · Written: · Reviewed:

Q&A-73A customer asks you to review an architecture their previous supplier delivered. How do you approach it?

I run it as a fitness assessment against the customer's current requirements, and I say the incentive problem out loud in the first meeting: I'm a supplier grading a competitor, so anything I find gets discounted unless I'm equally specific about what works. Then I ask for four things before reading any code: the requirements the system was built against (original SOW or design doc), the last six months of incident tickets, the on-call rota, and the change log. Much of what looks wrong in an inherited design was correct for a constraint that has since moved. A single-AZ database is a reasonable call for a pre-revenue product and a serious one for a system now carrying payroll.

Then two passes, kept separate. Fitness: does it meet today's load, availability target and compliance boundary. Condition: can the team that owns it operate, patch, document and change it. I deliver the findings in a walkthrough with the owning engineers in the room, not as a document drop: their reaction to each item tells you more about operability than the architecture diagram does.

Findings get sorted by consequence, not by severity adjectives: risk now, problem within a year, difference of style. Style goes at the bottom or nowhere; it's what makes a review read as posturing.

Worked example: figures illustrative, shape taken from a real engagement. A 3-tier app on EC2 behind an ALB, PostgreSQL 11.22 on a single m5.2xlarge, Terraform state in a local file on one engineer's laptop, CloudWatch alarms on CPU only.

FindingEvidenceConsequenceOptionRough cost
PostgreSQL 11.22, EOL 9 Nov 2023, no minor upgrade in 14 monthsSELECT version(); RDS maintenance window logRisk now: unpatched CVEs on a database holding customer PIILogical replication to PG 15, cut over in a maintenance window3–4 engineer-weeks
Single-AZ, 400 GB, no replicaRDS console; a restore drill they ranRisk now: measured restore ~50 min vs AWS's documented 60–120 s Multi-AZ failoverEnable Multi-AZdoubles instance cost, +$280/mo at m5.2xlarge on-demand list
Terraform state local, 18 months of console changesterraform plan reports 60+ drifted resourcesProblem within a year: nobody can rebuild the environmentMove state to S3 + DynamoDB lock, import the drift2 engineer-weeks
No tracing, CPU alarms onlyDatadog not installed; alarm listProblem within a year: incidents take hours to localiseOpenTelemetry SDK + APM1 week + per-host

Every real finding gets an option and a rough cost, so it's actionable rather than an assessment. And I check whether the actual question is technical: sometimes it's "should we keep this supplier", and that's worth surfacing directly rather than burying it in section 4.

Curated: · Written: · Reviewed:

Q&A-77A customer's nightly batch job has grown until it no longer finishes before the business day. What are the options?

Establish whether the job is CPU-bound, I/O-bound, or blocked on another system before changing anything. The remedies differ and the first guess is usually wrong.

Instrument the phases: extract, transform, load, and any wait on a downstream system. In PostgreSQL, pg_stat_statements plus EXPLAIN (ANALYZE, BUFFERS) on the top queries by total time. In MySQL, the slow query log and performance_schema. You want the wall-clock share per phase, not per statement count.

If one or two queries dominate, indexing or rewriting them is the cheapest fix by a wide margin. Say the job now runs 9 hours against a 6-hour window, and EXPLAIN shows a sequential scan over a 200M-row table eating 5 of those hours. An index that turns it into an index scan can bring that to minutes, and the job fits. That buys years of headroom at the cost of one migration.

The failure mode: the planner ignores the index if statistics are stale or the predicate isn't selective. Check pg_stat_user_indexes a week later: idx_scan = 0 means you're paying write amplification for nothing.

If the volume itself outgrew the window, process incrementally. This needs a reliable change marker. updated_at is the usual choice and the usual bug: clock skew, or an UPDATE that doesn't touch the column, silently drops rows. A CDC log, Debezium reading the WAL or binlog, is the marker that doesn't lie. Detect missed rows by reconciling per-day counts against the source; a drift of even 0.1% is a bug, not noise.

Parallelising across independent partitions works when the work is separable by key. Eight workers over eight shards is not 8x: the wall clock is the slowest shard. One hot key can eat the entire gain. Measure p99 per shard, not the mean, and watch for connection-pool exhaustion and contention on shared sequences.

If the transform is the cost, move it to a store built for it: Snowflake, BigQuery, Redshift, or DuckDB if it fits on one machine. That's a new system to operate and a new failure surface, so it's a bigger commitment than it looks.

There's a scoping question worth putting to the customer: does all of it need to run nightly? Batch jobs accumulate steps. Some produce outputs nobody reads, or could run weekly. Auditing consumers sometimes removes more work than optimisation does.

The structural answer, if the window keeps shrinking, is incremental or streaming processing: Kafka with Flink or Spark Structured Streaming. It removes the window but hands you checkpointing, exactly-once semantics, and backpressure as things you now own. Don't recommend it until the cheaper options are exhausted and the growth curve is understood.

Curated: · Written: · Reviewed:

Q&A-97A customer wants to move from quarterly releases to weekly. What has to change first?

Frequency is the last thing to change, not the first. Weekly releases multiply whatever per-release risk you already carry, so the work is to make a release detectable and reversible, then raise the rate. DORA's Accelerate State of DevOps reports (2018–2023) put elite performers at deploy-on-demand with a change failure rate around 5% and recovery under an hour: frequency and safety move together, they don't trade off.

GateTypical quarterly stateWeekly-ready target
Gating testsmanual regression pass, 2–3 daysautomated, <30 min on every merge to trunk
Deployrunbook + DBA applies SQL by handone artifact built once, promoted; versioned migrations in CI
Rollbackredeploy previous build, hoursflag kill switch <1 min; artifact rollback <15 min
Detectioncustomer reportcanary at 5% traffic for 10 min, SLO burn alert
ApprovalCAB reviews each releaseroutine deploys are pre-approved standard changes

1. A test suite that can gate in minutes. The quarterly cadence is almost always propped up by a manual regression pass that takes days. Split it: a fast suite (unit + contract + a thin smoke path) runs in under 30 minutes on every merge and is the gate; the 4-hour end-to-end suite runs nightly and on release candidates. This forces trunk-based development: a weekly release cannot absorb a two-week-lived branch.

2. Automated deploy, including migrations. Build the artifact once and promote it through environments. Migrations go through Flyway or Liquibase in CI, no manual DBA step, and use expand/contract so old and new code run against the same schema during rollout: add the column nullable, deploy code that writes both, backfill, switch reads, drop the old column a release later.

3. Tested rollback. Two mechanisms, both exercised: redeploy the previous artifact (blue-green swap, or kubectl rollout undo), and a feature-flag kill switch that takes effect in seconds without a deploy. Measure it: recovery from a bad deploy should be under 15 minutes. If it's an hour, that's the first thing to fix, because a team that can reverse in minutes tolerates far more change.

4. Observability per deployment. Deployment markers in Datadog or Grafana, error rate and p99 latency broken out by version, plus one business metric (checkout conversion, signups). Put an SLO behind it: 99.9% availability over 30 days is 43 minutes of error budget, and a burn-rate alert on that budget is what tells you a release is bad before a customer does. A canary at 5% traffic for 10 minutes catches most bad releases before full rollout.

5. Decouple deploy from release. Feature flags (LaunchDarkly, Unleash, OpenFeature) let unfinished work ship dark. Give every flag an owner and an expiry date; flag debt is the predictable cost.

6. Then raise frequency gradually (monthly, then fortnightly, then weekly) watching change failure rate and recovery time at each step. If change failure rate climbs past roughly 15%, stop and fix rather than push on.

7. Change the approval model. A CAB that meets monthly cannot approve weekly releases. Move to change enablement: routine deploys become pre-approved standard changes with the pipeline as the control, and the board keeps only high-risk changes. In regulated shops this is usually the real blocker, and it's a policy change, not an engineering one.

Curated: · Written: · Reviewed:

Q&A-4Walk me through how you build a total cost of ownership estimate for a proposed architecture.

Infrastructure is the part everyone estimates and it is rarely the largest number. A credible TCO has four categories.

Infrastructure: compute, storage, network egress, and managed service fees, projected across the growth curve rather than at today's volume. Egress and cross-zone transfer deserve their own lines because they scale with usage and are invisible in per-instance pricing.

Licensing and vendor fees, including how the price changes at the next tier. Many vendor contracts are cheap at pilot volume and step sharply at production volume, and finding that out after signing is a common and avoidable failure.

People: the operational load. How many engineers, at what fraction of their time, to run this? A self-hosted database might save a meaningful monthly figure and consume a quarter of an engineer's year, which is usually a net loss. This is the category that decides most managed-versus-self-hosted arguments honestly.

Change: the cost of the migration itself, the parallel-run period where both systems are paid for, training, and the productivity dip while a team learns something new.

Then present it as a range with the assumptions visible, over a three to five year horizon, and identify which assumption the total is most sensitive to: usually the growth rate. A single number invites false confidence and gets quoted back to you eighteen months later.

And include the do-nothing option's cost, since that is what the spend is actually being compared against.

Curated: · Written: · Reviewed:

Q&A-16How do you present a cost estimate you know is uncertain without losing credibility?

State the range, the assumptions, and what would move it, and be specific about which parts are firm and which are not.

A single number implies a precision you do not have, and when reality lands outside it your credibility goes with it. A range with visible reasoning survives being wrong, because the customer can see which assumption failed and that you had flagged it.

Structure it in confidence tiers. The infrastructure cost for a known workload is firm within perhaps ten percent. The licensing is firm if the volume assumption holds, and the volume assumption is the customer's. The integration effort with an unexamined legacy system might be plus or minus fifty percent, and that is the number to draw attention to rather than bury.

Name the sensitivity explicitly: this total is dominated by the growth assumption, so if volume is double the projection the figure rises by roughly this much, and here is the point at which the design itself would need to change.

Then propose how to narrow it, which converts the uncertainty from a weakness into a plan: a discovery phase, a load test against their real data, or a vendor quote at the actual tier.

And separate one-off from recurring. Customers routinely conflate them, approve a project on the build number, and then discover a run cost nobody highlighted. Presenting the five-year picture with both lines prevents a specific and common failure of trust twelve months in.

Curated: · Written: · Reviewed:

Q&A-47How do you compare two designs when one is cheaper to build and the other cheaper to run?

Over a horizon the customer agrees to, with the crossover point identified, and with the reversibility of each stated.

Model both across three to five years including the operational cost: infrastructure, licences, and the staff time to run it, which is the line most often omitted and frequently the largest. Then find the crossover: the month at which the cheaper-to-run option overtakes. If that is eighteen months, the decision is usually easy; if it is seven years, the cheaper build is probably right and the run cost is being over-weighted.

Then adjust for the customer's actual situation rather than the arithmetic alone. A startup with eighteen months of runway should optimise for build cost and speed, because the run cost in year four is a problem they will be fortunate to have. An established organisation with a stable, long-lived workload should weight the run cost heavily. Their cost of capital and whether the spend is capital or operating expenditure also matter more than engineers usually expect, and their finance director will have a view.

Then weigh reversibility, which often decides it. The cheaper build that can be replaced later is a low-risk bet; the cheaper-to-run option that locks in a platform or a data model is a larger commitment and should be held to a higher standard of evidence.

Present it as a model with visible assumptions and a sensitivity on the growth rate, so the customer can test their own beliefs against it rather than accepting a conclusion.

Curated: · Written: · Reviewed:

Q&A-24How do you handle a request for a fixed-price quote on work with significant unknowns?

First 90 seconds. Ask one question before you quote: is this number for a budget approval, or is it a hard cap you cannot exceed? Those need different contracts. If it is approval, a range plus a paid discovery phase works. If it is a cap, sell a not-to-exceed: the customer gets a ceiling, you get paid for hours actually worked up to it. Then say plainly that a fixed price on unknown scope transfers risk neither side can price, and that you will price the part you do understand.

Pick the structure that matches the constraint.

StructureCustomer getsYou carryUse when
Fixed-price discovery, then fixed-price buildA firm number before the large spendOnly the discovery riskUnknowns sit in requirements or design
T&M with a not-to-exceed ceilingA hard cap, no overrunEfficiency risk up to the capThe real constraint is a cap, not a number
Fixed price per iteration or modulePredictable unit costPer-unit estimate riskScope decomposes into similar slices
Fixed-price incentive fee (target cost, target fee, ceiling, 70/30 overrun share)Lower price if it goes wellA share of the overrunBoth sides can influence cost

Discovery is the cheapest lever. A three-week discovery at a blended rate, roughly 8-10% of the estimated build, buys a design and a firm estimate. If the project is unwise, the customer learns that for the price of discovery instead of the price of the build.

Price the risk where it can be seen. Take a three-point estimate: optimistic 10 weeks, most likely 14, pessimistic 24. PERT mean is (10 + 4×14 + 24)/6 = 15 weeks, and σ = (24−10)/6 = 2.33 weeks. Price at P80, roughly mean + 0.84σ ≈ 17 weeks. That is about 13% over the midpoint, and it goes in the contract as a named contingency line, refundable if unused. A competitor who quotes the 15-week midpoint is not cheaper; they are unpriced, and they will recover it through change orders.

Every assumption gets a trigger and a price. Not prose: a line item. "Client supplies production-like test data by week 2; if not, integration testing adds 3 weeks at $X." A contract with twenty such lines is honest and enforceable. The same twenty assumptions written as narrative is a dispute scheduled for month four, because nobody can point to the clause that was breached.

Failure modes and how you see them.

  • Change-order treadmill: the fixed price was wrong. Detect it when you issue more than one change order per month, or when change orders pass 15% of contract value.
  • Winner's curse: a rival bids below your P50. They will claw it back later; tell the customer that pattern exists rather than matching the number.
  • Silent overrun: the team absorbs the gap and morale goes first. Detect it by tracking burn against plan weekly, not at the milestone.

Say the incentive problem out loud. A fixed price on unknown scope pushes both sides to argue about what was included rather than about the outcome. Naming that in the room usually gets you to a capped structure or a discovery phase faster than any amount of padding. The customer's budgeting constraint is real; the fixed price is usually just the easiest way they know to express it.

Curated: · Written: · Reviewed:

Q&A-10How do you decide what belongs in a statement of work versus what stays flexible?

Fix what decides acceptance and price. Leave flexible what you cannot yet price, but never leave it silent: attach a rate and an approval path, so flexibility has a number instead of becoming a dispute.

ItemFixed in the SOWFlexibleCost of getting it wrong
Outcome and acceptance criteriaYes: measurable, e.g. reconciliation mismatch under 0.1% across 30 days of backfilled data—"Done" argued for weeks
Systems in scopeYes: named, with a count—Scope creep priced at zero
Assumptions behind the estimateYes: each with a number—The bid is unfalsifiable
Customer obligationsYes: with response times—Schedule slips with no remedy
Internal design, library choice, sequencing inside a phase—YesA change request per design decision
Staffing mix—Yes, within a named capMargin quietly eaten

Worked example, hypothetical figures but the shape is the point. A 12-week migration: 5 people, 60 working days each, 300 person-days at a blended $1,600/day = $480,000. The estimate assumes 4 upstream source systems. Each source costs roughly 9 person-days: connector, field mapping, reconciliation tests. If the real number is 11, that is 7 × 9 = 63 person-days = $100,800, or 21% of contract value.

Two ways to handle it. Pad the bid: add 63 person-days up front, the price becomes $580,800, and the customer who genuinely has 4 sources pays 21% over the odds, and you still have no answer if it turns out to be 15. Or fix the count at 4 in the SOW and write the rate next to it: "each additional source is a change order at 9 person-days." Now the unknown is priced, the customer decides how many sources to include, and nobody negotiates the rate mid-project.

Change process thresholds, again hypothetical and tuned to contract size: under 5 person-days, the delivery lead approves and no amendment is needed; 5–20 person-days, the customer's product owner signs a change order; over 20 person-days, or any movement of the end date, re-baseline and re-sign. Agree these before the first change, not during it.

Customer obligations get numbers too: access requests answered within 5 business days, a named decision-maker, test data by week 2. A six-week wait for firewall access is a six-week slip, and the SOW should say the dates move with it: otherwise the slip lands on your margin.

Write the exclusions down. "Data quality remediation in source systems", "training beyond two sessions", "production support after handover": each one is a dispute that otherwise arrives disguised as an assumption.

Phase high-uncertainty work. A 3-week fixed-price discovery at, say, $60,000 that produces a firm estimate for the build: if the build lands at $480,000, the discovery is 12.5% of it and buys a real decision point. The alternative is a padded fixed bid that hides the same risk at a worse price.

Curated: · Written: · Reviewed:

Q&A-22How do you sequence a multi-phase delivery so the customer gets value before the end?

Order phases by value delivered and risk retired, not by architectural layers. The tempting sequence (infrastructure, then platform, then services, then features) delivers nothing usable until the end and gives the customer no evidence that any of it works.

The useful shape is a thin vertical slice first: one real user journey, end to end, in production, however narrow. It proves the architecture, exercises the deployment path, surfaces the integration problems, and gives the sponsor something to show. It also converts a large number of assumptions into facts while there is still time to act on them.

Then extend along whichever axis carries the most risk or value: more journeys, more volume, more of the estate migrated.

Two constraints shape the order. Dependencies that cannot be faked, an identity integration or a network path that everything else needs, have to come early even without visible value, and that should be stated rather than hidden. And the customer's own calendar matters: a retailer cannot cut over in November, and a finance system has quarter-end.

Each phase needs a definition of done that stands alone, and a state that is stable if the next phase is delayed or cancelled. Funding gets cut, priorities change, and a phase boundary that leaves the customer with a half-migrated system is a liability you created.

I would also front-load anything that reduces the estimate's uncertainty, because a firmer number for phase three is worth a great deal at the end of phase one.

Curated: · Written: · Reviewed:

Q&A-92How do you decide what belongs in the first release versus later, when the customer wants everything?

By separating what is needed for the system to be usable at all from what is needed for it to be complete, and by making the trade visible rather than deciding it privately.

The test I apply to each item: can a real user complete a real task end to end without it? If yes, it is a candidate for later. That is stricter than it sounds and it eliminates a great deal, because much of what feels essential is refinement of a path that already works.

The exceptions that must be in the first release regardless of that test: anything that is expensive to retrofit, and anything a user would be harmed by its absence. Authentication and the permissions model, the audit trail if one is required, and the data model itself belong in the first category: retrofitting a tenancy boundary or an audit requirement is far more expensive than building it in. Data protection obligations belong in the second.

Then put the ranked list in front of the customer rather than presenting a scope decision as made. Ranking their own priorities is a conversation they can have; being told what was cut is one they resist.

Two techniques that help. Ask what they would ship if the date were two months earlier, which produces a sharper ranking than asking what is important. And identify what can be handled by a manual process initially (a report someone runs, an approval by email) because that often defers real engineering effort at very low cost for a low-volume workflow.

Curated: · Written: · Reviewed:

Q&A-2How do you scope a proof of concept so it actually reduces risk rather than becoming a small version of the whole system?

Start by naming the single assumption that, if wrong, kills the project. A POC exists to test that assumption and nothing else.

That framing does most of the work. If the risk is whether the legacy mainframe can sustain 500 lookups a second, the POC is a load harness against that interface: no user interface, no authentication, no data model. If the risk is whether the customer's users will accept an asynchronous workflow, the POC is a clickable prototype with no backend at all. These are entirely different artifacts, and confusing them is why POCs run long.

Write the success criterion before building: a number and a threshold, agreed with the customer, that determines pass or fail. Without it the POC ends in a discussion about whether it worked, which the most senior person in the room wins.

Time-box it, and state explicitly what is excluded: error handling, security, scale, operability. Then say clearly, in writing and more than once, that the POC is not the foundation of the production system. The most common failure in this role is a POC that succeeds so visibly that the customer asks to ship it, and the team spends a year paying for shortcuts taken to answer a question that has already been answered.

If the answer turns out to be no, that is a successful POC. Cheap disconfirmation is the entire point.

Curated: · Written: · Reviewed:

Q&A-14How do you decide whether a proof of concept has succeeded?

Against the criterion written before it started, and only against that.

The criterion should be a measurement with a threshold: the legacy interface sustains 500 lookups a second at under 200ms p95 for a sustained hour, or eight of ten pilot users complete the task without assistance. Written down, agreed with the customer, and not revised after seeing the result.

That last constraint matters more than it sounds. The strong pull at the end of a POC is to reinterpret the result favourably, because a POC that fails feels like wasted money. It is not: a cheap disconfirmation has just saved the cost of building the wrong thing, and framing it that way with the customer beforehand is what makes an honest result possible.

Three outcomes are worth distinguishing. The assumption holds: proceed, and be explicit that the POC code is not the production foundation. The assumption fails: the design needs to change, and you now know that for the price of a few weeks. The result is ambiguous: usually the criterion was not sharp enough, or the test was not representative. A load test against empty tables, a pilot with unusually motivated users.

The POC should also produce a written record of what was learned beyond the headline result, because the incidental discoveries (an undocumented rate limit, an authentication quirk, real latency numbers) are often worth as much as the answer itself and are otherwise lost when the branch is deleted.

Curated: · Written: · Reviewed:

Q&A-87How do you make sure a customer understands the difference between a pilot and production readiness?

Core answer, first 90 seconds. Write the gaps down before the pilot starts, attach them to the pilot's exit criteria, and re-read them the moment the pilot succeeds. The distinction isn't a briefing you deliver once: it's a document that has to survive the pilot going well, because success is exactly when admitting the gap gets expensive.

The artefact is a one-page pilot scope table, agreed with the sponsor at kickoff. Every row is a production requirement, the pilot's actual state, and what it costs to close. Worked example: a claims-triage tool piloted with 40 internal users; figures illustrative:

Production requirementPilot stateCost to close
HA, multi-AZ, failoverone instance, one AZ, ~4 min cold restart2 eng-weeks
DRnightly snapshot, never restored1 eng-week + a restore drill
Security review (threat model, pen test)none; SSO only3-week lead time, 1 eng-week
Load test at production volume40 users, peak 12 rps1 eng-week; target 120 rps
Monitoring, SLOs, pagingapp logs only1 eng-week
Support commitmentbest-effort, business hours0.5 FTE on-call
Retention, PII handlingreal claim data, no retention policylegal review

That's roughly 6 engineer-weeks plus a 3-week security lead time. With two engineers, the earliest defensible production date is about six weeks out. The arithmetic is the point: it turns "why can't we just open it up?" into a date and a number, and a number can be argued with, funded, or cut. A judgement about caution cannot.

The conversation at the moment of success. The sponsor asks to open it up. The answer is the date, plus the two levers that move it: dropping the pen test is not one of them; cutting scope to read-only for a named audience takes it from six weeks to three. Naming the levers in advance is what keeps the exchange a planning meeting rather than a negotiation about whether you're being overly careful.

The middle path, when pressure is real. Extend to a limited, informed audience with constraints written into the same document: read-only, no PII, named users only, an end date, and the production work running in parallel. That is better than refusing outright or letting the pilot quietly become production.

Failure modes, and how you'd see them.

  • Scope creep inside the pilot. Features get added, so the pilot stops testing the question it was built for. Detect it by holding the pilot to its original success metric; if the metric changed, it's a different pilot and needs its own scope table.
  • The pilot becomes production by attrition. Detect it by watching for real customer PII, traffic above the pilot cap, or support tickets landing on the on-call rota. Any of those means the gap is already being paid for by users rather than by the plan.
  • "We'll harden it later." Later never arrives, because the pilot's success removes the urgency that would have funded the work. Close this by tracking the gap-closing items as their own dated backlog with an owner, not as a follow-up to the pilot.

The one thing that makes all of this work is that the list exists before anyone is invested in the pilot succeeding. Written afterwards, it reads as an excuse; written first, it reads as a plan.

Curated: · Written: · Reviewed:

Q&A-88What is your approach to designing for a customer whose requirements you expect to change substantially?

What I'd say first

"Substantially change" is doing a lot of work in that question, so I'd pin it down before answering: which axis is expected to move. The data model, the business rules, the integration surface, or the volume? Then I'd say the thing I actually believe: I optimise for the cost of change on that axis, and I pay for it by accepting a worse fit to today's spec. If the customer can't name the axis, I don't build flexibility; I build the simplest thing and keep the exit cheap.

Where the money actually goes

Cost of change is not uniform. Roughly, cheapest to most expensive:

DecisionHow it changesCost
Volatile attribute stored in a JSONB columnUPDATE a rowminutes, no deploy
New pricing tierINSERT a rowminutes, no deploy
New nullable columnALTER TABLE ... ADD COLUMN (Postgres 11+, metadata-only)seconds
Column type change on a 200M-row tablefull rewrite under ACCESS EXCLUSIVE~33 min of blocked writes at a hypothetical 100k rows/s
Internal module boundaryrefactor + one deployhours
Published API / network boundaryversion, dual-run, migrate clientsweeks to months

The gap between row one and row six is the whole argument. So the design rule is: push the volatile thing as far up that table as it will go without lying about its structure.

The volatile thing, as data

Pricing is the classic case. Instead of a switch on customer type, tiers are rows:

# Python 3.12
from dataclasses import dataclass
from decimal import Decimal

@dataclass(frozen=True)
class Tier:
    min_qty: int
    unit_price: Decimal

def price(qty: int, tiers: list[Tier]) -> Decimal:
    applicable = max((t for t in tiers if t.min_qty <= qty), key=lambda t: t.min_qty)
    return applicable.unit_price * qty

With tiers = [Tier(1, Decimal("10.00")), Tier(100, Decimal("8.50")), Tier(1000, Decimal("7.00"))] and qty = 250, the generator yields the first two tiers, max picks Tier(100, 8.50), and the call returns 2125.00. Adding a 500-unit tier is an INSERT, not a release.

What I would not do

I would not make everything configurable. A rules engine that encodes business policy in a DSL is a second programming language with no debugger, and it usually guesses wrong about which axis varies. The test I use: ask the customer what changed in the last 12 months and count. If none of the last dozen pricing changes were structural, the tiers table is enough and a rules engine is a liability.

Failure modes and how I'd spot them

  • Config sprawl: nobody can answer "why was this order priced at X?". Detect by trying to answer it from logs alone.
  • Premature network boundary: a service split that has to be re-split. Detect by counting cross-boundary calls per request; if it's chatty, the boundary is wrong.
  • Migration lock: a type change that blocks writes. Detect with pg_locks during a staging run, not in production.

Process

Short phases with a decision point at each, and a change process agreed before the first one: a named owner, a stated turnaround, and a change budget as a percentage of the phase. That turns a requirement change into a plan revision rather than a contract dispute.

The close

I'd tell them plainly: this costs a bit more to build and a lot less to change, and here's the axis I'm betting on. Then it's their call, made knowingly.

Curated: · Written: · Reviewed:

Q&A-94How do you handle discovering, mid-project, that the architecture you proposed will not meet a key requirement?

Raise it immediately, with the evidence, an assessment of the options, and a clear statement of what it costs: in that order, and to the sponsor rather than only within the team.

Immediately is the whole thing. The instinct is to work the problem quietly and hope it resolves, and every week of delay narrows the options and increases the cost of the eventual conversation. A problem raised in month three with three viable responses is a manageable event; the same problem raised in month six with one is a crisis.

Bring evidence rather than a concern, so the discussion is about what to do rather than whether it is real. A load test result, a measured latency, a limit in a vendor's documentation.

Bring options with costs and consequences: change the design and what that adds; change the requirement, which is sometimes possible once the price of meeting it is visible; accept a limitation, if it is tolerable and documented; or stop, if the project no longer makes sense.

Be straightforward about responsibility without either deflecting or over-apologising. If the assumption was mine and it was wrong, say so once, then spend the time on the remedy: the customer's interest is in what happens next.

Then look at why it was found now rather than earlier. Usually the answer is that the riskiest assumption was not tested first, and the correction for the remainder of the project is to sequence the untested assumptions ahead of the comfortable work.

Curated: · Written: · Reviewed:

Q&A-89A customer wants to know why the estimate for the second phase is higher than the first. How do you explain it?

By identifying which of the three usual reasons applies, because they need very different responses and conflating them damages trust.

The first is that the second phase is genuinely harder: the first delivered the straightforward, well-understood journeys, and what remains is the complicated integrations, the edge cases, and the data migration. That is normal and should have been signalled at the start, but frequently was not. Explaining which specific work makes it harder is what makes this credible.

The second is that the first phase revealed something: data quality worse than assumed, an undocumented consumer, a legacy interface that cannot sustain the required rate. This is the estimate improving rather than growing, and framing it that way is honest: the first number was made with less information, and here is what was learned. Customers accept this reasonably well when the finding is specific.

The third is that the first phase was underestimated, and the second is being priced realistically. If that is the truth, say it. It is uncomfortable and it is far better than a fabricated technical justification, which an informed customer will see through and which poisons everything else you tell them.

In every case, show the composition rather than the total, so they can see where it goes and challenge specific lines.

And if the increase changes whether the project is worth doing, say that too. The customer's ability to stop is part of what a phased engagement is for.

Curated: · Written: · Reviewed:

Q&A-53How do you evaluate whether a customer's proposed timeline is realistic before agreeing to it?

The 90-second answer

Don't start with the build estimate. Start with the date's driver and the customer-side lead times, because those are the parts you don't control and can't compress by adding engineers. Before agreeing to anything, ask three things: who set the date and why, what happens if it slips, and what the customer's last three comparable approvals actually took, not their policy SLA.

Then work backward from the deadline through every gate and see whether the arithmetic leaves room for the build. If it doesn't, present the arithmetic and options, not an opinion.

Why the gates dominate

Build effort scales with people; approval queues don't. A security review is a queue with a fixed number of reviewers, and adding engineers to the build doesn't move it. So elapsed time ≈ build + serial customer-side gates + freeze windows, and the gates are usually the larger term.

Planning defaults I use, to be replaced with the customer's actuals:

GatePlanning figureCompressible?
Procurement / signature3 weeksRarely; legal cycles
Security review4–6 weeksOnly by pre-submitting artefacts
Pen test + remediation2–3 weeksRemediation can be scoped down
Environment / access provisioning1–2 weeksSometimes
UAT3 weeksOnly by cutting scope
Data migration + reconciliation2–4 weeksNo: data quality sets it
Parallel run2–4 weeksNo, it's calendar time
Training1 weekYes
Change freezecustomer-specificNo

Worked backward pass

Target 8 Dec go-live (before a 15 Dec–5 Jan freeze), today 1 June:

  • Parallel run 4 weeks → starts 10 Nov
  • Migration prep 2 weeks → 27 Oct
  • UAT 3 weeks → 6 Oct
  • Pen test + remediation 2 weeks → 22 Sep
  • Security review 6 weeks → 11 Aug
  • Procurement 3 weeks → 21 Jul
  • Build 10 weeks → must start 12 May

Today is 1 June. The build is already three weeks late, and the build is only 10 of the 30 weeks. That's the conversation: not "can you go faster", but "which of these gates moves".

What I test with the customer

Each gate has an owner and a history. "Security review takes six weeks" is a policy number; "your last three vendor reviews took 4, 9 and 11 weeks" is a planning number. Ask for the actuals. Name the individual who signs, not the team.

Ask what the date is for. A regulatory date or a contract renewal is immovable; a board update or a budget-year boundary often isn't. If the answer to "what happens if we miss it?" is "nothing much", the date is soft and the negotiation is easy.

Failure modes and how I catch them

  • Stated lead times are policy, not practice → ask for the last three actuals with dates.
  • A gate has no named owner → it will sit. Ask who signs and when they're next available.
  • Migration looks like a task but is a data-quality problem → dry-run on a production-sized copy before committing to a date.
  • The date is a proxy for a budget or political deadline → ask what happens on slippage; the answer changes the options.
  • Parallel run gets squeezed to protect the date → reconciliation defects surface in production. Protect it or cut scope.

If it doesn't fit

Bring three options with the arithmetic attached: reduced scope for the fixed date, phased delivery (subset live on the date, rest after), or a later date. Say which gates you'd need the customer to shorten and by how much.

What I commit to is a date plus the assumptions it rests on and a re-baseline checkpoint. I won't sign a date I don't believe: the cost of a missed commitment is paid in every subsequent conversation.

Curated: · Written: · Reviewed:

Q&A-36A customer's board asks why the migration will take nine months when a competitor claims six weeks. How do you answer?

By making the comparison specific rather than defending the number. The six-week claim is almost certainly for a different scope, and the useful response is to establish which.

The questions that resolve it: does that six weeks include the data migration, or only standing up the new platform? Does it include the integrations with their existing estate, and how many are there? Does it include the parallel-run period where both systems operate? Does it include decommissioning, or does the old system stay funded? Does it assume the customer's team is available, and to what extent?

In most cases the competitor's number is a platform stand-up and yours is an end-to-end migration, which is a real difference rather than a difference in efficiency.

Then be honest about where the time genuinely goes, because it is rarely where boards assume. It is not writing code; it is data reconciliation, discovering undocumented consumers, waiting for access and decisions, and the parallel-run period that exists to prove the new system before the old one is switched off.

If a shorter timeline is genuinely achievable by reducing scope, put that on the table as an option with its consequences: a phased migration that delivers a subset in ten weeks and the rest later, with the cost of running both for longer.

What I would avoid is competing on the number by compressing the estimate. A schedule agreed under that pressure fails visibly at the end, which is worse for everyone than an uncomfortable conversation now.

Curated: · Written: · Reviewed:

Q&A-44What is your approach when the customer's requirement is genuinely unachievable within their budget?

Core answer (first 90 seconds):

First I separate "unachievable" from "unachievable at this price." Those are different conversations and only one of them is a no. Then I bring the arithmetic, not the conclusion, and I bring priced options rather than a problem. If none of the options fit, I say the project shouldn't proceed, and I say it while there's still budget left to spend on something that works.

The assumptions I state up front: the budget is fixed, the timeline is fixed, and the requirement is the customer's stated one, not my interpretation of it. If any of those three is actually negotiable, the conversation is different.

Worked example (hypothetical figures, labelled as such):

B2B SaaS customer, six-month launch, $400k build budget. Stated requirements: 99.99% availability, p99 API latency under 100 ms at 5k rps, SOC 2 Type II and HIPAA.

ComponentWhat drives the costEstimate
Multi-region active-active, automated failoverfour nines (52 min/yr) vs three nines (8.76 h/yr)$180k
Read replicas, cache tier, query tuningp99 <100 ms at 5k rps$95k
HIPAA + SOC 2 controls, audit, BAAs, audit loggingcompliance regime$140k
Core product buildthe feature set itself$210k
Total$625k

Gap: $225k, 56% over. That number alone is not actionable. The split is.

The levers, priced:

  1. Relax availability to three nines. This is the single biggest lever and the one customers rarely question. It removes multi-region active-active; single-region with a warm standby is ~$45k. Saves $135k.
  2. Cut the two lowest-value features from the core build. $210k → $150k. Saves $60k.
  3. Defer the HIPAA audit to phase 2: build HIPAA-ready, no audit or BAAs until month 12. $140k → $70k in phase 1. Saves $70k.

1+2 = $430k, still $30k over. 1+2+3 = $360k, under budget, with $70k landing in the next budget year.

Buy instead of build is the fourth option and often the cheapest: a vendor already holding SOC 2 and HIPAA at $8k/month is $96k/year against $140k to build the controls plus the ongoing audit cost.

Failure modes and how I catch them:

  • Deferring the conversation. The same gap found in week 2 costs a meeting; found in month 5 it costs the sunk spend plus rework. I check estimate-vs-budget variance at every milestone; if the gap is over 20% at design review, it goes to the customer that week.
  • Presenting the total instead of the split. "$625k vs $400k" invites a price negotiation. The per-lever table invites a decision about one thing.
  • Saying no without options. The customer shops it, someone else says yes, and it fails in month 8 with your estimate in the file.

When I say don't proceed: if the customer needs four nines and HIPAA and has $150k, no combination of levers gets there. Saying that plainly is what makes the other four conversations credible.

Curated: · Written: · Reviewed:

Q&A-99A customer's finance lead says your proposed architecture is too expensive and asks you to cut the bill by 40% without accepting downtime. How do you respond?

Start with the invoice, not the architecture. A 40% target is arithmetic against a breakdown, and in most estates the top three line items carry 70% or more of the spend, so the cut lands in one or two places, not spread evenly across everything. Two questions come first: is the 40% on the total run-rate or on the infrastructure line only, and is the baseline a full business cycle or one peak month? A bill that includes 40 TB of one-off migration egress is not the bill we are optimising.

Then decompose it. Hypothetical but typical, a $96,000/month estate looks like this:

Line | Monthly | What usually sits in it Compute, all on-demand | $38,000 | Oversized instances, non-prod running 24/7, zero commitment coverage Managed database, Multi-AZ plus replica | $21,000 | Provisioned for peak, 20% average utilisation Object and block storage | $9,000 | Everything in hot tier, old snapshots, unattached volumes Internet egress | $12,000 | BI extracts and API responses leaving over the public path Logging and observability | $8,000 | Debug-level logs retained 30 days hot Load balancing, NAT, licences, misc | $8,000 | NAT data processing is the classic hidden item

Now the levers, in order of what they cost the customer in behaviour. Waste first: unattached volumes, orphaned snapshots, idle NAT gateways, load balancers with no targets. Normally 5–8%, so about $7,000 here, and it costs nothing but a day of someone's time. Second, rightsizing against measured p95 utilisation over a full business cycle: $8,000, but do not rightsize against a quiet fortnight or the peak day fails. Third, commitment on the steady floor: say $24,000 of the remaining compute is genuinely steady, and a one-year no-upfront Savings Plan takes 15–20% off it, so roughly $4,300. Note that the marketing figure of up to 72% is three-year all-upfront; I quote the number out of the pricing calculator at their actual rates. Fourth, scheduling non-prod: if it is a third of compute and you run it 12 hours on weekdays, that is 60 of 168 hours, so you keep 36% and save $6,400. Fifth, egress shape: 80 of 120 TB behind a CDN at a blended $0.03–0.05/GB instead of $0.09, and the BI tool moved in-region so extracts stop crossing the internet: about $5,000. Sixth, lifecycle policies and gp2 to gp3: 60 TB to Glacier Instant Retrieval at $0.004/GB-month instead of $0.023, plus block storage at $0.08 instead of $0.10: $1,600.

That is $32,300 of resilience-neutral savings, roughly 34%. The remaining $6,000 is where the trade-offs actually live, and this is the part you present rather than perform: drop the read replica and reporting p95 rises from 40 ms to perhaps 300 ms under load; move the reporting store to single-AZ and its RPO becomes the last backup. Acceptable, because it is derived data you can rebuild; run the overnight batch on spot capacity and save 60% on it, but only if it is checkpointable and start time is allowed to drift. Deliberately untouched: backup retention, security controls, and the observability floor required to meet the incident response commitments in the contract.

For the two audiences: the engineer gets the mechanism. Instance families, commitment coverage percentage, lifecycle rules, the peak-day headroom we are protecting and the number we are protecting it to. The finance lead gets a one-page ladder with three columns: what we cut, what it saves per year, and what it costs in RTO, RPO, latency or staff time. Put the money in the risk column: Multi-AZ at $7,000/month buys failover in around a minute against a snapshot restore measured in hours, and at their transaction rate that outage is worth more than the saving. And be explicit that a Savings Plan is a commitment they owe even if the workload is retired in month seven.

Two failure modes to name out loud. Savings erode without an owner: in my experience 5–10% drifts back per quarter unless someone owns a monthly review, so the deliverable includes an owner and a tag policy, not just a slide. And the over-correction: rightsizing to the point where autoscale cannot react in time, or cutting logs below what the support team needs, turns a cost problem into an availability problem. If the honest ladder tops out at 35%, say so, show where it stops, and let the finance lead choose the last $5,000 with the downtime cost printed next to each option.

Curated: · Written: · Reviewed:

Q&A-85How do you approach pricing an ongoing managed service for a customer after the build is complete?

Price it from the operational load the design creates, measured in hours, then multiply by a loaded rate. Not a percentage of build cost: that number has no relationship to what the system costs to run, and it collapses the first time the customer asks why a quiet month costs the same as a loud one.

The arithmetic. Take a mid-size estate: a dozen services, ~40 instances, managed Postgres, Kubernetes. Steady-state monthly hours, illustrative:

FunctionHours/month
Patching, dependency and base-image upgrades8
Certificate and secret rotation2
Backup verification + quarterly restore drill (amortised)3
Capacity and cost review3
Alert tuning, dashboard upkeep4
Change support (deploys, config)10
Reporting, account management3
Routine subtotal33

Incident response comes from measured alert volume, not a guess. If the system pages 12 times a month at 20 minutes to acknowledge and 45 to resolve, that is 12 × 65 min ≈ 13 h. Another 48 non-actionable alerts at 5 minutes of triage each adds 4 h. Total ≈ 50 h/month.

Loaded engineer cost: $150k salary + 25% benefits and overhead = $187.5k, ÷ 1,600 productive hours (2,080 less holiday, sick, training, internal work) ≈ $117/h. Bill at $195/h and the gross margin is 40%: the band most managed-services practices target. So 50 × $195 = $9,750/month for business-hours coverage.

Coverage is the biggest lever. 24×7 with a 15-minute response commitment cannot be bought for less than the cost of the rotation that delivers it. A sustainable rotation is 4–5 engineers (one week in four, plus a backup); at 25% attributable time each, 4 × 0.25 × $187.5k = $187.5k/year ≈ $15,600/month before a single page is answered. Add the incident work and out-of-hours premium and this estate lands around $26–30k/month: roughly 3× business hours, and the ratio narrows as the estate grows because the rotation cost is fixed. Customers routinely ask for the 15-minute commitment while budgeting for business hours; put both numbers in front of them in the same conversation.

Incentive. Billing per alert or per incident means you profit from the system staying noisy. Price incident response as a fixed monthly allowance (say 20 h) with an overage rate, and price improvement work (tuning, deduplication, runbook automation) as a separate fixed-fee SOW. Then write the step-down in: at the 90-day review, if paging volume is below the agreed baseline, the incident line drops by the measured hours saved. Tuning 60 alerts/month down to 20 saves about 11 h ≈ $2,100/month, and the customer keeps it. That clause is what makes the arrangement defensible when they audit it.

Cadence and obligations. Price the first 90 days at measured load, review at 90 and 180 days, then annually. Name what the customer must supply: a technical contact who can grant access within 4 business hours, change approvals within one business day, and a business owner who can call priority during an incident. If those are not met, the response SLA is suspended for the duration: otherwise you become the first responder for their application bugs and their slow approvals, and the margin is gone.

Failure modes to price for. A bad deploy that fires 400 alerts in an hour is one incident, not 400: define incident, not alert, as the billing unit. Anything that changes the running system is chargeable change; anything that restores it to the documented state is support. And a customer with no on-call of their own will route every application bug to you within a month; either price a triage allowance for it or exclude it in writing.

Curated: · Written: · Reviewed:

Q&A-101The customer asks what your design will cost them to run each month. How do you give them a number you can stand behind?

The number you can stand behind is not a number: it is a model with named assumptions, a range, and the two or three drivers that move it. Say that in the room, then show the model.

First, get usage. If the customer has no data, ask for what they do have: transaction volumes, peak-to-average ratio, data growth per month, and how much data leaves their network today. If they have nothing, state the assumption out loud and mark it as the thing to validate in month one.

Then build the estimate from drivers, not from a product list. An illustrative AWS us-east-1 model (list prices, rounded, early 2026: re-check before quoting):

DriverBase assumptionMonthlyIf the assumption is wrong
Compute96 vCPU steady, 1-yr Savings Plan$2,350$3,360 on-demand (+43%)
Object storage12 TB, S3 Standard$283+$23 per extra TB
Internet egress40 TB/month$3,450±$900 per 10 TB
NAT gateway6 TB/month processed$310linear with traffic
Logs1.5 TB/month ingest, 30-day retention$8151-year retention ≈ ×3
SupportBusiness plan, % of spend$720—
Total~$7,900$6.5k–$12.4k

Two rows are where estimates usually die. Egress is tiered and asymmetric: 40 TB/month out is 10 TB × $0.09 + 30 TB × $0.085 = $900 + $2,550 = $3,450/month, about $41k/year, and the same 40 TB coming in is free. NAT gateway data processing is the other: $0.045/GB looks trivial until a chatty service pushes 6 TB/month through it, which is $310/month for a component nobody drew on the diagram.

Present a range with the drivers named, not a point estimate with a caveat. "Base case $7,900/month. If egress doubles and you stay on on-demand compute, it's $12,400. The two levers that matter are egress volume and the compute commitment; everything else moves the number by less than 10%."

Then separate one-off from run-rate, because their finance lead will. One-off: migration effort, dual-running both estates during cutover, the egress bill for the migration itself, licence exit, training, decommissioning. Run-rate: monthly, annualised, with a growth assumption attached. Compare run-rate against their fully loaded current cost (hardware refresh, data centre power and space, ops headcount) not against last month's invoice, or cloud looks like a price rise when it is not.

Pitfalls worth naming out loud: quoting on-demand for a steady workload, typically 30–40% more than a one-year commitment; buying that commitment before the workload shape is known, so start on-demand and commit after three months of real usage; forgetting non-production, often 30–40% of production; forgetting the support plan, which on AWS Business is a percentage of spend (10% of the first $10k of monthly usage, tapering to 3% above $250k); and forgetting that a savings plan is a commitment the customer, not you, has to live with.

The failure mode to avoid is the confident single number. It becomes a commitment in the customer's head, and the first invoice that exceeds it costs more credibility than the range would have.

Curated: · Written: · Reviewed: —

Q&A-93A customer's cloud bill has doubled in twelve months and they want to cut it. How do you find where the money is going, and what do you change first?

Start with the cost-and-usage data and propose nothing until you can name where the growth came from. A bill that doubles in twelve months almost always has two or three dominant causes, and a generic optimisation checklist is worthless until you have found them.

I read the data in three cuts. By service: what grew. By account or project, who to talk to. By tag, which workload, and whether it is production or a dev environment nobody has stopped since March. AWS gives me the Cost and Usage Report into Athena, or Cost Explorer grouped by linked account and cost-allocation tag; GCP is a billing export to BigQuery; Azure is a Cost Management export. The first thing I check is whether the tags exist at all. If a third of the spend is unallocated, that is finding number one, because every conversation after it is guesswork. Then I ask what changed: a doubling is usually a new environment, an unbounded log stream, a storage class nobody revisited, or a data-transfer path that arrived with an architecture change, not prices drifting.

Then the fixes, in rough order of yield per hour of effort, which falls off sharply. Idle and oversized compute first: dev/test running 168 hours a week and used for fifty, production instances at 8% p95 CPU. Storage lifecycle second: everything in hot tiers regardless of age, orphaned snapshots, incomplete multipart uploads. Data transfer third: cross-AZ chatter between services calling each other thousands of times a second, logs shipped uncompressed across boundaries, bytes pushed through a NAT gateway that a gateway endpoint would move at no charge. Autoscaling and scheduling fourth. Commitments last.

Worked example, a B2B SaaS platform of roughly 200 workloads, bill $85k/month growing to $171k (figures rounded and illustrative):

LineMonth 12CauseChangeAfter
Non-prod compute$46kran 24/7, 30% oversizedstop/start 12h weekdays, rightsize$19k
Object storage$38kall hot tier, 40% untouched >180 dayslifecycle to infrequent/archive, purge snapshots$24k
Transfer + NAT$27kcross-AZ chatter, uncompressed logsbatch and filter logs, gateway endpoints$12k
Databases$31ktwo idle read replicas, oversizedretire one, rightsize$26k
Logs and monitoring$18kdebug verbosity in prod30 days hot, archive after$11k
Everything else$11k——$11k

$171k down to $103k, about $816k a year. Only then commitments: with the right-sized steady-state compute floor near $55k/month, a one-year no-upfront savings plan is roughly 25% on the committed portion. Another $14k/month, and the equivalent of a GCP committed use discount or Azure reservation. The order matters. Buying that commitment against the original $95k of compute would have locked in $40k/month of waste for a year. I cover only 60-70% of the observed floor first and re-measure after a month.

Making it stick is governance, not engineering. Baseline one full month before changing anything, or you cannot prove anything afterwards. Enforce the tag policy through policy-as-code in the pipeline, not a wiki page. Budgets and anomaly alerts per team: a 30% week-on-week anomaly alert finds the rogue training job in days instead of at month end. Publish monthly showback by team; I start with showback rather than chargeback because chargeback needs finance to agree a cost-allocation method, and that stalls the work for a quarter.

On the people: finance gets the number, the date and the named owners in their own worksheet. Engineering gets the change list as work items with an owner on each, and I say explicitly that missing tags are a platform problem, not a team failing. Savings nobody owns come back within two quarters.

Where I weigh cost against reliability and speed, I will not take a saving that quietly moves risk onto the customer. The largest line I refused in that engagement was retiring the second read replica of the orders database: $3k/month, and half their recovery capability for a tier-1 service. I wrote the alternative down instead: keeping that availability target costs about $36k a year, and losing it in an AZ event costs several times that in an afternoon. I have also refused to shrink test environments until engineers stop testing against realistic data, and refused to cut log retention below the compliance window. The rule I give the customer: if the saving is real but it buys back an outage or a slowdown in delivery, name it as a risk with an owner and let them decide with the number visible, rather than deciding quietly on their behalf.

Curated: · Written: · Reviewed:

Q&A-102A customer wants to cut cloud spend. How do you sort the savings that cost nothing from the ones that trade away reliability or performance?

The first 90 seconds

Don't open the billing console. Open the customer's unit economics. A cloud bill is only meaningful divided by something the business sells: cost per order, per active tenant, per 1,000 API calls. If spend grew 40% and volume grew 45%, there is no cost problem, there's a pricing conversation. If spend grew 40% and volume grew 5%, you have an architecture problem, and you can now say so with a number instead of an opinion.

Then get allocation before optimisation. Untagged spend can't be attributed, and unattributable spend can't be defended or cut. I pull the Cost and Usage Report and check tag coverage on the top 20 services by cost. Below roughly 80% coverage, the first deliverable is tagging, not savings.

Classify the levers before touching anything

Every saving falls into one of three buckets, and the bucket decides who has to approve it:

BucketExamplesWho signs off
Free: no SLO changegp2→gp3, idle EIPs and load balancers, orphaned snapshots, log level at source, S3 lifecycle to IA/Glacier, Graviton where the build already produces arm64Engineering
Trades performanceSmaller instance families, lower provisioned IOPS, shorter cache TTLs, moving batch to off-peakProduct owner
Trades reliabilitySingle-AZ, dropping warm standby, spot for stateful work, shorter backup retentionBusiness owner, in writing

On a typical estate, 15–25% of the bill sits in the first bucket. That's the number I lead with, because it costs the customer nothing and buys credibility for the harder conversation.

Arithmetic that survives the meeting

200 × m5.xlarge on-demand in us-east-1 at $0.192/hr = 200 × 0.192 × 730 = $28,032/month. A 3-year no-upfront Compute Savings Plan against that steady baseline lands near 50% off: call it $14,000/month, $168k/year, with no architectural change. Free bucket.

CloudWatch Logs at $0.50/GB ingest: 40 TB/month = 40,000 GB × $0.50 = $20,000/month. Dropping debug logging at source typically removes 60–80% of that volume: $12k–16k/month, no SLO touched.

Cross-AZ traffic at $0.01/GB in each direction: 100 TB/month of chatty service-to-service calls is $2,000/month. Fixing it means topology changes, so that's bucket two.

The trade-off, stated honestly

The cost/reliability curve is not linear. The first slice is free. The next slice costs an SLO step: going from 99.99% to 99.9% is roughly a 10x reduction in the redundancy you're paying for. Past that you're buying back pennies with outages. I put the SLO price tag next to the dollar figure: "this saves $9k/month and takes your RTO from 15 minutes to 4 hours." Then the customer decides, because risk appetite is theirs, not mine.

Presenting it

Three options, not one: a free-savings plan, a moderate plan with named SLO deltas, and an aggressive plan with the risk written down. Each with monthly saving, one-off effort, and what breaks. Recommend the first, sequence it, and put a date on re-measuring. A single number invites "cut more"; a portfolio invites a decision.

Failure modes

  • Savings Plans or RIs bought against a footprint that's about to shrink: you've locked in the waste. Check the 90-day trend first.
  • Cutting log retention below an audit or regulatory obligation. Check the retention requirement before the retention setting.
  • Spot for anything stateful without checkpointing; the interruption notice is 2 minutes.
  • Cutting cloud spend while adding headcount to manage it. Compare fully loaded cost, not the invoice.
  • Optimising a workload that's 2% of revenue. The engineering time costs more than the saving.

Curated: · Written: · Reviewed: —

Q&A-32How do you handle scope creep on a customer engagement without damaging the relationship?

Make it visible and priced, immediately and without drama, every time. The relationship damage comes from surprise, not from saying no.

The mechanics: when a new request appears, acknowledge it as reasonable, then state its impact in the currency the customer cares about. Cost, schedule, or something else being dropped. "We can do that; it adds about two weeks and this much, or it fits in the current phase if we defer the reporting module. Which would you prefer?" That is a decision for them rather than a refusal from you, and it usually gets a sensible answer.

The failure mode is absorbing small requests silently to be helpful. Each one is genuinely small, none is worth a conversation, and eventually the schedule slips for reasons nobody documented: at which point the customer is surprised and the team is blamed. Absorbing scope quietly feels generous and produces exactly the dispute it was meant to avoid.

Keep a visible list of everything requested and its disposition: in, deferred, or declined with a reason. Reviewing that list at the regular checkpoint means the conversation happens in small pieces instead of once at the end.

Distinguish creep from discovery. A requirement nobody knew about is not the customer being unreasonable; it is the estimate meeting reality, and it should be handled through the same visible process but framed accurately, because calling it creep implies fault that is not theirs.

And be willing to absorb something genuinely trivial. Insisting on a change request for an hour of work is its own kind of relationship damage.

Curated: · Written: · Reviewed:

Q&A-45How do you handle a customer who wants to see progress weekly on a project with a long lead time?

Give them something real every week, and be honest that early weeks produce evidence rather than features.

The underlying need is usually not features: it is confidence that the money is producing something and that problems will surface early rather than at the end. Meeting that need is straightforward once it is named.

What to show in the early phase: discovery findings, which are genuinely interesting to a customer because they are learning about their own estate; decisions made and the options rejected; risks found and retired; and the environment and pipeline standing up, which is visible progress even without a feature.

Then get a thin vertical slice into a demonstrable state as early as the design allows, even if it is narrow and ugly. One real journey working end to end changes the relationship, because from that point every week has something to show and the customer can see the shape of what they are buying.

Be equally consistent about reporting bad news weekly. A status that is green for four months and then red is worse than useless. Small, early disclosure of a slipping dependency is absorbed easily; the same information at the end is a crisis.

Keep the format light. A weekly demonstration of whatever exists, a short written note of decisions, risks, and what is needed from them: that last item matters, because customer-side dependencies are a common cause of delay and a weekly reminder is the cheapest way to manage them.

Curated: · Written: · Reviewed:

Q&A-49How do you approach the first architecture review with a customer whose previous project failed?

Find out why it failed before proposing anything, and expect the stated reason to be incomplete.

Ask directly and without judgement: what happened, what did people conclude at the time, and what would they do differently. The stated cause is usually technical (the wrong platform, an underestimate) and the actual cause is frequently organisational: a decision nobody would make, requirements that kept moving, a sponsor who left, a team that was never available. Those causes are still present, and a technically excellent proposal that ignores them fails the same way.

Then design the engagement around whatever the real failure mode was. If requirements moved constantly, the proposal needs a change process and a phased structure with decision points. If the sponsor left, it needs a broader base of support and value delivered early enough to survive a change of leadership. If the team was never available, the plan must reflect their real capacity rather than the stated one, or that dependency must be removed.

Expect scepticism and treat it as reasonable rather than as an obstacle. Credibility here is earned by early, small, visible delivery rather than by a better document: the previous project also had a good document.

Two things to be careful about: do not criticise the previous work, because the people who did it are usually in the room; and find out whether anything from it is salvageable, since writing off an investment they defended is a poor way to begin and occasionally wrong on the merits.

Curated: · Written: · Reviewed:

Q&A-29What is the most common way a technically sound architecture fails in delivery?

Nobody owns it after the people who designed it leave.

The mechanism is consistent. The design is right, the initial build follows it, and then the pressure of delivery meets a team that understands what the components do but not why the boundaries are where they are. Each individual shortcut is reasonable in isolation: a direct database read across a service boundary because the API does not have the field yet, a shared table because the migration is due Friday. Two years later the architecture on the diagram and the system in production are different things, and no single decision was wrong.

The preventions are unglamorous. Record the decisions with their reasoning and their reversal conditions, so a future engineer meets an argument rather than a rule. Encode boundaries where they can be enforced automatically (module import rules, schema ownership, pipeline checks) because a boundary that depends on discipline erodes at the first deadline. Name an owner for each component before handover, and make sure that owner was in the room during the design rather than receiving it.

The second most common failure is closely related: the design assumed an operational capability the customer's team does not have, so it degrades toward whatever they can actually run.

Both are failures of the handover rather than of the design, which is why in this role the transition out is worth as much attention as the architecture itself, and usually gets far less.

Curated: · Written: · Reviewed:

Q&A-100How do you communicate an incident during a customer engagement when your own work caused it?

Early, factually, and with the remedy, and take responsibility without spending the customer's time on contrition.

The sequence in the moment: tell them it is happening and what the impact is before they discover it, even when you do not yet know the cause. A customer who learns about an outage from their own users while their supplier is silently investigating loses confidence in a way that is hard to recover.

Then give them what they need to run their own business: what is affected, what is not, what the workaround is, and when the next update will come, and then actually send the update at that time, even if it only says the investigation continues. Predictable communication is what reduces the anxiety that produces escalation.

When the cause is your work, say so plainly and once. "Our change caused this" is worth more than any explanation, and it should be followed immediately by what is being done, not by a defence. Detailed causal explanation belongs in the postmortem, not in the incident channel.

Afterward, a written postmortem with the mechanism, why it was not caught, and specific actions with owners and dates, and then evidence that the actions happened, because the follow-through is what actually restores trust rather than the document.

The thing to avoid is minimising. Customers usually know roughly how bad it was, and a description that understates it converts a technical failure into a credibility problem that outlasts the incident by a long way.

Curated: · Written: · Reviewed: