Top 100 Identity and Access Management (IAM) Engineer Interview Questions and Answers
The questions most likely to come up in your Identity and Access Management (IAM) Engineer interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.
Curated: · Written: · Reviewed:
Q&A-1A team says they will use OAuth 2.0 to log users in. What do you tell them?
OAuth 2.0 does not log users in. It is a delegation protocol: the client receives an access token that says "this client may call this API on behalf of whoever granted it," and the token carries no statement about who that person is or when they authenticated. If the team's goal is a sign-in decision (establish identity, start a session) they want OpenID Connect, which is an identity layer on top of OAuth 2.0.
The distinction that decides the design is which token answers which question. The ID token is a signed JWT the client validates and uses to decide "this user authenticated to this client at this time." The access token is a bearer credential for the resource server; the client should treat it as opaque and never as proof of identity.
What to require:
- Request the
openidscope. Without it you get plain OAuth and no ID token. - Validate the ID token before trusting it: signature against the issuer's JWKS,
issmatches the expected issuer,audequals yourclient_id,expnot passed, andnoncematches the one you sent. Use the authorization code flow with PKCE; skip implicit. - Keep the access token for API calls only. Do not call the userinfo endpoint and treat a 200 as "the user is logged in": userinfo authenticates the token, not the user's presence at your app.
- Bind the session to the ID token's
subplusiss, not to the access token string.
Failure modes worth naming:
- Cross-client token replay. An access token minted for client A is presented to client B, which accepts it because it checks only that the token is valid, not that it was issued for it. Detection: the app must reject tokens whose
audis not its ownclient_id; test by replaying a token from another client. - Missing
audcheck on the ID token. Same class of bug, same fix. - Long-lived access tokens used as session cookies. Access tokens are commonly valid for an hour and are not revocable in the general case; a session should expire on its own schedule and be revocable server-side.
| Question | Access token | ID token |
|---|---|---|
| Names who authenticated | no | yes |
| Carries authentication time | no | auth_time when requested |
| Audience checked by | resource server | client app, required |
| Typical lifetime | ~1 hour, provider-configured | minutes, provider-configured |
Before shipping, take a valid access token issued to a different client and present it to the sign-in path; the app must refuse it. Then take an ID token with a wrong aud and confirm refusal. Delegation is not authentication, if the team wants login, they want OIDC.
Curated: · Written: · Reviewed:
Q&A-2Which OAuth flow do you pick for a public client that cannot hold a secret, and why not implicit?
Authorization code flow with PKCE. A public client can't hold a secret, so the authorization code has to be bound to the sender rather than to a credential the client can't keep. PKCE does exactly that: the client proves at redemption that it is the same party that started the request.
Assumptions: the client is a SPA or native app with no confidential backend, and the authorization server implements RFC 7636 and supports S256.
Mechanism: per authorization request, generate a high-entropy verifier (43-128 characters, RFC 7636 §4.1). Send only its S256 challenge on the front channel. Redeem the code on the back channel with the verifier. An attacker who captures the code from the redirect can't redeem it without the verifier, which never left the client.
// Browser, Web Crypto (SubtleCrypto)
const b64url = (buf) => btoa(String.fromCharCode(...new Uint8Array(buf)))
.replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');
const verifier = b64url(crypto.getRandomValues(new Uint8Array(32))); // 32 bytes -> 43 chars
const digest = await crypto.subtle.digest('SHA-256', new TextEncoder().encode(verifier));
const challenge = b64url(digest);
32 random bytes encode to 43 base64url characters (44 with padding, minus the stripped =), inside the RFC's 43-128 range. The authorize request carries response_type=code, code_challenge=<challenge>, code_challenge_method=S256; the token request carries grant_type=authorization_code, the code, the code_verifier, and client_id: no secret.
Why not implicit: implicit returns the access token in the URL fragment. The fragment is readable by browser history and by any script on the page, and it never touches a back channel, so there is nothing to bind to the sender. It also can't issue a refresh token safely. OAuth 2.1 drops implicit for these reasons.
| Leak path | Implicit | Code + PKCE |
|---|---|---|
| token in URL fragment | yes | no |
| token in browser history | yes | no |
| code replay without verifier | n/a | rejected (invalid_grant) |
| refresh token | not issued | issued, sender-bound |
Failure modes to check, not assume:
- AS accepts
code_challenge_method=plain: the challenge equals the verifier, so an interceptor can redeem. Require S256 and reject plain. - Code replay: the AS must reject a second redemption of the same code and revoke tokens already issued from it (RFC 6749 §4.1.2). If it doesn't, a captured code is a bearer credential.
- Redirect URI matching: must be exact-match against a registered value; a loose or open redirect hands the code to the attacker.
- Verifier reuse across requests: weakens the per-request binding; generate a fresh one each time.
Verification: capture a code, replay it at the token endpoint without the verifier, and require invalid_grant; then replay with the verifier and require the AS to reject the second use.
Curated: · Written: · Reviewed:
Q&A-7Your authorization server still accepts a code challenge method of plain. Why does that matter?
I would settle PKCE downgrade and verifier reuse by attempting the action after the identity event that was supposed to stop it.
PKCE is only as strong as the weakest method the server accepts and the freshest verifier the client generates. If plain is accepted, the challenge equals a verifier the attacker may already hold; if a verifier is reused, the binding is to a constant rather than to a request.
Concretely, refuse plain at the authorization server, require S256 in the client registration, generate a fresh verifier from a cryptographic source for every authorization request, and tie the verifier to the single-use request record so a second redemption finds nothing to compare against.
The reason for that specificity is a failure I have seen: A mobile SDK cached one verifier per install to save an allocation. An attacker who captured that verifier once redeemed 7 later codes over 19 days, and every redemption looked correct to the token endpoint.
Verifier lifecycle in the SDK before and after.
| Behaviour | Before | After |
|---|---|---|
| verifiers generated per install | 1 | 1 per request |
| codes redeemable with a stolen verifier | 7 | 0 |
| plain method accepted | yes | refused, 100 of 100 |
I would not consider it settled without evidence: attempt authorization with the plain method and with a previously used verifier, and require both attempts to be refused.
A constant verifier binds nothing.
Curated: · Written: · Reviewed:
Q&A-5Are state, nonce, and PKCE not three names for the same protection?
No. They are three separate checks performed at three different points in the flow, and only one of them is enforced by a server. The "three names for one thing" reading falls apart as soon as you ask which value each one compares and who does the comparing.
Assumptions I'd state first: OpenID Connect authorization code flow, public client (SPA or native app), S256 PKCE, and one pending-auth record per login, keyed by state and tied to the session cookie. On plain OAuth 2.0 with no ID token there is nothing for nonce to bind: that is why it is missing from some deployments, not because state covers it.
| state | nonce | code_verifier | |
|---|---|---|---|
| Generated by | client | client | client |
| Travels | front channel, echoed in the redirect query | front channel in the auth request, returned inside the signed ID token | back channel only, to the token endpoint over TLS |
| Checked by | client, before any token request | client, after the token response | authorization server, at the token endpoint |
| Compared against | the pending record selected by the login session | a claim inside a signed JWT | BASE64URL(SHA256(ASCII(verifier))) against the challenge stored with the code |
| Alone catches | a response no session here started | a token minted for a different request | a code redeemed by whoever captured it |
RFC 6749 section 10.12 is where state's CSRF role is defined; OpenID Connect Core 1.0 requires the nonce claim to match the value sent in the request; RFC 7636 requires the authorization server to reject a verifier that hashes to something other than the stored challenge.
Trace of one sign-in (values illustrative).
- Client generates a verifier of 43 unreserved characters, computes the challenge, generates state S and nonce N, and writes {S, N, verifier} to a pending record.
GET /authorize?response_type=code&scope=openid&state=S&nonce=N&code_challenge=C&code_challenge_method=S256.- Authorization server stores C alongside the code it issues, plans
nonce: Nin the ID token, and redirects toredirect_uri?code=CODE&state=S. - Client looks up the pending record by S. No record, or a value that does not match, and it stops: no token request, no error page that leaks anything. This is a lookup, not a signature check, so it rejects a callback that no session here initiated.
POST /tokenwith the code and the verifier. The server recomputes the hash, compares it to the stored challenge, and returnsinvalid_granton mismatch. This is the only check the server performs, and the only one that still holds if the client's own code is sloppy.- Client validates the ID token: signature, iss, aud, exp, and nonce equal to N. That is what makes a token minted for another request useless here.
Failure modes and how I'd detect each.
- state dropped, or compared against nothing: POST any code to the callback URL from a fresh curl session. If it is accepted, the response was never bound to a browser session; two tabs open at once also break, because the client can no longer tell which pending record a callback belongs to. PKCE blunts this when the server genuinely enforces the challenge: the injected code's challenge will not match the verifier the client holds, so the exchange fails, but that rejection arrives after the client has already committed to the response, and it vanishes entirely against a server that ignores
code_challenge. - nonce dropped or unchecked: feed an ID token minted for an earlier login into the token-response path. If the client accepts it, a session can be pinned to another principal's token. Neither state nor PKCE reads token contents, so neither covers this.
- PKCE dropped: on a native or SPA redirect, a second app registered for the same custom URI scheme, or anything that can read the redirect, redeems the code first. Detect by exchanging a captured code from a second machine, or by redeeming the same code twice.
- Test that keeps all three honest: disable one at a time in the harness and require a different rejection reason each time. Missing pending record, nonce mismatch,
invalid_grant. Getting the same rejection three times means only one check is actually running.
Curated: · Written: · Reviewed:
Q&A-6A developer asks for a wildcard redirect URI on the staging client. What is your answer?
No: register the exact staging callback URIs, or give staging its own client registration.
Assumption: the staging client is an ordinary web, SPA or mobile client, and the IdP supports exact-match registration (essentially all do). If the developer's real problem is "we spin up a preview host per branch and can't pre-register", that is a provisioning problem, not a reason to widen the redirect URI.
What I'd offer, in order:
- Add the concrete staging URIs to the existing client. They are finite and known: usually two or three.
- If preview hosts are per-branch, provision a client registration per environment from CI (RFC 7591 dynamic client registration) with exact URIs, rather than one client holding a pattern.
- For local dev, use a loopback redirect:
http://127.0.0.1:PORT/callback. RFC 8252 §7.3 requires the authorization server to accept any port on loopback, and the loopback interface is not reachable from off-box.
Why exact: RFC 6749 §3.1.2.2 has the authorization server compare the redirect URI against the registered value using simple string comparison. Once you accept a pattern, the accepted set is every string the pattern matches, and the server no longer knows which one the client meant. RFC 9700, the OAuth 2.0 Security Best Current Practice, tells authorization servers not to accept wildcards in redirect URIs. Some IdPs still expose a wildcard or prefix-matching toggle; that is a config choice, not a safe default.
| Registered value | What the AS accepts | What an attacker needs |
|---|---|---|
https://app.example.com/cb | that exact string | control of app.example.com, or an open redirect at /cb |
https://app.example.com/* | any path on that host | an open redirect anywhere on the host |
https://*.example.com/cb | any subdomain | a subdomain takeover, or XSS on any subdomain |
PKCE does not cover this. PKCE binds the code to the code_verifier of whoever started the flow. In a wildcard attack the attacker starts the flow, so they hold the verifier. PKCE stops a third party who intercepts a code in transit; it does nothing when the client's own redirect URI is attacker-controlled. Public clients (SPA, mobile) have no client secret to fall back on either.
Worked example (figures illustrative):
- Client registered
https://www.example.com/*: path wildcard on the marketing host. - That host has an open redirect:
/go?next=…302s anywhere. - Attacker sends the victim to
/authorizewithredirect_uri=https://www.example.com/go?next=https://evil.example/cband the attacker's own PKCE challenge. - Victim authenticates and consents; the AS matches the pattern, issues a code, redirects to the marketing host.
- The marketing host 302s to
evil.example/cbwith the code in the query string. - Attacker redeems the code at the token endpoint with their verifier. 82 accounts over 5 hours before the registration was corrected.
Detection: alert on redirect_uri values that match a registered pattern but are not in the exact-URI allowlist; alert on token-endpoint redemptions from ASNs the client never uses; and audit the registered URIs themselves for open redirects, because an exact match to a redirector is still a leak. Codes are single-use and RFC 6749 §4.1.2 recommends a maximum 10-minute lifetime, which bounds the window but does not close it.
Before calling it fixed: send an authorization request with redirect_uri=https://staging.example.com/not-registered and require the AS to return invalid_request before any code is issued. Repeat with a trailing slash and with a case-changed path. If any of those returns a code, the client is still pattern-matching.
Curated: · Written: · Reviewed:
Q&A-3Your API accepts an ID token as a bearer credential. What breaks?
This is an area where a successful federation and a current authorization decision are different events.
An ID token is a statement to one client that a user authenticated, and its audience is that client rather than the API. An API that accepts it is trusting an artifact minted for someone else, and it loses the scope and consent an access token carries.
Concretely, have the API compare the audience against its own resource identifier and refuse tokens whose audience is a client id, and have clients obtain an access token with the scopes the API publishes rather than forwarding whatever the login returned.
The reason for that specificity is a failure I have seen: A reporting API accepted ID tokens for convenience. A low-trust internal client's ID token, carrying no scopes at all, read 12,400 payroll rows in one afternoon because the scope check had nothing to evaluate.
What the API lost by accepting the wrong artifact.
| Property | ID token presented | Access token expected |
|---|---|---|
| audience value | a client id | the API resource id |
| scopes present | 0 | 3 |
| consent recorded | no | yes |
I would not consider it settled without evidence: present an ID token to the API and require a rejection that records an audience mismatch as the reason.
An ID token proves a login happened; it does not authorise a call.
Curated: · Written: · Reviewed:
Q&A-4Walk me through validating an ID token. What do you check beyond the signature?
The 90-second version. Signature verification is the first check, not the answer. An ID token is a bearer assertion about a user, minted by an issuer for a specific client, in response to a specific authorization request. So after the signature you check: iss equals the issuer you configured; aud contains your client id, and azp equals it when several audiences are present; exp and iat sit inside a clock-skew window; nonce equals the value you stored for that request and is then burned; auth_time is recent enough if you sent max_age. If the token arrived from the authorization endpoint next to a code, also verify c_hash/at_hash. That is the list in OpenID Connect Core 1.0 §3.1.3.7. Anything beyond it (sub format, tenant claims, custom roles) is application policy, not token validation.
// Node 20, jose v5
import { createRemoteJWKSet, jwtVerify } from 'jose';
const JWKS = createRemoteJWKSet(
new URL('https://idp.example.com/.well-known/jwks.json'),
{ cacheMaxAge: 10 * 60 * 1000, cooldownDuration: 30 * 1000 }
);
export async function validateIdToken(token, { issuer, clientId, nonceStore }) {
const { payload } = await jwtVerify(token, JWKS, {
issuer, // exact string match, no normalisation
audience: clientId, // aud must contain clientId
algorithms: ['RS256'], // pin; never 'none', never HS256
clockTolerance: 60, // seconds of skew
maxTokenAge: '1h', // requires iat
});
if (payload.azp !== undefined && payload.azp !== clientId) {
throw new Error('azp mismatch');
}
const expected = await nonceStore.take(payload.sub); // single use
if (!expected || expected !== payload.nonce) throw new Error('nonce mismatch');
if (payload.auth_time && Date.now() / 1000 - payload.auth_time > 900) {
throw new Error('auth too old');
}
return payload;
}
Trace: a token minted for a different client at the same IdP. jwtVerify fetches the JWKS, finds the kid, verifies the RS256 signature. Passes. The issuer option compares payload.iss to https://idp.example.com: matches, because it is the same IdP. The audience option sees aud: ["other-app"], which does not contain clientId, and throws JWTClaimValidationFailed: unexpected "aud" claim value. The function exits before nonceStore.take runs. That ordering is deliberate: burning the nonce on a failed validation would break the user's legitimate retry.
| Check | What skipping it admits | Where the snippet catches it |
|---|---|---|
iss exact match | token from a different issuer | issuer option |
aud / azp | another app's token at the same IdP | audience option, then azp guard |
nonce | replayed authorization response | nonceStore.take |
exp / iat + skew | expired or long-lived token | clockTolerance, maxTokenAge |
auth_time | a session older than your policy allows | explicit age check |
alg pin | alg: none, or HS256 signed with the RSA public key as the HMAC secret | algorithms option |
Trade-offs. Local JWKS verification is stateless and costs a signature check per request; it cannot see revocation, so a revoked session stays valid until exp. RFC 7662 introspection sees revocation but adds a network round trip per request and a dependency on the IdP being up: worth it for opaque access tokens, rarely for ID tokens, which are always JWTs. JWKS caching is the same tension: cache with a TTL and refetch on an unknown kid, but rate-limit that refetch (cooldownDuration above) or an attacker spraying random kid values turns your cache miss into a stampede against the IdP. Clock tolerance cuts both ways: 60 s is common, tighter causes intermittent 401s at token boundaries, looser widens the replay window. And strict max_age is a UX decision: forcing re-authentication every 15 minutes breaks long forms; use prompt=none with a refresh token, or step up only for sensitive operations.
Failure modes worth naming. alg confusion (accepting HS256 with the public key as secret) is caught only by pinning algorithms. Key rotation without a refetch path fails every login at once: the symptom is a wall of signature errors with an unfamiliar kid. Skipping aud is the quiet one: a shared IdP issues a perfectly valid token for a sibling application, and nothing in the signature tells you it was not meant for you. Nonce reuse is caught by making the store single-use, not by comparing strings twice.
Curated: · Written: · Reviewed:
Q&A-17Why is reading the algorithm header from the token itself a vulnerability?
The JWT header is attacker-controlled input. A verifier that selects its algorithm from header.alg lets the attacker choose how the signature is checked, so a token that verifies proves nothing about who signed it. The signature check is only meaningful when the relying party fixes the algorithm and the key independently of the token.
Two outcomes follow. First, alg: none: the token carries no signature and a permissive library skips verification entirely. Second, algorithm confusion: the server expects RS256 and holds the issuer's RSA public key; the attacker re-signs the same claims with HS256, using that public key (which is public) as the HMAC secret. A library that reads alg from the header sees HS256, treats the RSA public key as a shared secret, and the HMAC matches. The attacker now holds a token with whatever claims they wrote.
A third variant abuses the jwk/jku headers: the token embeds or points at the verification key, so the attacker signs with their own key and supplies it.
Pin the algorithm and resolve the key by kid:
import jwt
from jwt import PyJWKClient
jwks = PyJWKClient("https://issuer.example/.well-known/jwks.json")
def verify(token: str):
header = jwt.get_unverified_header(token)
if header.get("alg") != "RS256":
raise jwt.InvalidAlgorithmError(header.get("alg"))
key = jwks.get_signing_key_from_jwt(token).key # lookup by kid
return jwt.decode(
token, key,
algorithms=["RS256"], # allowlist, not header-driven
issuer="https://issuer.example/",
audience="api",
)
PyJWT 2.x requires algorithms and rejects none unless it is explicitly listed; the header check is belt-and-braces for libraries that do not.
Trace the confusion attack against a server that pinned RS256. Attacker token header {"alg":"HS256","kid":"k1"}, payload {"sub":"u1","scope":"admin"}. get_unverified_header returns HS256, the check raises, decode never runs: refused. Against a server with algorithms=["RS256","HS256"] that passes the RSA public key PEM as the HMAC key: header HS256 is in the allowlist, HMAC-SHA256 over base64url(header).base64url(payload) with the PEM bytes matches the attacker's signature, and decode returns the admin claims. Same token, opposite result, decided entirely by whether the algorithm came from the token.
Trade-offs. Pinning one algorithm is the strongest and cheapest; the cost is that rotating the issuer's algorithm (RS256 to ES256) requires a coordinated deploy, so many teams allowlist two during a migration window. An allowlist is weaker: it must be paired with a key-type check, because allowing HS256 alongside RS256 reintroduces the confusion attack if the same key material is reachable as both a public key and an HMAC secret. Resolving keys from JWKS by kid adds a cache and a rotation problem: on an unknown kid you must refetch, and you must not fall back to "try every key". That fallback is itself a confusion vector. Cache JWKS with a short TTL and refetch on miss, bounded to one retry.
The fix is insufficient when the algorithm is pinned but the key still comes from the token (jwk/jku), when kid lookup falls back to any key in the set, or when the verifier accepts multiple algorithms without checking that the key type matches. Detection: log the alg and kid of every rejected token; a spike in HS256 tokens against an RS256-only endpoint is the attack in progress.
Curated: · Written: · Reviewed:
Q&A-25A token arrives with a key id your relying party has never seen. What should happen?
What a strong answer sounds like
An unknown kid means you have no key to verify the signature with, so the token is rejected: you never accept a token because its key looks unfamiliar. But it is also the ordinary signal that the issuer rotated signing keys and your cached JWKS is stale. So: reject, do one bounded refresh of the key set, retry validation once against the refreshed set, and if the kid is still missing, return 401 and stop.
The mechanism
JWKS rotation is routine: providers publish overlapping keys and retire old ones on their own schedule. Your cache is a snapshot, so there is always a window where a token signed with a key you have not fetched yet arrives. That window is short and self-healing if you refresh on a schedule you control.
Concrete policy:
- Refresh the key set at most once per fixed interval per issuer, regardless of how many unknown
kids arrive in between. Pick the interval short relative to the issuer's rotation cadence; 5 minutes is typical. - Keep last-known-good keys serving during the refresh. Do not clear the cache before the new set lands.
- Negative-cache identifiers that were not found, so a replayed
kiddoes not trigger another fetch. - Retry validation exactly once after a successful refresh. No loops.
- Alert on the unknown-
kidrate, not on individual occurrences: a rotation is one or two tokens, a probe is thousands.
The failure mode
A relying party refreshed the key set on every unknown kid. A loop of forged tokens produced 240 key-set requests per second, the provider rate-limited the caller, and legitimate sign-ins failed for 22 minutes. The attacker never forged a signature; they exhausted the key-fetch path.
| Policy | Outbound key requests | Valid sign-ins failing |
|---|---|---|
refresh on every unknown kid | 240/s | 22 min |
| at most 1 refresh per 5 min per issuer | 12/hour | 0 |
1 per 5 min + 10-min negative cache per kid | 12/hour under random kids; 6/hour when one kid is replayed | 0 |
Detection: count unknown-kid rejections per issuer per minute and alert when the rate exceeds the rotation rate you have observed. A rotation shows up as one or two distinct kids; a probe shows up as many distinct values, or one value repeated at high rate.
What not to do
- Do not fetch synchronously on the request path without a rate limit. That is the incident above.
- Do not fall back to a previous key set and accept a token whose
kidis not in it. Verifying against the wrong key is not verification. - Do not disable signature checking "temporarily" during a rotation. That is the whole attack.
- Do not, on an unknown
kid, try every key in the set. That turns one signature check into N and hands an attacker a cheap CPU amplifier.
Verification
Send a burst of tokens carrying random kids and confirm outbound key fetches stay within the configured rate while valid sign-ins continue. Then send a token signed with a genuinely rotated key and confirm it validates after exactly one refresh.
Refresh on a schedule you control, not on an attacker's cue.
Curated: · Written: · Reviewed:
Q&A-16Your relying party fetches the key set at runtime. What are the failure modes?
The 90-second answer. Fetching the key set at runtime couples your authentication path to the provider's availability and to your own decisions about where the keys came from. The failure modes cluster into three groups: you can't reach the keys, you reach the wrong keys, or your refresh behaviour turns a routine rotation into an outage. Assumptions: the JWKS is fetched over HTTPS from a configured issuer, cached in-process, and tokens carry a kid.
1. Availability coupling. Every verification needs a key. If the JWKS endpoint is unreachable and you hold no cache, every sign-in fails: a 30-second provider blip becomes a 30-second auth outage. Detect it with the JWKS fetch error rate and cache age (now - fetched_at) as a gauge; alert when cache age exceeds the TTL and the last fetch failed. Mitigate with a bounded TTL, a last-known-good copy, and serving stale keys on fetch failure. You still fail closed on signature: stale keys are not a licence to skip verification.
2. Trust boundary. If the fetch follows a redirect to another host, runs over plain HTTP, or you never check that the discovery document's issuer equals your configured issuer, you accept keys the attacker chose. Log the resolved host and issuer on every fetch and assert the host matches the pinned one. Pin the host, require TLS, reject off-host redirects, validate issuer.
3. Rotation race. The provider rotates, publishes a new kid, and starts signing with it while your cache still holds the old set. Tokens with the unknown kid fail until the TTL expires. Count unknown kid verification failures. On an unknown kid, force one refresh: rate-limited.
4. Refresh storm. That forced refresh, without a cooldown, is a DoS vector: an attacker sends tokens with random kids and every request triggers a fetch. Watch JWKS fetch QPS. Use single-flight plus a per-kid cooldown.
5. Stale keys. Caching forever means a revoked or compromised key still verifies tokens. Bound the maximum age and honour Cache-Control.
6. Algorithm confusion. Match kid and alg, restrict the allowed algorithms, and reject a JWK whose kty/use don't fit the expected shape.
| Failure | Symptom | Detection | Mitigation |
|---|---|---|---|
| Provider unreachable, no cache | all sign-ins fail | fetch error rate, cache age | last-known-good, serve stale |
| Redirect to another host | attacker keys accepted | resolved host ≠ pinned host | reject off-host redirect |
| Rotation, stale cache | unknown-kid failures | unknown-kid counter | refresh on unknown kid |
| Random-kid flood | fetch QPS spike | JWKS fetch QPS | cooldown + single-flight |
| Unbounded cache | revoked key still valid | cache age vs max | bounded TTL |
import time, threading
class JwksCache:
def __init__(self, fetch, ttl=300, cooldown=30):
self._fetch = fetch # returns {"keys": [{"kid": ..., ...}, ...]}
self._ttl = ttl
self._cooldown = cooldown
self._keys = {}
self._fetched_at = 0.0
self._last_attempt = 0.0
self._lock = threading.Lock()
def key(self, kid):
now = time.monotonic()
if kid in self._keys and now - self._fetched_at < self._ttl:
return self._keys[kid]
with self._lock:
now = time.monotonic()
if kid in self._keys and now - self._fetched_at < self._ttl:
return self._keys[kid]
if now - self._last_attempt < self._cooldown:
return self._keys.get(kid) # stale, or None
self._last_attempt = now
try:
doc = self._fetch()
except Exception:
return self._keys.get(kid) # serve stale on outage
self._keys = {k["kid"]: k for k in doc["keys"]}
self._fetched_at = time.monotonic()
return self._keys.get(kid)
Trace: a token arrives with an unknown kid at t=0. The fast path misses, the lock is taken, _last_attempt is 0 so the fetch runs, the cache is replaced, and the key is returned. A second unknown kid at t=5 hits the cooldown branch and returns None without touching the provider. At t=31 the cooldown has elapsed and the next unknown kid fetches again. So a flood of random kids costs at most one fetch per cooldown window per process.
Arithmetic: 200 pods, TTL 300 s, no jitter → 200 fetches / 300 s ≈ 0.67 fetches/s against the provider. With a 30 s cooldown and a random-kid flood, worst case is 200 / 30 ≈ 6.7 fetches/s: bounded, and jittering the TTL spreads the steady-state refreshes.
Verify it by pointing the cache at a JWKS that returns an unknown kid and confirming exactly one refresh per cooldown, then killing the endpoint and confirming cached keys still verify while the fetch error rate climbs.
Curated: · Written: · Reviewed:
Q&A-27A mobile app sends the provider's access token to your backend as proof of sign-in. What is wrong?
What a strong candidate says first
The access token is a bearer credential addressed to a resource server, not to your backend. Nothing in it is bound to your application, so your sign-in endpoint ends up trusting every client the provider has ever issued a token to. Any party that can obtain a token for the same user: a second app on the same provider, a leaked log line, a malicious app on the device: can replay it and get a session.
Sign-in needs an artifact addressed to you: an OIDC ID token whose aud is your client id, or the authorization code redeemed server-side.
Why the access token fails
- Bearer semantics: possession is authority. There is no proof-of-possession binding to the app.
- Audience: the token's
aud, when present, is the resource server, not your client. Opaque tokens carry no audience you can read at all. - No authentication event: no
auth_time, nononce, noacr/amr. A token refreshed last week says nothing about when the user actually signed in. - Lifetime and scope: access tokens are often long-lived and refreshable; they authorize API calls, not identity.
- The userinfo endpoint makes it worse: it returns the user for any valid token, regardless of which client the token was issued to. "I called userinfo and got a
sub" is not a client check.
The attack, step by step (hypothetical)
- Victim signs into a fitness app registered with the same provider as client
fitness-app-123. - That app's backend logs the access token, or the token leaks through a misconfigured redirect.
- Attacker POSTs the token to
https://yourapp.example.com/session. - Your backend calls
GET /userinfowithAuthorization: Bearer <token>; the provider returns{"sub": "victim"}. - You mint a session for
sub=victim. No step asked whether the token was issued to your client.
What to do instead
Preferred: Authorization Code + PKCE (RFC 7636). The app sends the code to your backend; the backend redeems it at the token endpoint with its own client credentials and receives an ID token and an access token. Verify the ID token:
import { createRemoteJWKSet, jwtVerify } from 'jose';
const JWKS = createRemoteJWKSet(new URL('https://idp.example.com/.well-known/jwks.json'));
export async function verifyIdToken(idToken, expectedNonce) {
const { payload } = await jwtVerify(idToken, JWKS, {
issuer: 'https://idp.example.com/',
audience: 'yourapp-client-id',
clockTolerance: 5,
});
if (payload.nonce !== expectedNonce) throw new Error('nonce mismatch');
return payload; // sub, auth_time, acr, amr
}
jwtVerify checks the signature, iss, aud, exp and nbf. The aud check is the one that stops cross-client replay; the nonce check stops replay of a token minted for a different login attempt.
If you must accept an access token, introspect it (RFC 7662) and check the client it was issued to:
const res = await fetch('https://idp.example.com/oauth2/introspect', {
method: 'POST',
headers: {
'content-type': 'application/x-www-form-urlencoded',
authorization: 'Basic ' + Buffer.from(`${clientId}:${clientSecret}`).toString('base64'),
},
body: new URLSearchParams({ token, token_type_hint: 'access_token' }),
});
const info = await res.json();
if (!info.active || info.client_id !== 'yourapp-client-id') throw new Error('not our token');
client_id is the client the token was issued to; aud is the resource server. Check both when the provider returns them. Not every provider offers introspection (Google, for example, does not) which is one more reason to use the ID token.
Failure modes to name
- Accepting
active: truealone. That is the whole bug. - Verifying the JWT signature but not
aud: a token signed by the right issuer for the wrong client still passes. - Trusting
emailfrom userinfo as identity. Emails are not unique across providers and can be reassigned. - Long-lived access tokens landing in logs, crash reports, or
Refererheaders.
How to test it
Get an access token issued to a different client of the same provider and require your sign-in endpoint to reject it. If it returns 200 with a session, the bug is live.
Curated: · Written: · Reviewed:
Q&A-18A user clicks sign out. What has to happen, and in what order?
The useful question for local logout before federated logout is what still holds at the relying parties nobody has looked at this week.
The application's own session is the one the user is standing in, so it is destroyed first and unconditionally. Federated logout is a best-effort request to another party that can fail, redirect, or be blocked, and making it the first step leaves a live local session whenever it does.
Concretely, invalidate the server-side session record and clear the cookie before any redirect, revoke the refresh token tied to that session, then attempt the provider's end-session endpoint or back-channel logout, and record the outcome per relying party rather than assuming propagation.
The reason for that specificity is a failure I have seen: A sign-out button redirected to the provider first. A third-party cookie block ended the flow silently at the provider, and 1 in 6 sign-outs on shared kiosks left the local session alive for the remaining 45 minutes of its idle timeout.
Sign-out outcomes on kiosk browsers.
| Order | Local session ended | Kiosk sessions left live |
|---|---|---|
| provider first | sometimes | 1 in 6 |
| local first, then provider | always | 0 |
| local only | always | 0, provider session remains |
I would not consider it settled without evidence: sign out with the provider unreachable and confirm the application session is already gone when the browser returns.
End what you own before you ask anyone else.
Curated: · Written: · Reviewed:
Q&A-9What do you store as the primary key for a federated user, and why not email?
The 90-second answer
Key the account on the pair (issuer, subject). Email is an attribute you store, verify and display: never the thing you match on.
OIDC Core defines sub as "locally unique and never reassigned within the Issuer for the End-User". The qualifier is the whole point: sub is only meaningful scoped to the issuer that minted it, so the durable key is the tuple, not the subject alone. SAML's equivalent is a persistent NameID (urn:oasis:names:tc:SAML:2.0:nameid-format:persistent), which carries the same non-reassignment promise.
Email carries no such promise. Providers recycle mailboxes, and email_verified is a point-in-time statement about control of the address, not a statement about who the person is. A verified email from last March tells you nothing about who holds that mailbox today.
What that looks like in the schema
identity(issuer, subject): unique constraint on the pair, this is the account's identity row.email: a mutable attribute on the profile, with its ownverified_attimestamp.- Linking an incoming assertion to an existing account on email alone is a separate, explicit, logged operation. Never a fallback in the login path.
The failure this prevents
A representative incident: a mailbox was reissued 90 days after the previous holder left. The new joiner's first sign-in matched the old account on email and inherited 3 administrative group memberships plus 4 years of the previous holder's records. Nothing in the IdP dashboard looked wrong: the assertion was valid, the email was verified, the signature checked out. The bug was entirely in the matching rule.
Detection is cheap if you build for it: alert on any login that resolves an account via the email-linking path, and log the issuer, subject, email and resulting account id on every link. A spike there is either a migration you planned or an account takeover you didn't.
Attribute churn over 12 months, one 5,000-account tenant (illustrative figures)
| Attribute | Changed in 12 months | Safe as a key |
|---|---|---|
| email address | 318 | no |
| display name | 502 | no |
| issuer + subject | 0 | yes |
How I'd confirm it rather than assume it
Change a test user's email at the provider, sign in again, and check that the application resolves the same internal account id. Then do the harder one: create a second account with the old address and confirm it lands on a new identity row, not the first user's.
Follow-ups an interviewer will push on
- Pairwise subjects. If your client uses pairwise
sub, the same human gets a different subject per client. Either pin asector_identifier_uriso the subject is stable across your clients, or correlate through your own account id: don't fall back to email to stitch them together. - Issuer migration. If you move IdPs, every
(issuer, subject)pair changes. That's a deliberate, audited linking exercise, not something the login path should guess at. - Sloppy issuers. The non-reassignment promise is a promise, not a guarantee. If you ever see a subject reused for a different human, that's an incident against the issuer, and the mitigation is the same: your own account id is the thing you own, and the pair is how you find it.
Curated: · Written: · Reviewed:
Q&A-20When would you issue pairwise subject identifiers instead of a public subject?
The judgement in pairwise subject identifiers is which identifier is durable and which is only a convenient claim.
A public subject is the same string at every relying party, so any two applications can correlate the same person by comparing tokens. Pairwise identifiers give each relying party its own opaque subject, which keeps identity durable inside an application while removing the join key between applications.
Concretely, set pairwise as the subject type for external and multi-tenant relying parties, derive it from a sector identifier so several hosts of one product share a subject, and keep a provider-side mapping so support and audit can still resolve the person.
The reason for that specificity is a failure I have seen: A public subject shared across a consumer directory let two unrelated partners join their datasets. 74,000 accounts were correlated across the two products before the subject type was changed, and the join could not be undone.
Subject values for one person.
| Relying party | Public subject | Pairwise subject |
|---|---|---|
| partner A | identical string | distinct, 43 characters |
| partner B | identical string | distinct, 43 characters |
| correlation possible | yes | no |
I would not consider it settled without evidence: authenticate one test user at two relying parties and confirm the subject values differ while both resolve to the same internal account.
Durable inside an application, meaningless between applications.
Curated: · Written: · Reviewed:
Q&A-10Your application grants roles straight from a groups claim. How do you make that safe?
I would answer allowlisted claim-to-role mapping by separating authentication, authorization, and lifecycle into three control planes.
A claim is an assertion from the provider about the user, not a decision about your application. Mapping every group name straight through means whoever can create a group at the provider can create an administrator at the relying party.
Concretely, keep an explicit allowlist of group values that map to application roles, ignore unmapped values instead of passing them through, put mapping changes through the same review as a permission change, and log the claim values that were dropped.
The reason for that specificity is a failure I have seen: A self-service group feature let any employee create a group. Someone created one whose name matched a pass-through rule and held 6 privileged operations for 26 hours before the mapping was corrected.
Claim values seen in one synchronisation.
| Group value | In the allowlist | Role granted |
|---|---|---|
| finance readers | yes | reader |
| a self-service group named for admins | no | none |
| engineering on-call | yes | operator |
| 41 remaining values | no | none |
I would not consider it settled without evidence: create an unmapped group at the provider, sign in, and confirm the relying party grants nothing new.
The provider says what a user is in; you decide what that buys.
Curated: · Written: · Reviewed:
Q&A-8Your application federates to four identity providers through one callback. What can go wrong?
Assumptions: authorization code flow, four OIDC providers, one registered redirect URI (https://app.example/callback), a client_id/secret per provider.
What goes wrong, in one line: the callback cannot tell which of the four issuers sent the response, so a code issued by one provider can be redeemed at another's token endpoint, and the client will read the resulting identity as coming from the provider it originally chose.
The mix-up, step by step. The attacker controls one of the four registered providers: call it Evil (https://evil.example); the victim's account lives at Acme (https://acme.example).
- The victim starts login. The client mints
state=S1, recordsS1 → https://acme.example, and redirects to Acme's/authorize. - The attacker gets the browser to complete a flow at Evil instead, or Evil proxies the request through to Acme. Either way Acme authenticates the victim and delivers
code=C1tohttps://app.example/callback. - The callback has no
issto read. If the client resolves the token endpoint from anything other than its ownS1record (a query parameter, a cookie, a default) it postsC1plus its Acme client credentials to Evil's token endpoint. - Evil now holds a valid Acme code and the client's Acme credentials. It redeems
C1at Acme and returns the resulting token to the client, which reads the victim's identity out of it.
Nothing in the response ever named the issuer, so the client never saw a mismatch. PKCE does not stop this: the client hands the code verifier to the same wrong endpoint.
The fix. Bind the issuer to the request, and require the response to confirm it.
# Python 3.12, Flask-style. PENDING maps state -> issuer chosen at /login.
@app.get("/callback")
def callback():
state = request.args.get("state", "")
expected_iss = PENDING.pop(state, None) # one-shot: a replay fails here
if expected_iss is None:
abort(400) # unknown or reused state
if "error" in request.args:
return handle_error(expected_iss, request.args["error"])
if request.args.get("iss") != expected_iss:
abort(400) # mix-up: not the AS we asked
token = exchange_code(request.args["code"], expected_iss) # endpoint from our record
return finish_login(expected_iss, token)
iss is the authorization response parameter defined by RFC 9207. The token endpoint is looked up from expected_iss, never taken from the request.
Design choices.
| Design | Stops | Still exposed to |
|---|---|---|
shared callback, no iss check | nothing | mix-up; wrong-endpoint redemption |
shared callback, iss checked against the per-request record | mix-up, where the IdP returns iss | IdPs that omit iss: login fails closed, so you need the next row |
| one redirect URI per provider | mix-up even without iss | four registrations to keep in sync |
The second failure: identity collision. sub is unique only within an issuer. Keying users on sub alone, or on email, lets a user at the weakest of the four providers claim an account at the strongest. Store (iss, sub) as the primary key and treat email as an attribute, not an identifier. Check email_verified before trusting it for anything.
Other things that bite. State minted for provider A must not be accepted on provider B's response: the PENDING lookup above enforces that. Decide which provider owns the session at logout: with four IdPs, "log out" has four possible meanings and only one of them is yours.
How I'd verify it. Send the callback an authorization response whose iss names a different provider than the one the state was minted for, and confirm it is refused before any token request leaves the process. Then confirm the token request went to the recorded issuer's endpoint.
Curated: · Written: · Reviewed:
Q&A-24Your application grants an eight-hour session after SSO. What is wrong with that number?
Before calling relying-party session lifetime after federation done I would write down the grant that the source-of-truth change does not touch.
A relying party's session lifetime is a promise it makes on its own, and after federation it usually outlives the provider's session and any change made there since. The lifetime should be chosen against how quickly this application must react to a revocation.
Concretely, set the session shorter than the revocation objective for that application's data, re-check the provider session or introspect the token on a fixed interval for sensitive applications, and force reauthentication for actions above a stated risk.
The reason for that specificity is a failure I have seen: A finance application issued eight-hour sessions with no re-check. An account suspended at 10:40 kept approving purchase orders until 18:05, and 6 approvals inside that window had to be reversed by hand.
Session policy against the revocation objective.
| Application | Session length | Worst-case stale access |
|---|---|---|
| finance approvals, old | 8 hours | 7 hours 25 minutes |
| finance approvals, revised | 15 minutes with re-check | 15 minutes |
| internal wiki | 12 hours | accepted in writing |
I would not consider it settled without evidence: suspend a test identity mid-session and measure the minutes until the application refuses the next sensitive action.
A session length is a revocation delay you have chosen.
Curated: · Written: · Reviewed:
Q&A-19How do you make sure a step-up challenge applies to the request that needed it?
I would settle step-up authentication binding by attempting the action after the identity event that was supposed to stop it.
A step-up that only sets a flag on the session upgrades everything the session can do until it expires. The stronger authentication has to be bound to the specific operation, with its own freshness, so it authorises that transfer and not the next thirty.
Concretely, issue a short-lived single-use authorisation for the named operation, carry a reference to it on the request, check the authentication method and time at the point of decision, and record which challenge authorised which action.
The reason for that specificity is a failure I have seen: A payments console raised a step-up once per session. After a single challenge, a stolen session moved 26 payments in 12 minutes, and every one of them looked correctly authorised in the log.
Actions authorised per challenge.
| Binding | Challenges | Sensitive actions allowed |
|---|---|---|
| session flag | 1 | 26 |
| operation-scoped, 90 seconds | 1 | 1 |
| operation-scoped, reused | 1 | 0 after the first use |
I would not consider it settled without evidence: complete a step-up, then attempt a second sensitive action on the same session and require a fresh challenge.
Bind the challenge to the act, not to the session.
Curated: · Written: · Reviewed:
Q&A-102After a user signs in, where should the session token live, and what makes a stolen one useless to an attacker?
Assume a first-party browser app with no cross-site embedding requirement. Then the answer is an opaque session identifier in a cookie JavaScript cannot read, backed by a server-side record you can delete.
Storage. localStorage is readable by any script on the origin, so one XSS, including one inside a third-party widget, exfiltrates every live session on the page. HttpOnly cookies are not reachable from JS. The price is CSRF, which you pay with SameSite plus a per-request token on state-changing calls.
| Where the token lives | Readable by XSS | CSRF exposure | Revocable server-side |
|---|---|---|---|
| localStorage | yes | none (you add a header) | only if you keep a store anyway |
| HttpOnly cookie | no | yes, mitigated by SameSite + token | yes |
| In-memory JS variable | yes, via hooking | none | yes |
Cookie attributes.
Set-Cookie: __Host-sid=8f3c1a...; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=1800
The __Host- prefix forces Secure, requires Path=/, and forbids Domain, so a compromised subdomain cannot set or overwrite the cookie. SameSite=Lax rather than Strict because Strict drops the cookie on a top-level navigation from an email link and users land logged out; Lax still blocks cross-site POSTs. Add Max-Age so it is not a session cookie that survives a browser restart.
The identifier. 128 bits from a CSPRNG, base64url-encoded. Store SHA-256 of it, not the value, so a read-only database leak does not hand over live sessions: you look up by hash.
Binding. Do not hard-bind to IP: mobile handover and CGNAT break it and the helpdesk drowns. Use signals instead: hash of user agent plus a long-lived device cookie, ASN, geo velocity. A mismatch escalates to step-up authentication rather than an immediate kill, because false positives are routine. Hard-bind only where you control the network, such as an admin console behind a fixed egress range.
Timeouts. Idle 15–30 minutes sliding, absolute 8–12 hours regardless of activity, both enforced server-side. A client-side timer is decoration. Sensitive operations (change password, add a factor, change payout details) require re-authentication within the last 5 minutes whatever the session age.
Revocation. A sessions table keyed by token hash with an index on user_id. On password change, factor add or remove, role change, or an admin "sign out everywhere":
UPDATE sessions SET revoked_at = now()
WHERE user_id = $1 AND id_hash <> $2; -- $2 = current session
Detection: if a session identifier is presented after it was rotated, treat it as theft and revoke the whole family. Refresh-token reuse detection applied to sessions. Concurrent sessions from two ASNs minutes apart is a signal to step up, not to silently kill.
If you use JWTs instead: 5–15 minute access token, refresh token in the same HttpOnly cookie with rotation and reuse detection, and a token-version claim checked against the user record so revocation is one write.
Failure modes worth naming: session fixation (rotate the identifier at login), a cookie set on a parent domain where subdomain takeover reads it, and "revocation" that only clears the cookie in the browser while the server record stays valid.
Curated: · Written: · Reviewed: —
Q&A-103A workload in one cloud calls an API in another. Walk me through how you give it an identity, and what the receiving service must verify.
Give the workload its own identity, and make the credential short-lived and audience-bound. Running a service as a human account, or handing it a static API key, both fail the same way: the credential outlives the reason it was issued, and the audit trail points at the wrong principal.
The mechanism is workload identity federation. The platform the workload runs on (Kubernetes, GitHub Actions, an EC2 instance, a GCP VM) already runs an OIDC issuer. The workload presents a token from that issuer to the target's STS, which validates it against the issuer's JWKS and a trust policy and returns short-lived credentials scoped to one role. Nothing long-lived is stored.
A Kubernetes pod exchanging its projected service account token for AWS credentials:
aws sts assume-role-with-web-identity \
--role-arn arn:aws:iam::123456789012:role/payments-reader \
--role-session-name payments-7f3a \
--web-identity-token file:///var/run/secrets/eks.amazonaws.com/serviceaccount/token
The projected token carries an audience of sts.amazonaws.com; its expiry is set in the pod spec (Kubernetes' TokenRequest default is 3600s). The trust policy is where the security actually lives: pin the subject, not the issuer alone:
"Condition": {
"StringEquals": {
"oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLE:sub": "system:serviceaccount:payments:api",
"oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLE:aud": "sts.amazonaws.com"
}
}
When service A calls service B directly, B verifies the token: signature against the issuer's JWKS (cached, refreshed when an unknown kid appears), iss equals the expected issuer, exp/nbf with 60s clock skew, and, the check people skip, aud contains B's own identifier. Then B authorizes on the workload identity: which service account, which tenant, which operation.
Audience is the confused-deputy defence. A token minted with aud=reports-api and presented to payments-api must be rejected even though the signature is valid and the issuer is trusted. Without that check, any workload that can get a token from the shared issuer can call any other service.
| Credential | Secret at rest | Lifetime | Blast radius on leak |
|---|---|---|---|
| Shared API key | yes | until rotated | every caller |
| Service account key file | yes | months | the whole role |
| Client credentials + secret | yes | token 1h, secret long | the client's full scope |
| Workload identity federation | no | 1h default | one workload, one role |
Scopes are not user authorization. payments:read on a machine token is a capability; it says nothing about which account's payments. B still checks tenant and object ownership. If the call is on behalf of a user, the machine identity alone is not enough: either exchange the user's token for a downscoped one (RFC 8693) or have B validate the user's token alongside the workload's.
Failure modes worth naming: a trust policy with sub = repo:myorg/* lets any repository in the org assume the role; pin repo and ref or environment. A long-lived key left in CI as a fallback silently bypasses the federation. One service account shared by five workloads means one compromise is all five. JWKS cached with no refresh turns a key rotation into a 401 outage. And a JWT has no revocation list, so a 12-hour token is a 12-hour window: keep internal hops at 5-15 minutes and cloud STS at the 1-hour default.
If the workload genuinely cannot do OIDC (a legacy appliance, an old database driver) you are stuck with a secret. Put it in a vault, rotate it, scope the role narrowly, and log it as debt rather than pretending it is equivalent.
Curated: · Written: · Reviewed: —
Q&A-13The SAML response signature verifies. What else must be true before you sign the user in?
Signature verification proves one thing: the bytes covered by the signature reference were signed by a key you trust and haven't changed since. It says nothing about who the assertion is for, whether it's still valid, or whether you've already consumed it. Four checks close that gap, and each one has a decision behind it.
1. The signature covers the assertion you actually read. This is the check people skip. XML signature wrapping exploits the gap between "the document contains a valid signature" and "the element I extracted is the element that was signed." An attacker moves the signed assertion elsewhere in the document and appends an unsigned one; a parser that grabs the first Assertion element signs in the attacker. The fix is to resolve the signature's Reference URI to a specific element ID and consume that node. If your library doesn't expose the resolved reference, that's a reason to change libraries, not to add a check on top.
2. Destination and Audience. Destination must equal your ACS URL exactly: scheme, host, port, path. Audience must equal your entity ID. These stop an assertion minted for a different SP from being replayed at yours, which matters when one IdP serves many SPs or when staging and production share an entity ID.
3. Conditions window. Enforce NotBefore and NotOnOrAfter on Conditions, plus NotOnOrAfter and Recipient on SubjectConfirmationData. The deciding number is your clock-skew tolerance, and it should come from measurement rather than a copied default: compare your host's clock against the IdP's (NTP offset, or the gap between the IssueInstant you observe and your own clock). If both sides run NTP, the offset is milliseconds and the window only needs to cover issuance-to-validation latency. A 120-second allowance is a reasonable starting point when drift is unmeasured, but it also widens the window in which a captured assertion is replayable: tighten it once you know the real offset.
4. InResponseTo and replay. If you initiated the login, the response must carry the ID of the AuthnRequest you sent, and you must consume that ID exactly once. This is what makes a captured response useless. The trade-off is IdP-initiated SSO: those responses carry no InResponseTo, so requiring it disables that flow. Decide explicitly. If you need IdP-initiated, you're accepting unsolicited assertions and should lean harder on Audience, Destination, and a short Conditions window.
| Check | What it stops | Deciding input |
|---|---|---|
| signature reference resolves to the consumed assertion | XML signature wrapping | library must expose the resolved reference |
| Destination equals ACS URL | assertion replayed at another endpoint | exact URL match |
| Audience equals entity ID | assertion minted for another SP | one IdP serving many SPs |
| Conditions and SubjectConfirmation window | expired assertion | measured clock offset |
| InResponseTo consumed once | replay of a captured response | whether IdP-initiated SSO is supported |
I wouldn't call it done without negative tests: a signature-wrapping payload, an assertion addressed to another audience, an expired assertion, and a replayed response should each be refused, and ideally with distinct log reasons so you can tell which check fired.
Curated: · Written: · Reviewed:
Q&A-14How do you stop a captured SAML assertion from being used twice?
I would start SAML assertion replay from the session, token, and grant that outlive the login, not from the login itself.
An assertion is a bearer artifact that stays valid for its whole condition window, so anyone who captures it can present it again until the window closes. Replay defence is a shared record of assertion identifiers plus a short window, not a stronger signature.
Concretely, record the assertion identifier with its expiry in a store every service-provider node reads, refuse a second presentation of the same identifier, keep the condition window to a few minutes, and make the store fail closed when it is unreachable.
The reason for that specificity is a failure I have seen: A load-balanced service provider kept its replay cache in each node's memory. One captured assertion was accepted 4 times across 6 nodes inside a 5-minute window, and each acceptance created a full session.
Replay attempts across the node pool.
| Cache design | Nodes | Successful replays |
|---|---|---|
| per-node memory | 6 | 4 |
| shared store, fail closed | 6 | 0 |
| no cache at all | 6 | 11 |
I would not consider it settled without evidence: present the same assertion twice against different nodes and require the second presentation to be refused.
One-time use needs shared memory of what was used.
Curated: · Written: · Reviewed:
Q&A-21Your service provider accepts identity-provider-initiated SAML. What do you have to add?
What you add, in order
-
A second ACS endpoint. Register a dedicated AssertionConsumerService URL for unsolicited responses in your SP metadata: its own index, HTTP-POST binding, and its own policy, logging and rate limits. Leave the SP-initiated ACS as the default. This keeps the unsolicited path auditable and lets you turn it off without touching normal SSO.
-
Per-connection enablement. Unsolicited acceptance is a flag on the IdP connection, off by default. A response arriving on the unsolicited ACS from a connection that has not enabled it is rejected.
-
RelayState mapping. In SP-initiated SSO the SP sets RelayState to the deep link it wants back. In IdP-initiated SSO the IdP sets it, it is opaque, and the bindings spec caps it at 80 bytes. You must not redirect to it. Add a table mapping known RelayState values to internal routes; anything unmapped lands on a default page.
-
Validation changes. The Web Browser SSO profile's strongest anti-injection check, InResponseTo matching an AuthnRequest you issued, is unavailable, because there is no request. You compensate everywhere else:
| Check | SP-initiated | What changes for IdP-initiated |
|---|---|---|
| InResponseTo | must match an outstanding AuthnRequest ID | absent; require the response to arrive on the unsolicited ACS, and reject a present-but-unmatched value |
| Signature | Response or Assertion signed | require the Assertion itself signed, so the signature covers the subject and conditions you are about to trust |
| Destination | equals the ACS URL | equals the dedicated unsolicited ACS URL |
| Audience | equals the SP entityID | same |
| Recipient (SubjectConfirmationData) | equals the ACS URL | same |
| NotOnOrAfter | short window | same: e.g. 2 minutes, with clock skew allowed up to 5 |
| AssertionID replay cache | recommended | mandatory, since request correlation is gone |
- A confirmation step. Before an unsolicited assertion replaces an existing session, show an interstitial naming the IdP and the account. This is what stops login CSRF, an attacker forcing a victim's browser into the attacker's session, from being silent.
// ACS handler, Java 17
if (response.getInResponseTo() == null) {
if (!connection.isUnsolicitedEnabled()) return reject("unsolicited disabled");
if (!unsolicitedAcsUrl.equals(response.getDestination())) return reject("destination");
if (!spEntityId.equals(assertion.getAudience())) return reject("audience");
if (!unsolicitedAcsUrl.equals(sc.getRecipient())) return reject("recipient");
if (now.isAfter(sc.getNotOnOrAfter())) return reject("expired");
if (replayCache.contains(assertion.getId())) return reject("replay");
replayCache.put(assertion.getId(), sc.getNotOnOrAfter());
String route = relayStateMap.getOrDefault(relayState, DEFAULT_LANDING);
return confirmThenLogin(assertion, route);
}
Failure modes to name
- Open redirect via RelayState. Treating RelayState as a URL is the most common bug; the mapping table is the fix.
- Replay. Without the AssertionID cache, a captured assertion is valid until NotOnOrAfter. Cache for the full window, not a fixed TTL.
- Session fixation. Replacing an existing session without the interstitial lets an attacker plant their identity in a victim's browser.
- Auto-provisioning. Unsolicited assertions should bind to an existing account by NameID; creating accounts on first sight turns a forged assertion into a new tenant.
How to verify
Send an unsolicited response to a connection that has not enabled it and confirm the SP refuses. Then send one with a RelayState outside the map and confirm it lands on the default page rather than redirecting.
Curated: · Written: · Reviewed:
Q&A-15You need to rotate the identity provider signing key. How do you do it without an outage?
The 90-second answer
Ask first whether this is a scheduled rotation or a response to a suspected compromise: the compromise case inverts the procedure. Assume scheduled: OIDC with a JWKS endpoint, access tokens 15 minutes, refresh tokens opaque, relying parties caching JWKS per Cache-Control: max-age.
Rotation is a two-key problem, not a swap. The new key has to be in every relying party's trusted set before it signs anything, and the old key has to stay trusted until the last artifact it signed has expired everywhere. So: publish both keys, wait out the cache and token windows, start signing with the new key, then withdraw the old one after a second wait.
The overlap is arithmetic, not a convention:
overlap ≥ longest metadata cache TTL + longest lifetime of an artifact signed by the old key + clock skew
With a 24-hour metadata cache, 15-minute access tokens and 5-minute skew, that is about 24.5 hours. If refresh tokens are signed JWTs the relying parties validate, the second term is the refresh token lifetime instead: 30 days, 90 days, whatever you issued.
| Phase | Keys in JWKS | Minimum duration |
|---|---|---|
| Publish new key, keep signing with old | old + new | longest metadata cache TTL + skew (24 h 5 m here) |
| Sign with new key, keep old published | old + new | longest lifetime of an artifact signed by the old key + skew (15 min here; 30 days if signed refresh tokens) |
| Withdraw old key | new | — |
Alternatives, and when 30 days is wrong
- Shrink the window before you rotate. Drop JWKS
Cache-Control: max-agefrom 86400 to 300, wait one old TTL (24 h) for caches to lapse, then rotate with a 5-minute overlap. Costs a day of extra JWKS traffic; buys a rotation you can finish in an afternoon. - Forced refresh. You cannot force a cache to refetch, but a new
jwks_uripath is a cache miss by construction, and SAML metadata carriesvalidUntil/cacheDurationyou can shorten. Only works if relying parties re-read metadata on a schedule you control. - Per-client key pinning. Clients that pin the certificate or key out of band never fetch metadata, so no overlap length helps them. They need a coordinated change window each; find them by looking for clients that never appear in JWKS fetch logs.
- Shorter token lifetimes. Cutting access tokens from 15 minutes to 2 minutes shrinks the second phase proportionally, at the cost of more refresh traffic.
- Compromise. Invert it: publish the new key, sign with it immediately, accept that clients with stale caches fail until they refetch, and notify them ahead of time if you can. The old key must be distrusted, not overlapped.
30 days is wrong in both directions. Too long when the longest signed artifact lives minutes: you are keeping a key that can still forge tokens inside the trusted set for a month. Too short when refresh tokens or SAML assertions live 90 days, when a relying party caches metadata for 7 days, or when a client pins keys on a quarterly change cycle.
Failure modes and how you detect them
- Library ignores
Cache-Controland caches JWKS for the process lifetime: fails the moment the newkidappears. Detect with a canary: sign for one low-risk client with the new key first and watch its error rate before flipping the rest. - No refetch on unknown
kid: same failure. Detect by checking whether the client's JWKS fetch count rises when a newkidis introduced. - Pinned keys: the client never fetches JWKS at all. Detect by absence from fetch logs.
kidreused across two different keys: caches keyed bykidreturn the wrong key. Always mint a freshkid.- Clock skew:
iat/nbfin the future, 401s. Detect from the error string.
Verification before withdrawal
Fetch jwks_uri and confirm both kids resolve. Sign a token with each key and validate at a sample relying party. Check IdP logs for any fetch that still requests the old kid, if there is one, the second window is not over.
Worked example, illustrative figures: a rotation done inside a single 2-hour change window left 38 relying parties on a 24-hour metadata cycle validating against a key that was already gone; failures ran up to 26 hours and the change was rolled back. The overlap is what prevents that, and its length comes from the two numbers above.
Curated: · Written: · Reviewed:
Q&A-28You suspect the identity provider signing key has been exposed. What happens in the first hours?
First 90 seconds. Assume an OIDC or SAML IdP whose asymmetric signing key (RS256 private key, or the X.509 cert in a SAML KeyDescriptor) has leaked. That key is the trust anchor: any relying party that trusts it will accept a forged assertion for any subject, any group, any expiry. So the work is rotate, remove the old key from the published key set, and revoke live sessions, but the order and the overlap are a decision with two bad outcomes, not a default.
The trade-off.
- Distrust the old key immediately: every RP that hasn't refetched its key set rejects every token, legitimate ones included. Users drop mid-session, and if the IdP is your only path to the admin console you have locked yourself out too.
- Keep the old key published during a staged rotation: forged tokens validate for the whole overlap. If the attacker is active, that window is the one they use.
Deciding criteria, in order:
- Evidence of use. If IdP sign-in logs show issuances you can't account for, treat it as active and cut over now. Exposure alone (key found in a repo, no anomalous issuance) buys a short staged window.
- Can you force RP refresh? If you can push metadata/JWKS to RPs out-of-band (config management, coordinated deploy), the lockout is minutes and immediate distrust is cheap. If RPs poll on a cache TTL you don't control, "immediate" means "whenever each RP happens to refetch," which can be hours.
- Blast radius. If any RP is privileged (admin console, CI, cloud control plane) weight toward immediate distrust.
- Second auth path. Break-glass local accounts or a second IdP make the lockout survivable. Without one, an outage is an outage.
Default when exposure is confirmed and you cannot force refresh: distrust immediately and take the outage. A forged admin session is worse than a login outage.
Sequence. Timings below are from a worked incident, hypothetical figures:
| Step | Target | Actual |
|---|---|---|
| new key published | 30 min | 41 min |
| old key distrusted at every RP | 2 h | 5 h |
| sessions and refresh tokens revoked | 2 h | 2 h |
| issuance reconciliation complete | 5 days | 3 weeks |
What rotation alone does not do. Already-issued tokens stay valid until exp; the signature was checked at issuance. Rotating the key logs nobody out. Session and refresh-token revocation is a separate operation, and for SAML, sessions already established at RPs are separate again (back-channel logout where supported, otherwise the RP's own session TTL).
Failure mode. An exported key sat in a configuration repository. Forged assertions produced 12 administrative sessions across 4 relying parties over a weekend, and reconstruction took 3 weeks because relying-party logs recorded no key identifier. The fix is cheap: log the kid from the JWT header, or the signing cert thumbprint for SAML, on every accepted assertion, and alert on any identifier not in the currently published set.
Done when. The old kid is refused at every RP (test it, don't assume it) and the IdP's issuance record accounts for every accepted assertion.
Curated: · Written: · Reviewed:
Q&A-26How do you make federation debuggable without storing tokens in your logs?
What a strong answer sounds like
Decide first what the audit record is for: reconstructing a validation decision, not replaying the request. If the log can answer "which issuer, which subject, which key, which check failed, and when", the token itself adds nothing. So the default is to log the claims that are already non-secret: iss, sub, aud, kid, iat, exp, jti (RFC 7519 §4.1, RFC 7515 §4.1), and redact signature and payload in the logging serializer, so no call site can opt back in.
| Field | Stored | Why |
|---|---|---|
iss, aud, kid | yes | reconstruct which key validated what |
sub, iat, exp | yes | who, and whether it was in window |
| failing check name | yes | distinguishes the reasons a validation fails |
| signature, raw payload | no | credential material |
That works for JWTs. For opaque tokens there are no claims to log, and that is where the real design choice is.
Getting a correlator when the artifact is opaque
| Option | Gives | Cost | Use when |
|---|---|---|---|
| RFC 7662 introspection, log the response | active, scope, client_id, sub, exp, jti: real claims, no token | a round trip per validation (cache for the token's lifetime); the introspection endpoint is now on the hot path | you control the AS and the token is opaque |
| Plain SHA-256 of the token | cheap, deterministic, correlates across systems with no shared secret | anyone with log access can enumerate candidate tokens offline and match the hash | the artifact is high-entropy and unguessable: a 256-bit random opaque token, not a JWT with a small claim set |
| HMAC-SHA-256, key in KMS | same correlation, offline enumeration gone | every correlating system needs the same key, so rotation breaks correlation across the boundary and you must retain old keys to join historical events | the artifact is low-entropy, or log readers are less trusted than the services producing the events |
When you genuinely need the bytes
Two escape hatches, both time-boxed:
- Short-TTL debug sink. Full assertion bodies go to a separate store with a 15-minute TTL, its own IAM, and no replication into the main log index. Use it while you are actively debugging a live federation break. The TTL is the only thing between a token and long-term exposure, so alert on the sink's oldest object age, not on its size.
- Sampled full-trace mode. Capture the full assertion for a fixed fraction of requests, or for one canary tenant, and log the sampling decision so you know what you are looking at. Use it for intermittent failures you cannot reproduce. It misses the rare failure you are chasing, and the sample is still credential material: same TTL and access controls as the sink.
Failure modes and how you would see them
- A library logs the raw header at DEBUG while your own code redacts correctly. Detect with a canary: send a request carrying a known-format fake token in a test environment and alert if it appears in the log index.
- The debug sink's deletion job stalls. Detect by alerting on max object age in the sink, not on the job's exit code.
- Key rotation silently breaks correlation. Detect by monitoring the join rate between validation events and downstream events; a step change after a rotation is the signal.
The incident that makes this concrete: a debug flag left assertion bodies in an application log for six weeks. 11,000 assertions were readable by every engineer with log access, and 90 were still inside their validity window when the flag came off. None of that showed up in log volume or error rate: only in a search for the token format.
Verification
Search the log store for the token prefix pattern and require zero matches after the redaction change, and add a CI test that feeds a sample assertion through the real logging path and asserts the output carries no signature or payload. Correlation is the thing you traded away, so measure it: if the join rate between validation events and downstream events drops, the correlator is too lossy and you have over-redacted.
Curated: · Written: · Reviewed:
Q&A-22A user changes their surname and their email address. What must not change in your system?
What a strong candidate says first
Clarify one thing before answering: which identifier does the app use to find the account on each login? If it is the email, the rename is an incident. If it is the provider's subject identifier, the rename is a row update.
Assumption: OIDC login against one or more external IdPs, and the application owns its own account table.
Answer: the internal account id and the (issuer, subject) pair must not change. Roles, group memberships, resource ownership, audit actor references and API tokens all hang off those. Email, display name and surname are mutable attributes that update in place.
| Must not change | May change |
|---|---|
| internal account id (surrogate PK) | |
(issuer, subject) from the IdP | email_verified (can flip to false mid-change) |
| role and group grants | display name, given/family name |
| resource ownership, audit actor refs | locale, avatar URL |
| account creation date |
Why
OIDC Core defines sub as locally unique and never reassigned within the issuer. Email is neither unique nor stable: one person can hold several addresses, and a provider can re-register a deleted account's address to a different person. sub may be pairwise per client, so key on the pair (iss, sub), not sub alone.
Trade-off: email as key vs surrogate key
| email as key | (iss, sub) surrogate | |
|---|---|---|
| works when | one IdP, one tenant, no self-serve email change, no account linking | multiple IdPs, self-serve changes, linking, recycling matters |
| rename | duplicate account or manual merge | row update |
| provider recycles the address | new person inherits the old account | new person gets a new account |
| cost | zero schema work | one extra column plus backfill |
Email-as-key is defensible for a small in-house app with a single verified provider and no self-serve email change, but store sub alongside from day one so you can migrate without a merge.
Failure modes and how they are detected
- Duplicate account on rename: a unique constraint on
(iss, sub)plus an alert when an insert targets a pair that already exists. - Takeover via recycled email: require
email_verifiedand re-verification when the address changes. - Lockout during the change window: providers can emit
email_verified: falsewhile the new address is unconfirmed; don't gate login on it, gate sensitive actions. - Case and alias collisions: normalize (lowercase, strip plus-addressing) for display, never for identity.
Trace: alice@example.com → alice.smith@example.com
- IdP token:
iss=https://idp.example.com,sub=00u1a2b3c,email=alice.smith@example.com,email_verified=true.subis unchanged. - Callback:
SELECT id FROM accounts WHERE issuer=$1 AND subject=$2→42. UPDATE accounts SET email=$1, email_verified=$2 WHERE id=42.INSERT INTO account_attribute_history(account_id, attr, old_value, new_value) VALUES (42,'email','alice@example.com','alice.smith@example.com').- The existing session still resolves to account 42; if the session caches the email, refresh it.
Wrong path: SELECT id FROM accounts WHERE email=$1 → no row → INSERT → the user lands in an empty account with none of their grants.
-- PostgreSQL 16; requires UNIQUE (issuer, subject)
INSERT INTO accounts (issuer, subject, email, email_verified)
VALUES ($1, $2, $3, $4)
ON CONFLICT (issuer, subject) DO UPDATE
SET email = EXCLUDED.email,
email_verified = EXCLUDED.email_verified,
updated_at = now();
The person is the subject; the address is a label.
Curated: · Written: · Reviewed:
Q&A-12Just-in-time provisioning created the accounts. What removes them?
Before calling JIT provisioning versus SCIM deprovision done I would write down the grant that the source-of-truth change does not touch.
Just-in-time provisioning is a side effect of a sign-in, so it has a create path and no delete path. Without a provisioning protocol driving the other direction, an application accumulates accounts that only stop being used rather than stopping existing.
Concretely, run a provisioning protocol for lifecycle in both directions, keep just-in-time creation only where that is impossible, and reconcile the application's account list against the source weekly so accounts with no matching identity are surfaced and closed.
The reason for that specificity is a failure I have seen: An analytics tool created accounts on first sign-in for 3 years. A reconciliation found 214 accounts whose owners had left, 19 of them holding workspace administration, and the oldest had been unowned for 22 months.
Reconciliation of one application.
| Account state | Count | Action |
|---|---|---|
| active with a matching identity | 1,286 | keep |
| no matching identity | 214 | close |
| no matching identity, administrative | 19 | close and review |
I would not consider it settled without evidence: remove a test user at the source and confirm the application's own account list no longer contains them inside the stated window.
A create path without a delete path is an accumulation.
Curated: · Written: · Reviewed:
Q&A-101Walk me through what SCIM 2.0 buys you over a bespoke user API, and what breaks when the target application only implements half of it.
First 90 seconds
SCIM 2.0 is two RFCs: 7643 defines the resource schemas (User, Group, plus extension schemas), 7644 defines the protocol. Endpoints, filter grammar, PATCH semantics, optional bulk, and a discovery surface. What it buys you is not elegance. It buys you one connector instead of N bespoke ones: your joiner/mover/leaver pipeline speaks /Users and /Groups to every target that implements it, and the per-app work drops to attribute mapping plus handling the gaps.
The discovery surface is the part people skip. GET /ServiceProviderConfig tells you whether PATCH, bulk, filter, sort, ETag and change-password are supported, and the limits (bulk.maxOperations, bulk.maxPayloadSize, filter.maxResults). GET /Schemas and GET /ResourceTypes tell you which attributes exist and which are required. A connector that probes these at connect time and stores a capability profile behaves correctly against a strict implementation and a half-broken one.
Deprovisioning is a PATCH, not a DELETE
PATCH /scim/v2/Users/2819c223-7f76-453a-919d-413861904646
Content-Type: application/scim+json
If-Match: W/"e180ee84f0671b1"
{"schemas":["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
"Operations":[{"op":"replace","path":"active","value":false}]}
active:false is the soft delete. The row stays, the identifier stays, group memberships can stay, and a rehire flips it back, which is why you never DELETE a user to offboard. If-Match with the ETag is what stops two concurrent writers from silently clobbering each other; without it, last write wins and one of the two changes disappears.
What breaks when the target half-implements it
| Gap | What you see | What you do |
|---|---|---|
active accepted but ignored | Leaver's account still authenticates | Read back after write: GET the user, assert active == false. Fail the workflow and open a ticket. Never mark a leaver complete on a 200 alone. |
| No PATCH, PUT only | GET-modify-PUT races | Serialise per user; require ETag, or accept lost updates and reconcile nightly |
Filter only eq on id | Cannot resolve by userName | List plus client-side match, or keep a local id map |
No /Groups write | Membership drifts | Read-only reconcile, raise drift to the app owner |
| No bulk | 500 users × 3 calls = 1500 requests | Token bucket, honour Retry-After on 429, checkpoint the queue |
That last row is the one that bites in production. A 500-person onboarding at 10 requests/second is 150 seconds of wall clock before retries; a single 429 without backoff turns it into a partial sync, and partial syncs are how you get accounts that exist in the source of truth and not in the target.
The framing that matters
SCIM is best-effort delivery, not a transaction. Treat the target as eventually consistent: every write is followed by a read-back assertion, and a reconciliation job diffs source of truth against target on a schedule. Correlate on externalId, not userName: userName is mutable and is usually the email address. The capability profile from /ServiceProviderConfig decides which features you can rely on; everything else is a documented gap with a named owner.
Curated: · Written: · Reviewed: —
Q&A-11You disable a leaver at the identity provider. Which of their sessions are actually over?
The 90-second answer
Disabling an account at the IdP is an authentication gate, not a revocation. It stops the next sign-in. It does not reach into cookies, refresh tokens, or application sessions that were already issued, because those objects are held by other parties and run on their own clocks. So the direct answer to "which sessions are over?" is: none of them at the moment you click disable. Except the IdP's own session, if the IdP clears it as part of deactivation.
Assume the IdP is the only auth path, resource servers validate access tokens locally, and no relying party has registered a back-channel logout endpoint. Under those assumptions, here is what each object is and what actually ends it.
| Object | Held by | Typical lifetime | What ends it |
|---|---|---|---|
| IdP session cookie | IdP | 8–12 h | deactivation (if the IdP clears it), explicit session revocation, expiry |
| Access token (JWT) | RP / resource server | 5–60 min | expiry only, when validated offline |
| Refresh token | RP's token store | per-app policy, days to weeks | RFC 7009 revocation call, or IdP-side session revocation |
| RP session cookie | each app | app policy, often 8–12 h or browser-session | app logout, back-channel logout, expiry |
| VPN / remote access session | network edge | re-auth interval, often 8–24 h | re-auth, or explicit disconnect |
| API keys, PATs, SSH keys | user-created | until rotated | manual rotation only |
What actually revokes
- OIDC Back-Channel Logout 1.0. The IdP POSTs a logout token to the RP's registered
backchannel_logout_uri; the RP validates it (iss,aud, the backchannel-logout event claim,sid/sub) and kills the session. Requires RP support and the RP having storedsidfrom the ID token. - SAML 2.0 Single Logout (Profiles spec §4.4). Same idea, poorly supported in practice.
- RFC 7009 token revocation. Kills refresh tokens at the authorization server. Someone has to call it: the IdP on the user's behalf, or the RP.
- RFC 7662 introspection, or a session-check call per request. Makes revocation immediate, at the cost of a network hop on every request.
- Short access-token TTL. The blunt lever: a 5-minute access token bounds the offline-validation window to 5 minutes.
- Entra ID CAE. Supporting resource servers (Exchange Online, SharePoint, Teams) react to revocation events in near real time instead of waiting for token expiry. Note that in Entra ID, disabling the account blocks new sign-ins; "Revoke sessions" is the separate action that invalidates refresh tokens. Okta's
DELETE /api/v1/users/{id}/sessionsclears Okta sessions; app sessions still need app-level logout or Universal Logout.
Worked example (hypothetical figures: one IdP, three RPs, leaver disabled 09:12)
- Access token, 15 min TTL, validated locally → dead by 09:27.
- Wiki, SAML SP, 12 h session cookie, no SLO → session ends 21:12.
- Customer system, OIDC RP, 14-day absolute refresh token, no back-channel logout → keeps minting 15-minute access tokens until day 14.
- VPN, re-auth every 8 h → ends 17:12.
At 09:12, zero sessions are over. The first thing to die is the 15-minute access token.
Failure modes
- Offline JWT validation leaves no revocation path until
exp. This is the common case, not the exception. - Back-channel logout silently no-ops if the RP never stored
sid, or its endpoint returns 500. Delivery is best-effort with retries, not a guarantee. - Front-channel logout breaks under third-party cookie blocking.
- Refresh-token rotation without reuse detection: the leaver's client keeps rotating and stays alive.
- Kerberos TGTs are a separate surface: a TGT issued before disablement has its own lifetime (10 h default in AD, renewable up to 7 days). Whether your KDC rejects service-ticket requests for a disabled account is worth testing in your domain rather than assuming.
Verification
Disable a test identity, attempt a privileged action at each relying party, and record minutes-to-denial per surface. That number is the only one that answers the question.
Curated: · Written: · Reviewed:
Q&A-23Provisioning set the user to inactive in the target application. Are their API clients dead?
The engineering content of SCIM deactivation versus refresh tokens is the revocation path and its measured latency, not the protocol name.
Provisioning changes an attribute in the target's user store; it does not enumerate and destroy the grants that application issued. Refresh tokens, personal access tokens, and installed integrations each survive until something revokes them by name.
Concretely, make deactivation trigger a token revocation sweep for that subject at every authorization server and application, verify with an introspection call rather than by reading the user record, and treat a target with no revocation interface as an exception carrying a compensating control.
The reason for that specificity is a failure I have seen: A contractor was deactivated on a Friday. A refresh token inside a build agent kept exchanging for access tokens and pulled 3 repositories every 30 minutes for 16 days, while the user record read inactive the entire time.
State after deactivation, before the sweep.
| Object | State | Still usable |
|---|---|---|
| user record | inactive | no |
| refresh token in a build agent | 16 days remaining | yes |
| 2 personal access tokens | valid | yes |
I would not consider it settled without evidence: deactivate a test user, then introspect their outstanding refresh tokens and require every one to return inactive.
Deactivation writes a field; revocation ends a grant.
Curated: · Written: · Reviewed:
Q&A-86You create accounts a week before the start date. What keeps them safe until then?
Assumptions: the directory can hold an account in a disabled state separate from credential issuance; the personnel system emits a confirmed-start event; you control group membership and licence assignment.
The answer: create the account disabled with no usable credential, hold the initial factor for supervised enrolment on day one, activate on a confirmed-start event from the personnel system, and delete the record automatically if the start never happens. Safety comes from the session, token and grant layer that outlives the login, not from the login form.
Why: an account that exists before its owner arrives is a credential nobody is watching. A disabled account cannot obtain a session, so there is nothing to steal that grants access. No credential means nothing to phish or guess. Tying activation to a proofing event rather than a date means the human proves identity before the account becomes usable, and a withdrawn offer leaves a record that never activates.
Alternatives, and when to pick them.
| Approach | Dormant window | Cost |
|---|---|---|
| JIT provisioning at first login | none | nothing pre-staged: no mailbox, groups or licences before day one |
SCIM with staged activation (active=false, flipped on the start event) | same as pre-provisioning, but standardised | depends on IdP/SCIM support and a reliable HR feed |
| Pre-provisioned and enabled early | the whole gap | the failure below |
Pre-provisioning earns its keep when you must stage a mailbox, group memberships, licences or hardware-bound credentials before day one. If the start date is firm and the role needs none of that, just-in-time provisioning is simpler and removes the dormant window entirely: that is the case where pre-provisioning is not worth it.
Failure modes and detection.
- Enabled early with a predictable initial password. Detection: alert on any authentication attempt against an account whose activation event has not fired.
- Offer withdrawn, account never deleted. Detection: reconcile the directory against the HR active-employee feed daily and alert on accounts with no matching active record.
- Activation driven by a date rather than an event. Detection: assert the activation source is the personnel event; a scheduled job that flips accounts live is the bug.
Worked example. Accounts created 10 days early with a predictable initial password and enabled immediately. Two offers were withdrawn; the accounts stayed live for 5 weeks and one was used to read an internal document library. That is 35 days of exposure per account. Under the design above the same two accounts never authenticate, and daily reconciliation caps the stale-record window at 24 hours.
Verification. Attempt to authenticate as a pre-provisioned account before its activation event and require the attempt to fail: no session, no token. Existence is not usability.
Curated: · Written: · Reviewed:
Q&A-87Should a mover's new access land before or after the old access is removed?
Remove first, by default, and make the removal a triggered, verified step rather than a sequence you hope someone follows.
Assumptions that decide it: the receiving manager has already requested and approved the new entitlements before the move date; the old entitlements can be revoked without breaking a system the mover still owns; and there is a provisioning SLA for the new set (same-day for SSO-backed apps, longer where a manual approval chain sits in front of it).
Why the order matters: grants are additive, revokes are not. A grant is an event somebody wants: the new manager files it, the app team provisions it. A revoke is an event nobody owns. So "add then remove" doesn't fail at the add; it fails at the remove, silently, and the failure looks like success: the mover can do the new job, no ticket is open, and the old set just sits there. "Remove then add" fails at the remove, loudly: the mover can't open the thing they need, they file a ticket, and it's fixed the same day.
That asymmetry, silent over-permission versus loud under-permission, is the whole argument. Over-permissioning surfaces months later as a segregation-of-duties finding in an access review. Under-permissioning surfaces as a Slack message.
The exception. Some moves genuinely need overlap: handing over an on-call rotation, transferring ownership of a pipeline, a decommissioning where the old system still holds the records. Make that overlap explicit: a named window with an expiry date, scoped to the specific entitlements, approved by both managers, and reported. Overlap should be a decision with a date on it, not a leftover.
Failure modes, and how each is detected.
- Nested groups and role chaining. Removing the mover from the old group doesn't remove access granted through a nested group or a role that maps to the old app. Detect by diffing effective entitlements, not group memberships.
- Session and token lifetime. Revoking a group doesn't kill an active session or a long-lived OAuth refresh token. Check the app's session TTL; for anything holding a token longer than an hour, force a session revoke alongside the group change.
- Downstream provisioning lag. The IdP revokes in seconds; a SaaS app deprovisions on its next SCIM sync, often nightly. Detect by reconciling the app's user list against the IdP, not by trusting the IdP's view.
- Shared and break-glass credentials. The mover's personal access is gone but the shared credential they knew still works. Rotate shared credentials on mover events.
The mover event, as a sequence.
T0 IdP: drop old role assignment, revoke active sessions
T0 SCIM push to apps that support it; ticket to app owner for those that don't (SLA 3d)
T0+1d reconcile: effective entitlements == new set?
T0+30d exception window expires -> hard revoke; alert if the old set is still live
Illustrative figures from a hypothetical 12-month mover program, to show the shape of the outcome rather than to be quoted as benchmarks:
| Sequence | Moves | Both sets live past 30 days |
|---|---|---|
| add then remove | 340 | 51 |
| remove then add | 288 | 2 |
| explicit handover window | 46 | 0 (expiry enforced) |
The remove-then-add row isn't zero because the gap is harmless: it's near-zero because the gap is visible and gets closed. The add-then-remove row is the one that needs a control, because nothing in the process complains.
Verification. Sample recent movers, pull the effective entitlement set before and after, and confirm the old set is empty inside the stated window. Alert on any mover whose old and new sets are both live past the window. Without that check, the ordering is a convention, not a control.
Curated: · Written: · Reviewed:
Q&A-88How does access granted for a project come to an end?
Project access isn't one thing. In a typical cloud setup it is carried by one of four artefacts, and each ends differently: a short-lived credential, a conditional IAM binding with a time condition, a group membership, or a standing role assignment. Which one you are relying on decides whether access ends by itself or waits for someone to act. Assume the common shape: SSO groups mapped to cloud roles, with a project record in a tracker that the IAM layer knows nothing about.
The direct answer: access ends when the artefact carrying it stops being valid. Only two of the four do that on their own.
| Carrier | Ends by itself? | What ends it | Detection |
|---|---|---|---|
AWS STS session (AssumeRole, default 3600 s, max 43200 s) | yes | session duration | CloudTrail AssumeRole plus expiry |
GCP IAM binding with a request.time condition | yes | condition evaluates false | none at expiry: silent |
| SSO group membership (Okta / Entra / IAM Identity Center) | no | someone removes the member | membership export diffed against the live project list |
| Standing role assignment (AWS permission set, Azure role) | no | an admin revokes it | access review / attestation |
The weak link is group membership, because nothing in the system knows the project ended. The project record and the group are separate objects; closing the project does not touch the group. The fix is to make the project record the source of truth: a SCIM group rule keyed on a project attribute, or a reconciler that runs on project close and removes every membership derived from that project.
Trade-offs. Mandatory end dates are cheap and work when the reason for the grant is known at grant time. They hurt for standing platform, on-call and break-glass access, where the "project" is the org itself; forcing 90-day renewals there produces rubber-stamped renewals, which is worse than no expiry because it looks like control. Just-in-time elevation (Azure PIM eligible assignments, GCP conditions plus approval) is better for privileged access but adds a request step and latency during incidents, so break-glass has to bypass it. Attestation campaigns (Entra access reviews, quarterly) catch what automation misses, but they are periodic: a stale grant lives until the next cycle, and reviewers approve what they do not recognise. Short-lived credentials are strongest but only cover what can assume them; long-running batch jobs and third-party integrations often cannot.
Failure modes.
- Silent expiry. A GCP binding of
roles/viewerwith conditionrequest.time < timestamp("2025-03-01T00:00:00Z")grants nothing from 00:00:00Z onward, and no log line is written at that instant. A nightly job depending on it fails at 02:00 with no obvious cause. Alert on the first authorization failure for a principal that had access yesterday, not on the expiry date. - Orphaned group. Hypothetical: a 90-day migration grants 40 engineers read access through one group. Dated, exposure is 40 × 90 = 3,600 person-days. If the group outlives the project by five months, 40 × 240 = 9,600. The second number is the one an auditor finds.
- Contractor identity. Offboarding removes the employee account; a contractor identity in the same group survives it. Reconcile membership against the contractor roster, not just the employee directory.
- Extension without a reason. A one-click extension with no recorded justification turns a 90-day cap into a rolling permanent grant.
Verification: close a test project and confirm the membership count for its group reaches zero and the reconciler logs each removal. If it does not, the tie between project record and group is decorative.
Curated: · Written: · Reviewed:
Q&A-89A dismissal is happening at two in the afternoon. What is your sequence?
I would treat immediate leaver kill chain as a claim about effective access that has to survive a disablement.
An immediate termination is an ordered sequence, because the steps have different latencies and one of them warns the person. Ending authentication and live sessions comes first, and the tidying of records comes afterwards.
Concretely, disable the account and revoke sessions and refresh tokens at the source, revoke device certificates and remote-access sessions, disable the authenticators, sweep tokens at the major relying parties, then handle mailbox delegation, data ownership transfer, and physical access, with a named owner and a timestamp per step.
The reason for that specificity is a failure I have seen: A termination began with a mailbox transfer and reached session revocation 40 minutes later. Inside that window the person cleared a shared folder and forwarded 260 messages from an active session on a personal device.
Termination sequence and timing.
| Step | Target elapsed | Owner |
|---|---|---|
| disable account, revoke sessions | 0 to 2 minutes | identity on-call |
| revoke tokens at 6 major relying parties | 5 minutes | automation |
| device and remote-access certificates | 10 minutes | endpoint team |
| mailbox and data transfer | same day | the manager |
I would not consider it settled without evidence: run the sequence against a test identity and record the elapsed seconds to first denial at each tier.
End the live paths first; the paperwork does not run away.
Curated: · Written: · Reviewed:
Q&A-90Your provisioning create timed out but the user may already exist. What now?
What I'd say first. A timeout is an unknown outcome, not a failure: the create may have committed and the response was lost. So the first move is never a retry of the POST. It's a lookup by the source system's immutable identifier, and only then a create. If the lookup finds the user, adopt it and move on. If it doesn't, create, and treat a uniqueness conflict on the way back as success, because it means someone else's attempt won the race. The whole thing rests on one rule: retry against an identifier, never against hope.
Why the identifier has to be the source's, not the target's. SCIM's externalId is the client-assigned identifier for the resource (RFC 7644 §3.1), which is exactly what you want: it's stable across email changes and renames. userName is not: it's usually the email, it mutates, and the spec only permits providers to enforce uniqueness on it rather than requiring it. Two caveats worth stating out loud in an interview: the spec does not require the provider to enforce uniqueness on externalId, and filtering on externalId is optional support. So externalId is a lookup key, not a guarantee, and you check the provider's docs for filter support before you build on it.
The retry path.
def ensure_user(scim, ext_id, payload):
# The create may have landed before the timeout. Look before you leap.
found = scim.get(f'/Users?filter=externalId eq "{ext_id}"')
if found['totalResults'] == 1:
return found['Resources'][0]['id']
if found['totalResults'] > 1:
raise DuplicateExternalId(ext_id) # reconcile, never create
r = scim.post('/Users', json={**payload, 'externalId': ext_id})
if r.status == 201:
return r.json()['id']
if r.status == 409: # scimType: uniqueness
again = scim.get(f'/Users?filter=externalId eq "{ext_id}"')
if again['totalResults'] == 1:
return again['Resources'][0]['id'] # the other worker won; adopt
raise NeedsManualReconcile(ext_id) # 409 but unfindable
raise Retryable(r.status)
Trace it. If the original POST committed, the first GET returns one resource and no second create is issued. If it didn't commit, the GET returns zero and the POST creates. If two workers retry at once, one gets 201 and the other gets 409 with scimType: uniqueness (RFC 7644 §3.12), re-reads, and adopts the winner. The only path that escalates is a 409 whose filter finds nothing.
| POST result | Meaning | Action |
|---|---|---|
| 201 | created | persist source-id → target-id mapping |
| 409 uniqueness | already exists | GET by externalId, adopt |
| 409, GET empty | userName held by a soft-deleted or foreign record | escalate, do not retry |
| timeout / 5xx | unknown | GET by externalId with backoff, then create |
Failure modes that bite. Eventual consistency: a GET issued milliseconds after a timed-out POST can 404 because the write hasn't replicated, and the retry then creates a duplicate. Back off and re-read before concluding absence. Soft deletes: a deactivated user often still holds the userName, so the create 409s while a default filter hides the record. That's the escalate row, and it's why you don't loop on 409. Unsupported filters: if the provider rejects externalId filters, fall back to a local mapping table written after every successful create, and look up there first. And if duplicates already exist, don't delete one: the duplicate owns files and group memberships, so transfer ownership and memberships to the survivor before removing it.
Detection and verification. Alert on create timeouts and on 409 rate, and run a periodic scan for any externalId with more than one resource. To prove the path works, force a timeout on a create, kill the connection after the server has accepted the request, and confirm the retry resolves to one object, not two.
Curated: · Written: · Reviewed:
Q&A-91One malformed user record is blocking your provisioning queue. What is the design fix?
Isolate the bad record. A provisioning queue that halts on one malformed payload turns a single data problem into an outage for every identity change behind it: including terminations, which are the ones you cannot let sit.
Assume the queue carries identity events (joins, moves, leavers) into a downstream connector, and the bad record fails deterministically: same payload, same error, every time. That distinction matters, because it separates a poison record from a transient failure.
The fix is per-record error handling. Process each record in its own unit of work, catch the failure at that boundary, and move the record to a quarantine store with the payload and the error reason attached. The queue keeps draining. Repeated failures do not trigger a batch retry: a batch retry re-runs the good records and re-fails the bad one, which is how one record becomes 1,100.
Terminations get a priority lane that cannot be blocked by the main queue. A leaver's access removal is a security event; it should not wait behind a display-name validation error. Bound that lane so it cannot starve the main queue, and alert on its depth.
Alert on quarantine depth, not just on queue depth. A quarantine that nobody watches is a silent graveyard: records accumulate, nobody notices, and the "fix" has just moved the outage somewhere quieter. Clearing quarantine is a human action: a person looks at the reason, fixes the source data or the connector, and re-drives the record. A blanket auto-retry of quarantine is wrong for deterministic failures; it just re-fails. Auto-retry with backoff is right for transient errors, so classify the failure before deciding.
The failure I have seen: a user whose display name contained an unsupported character failed the connector, which retried the whole batch. 1,100 changes queued behind it, including 3 terminations that completed 26 hours late.
| Design | Records blocked | Late terminations |
|---|---|---|
| batch retry on failure | 1,100 | 3 |
| per-record isolation | 1 | 0 |
| priority lane for leavers | 1 | 0 |
Verification: inject a malformed record into a staging queue and confirm three things. The queue continues past it, the record lands in quarantine with its reason, and terminations still complete on the priority lane. Then check the alert fires when quarantine depth crosses its threshold.
One bad record should cost one record.
Curated: · Written: · Reviewed:
Q&A-92Someone rejoins after two years away. Do you reactivate the old account?
Reactivate the identity record. Do not reactivate the account's access. Keep the person and their history; rebuild entitlements from the new job's birthright set. If you cannot show that the old entitlements were stripped at departure, treat the account as untrusted and provision fresh.
What you are actually deciding
An identity record and an account are different objects. The identity carries the employee ID, payroll link, audit references, commit authorship, ticket history and on-call rotations: all of which you want to keep, and all of which break if you mint a new identifier. The account is a credential plus an entitlement container, and its value is entirely in what is attached to it. Two years is long enough for the entitlement model to have moved: groups renamed, roles split, applications migrated, the old job code retired.
Reactivation restores the attachment set wholesale. Enumerate before you decide:
- group memberships, including nested ones
- application role grants and cloud IAM roles
- delegations: mailbox, calendar, approval
- OAuth refresh tokens, long-lived API keys, SSH keys, code-signing keys
- MFA enrolments bound to a phone number or device that no longer exists
- distribution lists, shared mailboxes, VPN profiles
- anything the account owns: service accounts, groups, repositories
Failure modes and how they surface
Stale privileged membership. The dormant account is still in a group that grants approval rights. This is invisible on the account summary page; it appears only in an effective-access diff.
Live delegation. A mailbox delegate or forwarding rule still points at a former colleague. Detected by enumerating forwarding rules and delegates, not by reading the account.
Orphaned ownership. The account owns a service account or a group, so reactivation silently re-grants admin. Detected by reverse lookup: what does this principal own?
Dead MFA device. Enrolment is bound to a phone that is gone, the user cannot complete the challenge, and the helpdesk bypasses MFA to unblock them. Detected by comparing enrolment state against the device inventory.
Unrevoked tokens. Refresh tokens issued before departure may still be valid if offboarding never revoked them. Detected by a token-revocation audit.
The alternative, and when it wins
Minting a new account costs you split history, a duplicate person in HR and ITSM, broken references in commits and tickets, and an orphaned old account that nobody deletes. It is still the right call when the old account was compromised, when the IdP has changed, when the person returns as a contractor under a different legal entity, or when a legal hold requires the old account to stay frozen.
Otherwise the operational rule is: strip at departure, not at rehire. Departure is the moment you have a trigger and know what was attached. Rehire is two years later with no reliable memory of it.
Worked example (illustrative)
An account dormant 26 months is reactivated as-is. Effective access diff:
| Source | Entitlements | Kept |
|---|---|---|
| old account, as found | 18 | 0 |
| new birthright (job code) | 14 | 14 |
| requested additions | 2 | 2 |
Of the 18, one was a payroll approval role and one was a mailbox delegation forwarding to a colleague who had also left. Neither appeared on the account page.
Verification
Do not call it done until effective access equals the birthright set exactly. A flat membership query misses nested groups, so recurse:
WITH RECURSIVE eff AS (
SELECT group_id FROM group_membership WHERE principal_id = :account_id
UNION
SELECT gm.group_id
FROM group_membership gm JOIN eff ON gm.principal_id = eff.group_id
)
SELECT g.name, g.risk_tier FROM eff JOIN groups g ON g.id = eff.group_id;
Any row in that result that is not in the birthright set is a finding. Keep the person's history; rebuild the person's access.
Curated: · Written: · Reviewed:
Q&A-85A manager asks you to give the new starter the same access as a colleague. What do you say?
What I'd actually say to the manager
"Happy to get them productive on day one. Before I copy [colleague]'s account, can you tell me what the new starter needs to do in their first two weeks? A copy hands them everything [colleague] has accumulated, including access from roles they used to hold, and I can't defend any of it in an access review. If I provision from the birthright set for their job code, they get the standard baseline today, and anything extra goes through a request you approve. If there's something they need in the next hour, name it and I'll grant it directly with a ticket."
That's the conversation, roughly 30 seconds. The rest is what happens when the manager pushes back.
Assumptions I'd state out loud
- There's a source-of-truth HR record with job code, location and employment type.
- A birthright or role set exists, or can be defined, per job code.
- The colleague's current access is not itself the definition of the job.
If the first two aren't true, cloning is the pragmatic stopgap: see below.
Why cloning fails
Cloning copies exceptions. A colleague's account is their birthright plus every approved exception they've collected plus every role they held before. Once granted, none of that is visible as "extra": the new account looks identical to any other. The failure is silent: nothing flags it, and the next access review compares the new account against the colleague's, which is also wrong.
Hypothetical numbers, made up to show the shape: cloning a nine-year veteran onto a graduate produced 62 entitlements; 19 were justified by the graduate's job code. The other 43 included production database write and a finance approval role: a segregation-of-duties conflict nobody would have approved on request.
| Source | Entitlements | Justified by the job |
|---|---|---|
| Clone of a colleague | 62 | 19 |
| Birthright policy | 19 | 19 |
| Requested additions | 3 | 3, each approved |
When cloning is acceptable
- No role definitions exist yet and the team is small enough that the manager can name every entitlement.
- It's time-boxed: grant for 14 days, ticket attached, calendar reminder to convert to birthright or revoke.
- The colleague is in the same job code, location and employment type, and has held no other role.
Outside those, it's a defect to clear, not a provisioning method.
The cost of the policy path
Someone has to define and maintain birthright sets per job code. That's real work: a few hours per job code up front, then drift as roles change. The payoff is that reviews become a diff against a readable policy instead of a comparison between two accounts.
Exceptions
Anything beyond birthright goes through a request with a justification and an expiry. Standing exceptions get reviewed quarterly, and the reviewer sees the justification, not just the grant.
How I'd know it worked
Compare a new joiner's effective access to the birthright set for their job code. The difference should be zero or covered by an approved, unexpired request. If it isn't, that's the defect to clear.
Curated: · Written: · Reviewed:
Q&A-83Managers approved every entitlement in the last campaign. What went wrong?
The short answer: the campaign asked managers to certify things they had no way to evaluate, and made "approve" the cheapest action on the screen. Managers did the rational thing. A 98% approval rate is not evidence of clean access; it is evidence that the review form was unanswerable.
Three defects produced it.
Presentation. Reviewers saw entitlement labels (group names, role names, application codes) not the permissions those entitlements grant. APP-FIN-RO tells a line manager nothing. "Read access to the general ledger, including salary cost centres" tells them whether their team should have it. Without resolved permissions, a last-use date, or a peer comparison (who else in this job family holds it), the only honest answer available is "I don't know", and the form doesn't offer that.
Default. Silence meant keep. Every item the manager skipped, or never reached before the deadline, stayed granted. The path of least resistance and the path of least work were the same path, so the campaign's outcome was decided by its default, not by its reviewers.
Reviewer. Line managers were asked to certify privileged entitlements (domain admin, production database roles, cloud IAM admin) that they cannot see the use of. The resource owner is the only person positioned to say whether that access is still needed. Routing it to the manager produces a signature, not a decision.
What a working campaign changes: show resolved permissions, last-use date and peer comparison; make no response mean removal after a grace period; route privileged entitlements to the resource owner; and sample-test approvals afterwards by asking the approver to justify one specific item.
Old campaign versus redesigned campaign, illustrative figures:
| Measure | Old campaign | New campaign |
|---|---|---|
| Approval rate | 98% | 71% |
| Median seconds per item | 9 | 48 |
| Entitlements removed | 240 | 3,470 |
The arithmetic is the point. 12,000 entitlements at 9 seconds is 108,000 seconds: 30 hours of manager time spread across the population, spent producing 240 removals. At 48 seconds it is 576,000 seconds, 160 hours, producing 3,470 removals. The redesign costs roughly five times the reviewer time and removes roughly fourteen times the access. A campaign optimised for reviewer throughput is optimised for the wrong output.
Failure modes to watch, and how each shows up:
- Approval rate used as the campaign KPI. The metric rewards the failure. Track removals and the justification rate instead; a campaign above 95% approval with near-zero removals is not working, whatever the completion dashboard says.
- "No response means removal" shipped without a grace period and a notification. This is how a recertification takes out a service. The grace period is what makes the default safe, and it needs a named owner who can restore the entitlement.
- Sampling that never happens. Asking a sample of approvers to justify one approval they made is the detection mechanism. If most of them can't, the campaign is rubber-stamping again and the next one will look identical.
A review that costs nothing removes nothing.
Curated: · Written: · Reviewed:
Q&A-64Someone moves from support into engineering. What does your process usually miss?
What a strong candidate says first
The move gets processed as a join. HR fires a transfer event, provisioning adds the engineering role, and nothing subtracts the support role, because deprovisioning is keyed to termination, not transfer. So the miss is the subtraction, and for support→engineering the subtraction is unusually large, because support carries a class of access engineering never needs: customer impersonation, refund and credit tools, the support console, vendor admin seats, ticket-queue routing, and the support on-call rotation.
Two of those aren't in your IAM system at all. Queue membership and rotation membership live in the ticketing and paging tools, so even a correct role-bundle diff misses them.
Why it survives review
Every grant was justified once, when it was made. Access reviews check grants individually, so an inherited impersonation entitlement reads as "approved in 2023" and passes. Nothing fires on use either: the person impersonates a customer to reproduce a bug, which is legitimate engineering work now.
The diff, concretely
| Entitlement | Support birthright | Engineering birthright | Action on move |
|---|---|---|---|
| Customer impersonation (prod) | yes | no | revoke, or time-box 14 days with approval |
| Refund / credit tool | yes | no | revoke day 1 |
| Support console | yes | no | revoke day 1 |
| Ticket queue routing | yes | no | remove from routing (not in IAM) |
| Support on-call rotation | yes | no | remove (not in IAM) |
| Vendor admin seat | yes | no | reclaim seat |
| Repo write + CI | no | yes | grant day 1 |
| Deploy to prod | no | yes | grant after ramp, gated |
Arithmetic: 23 support entitlements, 19 engineering, 7 overlap → 16 to revoke, 12 to grant. A join-only process leaves 35 effective entitlements, 16 of them unjustified, until the next quarterly campaign: up to 90 days.
Trade-offs: how fast to revoke
- Immediate revocation of everything support-side. Safest, and correct for write access, break-glass, and anything that can move money or read customer data at scale. Cost: it breaks the handover. The person is often the only one who knows the escalation path for the first week or two, and you have just cut their access to the tool they would use to explain it.
- Time-boxed overlap. Keep read-only and impersonation-with-approval for a fixed window, 14 days is a common choice, with an expiry date and a named owner; revoke write and break-glass on day 1. Usually the right default for support→engineering, because the overlap is genuinely useful.
- Dual sign-off. The previous manager confirms the removal list separately from the new manager confirming the addition list. It is the control that gives the leave half an owner. Worth it when the support role touches customer data; ceremony when it does not.
When the whole thing is overkill: if the two roles' birthright sets are near-identical (small company, no customer data, no production access) a transfer-specific process is theatre. A role-bundle diff on the HR event is enough.
Failure modes and how you would see them
- Impersonation retained and used legitimately → no anomaly alert fires. Detect by diffing effective access against the new birthright set, not by watching usage.
- Queue membership retained → tickets keep routing to someone who no longer works support. Detect by auditing queue membership against the HR system.
- Rotation membership retained → they get paged for support incidents. Detect by paging-list audit; the symptom is a confused page at 3am.
Verification
Move a test identity from support to engineering, wait out the window, and diff effective access against the new birthright set. Expect zero delta outside the documented overlap. If the diff is non-empty, the leave half is still unowned.
Curated: · Written: · Reviewed:
Q&A-29A user's second factor is a code sent to the phone they are logging in from. Is that two factors?
Short answer: by the taxonomy, yes; by the property that matters, no.
Password is knowledge, the phone is possession, so a policy engine will happily log this as MFA. But the second factor is delivered to, and typed back into, the same device that holds the session. Compromise one and you have the other. Independence is the whole point of MFA, and this pair has none.
Why it collapses. Three paths, all cheap:
- Device malware. An app with SMS-read or notification-listener access sees the code arrive. No network attack required. On Android that is a runtime permission; on iOS it is notification access or SMS autofill on an unlocked device.
- Number port-out. The "something you have" is the phone number, not the handset, and a number is a bearer identifier the carrier will move to anyone who convinces support. The attacker needs the account PIN or date of birth, which is routinely sitting in breach dumps.
- Real-time phishing proxy. A reverse proxy in front of the login page relays the code inside its validity window. The code is a bearer secret; whoever presents it first wins.
A worked port-out (hypothetical, timings illustrative).
- T+0: attacker holds email and password from a credential dump.
- T+2 min: calls the carrier, passes the account PIN from the same dump, requests a port to a SIM they hold.
- T+35 min: port completes; the victim's handset drops to "no service".
- T+36 min: attacker logs in, the OTP lands on the attacker's SIM, session established.
- T+40 min: victim notices the dead phone.
The victim's phone was never compromised. The factor was simply moved.
Which pairs survive what.
| Pair | Survives device compromise | Survives number port-out |
|---|---|---|
| password + SMS to the same phone | no | no |
| password + TOTP app on the same phone | no | yes |
| password + passkey on the same phone | no: key is non-exportable, but an unlocked session can be driven | yes |
| password + hardware key on a separate device | yes | yes |
Note the second row: a TOTP app on the same phone is still one device, so malware that reads the seed or scrapes the screen gets both. It only fixes the port-out half of the problem.
What makes it genuinely two factors. The second factor has to arrive on a channel the first factor's device does not control, or be a credential that cannot be relayed: a hardware key, a passkey with user verification, or a code delivered to a separate device. NIST SP 800-63B classifies PSTN/SMS out-of-band as a restricted authenticator (permitted, but only after a documented risk assessment) which is the standards body saying the same thing.
Detection. Alert on port-out and SIM-change events, on OTP entry from a different ASN than the login, and on repeated OTP requests followed by a success. Expire codes aggressively; a 10-minute window is 10 minutes of relay.
When it is acceptable. Low-value accounts where the alternative is no second factor at all. For anything that can move money or read other people's data, it is not two factors: it is one factor with a delivery receipt.
Curated: · Written: · Reviewed:
Q&A-30Why can a phishing site not relay a WebAuthn assertion?
The 90-second answer. A WebAuthn assertion is a signature over a byte string that contains the browser's origin and a hash of the relying party's ID. The browser will not let a page request a credential scoped to an RP ID that isn't the page's own registrable domain, and the authenticator signs the RP ID hash it was handed. A phishing page is a different origin, so it can only obtain a signature over its origin and its RP ID. The real RP recomputes both and rejects. The attacker can proxy the challenge and the whole ceremony end to end; they cannot make the signature cover example.com.
What is actually signed. The RP verifies signature over authenticatorData || SHA-256(clientDataJSON).
clientDataJSONis built by the browser, not the page:{"type":"webauthn.get","challenge":"...","origin":"https://example.com","crossOrigin":false}. The page cannot setorigin.authenticatorDatastarts withrpIdHash= SHA-256 of the RP ID, then a flags byte (UP 0x01, UV 0x04, BE 0x08, BS 0x10, AT 0x40, ED 0x80), then a 4-byte sign counter.
So the origin and the RP ID sit inside the signed bytes. A relay proxy terminates TLS on https://examp1e.com; the browser reports that origin, and the only RP ID it will accept is examp1e.com or a suffix of it. The authenticator signs SHA-256("examp1e.com").
The check that fails. Node 20, no dependencies:
import { createHash } from 'node:crypto';
const sha256 = (s) => createHash('sha256').update(s).digest();
const clientDataJSON = '{"type":"webauthn.get","challenge":"...","origin":"https://examp1e.com","crossOrigin":false}';
const fromAuthenticator = sha256('examp1e.com'); // what the relayed assertion carries
const expected = sha256('example.com'); // the RP's own RP ID
console.log(fromAuthenticator.equals(expected)); // false
console.log(JSON.parse(clientDataJSON).origin === 'https://example.com'); // false
Trace: the two SHA-256 inputs differ, so the 32-byte digests differ (collision resistance), and Buffer.equals returns false. The second line parses the relayed clientDataJSON, whose origin is https://examp1e.com, and strict string comparison returns false. Both checks fail before the signature is verified at all.
What a real-time proxy still gets. Evilginx-class kits proxy the ceremony and forward the challenge, so the challenge matches: the challenge is not the defence. What survives:
| Step | Relay proxy | Why |
|---|---|---|
| Password | Yes | Typed into the proxy |
| TOTP / SMS code | Yes | Typed into the proxy |
| WebAuthn assertion | No | Signed over attacker origin and attacker RP ID |
| Session cookie after password+OTP | Yes | Proxy holds the real session |
Where it does break. The binding is only as tight as the RP ID.
- RP ID set to a single hostname (
login.example.com): moving sign-in toauth.example.cominvalidates every credential registered under the old RP ID. Users re-enrol or fall back to a weaker factor. - RP ID set to the registrable domain (
example.com): any subdomain takeover, XSS on a marketing subdomain, or dangling DNS record lets an attacker register and use credentials for the whole domain.rpIdHashmatches; only theorigincheck stops it, so an RP that verifiesrpIdHashand skipsoriginis relayable fromevil.example.com. - Same-device fraud: an attacker holding the unlocked device with the real site open gets a valid assertion. WebAuthn proves possession of the key, not the human.
- Synced passkeys versus device-bound keys changes portability, not origin binding.
Detection: log clientDataJSON.origin and the computed rpIdHash on every failed verification. A spike of mismatched origins from one ASN is a relay attempt, not a client bug.
Curated: · Written: · Reviewed:
Q&A-31Your users have a code app. Why is a modern phishing page still a problem?
The 90-second answer
A code app doesn't stop phishing because the code is a bearer credential the user can read out loud. A phishing page that proxies the real login can ask for it and forward it inside the validity window. Nothing in a six-digit code binds it to the site the user thinks they're on.
Assumptions: TOTP per RFC 6238, 30-second step, 6 digits, no device binding, and a session cookie issued on success.
Mechanism. TOTP is HMAC-SHA1(shared_secret, floor(unix_time / 30)) truncated to 6 digits. The server recomputes it from the same secret and accepts a match, usually with ±1 step of drift tolerance, so a code is often live for 60–90 seconds in practice, not 30. That check answers one question: does this person hold the secret? It never answers is this the site they intended to authenticate to? A relay proxy sits in the middle, so the user's browser really is talking to the proxy, the proxy really is talking to the real site, and the code is valid at both ends.
The deeper problem is that the session cookie is also a bearer token. Once the proxy has the code it gets a session; once it has the session, MFA is finished for that session's lifetime.
Timing on a representative relay (worked example, not a specific vendor's numbers):
| Step | Seconds elapsed | Attacker holds |
|---|---|---|
| user enters the code | 0 | the code |
| code replayed to the real site | 9 | a session |
| cookie exported to another host | 11 | full access |
| code would have expired | 30 | already irrelevant |
The 30-second lifetime never matters, because the attacker stops needing the code at step 2. The cookie is what carries access from there.
What actually helps. WebAuthn/passkeys: the credential is bound to the RP ID, and the signature covers the challenge and the origin, so a proxy on a different origin cannot obtain a usable assertion. Device binding, mTLS client certificates or a device-bound key, raises the cost of a pure relay. Where codes must stay, cap concurrent verification attempts per account, alert when a code is used from a new ASN, and shorten session lifetime for privileged roles.
Failure modes and detection. Relay proxies (Evilginx-style) are the primary one; session-cookie theft is the second, and it defeats any second factor. The signal to alert on is a login that succeeds from two different networks within seconds, or a session cookie presented from an ASN the user has never used.
Verification. Run a controlled relay against a test account and check whether the code and the resulting session are usable from another network. If they are, the code app is not a phishing control for that population.
Curated: · Written: · Reviewed:
Q&A-33Does the fingerprint used on a passkey sign-in travel to your server?
No. The fingerprint is matched inside the device's secure element (Touch ID, Windows Hello, Android's biometric stack) and the only thing about that check that reaches your server is a single bit.
When a platform authenticator creates a passkey it generates a key pair in hardware and keeps the private key there. The fingerprint template is enrolled in the same secure element and never leaves it; the OS matches it locally to authorize a signature. On sign-in, navigator.credentials.get() returns an assertion: authenticatorData, clientDataJSON, signature, and the credential ID. The sole trace of the biometric is bit 2 (UV, user verified) of the flags byte inside authenticatorData.
| Field | Where it lives | Size |
|---|---|---|
| rpIdHash | authenticatorData[0:32] | 32 bytes |
| flags (UP, UV, BE, BS) | authenticatorData[32] | 1 byte |
| signCount | authenticatorData[33:37] | 4 bytes |
| clientDataJSON | response | ~100–200 bytes (challenge, origin, type) |
| signature | response | 70–72 bytes for ES256 (DER, variable) |
| credential ID | response.id | 16–32 typical, spec caps at 1023 |
| fingerprint template | nowhere | 0 bytes |
The UV bit is only meaningful because the signature covers authenticatorData || SHA-256(clientDataJSON). Verify the signature against the stored public key first, then read the flag: checking the flag without verifying the signature lets anyone flip it.
// Node 20, no deps. Server-side, after the client JSON-serialized the assertion.
const authData = Buffer.from(credential.response.authenticatorData, 'base64url');
const rpIdHash = authData.subarray(0, 32);
const flags = authData[32];
const UP = (flags & 0x01) !== 0; // user present
const UV = (flags & 0x04) !== 0; // user verified
const BE = (flags & 0x08) !== 0; // backup eligible
const BS = (flags & 0x10) !== 0; // backup state
// A 37-byte buffer here is the whole story: no biometric payload exists to parse.
Three decisions the UV bit forces, and how to pick:
userVerification: "required"vs"preferred". Required makes the authenticator demand a biometric or PIN, so you get UV=1 and can treat the passkey as two factors (possession plus inherence/knowledge). Preferred lets the authenticator skip it: a security key with no PIN set returns UV=0 and the passkey is single-factor. Require UV when the passkey is the only factor; prefer it when it sits behind a password, since requiring it locks out authenticators that have no local check configured.- Attestation
nonevsdirect. Direct returns an attestation statement and the AAGUID, letting you allowlist device models or demand hardware-backed keys. The cost is privacy, attestation fingerprints the device, and Apple and Google anonymize it by default, so AAGUID allowlisting is coarse. Choose direct in a regulated environment that must prove a hardware authenticator; choose none in a consumer app optimizing for passkey adoption. - Synced vs device-bound. BE/BS tell you whether the credential is backed up (iCloud Keychain, Google Password Manager) or bound to one device. Synced survives device loss; device-bound can't be exfiltrated through a cloud account. Pick by whether recovery or blast radius matters more.
Failure modes worth naming:
- Parsing UV out of
clientDataJSON. It isn't there; it's inauthenticatorData. - Treating UV=1 as proof of a fingerprint. It could be a PIN or a face. If your compliance claim says "biometric", the bit does not support it.
- Using
signCountas a clone detector. Synced passkeys commonly report 0 and never increment. - Skipping the
rpIdHashcomparison, which accepts an assertion minted for another origin.
To confirm this on your own stack: log the base64url of authenticatorData from one assertion and decode it. You get 37 bytes, flags at index 32, and no biometric payload anywhere in the message.
Curated: · Written: · Reviewed:
Q&A-35The signature counter in an assertion went backwards. What does that tell you?
The engineering content of WebAuthn sign-counter anomalies is the revocation path and its measured latency, not the protocol name.
The counter is a cloning signal rather than an authentication decision. A value at or below the stored one is consistent with a duplicated authenticator, but many authenticators, including most synced passkeys, report zero permanently, so the check has to be applied per credential.
Concretely, record whether a credential has ever reported a non-zero counter, enforce monotonic increase only for those, treat a regression as a security event that suspends the credential and forces re-registration, and leave always-zero credentials to other signals.
The reason for that specificity is a failure I have seen: A regression check applied to every credential locked out 640 users on synced passkeys in one morning because their counters were permanently zero. The check was then switched off entirely, including for the 1,200 hardware keys where it had been working.
Counter behaviour by credential class.
| Credential class | Fleet size | Counter behaviour | Regression enforced |
|---|---|---|---|
| hardware key | 1,200 | increments | yes |
| synced passkey | 640 | always zero | no |
| platform key with a counter | 380 | increments | yes |
I would not consider it settled without evidence: replay an assertion with a stale counter for a counter-reporting credential and require that credential to be suspended.
Enforce the counter where the counter means something.
Curated: · Written: · Reviewed:
Q&A-34What do you verify when a user registers a new passkey?
Registration is the only moment a credential becomes trusted, so every later assertion inherits whatever registration accepted. I verify five things server-side, in order, and reject on the first failure.
1. The challenge is mine, unused, unexpired. I generate 32 random bytes, store them against the session with a 5-minute TTL, and delete on first use. The spec's floor is 16 bytes; the timeout I send to the client is 300000 ms, the value WebAuthn Level 2 suggests. Compare against clientDataJSON.challenge after base64url-decoding.
2. The ceremony and origin. clientDataJSON.type must be exactly webauthn.create: an assertion response replayed here fails. clientDataJSON.origin must equal the expected origin exactly (scheme, host, port), not a suffix match.
3. The RP ID. authenticatorData.rpIdHash must equal SHA-256 of the RP ID, and the RP ID must be the origin's registrable domain or a suffix of it. This is what stops a credential minted for evil.example being registered against example.com.
4. The flags. UP (bit 0) must be set. UV (bit 2) must be set if my policy claims user verification. AT (bit 6) tells me attested credential data follows, so I parse the AAGUID, credential ID and COSE public key from it. BE (bit 3) and BS (bit 4) tell me whether the credential is backup-eligible and currently backed up: I record both, because they change what counter checking means.
5. The credential ID is new. No existing row for this RP with that credential ID. I also pass excludeCredentials at creation time so the authenticator itself refuses to re-enrol.
Then: the COSE algorithm must be in my allowlist. -7 (ES256) is what nearly every authenticator returns, -257 (RS256) and -8 (EdDSA) are the other realistic ones. The key must parse, and I store it with the algorithm and signCount. Finally, registration runs inside an authenticated session or behind a step-up, never as an unauthenticated endpoint.
| Check | Stops |
|---|---|
| challenge issued, unused, unexpired | replayed registration |
| type == webauthn.create | assertion replayed as registration |
| origin exact match | cross-site enrolment |
| rpIdHash == SHA-256(rpId) | credential minted for another RP |
| credential ID unseen | credential grafting onto an existing account |
| UV flag per policy | silent enrolment without user verification |
Where I'd relax or tighten. Attestation conveyance defaults to none, and I keep it there for consumer sign-up: direct only helps if I verify the statement against FIDO MDS and allowlist AAGUIDs, and it breaks outright on privacy-preserving authenticators that return an all-zero AAGUID and a none statement. I'd only pay that friction in an enterprise deployment with managed devices. I allow multiple passkeys per user, capping at one turns a lost phone into a support ticket, but cap the count (say 10) and require an existing session to add one. userVerification: "required" is right for high-value accounts and wrong for a consumer sign-up where a security key with no PIN would be rejected; "preferred" is the default and usually the right call. Requiring a resident (discoverable) credential buys usernameless login but costs authenticator storage, a security key holds roughly 25, so I set residentKey: "preferred" unless usernameless login is a product requirement. And I do not hard-fail on signCount: synced passkeys report 0 forever, so a strict counter check locks out legitimate users.
Worked example (hypothetical). A registration endpoint that skipped check 1: an attacker captures a victim's registration payload, replays it 3 times against 3 accounts they control, and each replay binds the attacker's authenticator to a new account. Every later assertion for those accounts verifies correctly, because registration already blessed the credential. The fix is one line: the challenge row is deleted on first use, so replays 2 and 3 fail.
How I'd know it works. Replay a captured registration payload verbatim and require a 4xx on the challenge check; then re-run with a mutated origin and require the same. That's the test that proves the checks are load-bearing rather than decorative.
Curated: · Written: · Reviewed:
Q&A-32You cannot roll out security keys everywhere at once. Who goes first?
The judgement in phishing-resistant MFA for administrators is which identifier is durable and which is only a convenient claim.
Sequence by blast radius rather than by headcount. The accounts that can change identity itself: directory administrators, federation administrators, privileged-access operators, and break-glass holders: are the ones whose compromise makes every other control negotiable.
Concretely, enumerate the roles that can alter authentication policy, group membership, or federation trust, require hardware-backed origin-bound authenticators for those roles with no code fallback, and enforce it through a conditional policy that names the role rather than the person.
The reason for that specificity is a failure I have seen: A rollout ordered by department reached the directory administrators in month 5. In month 3 an administrator approved a fatigue prompt, and the attacker added a federation trust that survived the eventual key rollout by a further 2 weeks.
Rollout order by what the role can change.
| Population | Can alter federation trust | Wave and size |
|---|---|---|
| directory and provider administrators | yes | wave 1, 42 people |
| privileged-access operators | partly | wave 2, 130 people |
| general staff | no | wave 3, 6,400 people |
I would not consider it settled without evidence: attempt an administrative sign-in with a delivered code and require the policy to refuse before the rollout is called complete.
Start where a compromise rewrites the rules.
Curated: · Written: · Reviewed:
Q&A-36Should you accept synced passkeys for administrative access?
Before calling synced passkeys versus device-bound keys done I would write down the grant that the source-of-truth change does not touch.
A synced passkey is phishing-resistant, but its private key exists wherever the vendor account is signed in, so the credential's security becomes the security of that consumer account and its recovery path. A device-bound key keeps the private key inside one piece of hardware you can inventory.
Concretely, allow synced passkeys broadly to win the phishing resistance, and require device-bound attested keys for administrative roles, checking the authenticator model against an allowed list and recording the backup-eligible and backup-state flags at registration.
The reason for that specificity is a failure I have seen: An administrator's personal cloud account was recovered by an attacker through a support call. The synced passkey followed that account onto a new device, and the attacker held phishing-resistant access for 5 days without ever touching a corporate machine.
Administrator credential inventory.
| Credential type | Count | Allowed for the role |
|---|---|---|
| device-bound and attested | 38 | yes |
| synced passkey | 11 | no, migrate |
| delivered-code fallback | 2 | no, remove |
I would not consider it settled without evidence: read the backup-eligible flag on every registered administrator credential and require the count of syncable ones to be zero.
Phishing-resistant is not the same as bound to hardware you control.
Curated: · Written: · Reviewed:
Q&A-37An attacker holding a stolen session enrols their own passkey. What stops that?
The 90-second answer
Assume two things: the account already has at least one registered passkey, and the attacker has the session cookie, not the private key. Under those assumptions the fix is narrow: the registration endpoint must demand a fresh assertion from an existing credential, and a session cookie must never be sufficient to add one.
A session cookie is a bearer token. Anyone who copies it is you as far as the server is concerned. A passkey private key is not a bearer token: it never leaves the authenticator, and the signature it produces is bound to the RP ID and the challenge. So the only thing that separates the owner from the cookie thief at enrolment time is whether they can produce an assertion. Make that the gate.
What that looks like
- Store the time and method of the last strong authentication in the session record, not just "logged in".
/webauthn/register/beginrefuses to issuePublicKeyCredentialCreationOptionsunless that timestamp is inside a short window (5 minutes is a reasonable default) and the method was a passkey assertion.- If it is stale, return a step-up challenge:
navigator.credentials.get()withuserVerification: "required"against the existing credential. Only after the server verifies that assertion does it refresh the timestamp and issue the registration options. Rotate the session ID at that point. - Put the same gate on credential deletion, recovery-email change, and recovery-code regeneration. Otherwise the attacker deletes the owner's key instead of adding one.
- Refuse to remove the last credential, or require a second credential or recovery code first.
- Notify every previously verified channel, including the address on file before the session started, and let the owner revoke the new credential from that notification.
STEP_UP = timedelta(minutes=5)
def register_begin(session, user):
auth = session.get("strong_auth") # {"method": "passkey", "at": ts}
if not auth or auth["method"] != "passkey" or now() - auth["at"] > STEP_UP:
return 401, {"error": "step_up_required"}
return 200, build_creation_options(user, exclude=user.credential_ids)
excludeCredentials stops the same authenticator being enrolled twice. It does nothing about a second authenticator, which is the attack.
Trace of the attack against this
- t+0: attacker replays
sid=abc. Session valid,strong_auth.at = t-6h. POST /webauthn/register/begin→ 401step_up_required. No challenge issued, nothing to sign.- Attacker cannot answer the step-up: the private key is in the owner's authenticator, and the challenge is bound to the RP ID, so a phishing page cannot relay it either.
DELETE /credentials/{id}→ same 401.POST /account/email→ same 401.
The attacker's only remaining move is to wait for the owner to authenticate, which is exactly when the notification fires.
| Gate | Stops | Does not stop |
|---|---|---|
Step-up assertion before register/begin | Cookie replay adding a credential | Attacker who already holds a credential |
| Same gate on delete / email change | Lockout by removing the owner's key | — |
| Last-credential removal block | Permanent lockout | Attacker adding a second key |
| Notify prior channels + revoke link | Silent persistence | Owner who ignores the mail |
Where it breaks
- No existing passkey. Step-up falls back to the password, which the attacker may have. Then the gate has to be an OTP to a channel that cannot be changed in the same session, plus a delay before the new credential can be used.
- The session was created by the attacker's own login, not stolen. Step-up passes. That is credential compromise, not session theft, and needs a different control.
- Notification address changed in the same session. Send to all verified channels and the previous address, and gate the change itself behind step-up.
- Sign counter as a theft signal. Synced passkeys report a zero counter, so it tells you nothing.
- Attestation or AAGUID allowlists. Virtual authenticators make these trivially forgeable; do not lean on them for consumer accounts.
The cooling period is detection, not prevention: a patient attacker outlasts it. The step-up is what actually stops them.
Curated: · Written: · Reviewed:
Q&A-38How do you store account recovery codes?
What I'd say first
Store recovery codes the way you store passwords: generate each with at least 128 bits of entropy, show the set once, and keep only a memory-hard hash. No reversible copy anywhere, not in the database, not in a support tool, not in a log.
Assumptions: the codes are machine-generated, not user-chosen; the account already has a password and a second factor; support is not allowed to read a code back to the user.
Mechanism
Generate 10 codes per set, each 128 bits from a CSPRNG, rendered as base32 in groups of four so a human can transcribe them. Hash each with Argon2id at OWASP's minimum parameters (m=19 MiB, t=2, p=1) and store the hash plus a per-code salt. Display the plaintext once, at issue time, and never again.
Consume atomically:
UPDATE recovery_codes
SET used_at = now()
WHERE account_id = $1
AND code_hash = $2
AND used_at IS NULL
RETURNING id;
Zero rows means the code was wrong or already spent. Doing the check and the mark in one statement is what stops two concurrent presentations of the same code from both minting a session.
On reissue, delete the old set in the same transaction that inserts the new one, and show the remaining count in the account view so a user notices a code they did not spend.
Why memory-hard, and where it actually matters
With 128 bits of entropy, brute force is not the weak link: 2^128 guesses is out of reach whatever the hash. The weak link is reversibility and reuse. A 32-bit code, by contrast, is 4.3 billion candidates; at roughly 10^10 SHA-256 hashes per second on one GPU that is under a second offline. So the entropy does the heavy lifting, and the memory-hard hash is defence in depth: it costs little, and it means a leaked table is not a fast oracle if anyone ever shortens the codes or lets a user pick one.
The failure I'd design against
I have seen codes stored reversibly so support could read them back. A support-tool export covering 15,000 accounts contained every unused code, and two accounts were entered with codes generated 17 months earlier. Both problems share one root cause: the code outlived its set and was readable.
What I'd check before calling it done
Read the recovery-code column directly and confirm nothing usable can be recovered from it. Grep the logs and the support tool for the code format. Confirm the consume path is a single conditional update, and that reissue invalidates the prior set.
A code you can read back to a user is a second factor you have copied.
Curated: · Written: · Reviewed:
Q&A-43Why can you not store code-app seeds the way you store passwords?
I would settle TOTP seed encryption versus password hashing by attempting the action after the identity event that was supposed to stop it.
Verifying a time-based code requires the server to recompute it, so the seed has to be recoverable and hashing is not available. That makes the seed store a key-management problem rather than a hashing problem.
Concretely, encrypt seeds under a key held in a hardware module or a managed key service, keep that key out of the database and out of its backups, perform verification inside a service that returns only a yes or a no, and rotate the data key with a re-encryption job rather than by re-enrolling users.
The reason for that specificity is a failure I have seen: Seeds sat in plaintext in a table replicated to a reporting warehouse. A read-only analytics credential exposed 26,000 of them, and re-enrolment took 5 weeks during which the second factor was known to whoever held the export.
Seed store design.
| Property | Password hash | Code-app seed |
|---|---|---|
| recoverable by the server | no | required |
| storage | memory-hard hash | envelope encryption |
| impact of a table export | 0 codes usable | 26,000 accounts |
I would not consider it settled without evidence: read the seed column with a database credential and confirm the value cannot be used to generate a valid code.
A secret you must recompute with is a key-management problem.
Curated: · Written: · Reviewed:
Q&A-44How wide should the code acceptance window be, and what else does it need?
The judgement in TOTP clock-window and replay is which identifier is durable and which is only a convenient claim.
A wider window buys tolerance for clock drift and buys the attacker time, so the window and the replay defence are one decision. One step either side of the current interval is usually enough, and a code already used must never be accepted again inside its window.
Concretely, accept one step of skew, record the last accepted step per user and refuse anything at or below it, track per-user drift so a consistently early device is corrected rather than accommodated by widening, and rate-limit verification per account and per source.
The reason for that specificity is a failure I have seen: A window of five steps was configured to stop drift complaints. The acceptance span became 330 seconds, one relayed code was reused successfully 3 times, and every guess had 11 valid targets instead of 3.
Window width against exposure.
| Window | Acceptance span | Valid intervals per guess |
|---|---|---|
| one step either side | 90 seconds | 3 |
| five steps either side | 330 seconds | 11 |
| one step with a replay record | 90 seconds | 1 |
I would not consider it settled without evidence: present the same code twice inside the window and require the second attempt to fail.
The window is an attacker's budget as much as a user's.
Curated: · Written: · Reviewed:
Q&A-45Users are approving prompts they did not start. What do you change?
Where candidates lose the interview on MFA push fatigue is reading a green IdP dashboard as revocation.
A prompt that can be approved with one tap and no context turns the second factor into a doorbell the attacker can ring. The fix removes the blind approval rather than asking users to be careful at two in the morning.
Concretely, replace plain approvals with number matching, show the application, location, and requesting device, cap prompts per account per interval, lock after repeated denials, and give a one-tap route to report a prompt the user did not start.
The reason for that specificity is a failure I have seen: An attacker holding a valid password sent 63 prompts between 01:40 and 02:20. The 41st was approved, and mailbox rules were changed 4 minutes later.
Prompt behaviour after the change.
| Control | Before | After |
|---|---|---|
| prompts allowed per hour | unlimited, 63 observed | 3 |
| number matching | no | yes |
| account locked after denials | never | after 2 |
I would not consider it settled without evidence: drive repeated prompts at a test account and require throttling and a report path before the tenth arrives.
Remove the blind tap.
Curated: · Written: · Reviewed:
Q&A-46How do you throttle second-factor attempts without handing an attacker a lockout tool?
The 60-second answer
Never lock the account on second-factor failures. Anyone who knows a username can submit wrong codes, so a per-account lockout is a denial-of-service tool pointed at your own users. Throttle on three axes with different responses: the account axis gets growing delay (the attacker pays it too), the source axis gets a short hard block, and the code-value axis gets a short block plus an alert. Behind all three, alarm on the global failure rate: the attacker's real cost is the ~231,000 attempts needed to land one 6-digit code, and that volume is visible.
Why the account axis must not lock
A 6-digit TOTP with a ±1 step window accepts 3 values at any instant out of 10^6, so a random guess succeeds with probability 3×10^-6. Reaching a 50% chance of one success takes ln2 / 3e-6 ≈ 231,000 attempts. That number decides the design: the attacker cannot win by guessing, only by volume, so the job is to make volume expensive and visible, not to lock accounts.
Per-account: after n consecutive failures in a 15-minute sliding window, delay the response by min(2^n, 30) seconds, enforced server-side (sleep, or 429 with Retry-After). A legitimate user waits at most 30 seconds; the attacker gets roughly two attempts per minute per account. No lockout, so no lockout tool.
Per-source: 20 attempts per 10 minutes per IP, 200 per hour per /24. This axis can hard-block, because a source is not a victim, but keep the window short, since corporate egress and carrier CGNAT put thousands of legitimate users behind one address.
Per code value: if the same submitted value fails against 6 distinct accounts within an hour, block that value for an hour. That is the spray signature, one guess fanned across many accounts, and blocking it costs a legitimate user at most one 30-second step, because their next code is different.
| Axis | Limit | Window | Response |
|---|---|---|---|
| per account | delay min(2^n, 30) s after n failures | 15 min sliding | delay only, never lock |
| per source IP | 20 attempts | 10 min | hard block, short |
| per source /24 | 200 attempts | 1 hour | hard block |
| per code value | 6 distinct accounts | 1 hour | block the value |
Trace: password-valid attacker sprays one code
Attacker holds a valid password and sends code 481920 to 10,000 accounts, one attempt each, rotating 50 source IPs.
- Attempts 1–6 fail on six distinct accounts → per-value counter hits 6 → 481920 blocked for an hour.
- Each IP trips the 20/10min limit after 20 attempts → 50 × 20 = 1,000 attempts in the first 10 minutes, then the sources are blocked.
- No per-account counter fires. Nobody is locked out.
- Global: 1,000 failures across 1,000 distinct accounts in 10 minutes → alarm, and each affected account owner gets a "someone tried to sign in" notice.
A value-rotating attacker defeats the per-value counter: a fresh random code per account never trips it. That is why the per-source limit and the global alarm are the backstop, not the value counter.
Failure modes and how you see them
- CGNAT false positive: a carrier NAT trips the per-source block and thousands of users fail at once. Detect by block rate per ASN with many distinct accounts behind it; mitigate with the short window and by never letting a source block touch the account counter.
- Clock skew: one account fails repeatedly with a correct password from a device that worked yesterday. That is a skewed TOTP clock, not an attack. Widen to ±1 step and tell the user to check the device clock; do not let the delay stack into an apparent lockout.
- Recovery-path bypass: backup codes, email links and "resend SMS" are second-factor verification too. If they do not share the same counters, the attacker pivots to them.
- Push fatigue: with approve/deny push there is no code to guess, and the attack is 50 pushes until someone taps approve. Number matching (type the number shown) plus a cap of ~3 pushes per 10 minutes per account, then require the password again.
When not to apply this
WebAuthn and passkey assertions are not guessable, so OTP throttles there add only lockout risk. Skip them. SMS OTP uses the same 6-digit math but adds delivery retries and SIM swap, so treat the phone number as a weaker factor rather than tuning the counters harder.
Curated: · Written: · Reviewed:
Q&A-40Your risk engine rates this sign-in low risk. Can it skip the second factor?
Short answer: No, not if the session can reach anything you'd protect with a second factor. A low risk score may choose which strong factor is demanded; it must not remove the requirement. Risk is a routing signal, not a control.
Assumptions that decide it: what the token grants (read-only profile vs. admin console, payment, or the ability to enroll a new factor), the assurance level you've committed to, and whether account recovery runs through a weaker channel than the sign-in itself.
The alternative, and why it loses here. Consumer IdPs (Google, Microsoft Entra ID, Okta) do run risk-based step-up with no baseline: a known device on a known network with a good score gets password-only. That is a defensible design when the account's blast radius is a mailbox and the recovery path is the same strength as the sign-in. It stops being defensible when the session can change recovery, enroll a factor, or reach systems where one compromised credential is lateral movement. The deciding criteria are blast radius and assurance level, not the score's confidence. Under NIST SP 800-63B, AAL2 requires two authentication factors; risk signals can select among authenticators at that level, they cannot drop you to AAL1. If your policy says AAL2, a low score cannot waive it.
Why the score can't be the control. A risk score is a prior over what the platform observes: network, device fingerprint, geography, timing, velocity. Every one of those signals is observable to an attacker who has the endpoint, and most are forgeable: a residential proxy fixes the ASN, a stolen session cookie fixes the device fingerprint. The score is evidence about the channel, not about who is holding it. That is why it belongs in the routing layer.
Policy shape: floor plus escalations.
| Signal | Old outcome | New outcome |
|---|---|---|
| office address range | factor waived | key required |
| new device, 2 countries in 1 hour | factor required | key plus challenge |
| known device, low score | factor waived | key required |
| low score, request to enroll new factor | allowed | blocked, re-auth required |
Any request to drop below the floor needs a named exception with an owner and an expiry, not a config flag someone sets once.
Failure mode. An illustrative incident: a trusted-network rule waived the second factor across the office range. A compromised printer on that range, combined with dumped passwords, reached 7 internal applications over 11 days. Nothing alerted, because every sign-in was ordinary by the engine's own signals: the attacker was inside the trusted range on a trusted device. Detection for this class is not impossible-travel; it is a factor-usage ratio per network range (what fraction of sign-ins from this range present a second factor?) plus a step-up requirement on first access to sensitive applications regardless of score.
How I'd verify it. Authenticate from the location the engine rates lowest risk and confirm a strong factor is still required. Then try to enroll a new factor from that same low-risk session and confirm it is refused. If either passes without a factor, the floor is not a floor.
Risk can add friction. It cannot remove the floor.
Curated: · Written: · Reviewed:
Q&A-42Multi-factor authentication succeeded and the account was still taken over. How?
The 90-second answer
MFA authenticates a login event, not the session that event creates. The session artifact (SSO cookie, access token, refresh token) is a bearer credential: possession is authentication. Takeover after a successful MFA prompt happens in four places.
1. The artifact is stolen after the factor. An adversary-in-the-middle proxy on a lookalike domain relays the password and the OTP to the real IdP, then copies the SSO cookie out of the response before forwarding it to the victim. An infostealer gets the same cookie offline by reading the browser profile.
2. The IdP session cookie is replayed. That cookie is the master key. Replay it and the IdP issues fresh access and refresh tokens for every app in the tenant without another prompt.
3. The factor is relayed or fatigued. Real-time OTP relay defeats TOTP. Push bombing defeats approve/deny; number matching, default in Microsoft Authenticator since 2023, kills push bombing but not relay.
4. MFA is reset out of band. The helpdesk is the factor. In September 2023 Scattered Spider vished MGM's helpdesk into resetting MFA (CISA advisory AA23-320A); MGM's 8-K put the Q3 2023 EBITDA impact near $100M.
What actually stops it, and what each control costs
| Control | What it stops | Where it breaks |
|---|---|---|
| Short access-token lifetime (15 min) + refresh rotation with reuse detection | shrinks the replay window to minutes; a replayed refresh token revokes the whole family | parallel tabs and mobile clients race on rotation and cause false revocations; long-running jobs need re-auth |
| Sender-constrained tokens: DPoP (RFC 9449), mTLS-bound (RFC 8705) | replay of a stolen token from a different client | needs client support; browsers without DPoP-aware JS, legacy thick clients, and third-party SaaS you don't control can't do it |
| Device binding (TPM-backed key; token protection in Entra ID is preview) | cookie export to another machine | Windows/macOS only; breaks on VDI, shared workstations, and any app that can't run the ceremony |
| Continuous re-evaluation (CAE) / step-up on risk | a session continuing after a risk signal fires | needs IdP-app integration; adds a prompt; apps that ignore the signal keep the session |
| Phishing-resistant MFA (WebAuthn) | OTP relay and AiTM on a lookalike domain | does nothing for a session stolen after the ceremony; a same-origin proxy (compromised router, malicious extension) still relays |
Deciding criteria: if you control the client, sender-constrain the token. If you don't, short lifetimes plus rotation plus CAE. Device binding only where the fleet is managed and the app is first-party.
The flow, with timings
T+0:00 victim opens login.contoso-sso[.]com (lookalike); proxy fetches the real IdP page
T+0:20 victim submits password + TOTP; proxy relays both to the real IdP
T+0:21 real IdP sets ESTSAUTH and returns it to the proxy
T+0:21 proxy copies the cookie, forwards it to the victim — victim sees a normal login
T+0:45 attacker replays ESTSAUTH from a different ASN; IdP sees a valid session
T+0:46 IdP issues access + refresh tokens for every app in the tenant, no prompt
T+6:00 attacker registers a new MFA method and a mailbox rule
WebAuthn breaks this at T+0:20, because the RP ID is the lookalike domain and the ceremony fails. That is the whole reason to prefer it over TOTP.
The arithmetic that decides your exposure
With a 60-minute access token and a 90-day sliding refresh token, the attacker's window is 90 days. Add rotation with reuse detection and the first replay invalidates the token family, so the window collapses to the gap between theft and the victim's next legitimate refresh: often minutes, and the victim gets logged out, which is your detection.
How you know it works
Replay a captured session cookie from a different ASN and require the application to end the session rather than accept it. Watch for the same cookie used from two IPs concurrently, token issuance from a new ASN, and a new MFA method registered within hours of a login. Those three signals catch the flow above before the mailbox rule lands.
Curated: · Written: · Reviewed:
Q&A-41How do you decide which operations get a transaction-level confirmation?
The 90-second answer. Confirmation is a claim about effective access, so I scope it to the action, not the session. Three triggers put an operation in scope: it is irreversible, it moves value, or it changes who can authenticate. Everything else rides on the session.
The second half of the rule matters as much: the challenge must display the parameters being authorised, and the authorisation must be bound to them. A prompt that names nothing authorises anything: the user approves whatever the attacker asked for.
Assumptions. A human user is present, the session is already authenticated, and the confirmation is the last gate before the effect. If any of those is false, this is the wrong control: see below.
Alternatives, and when each wins.
| Control | Binds to | Cost | Fails when |
|---|---|---|---|
| Per-session step-up (re-auth at login or idle timeout) | the session | one extra prompt per session | the session is stolen or ridden; the attacker reuses the live session and never sees a prompt |
| Transaction-level confirmation (this answer) | the specific parameters | one prompt per in-scope action | volume is high, or the user is not the right approver |
| Out-of-band confirmation (push to a registered device, separate channel) | the action, on a channel the browser session cannot touch | device dependency, added latency, push fatigue | the user approves reflexively; the attacker can spam prompts |
| Delay window (hold N minutes, notify, allow cancel) | time | breaks time-sensitive operations | the user is asleep, or the operation is a market order |
Combining them is normal: out-of-band confirmation is transaction-level confirmation delivered on a second channel, and a delay window on top covers the case where the user approves under duress.
When transaction-level confirmation is the wrong answer. High-volume low-value operations: confirming each of 10,000 payouts a minute is a denial of service on the user. Read-only operations, where the session is enough. Machine-to-machine calls with no human present: there is nobody to confirm, so use a scoped, short-lived credential instead. And maker-checker flows, where the approver must be a different person from the initiator: that is a four-eyes control, not a confirmation.
Register. Keep it as data, not code, so it can be reviewed and diffed.
| Operation | Assurance | Parameters shown |
|---|---|---|
| Add a payee | key + confirmation | payee name, last 3 digits of account |
| Grant admin role | key + approval | role, scope, 90-day expiry |
| Read a report | session only | none |
Failure mode and detection. The failure I have seen is a challenge with no parameters: a bare approve button. In one incident, a transfer was confirmed by a user who believed they were approving a sign-in; 2 of the 4 payments in that hour were fraudulent before the prompt text was rewritten. (Illustrative figures.)
Detection: (1) a parameter-binding test. Issue a challenge, mutate one field before execution, require rejection; (2) challenge rate per user, to catch push fatigue; (3) approval-latency distribution. Sub-second approvals mean the user is not reading.
Trace of the binding check:
request = {payee:"ACME-4471", amount:1200.00, ccy:"GBP"}
H = sha256(canonical_json(request)) # shown in challenge, carried in token
# attacker mutates amount after approval
request' = {payee:"ACME-4471", amount:12000.00, ccy:"GBP"}
sha256(canonical_json(request')) != H -> reject
That is the whole decision: irreversible, value-moving, or identity-changing operations get a challenge that names its parameters and is bound to them; everything else gets the session.
Curated: · Written: · Reviewed:
Q&A-48A user cannot use the standard authenticator. What do you offer them?
First I'd pin down why the standard authenticator doesn't work, because the fix differs: no smartphone, a phone the employer won't let on the corporate network, a visual impairment that makes reading a rotating 6-digit code unreliable, or a motor impairment that makes tapping a push prompt hard. The principle doesn't change with the reason: issue a different strong factor, never share one.
What I offer, roughly in order of preference:
| Constraint | What I issue | Phishing-resistant? |
|---|---|---|
| No smartphone | USB/NFC security key (FIDO2/WebAuthn) | Yes |
| Can't read a code | Platform authenticator (Windows Hello, Touch ID) or a key with a tactile marker | Yes |
| Can't operate a key (dexterity, no usable port) | Hardware token with a display (OATH HOTP/TOTP) | No: code-based |
| Remote, no hardware at all | Push with number matching | Partly |
| Nothing above works | Support-assisted verification, time-boxed and audited | No |
The deciding axis is phishing resistance, and it should track the account's risk, not the user's convenience. WebAuthn binds the credential to the origin, so a proxy phishing page can't replay it; a 6-digit code can be relayed in real time by anyone who can read it off the screen. So an admin or anyone who can move money gets a key or a platform authenticator. A warehouse user reading a code off a display token is fine if that account can't approve a transfer.
The real cost isn't the hardware, a FIDO2 key runs roughly $25–50 and a display token $40–60, it's enrollment and replacement. Issue two credentials per user (primary plus backup) so a lost key is a re-enrollment, not a lockout, and keep the backup in the same inventory as the primary.
The failure mode I'd design against is the informal accommodation. Hypothetical but typical: an admin assistant held a display token for three executives; during a payment dispute nobody could prove which of four people approved a transfer, and the investigation ran seven weeks without an answer. Detection is an inventory where every issued authenticator maps to exactly one named person, plus an alert on any credential bound to more than one identity.
When nothing else works, the last resort is support-assisted verification: an agent confirms identity out-of-band (video call, government ID, manager confirmation), issues a temporary credential scoped to that account, expires it in 24 hours, forces re-enrollment of a permanent factor, and logs the whole exchange. That's an audited exception, not a waiver of the second factor.
Decision criterion: pick the strongest factor the user can actually operate; if that still falls below the account's risk tier, change the account's access. Don't weaken the factor.
Curated: · Written: · Reviewed:
Q&A-47What does your service desk need before it resets someone's second factor?
The engineering content of helpdesk MFA reset social engineering is the revocation path and its measured latency, not the protocol name.
A factor reset carries the same privilege as issuing a new second factor, so it needs proofing of comparable strength to enrolment. A caller who knows a start date and a manager's name has demonstrated access to a staff directory, not identity.
Concretely, require a verifiable channel such as a supervised check against a recorded document or a manager approval delivered through an authenticated application rather than a phone call, attach the proofing evidence to the ticket, and separate the person who approves from the person who performs the reset.
The reason for that specificity is a failure I have seen: A caller using facts taken from a public profile passed proofing in 6 minutes and had a new authenticator on a senior finance account. 3 payment changes were attempted within the hour, and the ticket recorded no evidence beyond the caller's answers.
Reset proofing before and after.
| Requirement | Before | After |
|---|---|---|
| knowledge questions | 4 | 0 |
| supervised identity check | no | yes |
| approver separate from performer | no | yes |
| median reset time | 6 minutes | 34 minutes |
I would not consider it settled without evidence: run an unannounced impersonation attempt against the reset process and require it to fail at the proofing step.
A reset is enrolment; prove it like one.
Curated: · Written: · Reviewed:
Q&A-39A user whose only factor is a security key loses it on a Friday. What is the recovery path?
Assumptions: one account, one registered authenticator (a FIDO2/WebAuthn security key), no password, no backup key, no pre-issued recovery codes, enterprise IdP, service desk staffed Monday–Friday.
The 90-second answer. The recovery path is the one you built before Friday. If the account really has a single authenticator and nothing pre-provisioned, the correct Friday-night answer is: the user stays locked out until a supervised identity-proofing session, and you eat the weekend outage rather than open a phone-based bypass. Anything a service desk can complete in ten minutes on a Friday night from a phone call is something an attacker can complete too.
The design that makes Friday boring is two authenticators at enrolment: a second key kept at home, or a platform passkey on a managed laptop. Most enterprise IdPs let you require more than one enrolled authenticator; turn that on. Then a lost key is a revocation, not an incident.
Why the key is not the target. An attacker with a name, an employer and a LinkedIn profile does not attack the key. They attack the recovery path, because it is reachable remotely and is usually the weakest link. Account strength is the minimum over all paths, not the maximum.
| Recovery path | Restores access in | Attacker's cost | Residual risk |
|---|---|---|---|
| second registered key, self-service | minutes, no human | needs the physical key | revoke the lost credential |
| supervised video proofing + manager attestation | same business day | forged documents plus a colluding manager | insider collusion |
| one-time bypass code, short TTL, single use | minutes | needs the code | interception, help-desk read-back |
| knowledge questions or a code mailed to the user | minutes | public records, one phone call | treat as no factor at all |
Worked trace (hypothetical, same Friday 18:40 event, two designs).
Design A: single key, service-desk reset: 18:40 key lost. 18:52 user calls. 19:05 agent clears four knowledge questions. 19:10 a bypass code is issued, valid 48 h. 19:12 an attacker who called at 18:50 with the same public facts also holds a valid code. Two live codes, no alert fired.
Design B: two keys at enrolment: 18:40 key lost. 18:41 user signs in with the backup key. 18:43 revokes the lost credential in the IdP. 18:45 registers a replacement. No service-desk contact, no bypass, no window.
Failure modes and how you detect them.
- Help-desk social engineering. Instrument recovery attempts per week and the proofing failure rate; alert on any recovery attempt from a new ASN or country, and on any account with two attempts inside 24 h.
- Bypass codes that outlive the session. Enforce a TTL in minutes and single use; alert on any use after the issuing session ends.
- Recovery loops. If the recovery mailbox is itself protected by the lost key, the path is dead. Map the dependency and keep one channel that does not depend on the factor being recovered.
- Silent re-enrolment. Whoever recovers also enrols their own key. Alert on any new authenticator registration within a few days of a recovery, and confirm the old key's revocation over a second channel.
Check it before you need it. Run the recovery flow yourself using only what is public about a test user: name, employer, role, social profiles. If it succeeds, the key was never the account's strength.
Curated: · Written: · Reviewed:
Q&A-49Can you list every authenticator registered to a departing employee and remove them?
What I'd say first
Not with one call, and not by disabling the account. "Every authenticator" spans at least three stores: the IdP's registered methods, the IdP's own recovery data, and credentials issued by downstream apps that never touch the IdP. You enumerate per store, revoke per credential, then re-list to prove zero.
Assumptions: one primary IdP (Entra ID or Okta), the leaver holds no admin role, and the account is being deleted rather than parked.
Enumeration and revocation, per store
| Store | List | Revoke |
|---|---|---|
| Entra ID | GET /v1.0/users/{id}/authentication/methods | DELETE /v1.0/users/{id}/authentication/methods/{methodId} |
| Okta | GET /api/v1/users/{userId}/factors | DELETE /api/v1/users/{userId}/factors/{factorId}, or POST /api/v1/users/{userId}/lifecycle/reset_factors for all |
| AWS IAM | aws iam list-mfa-devices, list-access-keys, list-signing-certificates, list-service-specific-credentials | aws iam deactivate-mfa-device --serial-number ..., delete-access-key, delete-signing-certificate |
Entra's methods endpoint returns one entry per registered method: FIDO2 key, Microsoft Authenticator push, software OATH, Windows Hello, phone, email: each with its own id. Graph scope UserAuthenticationMethod.ReadWrite.All; read-only is UserAuthenticationMethod.Read.All.
Connect-MgGraph -Scopes 'UserAuthenticationMethod.ReadWrite.All'
$id = (Get-MgUser -UserId 'leaver@contoso.com').Id
$methods = Invoke-MgGraphRequest GET "https://graph.microsoft.com/v1.0/users/$id/authentication/methods"
$methods.value | ForEach-Object {
Invoke-MgGraphRequest DELETE "https://graph.microsoft.com/v1.0/users/$id/authentication/methods/$($_.id)"
}
(Invoke-MgGraphRequest GET "https://graph.microsoft.com/v1.0/users/$id/authentication/methods").value.Count
The last line prints 0 when the loop worked. Run it before the account is deleted: the method list is scoped to the user object.
The trade-off
Disable-then-audit is one call, reversible, and needs no elevated scope. It is the right flow when the IdP is the only credential store, there are no duplicate account objects, and no downstream tokens exist: a small shop on one tenant with no SSO'd SaaS. It stops being right as soon as any of those is false, because disabling is a property of the account row and credentials are not.
Enumerate-and-revoke costs more: N+1 API calls per leaver, a read-write scope most helpdesk roles shouldn't hold, and a job that has to run before deletion. That is the price of the guarantee.
Failure modes
- Second account object. A duplicate from a migration or a second tenant holds a key the primary account's list never shows. Detect it by enumerating methods across every object matching the person (UPN aliases, mail, employeeId), not just the one on the ticket.
- Downstream credentials. GitHub PATs, app passwords, OAuth refresh tokens from consented apps, service-account keys. None appear in the IdP list. Detect them from the app-consent list and the token-issuance audit log.
- Recovery email or phone on the account itself, which survives method revocation and can drive a self-service reset.
- Okta deactivation is reversible and preserves factors; reactivation restores them. Delete the user or reset factors, don't just deactivate.
Verification
Run the loop against a test identity, confirm the re-list returns zero, then do the same for the leaver. Zero entries is the only evidence that counts.
Curated: · Written: · Reviewed:
Q&A-105What is segregation of duties, and how would you enforce it in an identity platform?
Segregation of duties means no single identity can complete a sensitive transaction end to end. It is not the same control as least privilege: least privilege bounds what one person can do, SoD constrains which combinations one person may hold. Two purposes: it forces collusion for fraud, and it makes errors visible because a second person has to look at the work.
The canonical example is payments. Whoever raises a payment must not be the one who approves it. If a single identity holds both payment.initiate and payment.approve, they can pay themselves and nothing downstream notices, because the approval is genuine. Same shape in infrastructure: whoever can write an IAM policy must not also be able to delete audit logs, or the change and its evidence disappear together.
Enforcement splits into preventive and detective.
Preventive is where you want to live. Three places:
- Role design. Keep a toxic-pair matrix: a maintained list of entitlement pairs that must never co-exist. Review it when a new entitlement is created, not when someone requests it.
- Assignment and request time. Before granting an entitlement, check the subject's current entitlements against the matrix and refuse the grant. Highest-value control, because it stops the conflict existing.
- Policy rules combining role and attribute conditions. Some conflicts are only conflicts in context: a support engineer may hold
refund.issueandrefund.approveas long as the amounts they can touch are capped, or as long as it is not the same ticket. That is a context-dependent conflict and it belongs in the policy engine, not the role model.
A Rego check at grant time:
# OPA / Rego. Deny a grant that would create a toxic pair.
deny[msg] {
conflict := toxic_pairs[_]
conflict[input.request.entitlement]
conflict[input.subject.entitlements[_]]
msg := sprintf("SoD: %v conflicts with an entitlement already held", [input.request.entitlement])
}
toxic_pairs := [
{"payment.initiate", "payment.approve"},
{"iam.policy.write", "audit.log.delete"},
{"vendor.create", "vendor.payment.approve"},
]
Detective is the backstop, because preventive checks miss things: entitlements granted outside the platform, legacy systems, role changes never re-checked. Access certifications, periodic reviews where a manager or resource owner confirms each entitlement, plus a standing SoD violation report that queries the entitlement store for subjects holding a toxic pair. The report is the honest measure. If it is non-empty and nobody can explain why, the preventive controls have a hole.
| Point | Control | Catches | Weakness |
|---|---|---|---|
| Role design | Toxic-pair matrix | Structural conflicts | Rots without an owner |
| Grant / request | Pre-commit check | Most conflicts | Blocks legitimate work |
| Runtime policy | Role + attribute condition | Contextual conflicts | Latency, complexity |
| Certification | Periodic review | Drift, orphaned access | Rubber-stamping |
| Monitoring | Violation report | What slipped through | After the fact |
Joiner-mover-leaver is where SoD actually breaks. Joiners are easy: check at assignment. Leavers are easy: remove. Movers are the hard case, because there is a window where the person holds both old and new access. The rule: compute the post-move entitlement set, run the SoD check against it, and if it conflicts, either block the move until the old access is gone, or grant the new access time-bound with an expiry that forces the old one out. Never leave both standing and hope.
Privileged access management is where SoD is most often waived and where it matters most. Standing admin rights are themselves an SoD problem: one account that can do everything. The controls that substitute for separation: vault checkout with dual approval (two-person rule), time-boxed elevation, session recording, automatic revocation at expiry.
Break-glass is the legitimate exception. When it is needed: the identity provider is down and the only person who can fix it is the one who would normally be blocked: you do not pretend the conflict away. You make the exception explicit: time-bound in hours, not days; dual-approved; fully logged; reviewed inside a defined window. The exception is itself an entitlement with an owner and an expiry. If it recurs, that is a signal the standing design is wrong, not that the exception should become permanent.
Curated: · Written: · Reviewed: —
Q&A-50When is role-based access the right model rather than an attribute expression?
The 90-second answer. Use RBAC when the permissions a principal needs are a function of their job function and not of the object they touch. If two engineers with the same title need the same access to every resource in a domain, a role is the cheaper and more reviewable encoding. Reach for attribute expressions when the decision depends on data a role cannot carry: who owns this document, which tenant it belongs to, how it is classified, what time it is, whether the request came from a managed device.
The deciding test is cardinality. Count the distinct permission sets your population actually needs. If N principals collapse into a handful of sets, RBAC. If the number of distinct sets approaches N, every user needs a slightly different slice, roles become per-person records with a name, and you have built ABAC with worse tooling.
| Criterion | RBAC wins | Attribute expressions win |
|---|---|---|
| Decision inputs | job function only | resource owner, classification, tenant, environment |
| Distinct permission sets | few, shared across many principals | roughly one per principal or per object |
| Change trigger | hire, transfer, promotion | data changes, not org-chart changes |
| Review method | read the role, predict the access | simulate the policy against real requests |
| Decision cost | set membership, cacheable in the token | attribute fetch per decision, or a cache with a freshness bound |
| Characteristic failure | role explosion, stale membership | policy sprawl, stale attributes |
Worked example (hypothetical figures). A document service with 4,000 employees and 12 job functions. RBAC: 12 roles, membership driven from the HR feed, roles baked into the access token, and the check is a set-membership test with no per-request I/O. Now add per-document sharing. The permission set now depends on the document's ACL, which differs per object, so encoding it in roles means one role per sharing pattern: thousands of them, each with one member. The workable split is the common one: RBAC for the coarse gate (may this principal reach the document service, may they see the finance domain), an ownership or ACL attribute check at the object.
Failure modes and how you catch them.
- Role explosion: track roles per distinct job function and the share of roles with exactly one member. When that share climbs, the role model has stopped describing job functions and started describing people.
- Stale membership: role changes only take effect when the token is reissued, so a revoked role lives until the token expires. Measure the time from an HR change to effective revocation; with one-hour tokens the worst case is an hour plus refresh, and that is a number you should know before an auditor asks.
- Stale attributes: if the policy reads group membership from a directory that syncs every 15 minutes, you have up to 15 minutes of stale allow. Detect it by comparing the attribute snapshot used at decision time against the source of truth.
- Policy sprawl: try to answer "who can read document X" by simulation. If you cannot, you have lost the property that made RBAC attractive in the first place.
When RBAC is the wrong call. Consumer-style file sharing, per-tenant custom roles, break-glass access, and any time-bound grant. In each case the deciding attribute lives on the resource or the request, not on the person, so a role would have to be minted per object or per window.
If you cannot answer "who can read this document" by reading a role, you are not doing RBAC: you are doing ABAC with a role table in front of it.
Curated: · Written: · Reviewed:
Q&A-63Where do you draw the line between a role and an attribute condition?
The line: does the fact change without anyone's job changing?
If a fact changes only when someone's job function changes (they move teams, get promoted, pick up on-call) it belongs in the role. If it changes per request while the person's job stays the same (which device, which network, what time, how sensitive the record is) it belongs in a condition.
That test matters because the two are reviewed, revoked and audited by different machinery. A role is a stored grant: subject → role, enumerable, so a data owner can list all 340 holders and sign off quarterly. A condition is a predicate evaluated at decision time against request context, and it is not enumerable per subject: you cannot ask "who does this condition permit?" without running it over the whole population.
Mechanism. Authorization evaluates in two stages: resolve the subject's roles from the directory, then evaluate the policy's conditions against request context. Resource classification, device posture, network, time. Roles answer "may this class of person ever do this"; conditions answer "may this particular request proceed now". Revocation differs too: removing a role takes effect when the subject's next token or session is issued, or immediately if the PDP reads the directory per request; a condition flip takes effect on the next request. That latency difference is why conditions are the right place for break-glass and compromised-device response.
The asymmetry that decides the split. Editing a role assignment affects one person. Editing a condition affects every holder of every role the condition is attached to. A condition on a 340-holder role has a 340-person blast radius per edit, so conditions need code-grade change control: review, tests, staged rollout.
Failure modes.
- Condition explosion: rules that grow to 14 nested predicates. No reviewer can state who they permit, and operator-precedence bugs hide inside them.
- Identity smuggled into a condition:
subject.email == "alice@example.com". That is a role assignment wearing a condition's clothes, and it is invisible to the quarterly access review. - Stale context attributes: device posture asserted at login and cached for the session means a condition on
device.managedis really a condition on "was managed at login". - Time-of-check/time-of-use: the condition passes, the resource's classification changes, the request proceeds anyway.
Worked trace. Same role, two requests, one condition flips:
| Step | Request A | Request B |
|---|---|---|
| roles(alice) | customer-support-reader | customer-support-reader |
| resource.classification == internal | true | true |
| device.managed | true | false |
| decision | allow | deny |
Request B's deny is attributable to exactly one named condition, and no role changed. That attribution is the point of the split: an incident review can say which condition failed without re-deriving the org chart.
How to tell you got it wrong. Write one sentence naming who a rule permits. If you cannot, the rule is over-conditioned. Then evaluate the policy over the full population and diff the permitted set against that sentence. Any subject in the diff is either a missing role or a condition doing a role's job.
Curated: · Written: · Reviewed:
Q&A-54Your endpoint checks that the caller holds a read scope. Is that enough?
No. A read scope answers may this caller read reports? It does not answer may this caller read report 8841? The scope is the verb; the object check is the noun. This is OWASP API Security Top 10 #1, Broken Object Level Authorization, and it is the most common serious flaw in APIs that otherwise look well-secured.
Assumptions I'd state out loud: a bearer token carrying a coarse scope, resource identifiers arriving from the client in the path or body, and more than one tenant sharing the same tables.
Why the scope can't cover it. The scope check runs at the edge, before the object exists. It can only compare a string in the token against a string in the route. The object check needs two things the edge does not have: the authenticated subject and the record. So it has to run after resolution, and it has to run on every path that resolves an object: including the ones that resolve it implicitly: batch endpoints, exports, search filters, nested resources.
Order and response shape matter. Resolve, then authorize, then return. If you return 403 for "exists but not yours" and 404 for "doesn't exist", you have built an enumeration oracle; return 404 for both.
Where to put the check.
| Approach | Where it lives | Good when | Breaks when |
|---|---|---|---|
| Centralized middleware | one function called per route | ownership is a simple subject-to-owner column | the decision depends on the object's own state (ACLs, sharing, field-level rules), or the record lives in another service |
| Per-endpoint checks | in the handler | rules genuinely differ per resource | it is the default; one forgotten handler is the whole bug |
| Policy engine (OPA/Rego, Cedar, Casbin) | external decision point called with subject, object, action | many resources, rules that change often, auditability required | you still have to pass the right object in; a wrong input is a wrong decision |
| Deny-by-default query scoping | the data access layer | you can push the predicate into the query | cross-store joins, or legacy code that bypasses the layer |
The strongest is the last: scope the query itself, SELECT ... WHERE tenant_id = :subject_tenant AND id = :id, so a missing check returns empty instead of leaking. Centralized middleware is the wrong choice when the decision depends on the object's own state or on data the middleware cannot see; then the check belongs next to the data.
Failure modes. TOCTOU between check and use, when the object's owner changes in between. Checking the parent ID but not the child ID in /reports/{id}/comments/{cid}. Batch endpoints where the first ID is checked and the rest are trusted. Treating unguessable identifiers as a control.
Worked example (figures from an audit, rounded). A report endpoint checked scope only. Incrementing the identifier returned 4,700 reports across 38 customers; the crawl ran 2 hours at 6 requests per second (roughly 43,000 requests) and tripped nothing, because every request was individually authorized by scope.
Endpoint audit for object checks:
| Endpoint group | Endpoints | Missing the object check |
|---|---|---|
| reports | 14 | 9 |
| exports | 6 | 2 |
| administration | 21 | 0 |
Evidence. One test per endpoint that requests another tenant's object with a valid token and asserts an empty body or 404. That is the evidence; a code review is not.
Curated: · Written: · Reviewed:
Q&A-55What is the difference between static and dynamic separation of duty, and when do you need each?
Separation of duty is a constraint on conflicting entitlements. Static and dynamic differ in where the constraint bites: static forbids holding both entitlements at all; dynamic permits holding both but forbids using both in the same session or on the same transaction.
Both come from the RBAC reference model. ANSI INCITS 359-2004 defines static separation of duty (SSD) as a constraint on the user-role assignment relation and on role hierarchies, and dynamic separation of duty (DSD) as a constraint on activation of roles within a session. That difference in enforcement point is the whole answer.
Mechanism. SSD is a set-membership check evaluated when you write an assignment: if user u already holds role r1 and (r1, r2) is in an SSD set, the grant of r2 is rejected. It is stateless (no session, no history) so the provisioning system can enforce it at grant time, and it holds even if the user never logs in. DSD is evaluated at decision time, per session. The user may be assigned both roles; the session may activate only one. That needs session state, and if the conflict is per-transaction rather than per-session it needs transaction state too: the policy has to know who created the record being approved. A stateless policy decision point cannot evaluate that alone: the request must carry the transaction context, or a policy information point must supply it.
| SSD | DSD | |
|---|---|---|
| Enforcement point | grant / assignment | session activation or request decision |
| State required | none | session, plus transaction history for per-record rules |
| Typical failure | blocks the business; users share accounts | rule cannot be evaluated and silently passes |
| Defined in | ANSI INCITS 359-2004 | ANSI INCITS 359-2004 |
When each. Use SSD when the conflict is permanent and the population can be split: no one may hold both "create user" and "approve access request" where there are enough admins to staff both. Use DSD when the population is too small to split, or when the conflict exists only within one transaction: a five-person finance team where everyone must be able to create vendors and approve payments, but no one may approve a payment to a vendor they created.
Worked trace (DSD, per-record).
Request: subject=alice, action=approve_payment, resource=payment_4471.
- PIP resolves payment_4471.vendor = vendor_88.
- PIP resolves vendor_88.created_by = alice.
- DSD rule: subject != resource.vendor.created_by.
- alice == alice → deny.
Under SSD the same rule is evaluated at grant time instead: assigning alice the payment-approver role while she holds vendor-creator is rejected outright, and the trace above never runs.
Failure modes. Static SoD applied to a team too small to split does not produce compliance; it produces shared accounts, which removes attribution and is worse than the conflict it prevented. Dynamic SoD deployed on a stateless PDP produces a rule that cannot see the record's creator and therefore passes: the conflict is live and the audit log shows a clean decision. Detect both by attempting the forbidden action: have the creator approve their own record and require a deny, then confirm the deny is logged with the rule that fired.
Curated: · Written: · Reviewed:
Q&A-53One policy allows and another denies the same request. What should your engine do?
Deny. When one applicable policy permits and another denies the same request, the engine returns deny, and it does so because the combining algorithm is deny-overrides, declared in configuration rather than inherited from a default.
Why deny and not permit: the two errors are not symmetric. A wrong deny is an availability incident: someone is blocked, they file a ticket, you fix it in minutes. A wrong permit is a confidentiality or integrity incident: data left a boundary and you hear about it from someone else. Deny-overrides makes the safe direction the one that needs no coordination: any team can add a prohibition that no other team's allow can silently undo.
Mechanism. The engine evaluates every policy whose target matches the request, collects the effects, then applies the combining algorithm to that set. Deny-overrides: if any collected effect is Deny, the decision is Deny. Two things this depends on that people get wrong:
- Only applicable policies count. A deny scoped to a different principal or resource is NotApplicable, not Deny. A deny-overrides engine whose deny rule never matches is a permit-overrides engine.
- Indeterminate is not Permit. If a policy errors (missing attribute, unreachable decision point) the engine must fail closed. Treating an evaluation error as "no deny found" turns a dependency outage into an open door.
Trace. Request: principal svc-reports, action PutObject, resource audit-logs/2024/.
| Policy | Target match | Effect |
|---|---|---|
A: allow svc-reports on audit-logs/* | yes | Permit |
B: deny any principal PutObject on audit-logs/* | yes | Deny |
Combining {Permit, Deny} → Deny, decided by B. Decision record: {request_id, principal, action, resource, effect: "Deny", deciding_policy: "B", evaluated: ["A","B"], algorithm: "deny-overrides"}. The deciding_policy field is what makes a denial explainable without re-running the engine by hand.
Alternatives, and when they're right.
| Algorithm | Same request | Use when |
|---|---|---|
| deny-overrides | Deny | Access control over shared resources; the cost of over-permission is unbounded |
| permit-overrides | Permit | Grant systems (entitlements, feature flags) where over-restriction is the main risk |
| first-applicable | Order-dependent | Policies are ordered by specificity and one team owns the whole set |
| only-one-applicable | Error if more than one matches | You want overlapping policy sets to fail at build time, not decide at runtime |
first-applicable is the one that bites. The decision is a function of list order, so a reorder is a silent permission change with no policy diff to review.
Failure modes and detection.
- Ordering dependency: two teams write overlapping policies, a deployment reorders them, access changes with no policy edit. Detect by diffing the effective decision set for a fixed corpus of (principal, action, resource) triples across every policy-set change.
- Blanket deny: a deny written on a resource prefix blocks everything under it, including the team that owns the prefix. Detect with a per-policy deny-rate metric; alert when one policy's deny count jumps by an order of magnitude.
- Unexplainable denial: the user is blocked, nobody can say which rule did it, and the answer is found by bisecting the policy set. Detect by requiring
deciding_policyin every decision record and exposing an explain endpoint.
What I'd verify before calling it done. Evaluate a principal covered by both an allow and a deny and require the response to be Deny naming both policy ids. Then reorder the policy set and re-run: the decision must not change. If it does, the algorithm isn't deny-overrides, whatever the config says.
Curated: · Written: · Reviewed:
Q&A-51Your attribute rule needs a clearance value and it is absent. What happens?
What a strong candidate says first
A missing required attribute is indeterminate, not false. The policy has to say what indeterminate means, and for a clearance attribute the answer is deny: with a reason code that names the missing attribute, so the decision is separable from a deny-by-rule. Deny is a choice, not a law: it buys confidentiality with availability, and on some paths that trade is the wrong one. So fail closed for clearance, and allow a bounded degraded mode (short-lived, resource-scoped, logged) only where a deny storm costs more than the exposure it prevents.
Mechanism
ABAC compiles policy into an expression over attribute data, so the policy inherits that data's availability. Three-valued evaluation is the honest model: present-and-sufficient → allow; present-and-insufficient → deny; absent → indeterminate, which the policy maps to a decision. Collapsing indeterminate into false is fine for the decision but throws away the reason, and the reason is what you alert on.
MISSING = object()
ORDER = {"public": 0, "internal": 1, "secret": 2}
def decide(attrs, resource):
clearance = attrs.get("clearance", MISSING)
if clearance is MISSING:
return Deny("missing_attribute:clearance")
if clearance not in ORDER:
return Deny("invalid_attribute:clearance")
if ORDER[clearance] < ORDER[resource.required]:
return Deny("insufficient_clearance")
return Allow()
Three deny reasons, three alerts. missing_attribute spiking means the attribute source is broken; insufficient_clearance spiking means someone is probing.
| attribute state | evaluation | decision | reason |
|---|---|---|---|
| present, sufficient | true | allow | clearance_ok |
| present, insufficient | false | deny | insufficient_clearance |
| absent | indeterminate | deny | missing_attribute:clearance |
| source down, cached value within TTL | stale true | allow (degraded) | clearance_ok_stale |
| source down, cache expired | indeterminate | deny | attribute_unavailable |
The trade-off
Fail-closed is right when the attribute is the only thing between the caller and exfiltration, money movement, or a write. Clearance is that kind of attribute.
Fail-open is defensible when the attribute is a soft signal rather than a gate (a region label used for routing, a team tag used for personalization) or when the resource is low-sensitivity and the deny storm takes down more than the exposure leaks. A public status page behind SSO whose missing attribute only affects layout is a fail-open case.
The middle option is usually right for hard gates: allow, but mint a grant that expires in minutes, scoped to the one resource, tagged degraded in the log, and alerted. The expiry bounds exposure regardless of how long detection takes; the scope bounds blast radius; the tag keeps the audit honest. Break-glass clinical access during an EHR outage is the canonical version of this.
Worked example (hypothetical figures)
A staff feed fails at T+0. Attribute cache TTL is 60 s, so cached clearances serve until T+60 s, then every request for the restricted dataset sees a missing clearance. 4,300 principals hold that clearance.
| Behaviour on missing | Decisions in the 90-min window | Exposure | Legitimate traffic denied |
|---|---|---|---|
| fail-open | 4,300 allow | up to 90 min unauthorized | 0 |
| fail-closed | 4,300 deny | 0 | 4,300 blocked |
| degraded, 5-min expiry | 4,300 allow, re-evaluated each 5 min | ≤5 min per grant | 0 |
The fail-open row is the one that hurts: those grants land in the log as ordinary allows, so nothing distinguishes them from real ones and the outage stays invisible until someone reads the feed's own metrics.
Failure modes
- Silent permissive default. Missing → allow, logged as a normal allow. Detect with a synthetic probe that strips a required attribute in a test tenant and asserts a deny naming it.
- Stale cache serving a revoked clearance. TTL too long means revocation doesn't take. Detect by recording cache age at decision time and alerting when p99 age approaches TTL.
- Deny storm. Fail-closed on a hot path turns an attribute-source outage into a full outage. Detect by deny rate per reason code;
missing_attributespiking is the signal. - Alert fatigue. If
missing_attributefires for optional attributes too, nobody reads it. Only required attributes page.
Curated: · Written: · Reviewed:
Q&A-52Access to a resource depends on a tag. Who is allowed to change the tag?
The short answer
A tag that gates access is a grant, so the set of principals allowed to write it must be a subset of the principals allowed to grant the access it controls. In practice that is a small named role, not the resource owner, not the platform team at large, and not the service account the workload runs as. If you cannot name the writer set for a decision-bearing key, the tag is a public grant.
Why it works that way
The policy engine reads the tag at request time; the tag write is a separate API call with its own permission. In AWS, an access policy conditions on aws:ResourceTag/<key>, and tag writes go through ec2:CreateTags / tag:TagResources. A principal holding the tag-write permission but no data permission can still reach the data by relabelling: that is the documented escalation path, and it is why the two permissions have to be reviewed together. Kubernetes is the useful counterexample: labels are not authorization at all, RBAC never reads them, so "tag-driven access" there is something you built on top (OPA/Gatekeeper, an admission webhook) and the writer set is whatever that policy says.
| Tag key | Drives access | Writers |
|---|---|---|
| data classification | yes | 6 named approvers |
| environment | yes, for administrative scope | 11 platform engineers |
| cost centre | no | any resource owner |
Alternatives, and when not to use tags at all
- Separate namespace. Put decision-bearing keys in their own namespace with their own IAM, so a cost-centre edit cannot touch classification. Cost: two tag APIs, and drift when a resource is created without the decision key.
- Immutable / set-on-create. The tag is fixed at creation and a change means delete-and-recreate. This kills relabel escalation outright, but it breaks legitimate reclassification and forces a data migration each time.
- Move classification out of tags. A dedicated attribute store or per-object ACL with its own audited API. More code, but the grant and the metadata stop sharing a permission surface. This is the right answer when the blast radius is large.
- Don't use tag-driven access. When the tag is genuinely user-editable metadata (cost centre, owner), or when the decision needs per-object exceptions, tags are too coarse: a per-object grant list is the honest model.
Failure modes
- Relabel to unblock. Illustrative worked example: an engineer flips classification from restricted to internal to unblock a batch job; 12 datasets become readable by 260 people for 6 days, and the trail shows a routine metadata edit. (Figures are hypothetical, chosen to show the shape of the incident.)
- Absent tag defaults to allow. Check what the policy does when the key is missing. A missing decision tag must deny, not fall through.
- Inherited tags. The resource inherits from a parent or account; changing the parent silently re-grants everything beneath it.
- Automation. A CI service account with broad tag permissions becomes the escalation path nobody reviews.
Detection: diff tag-change events against subsequent access by the same principal inside a short window, and alert on any decision-bearing key change outside the approver set.
Verification
As a standard engineer, attempt to write the decision-bearing key and require the change to be refused. Then delete the tag and re-request access; require denial, not a default allow.
Curated: · Written: · Reviewed:
Q&A-56Your role hierarchy is six levels deep. What does that cost you?
In the first minute
Assume a standard RBAC model: roles inherit transitively, and a principal's effective permissions are the union of every role on its ancestor chain. Six levels costs you in three places: the size of the effective-permission closure you compute and cache, the human cost of reviewing a grant, and the blast radius of editing a role near the root. The last one is the expensive one, because it is the one that gets approved without anyone seeing it.
The closure is not linear in depth
For a tree with branching factor 3, depth 2 gives 13 roles and 21 (role, ancestor) pairs. Depth 6 gives 364 roles and 6,015 pairs: 28x the roles, 286x the inheritance edges. The pair count is sum(k * 3^k) for k = 1..6.
Every authorization check either walks that chain per request or reads a materialized set. Walking costs hops per check; materializing costs invalidation. Editing the root role invalidates the cached effective set of all 363 descendants, not 13.
Blast radius
| Level touched | Roles below | People affected (illustrative) |
|---|---|---|
| L1 root | 363 | 3,100 |
| L3 | 39 | 210 |
| L6 leaf | 0 | 40 |
The same edit is a 40-person change or a 3,100-person change depending only on which level it lands on. Nothing in the grant itself tells you which.
The failure mode
A permission added to a root role reached 3,100 people through five inheritance hops. It was approved on the belief that it affected 40: the reviewer read the grant, not the effective access. It surfaced nine days later in an unrelated access review. Detection is a pre-merge effective-access diff that lists the principals who gain a permission, computed from the closure rather than from the role being edited.
What to do
- Cap inheritance at two levels. When a new need appears, split a role rather than add a level.
- Show effective access in the review UI, not the grant path.
- Require the effective-access delta, principals gaining and losing, on every hierarchy change, computed in CI.
- If you need six levels to express policy, the hierarchy is encoding your org chart. Move to attribute-based or policy-as-code (Cedar, OPA/Rego) where you can query "who can do X" directly instead of walking a tree.
When depth is fine
If the hierarchy is a human taxonomy and effective permissions are materialized and diffed on every change, depth costs storage and CI time, not correctness. It stops being fine the moment authorization is evaluated at request time against the live tree.
Curated: · Written: · Reviewed:
Q&A-58You cache authorization decisions. What goes into the key?
The key is the full set of inputs the policy actually read: nothing more, nothing less. If the policy consulted it, it belongs in the key; if it didn't, including it only shreds your hit rate. A decision cached under a key that omits one input is a wrong answer waiting for a hit.
| Component | Example | Why it must be in the key |
|---|---|---|
| subject | u:4417 | Stable principal ID, not email or username: those get reassigned and you inherit the previous holder's grants |
| tenant / org | t:acme | The classic leak: same user ID, different tenant, different object namespace |
| object | doc:8812, plus parent (folder:220) if the policy walks the hierarchy | doc:* collapses every document into one decision |
| action | read | Read and write are separate policy branches |
| request context the policy reads | MFA level, device posture, IP class, break-glass flag, time bucket | Only include what the policy branches on: raw client IP explodes cardinality for no gain |
| policy version | bundle hash or deploy tag | Without it, a policy fix doesn't take effect until TTL expiry |
| attribute version | snapshot ID for group membership, role bindings, resource labels | Without it, a revoked role keeps its allow |
import hashlib, json
def decision_key(req, policy_version, attr_version):
material = {
"sub": req.subject_id, # "u:4417", not "alice@acme.com"
"tenant": req.tenant_id,
"obj": req.object_id, # "doc:8812"
"act": req.action, # "read"
"ctx": req.context_hash, # mfa + device posture + ip class, bucketed
"pol": policy_version, # "authz-2024-11-04.3"
"attr": attr_version, # "groups-2024-11-04T09:12Z"
}
blob = json.dumps(material, sort_keys=True).encode()
return hashlib.sha256(blob).hexdigest()
sort_keys=True matters: two dicts with the same contents in different insertion order must hash identically, or you get phantom misses.
The trade-off is cardinality against hit rate. Every input you add multiplies the key space. Bucket context values the way the policy reads them, if the policy says "corp network or not", key on corp/external, not the /32. TTL is the backstop, not the mechanism: version bumps invalidate precisely, TTL catches whatever you forgot to version. Keep privileged decisions short (tens of seconds) and low-risk reads longer. Cache denies too, but with a shorter TTL than allows: a deny held on stale attributes blocks legitimate access and looks like an outage to the user.
Failure modes worth naming:
- Cross-tenant leak. Worked example (hypothetical figures): a key of
(user, action)served an allow minted for one tenant's object to the same user ID in another tenant. 78 reads over 12 minutes, and the audit log showed only the original decision, so the leak was invisible until someone diffed object IDs. - Stale attribute. A user is removed from a group; if
attr_versionisn't bumped, the allow survives until TTL. Detection: compare cache-hit decisions against a sampled live evaluation. - Policy version not bumped on deploy. The new policy is live but every request still hits old entries. Make the version part of the deploy artifact, not a manual step.
- Invalidation stampede. Bumping the version misses every key at once and the policy engine takes the full load. Jitter TTLs and warm the new version before cutting over.
To check the key is complete: send two requests differing in exactly one input, assert two distinct keys and two policy evaluations. Then bump attr_version and assert the old key misses. If either assertion fails, the key is missing an input.
Curated: · Written: · Reviewed:
Q&A-57An elevation was granted for two hours. What actually expires?
"Two hours" is a property of the broker's record, not of the access. Three clocks start when the elevation is granted and they do not stop together:
- the grant record: the broker's row saying who holds what until when
- the credential issued under it: token, session, key
- any session opened while the elevation was live
What expires is whatever enforces the expiry. If the target authorizes on group membership, and the group is what the target checks, the broker's clock is decorative.
Four designs, and the criteria that pick between them:
| Design | What enforces expiry | Works when | Fails when |
|---|---|---|---|
| Time-bound credential at the target (STS session, PIM activation, IAM Condition with an expiry) | the target itself | the target's authz model puts an expiry on the identity | authz is group-based: the group is the unit and carries no expiry |
| Broker-managed grant, confirmed revocation, scheduled reconciliation | the broker, verified against the target | the target exposes an API to remove the grant | the broker cannot reach the target, or removal is async and unconfirmed |
| Session-scoped elevation, no membership at all | the credential lifetime | the work is short and self-contained: one deploy, one query | the work outlives the window: batch job, SSH session, CI runner |
| Break-glass with alerting, no expiry | a human | the work legitimately outlives any window | you must prove expiry to an auditor |
Deciding criteria, in order: can the target enforce expiry itself? Is the unit of authorization a group or a credential? Does the work outlive the window? Can you prove expiry from the target's own logs rather than the broker's?
Where not to use the time-bound credential: a legacy target with local accounts and no management API. There the credential's expiry is a fiction, because nothing on the target reads it. Compensate instead: rotate the credential on a schedule and put the target behind a proxy that enforces the window.
Why the distinction matters, from a real incident: a broker marked an elevation expired at 120 minutes while the group membership at the target remained. 14 accounts held privileged membership for an average of 26 days, and the broker's report showed zero active elevations throughout.
State 24 hours after expiry.
| Location | Expected | Found |
|---|---|---|
| broker grant record | removed | removed |
| target group membership | removed | present in 14 cases |
| sessions opened under elevation | ended | 3 still live |
I would not call it settled without evidence: let an elevation expire, then query the target directly for the membership and for any live session. The broker's clock is not the target's state.
Curated: · Written: · Reviewed:
Q&A-66A batch endpoint accepts five hundred identifiers. Where does authorization go?
Authorization goes on each element, at the point where that element is acted on, not on the request, and not in a separate pass before it.
A batch is a transport envelope. The resource being authorized is each object inside it, against the caller's identity and scope. So the check belongs in the same loop that performs the effect, so no code path can act on an element without having decided on it:
MAX_BATCH = 500
def handle_batch(caller, ids):
if len(ids) > MAX_BATCH:
return 413
out = []
for oid in ids:
# scope the read to the caller's tenant: never load a row you can't touch
obj = store.get(oid, tenant=caller.tenant)
if obj is None or not can(caller, "delete", obj):
out.append({"id": oid, "status": "not_found"}) # one shape for both
continue
store.delete(obj) # effect only after the decision, same iteration
out.append({"id": oid, "status": "deleted"})
return 200, out
Three things that code does deliberately:
- The read is scoped to the caller's tenant, so another tenant's row is never loaded, let alone returned. Tenant scoping is a coarse filter, not the decision:
can()is still the per-object check, because ACLs are finer than tenancy. If you fetch by id alone and check afterwards, the object is already in memory and one carelessreturn objon an error path leaks it. - Forbidden and non-existent collapse to the same
not_found. Distinct 403/404 shapes turn the batch into an enumeration oracle: submit 500 ids, read off which ones exist. - The decision and the effect sit in the same iteration. A pre-pass that authorizes all 500 and then acts on them has a window where an ACL changes, a role is revoked, or the object is deleted and recreated: the check-then-act gap, multiplied by 500.
Failure modes and how they show up:
- First-element authorization. The check runs on
ids[0]and the loop acts on all of them. I have seen this: a bulk delete authorized on the first identifier removed 340 records across 12 tenants; restoration took 19 hours and the audit trail recorded a single authorized call. Detection: one decision in the audit log for a request that touched N objects. - Per-call rate limits. If the limit is 1 request/s, a 500-element batch is a 500x amplifier. Count the limit per element and cap batch size independently.
- All-or-nothing responses that hide partial success. Returning 200 with a summary count leaves the caller unable to tell which elements failed. Return per-element outcomes: 207 Multi-Status, or 200 with a body listing every element.
Trace, batch of 60 mixed identifiers, caller in tenant A:
| Element class | Count | Decision | Effect |
|---|---|---|---|
| tenant A, caller may delete | 41 | allow | deleted |
| tenant B | 12 | deny → not_found | none |
| id never existed | 7 | deny → not_found | none |
Response: 41 deleted, 19 not_found, all 60 present in the body. Audit: 41 allow decisions and 19 denials, each naming the caller, the element and the reason.
How I'd verify it: submit a batch that deliberately mixes permitted and forbidden ids, including ids from another tenant and ids that never existed, and assert that exactly the permitted ones changed state, that the response contains one entry per submitted id, and that the forbidden and missing entries are byte-identical in shape. Then repeat with the caller's role revoked mid-batch to confirm the check-then-act window is closed.
Curated: · Written: · Reviewed:
Q&A-67A user queues an export that runs an hour later. Whose authority does it run with?
I would settle async job carrying user authority by attempting the action after the identity event that was supposed to stop it.
A queued job carries the requester's authority as a narrow, time-bound delegation, and it re-checks at execution because that authority may have been removed while the job waited. Running it as a service account with broad rights turns every queue into an escalation path.
Concretely, record the subject and the requested scope on the job, mint a short-lived delegated credential limited to that operation at execution time, re-evaluate the subject's current entitlement before producing output, and fail the job with a clear reason when the authority has gone.
The reason for that specificity is a failure I have seen: An export queue ran under a service identity holding tenant-wide read. A job queued 50 minutes before a dismissal completed afterwards and mailed 8,200 customer records to the leaver's personal address.
Job execution against current authority.
| Design | Re-checks at run time | Records released in the drill |
|---|---|---|
| service identity with tenant read | no | 8,200 |
| delegated credential, no re-check | no | 240 |
| delegated credential with re-check | yes | 0 |
I would not consider it settled without evidence: queue a job, remove the requester's access, and require the job to fail at execution.
The queue must not outlive the authority.
Curated: · Written: · Reviewed:
Q&A-59The authorization service times out during a payment write. What does the caller do?
Assume the authorization check is a synchronous pre-write call: the caller asks "may subject S write payment P?" and commits only on an allow. A timeout is the absence of a decision, not a decision to allow. The caller aborts the payment write, rolls back, returns 503 with Retry-After to its client, and increments a fail-closed counter. It may retry the authorization call once, inside the deadline it has left. It does not retry the write inside that request: the client retries with the same idempotency key.
Why that split: the decision call is a read, so repeating it costs one round trip and cannot corrupt anything. The write moves money, so repeating it is only safe when it is keyed, and the retry belongs to whoever owns the key.
Concrete behaviour, with a 2,000 ms client deadline and a 150 ms authorization budget per attempt (hypothetical figures):
- 0 ms: POST /payments accepted, transaction opened.
- 10 ms: Check(payment:write, resource) sent, deadline 150 ms.
- 160 ms: deadline exceeded. No decision.
- 170 ms: one sequential retry against another replica, deadline 150 ms.
- 320 ms: second timeout. Abort.
- 325 ms: rollback; no payment row committed.
- 330 ms: 503 plus Retry-After: 1 returned, and authz_failclosed_total{op="payment_write"} incremented.
Worst case the caller spends 320 ms of a 2,000 ms budget and still fails closed, which is the point: the budget is there to absorb a retry, not to buy a permissive answer.
| Authorization response | Caller action | Client sees | Payment committed |
|---|---|---|---|
| allow | commit | 201 | yes |
| explicit deny | abort, no retry | 403 | no |
| timeout | retry once in budget, then abort | 503, Retry-After: 1 | no |
| connection error or 5xx from authz | same as timeout | 503, Retry-After: 1 | no |
| cached allow (low-risk reads only) | served if age under 60 s | 200 | not applicable: never used for a write |
There is one legitimate middle path: accept the request as pending and return 202, so the client is not blocked. Nothing moves money until a decision exists, and the pending record carries the idempotency key so a later retry cannot double-charge.
Failure modes worth naming:
Permissive default set during a capacity incident and never reverted. Writes commit with no decision. Detection is a counter of committed payments with no matching allow decision for the same resource; it should be zero on every dashboard, and the test is to stop the authorization service in staging and require every write path to refuse.
Retry amplification. Three attempts per caller across N callers triples load on a service that is already unhealthy, extending the outage. Cap at one retry with jitter, or fail fast and let the client's own backoff handle load.
Stale allow reused on the write path. A revocation that lands after the cached allow produces an authorized-looking write from a subject who no longer has the permission. Keep the read cache and the write path on separate decision sources.
Timeout after an allow. The authorization succeeded, then the commit timed out: the caller does not know whether the charge landed. That is what the idempotency key is for: the client retries and the server returns the original result instead of charging twice.
Surface the failure, not a generic error: the customer should be told the payment was not taken and can be retried, because failing closed means the money did not move.
Curated: · Written: · Reviewed:
Q&A-60How do you prove your multi-tenant authorization actually isolates tenants?
The 90-second answer
Isolation is a claim about what happens when a valid caller supplies another tenant's identifier. The only way to establish it is to do exactly that, per endpoint, in CI, and fail the build when any response returns another tenant's data or reveals that it exists. A design review cannot establish it, because the failure lives in whichever handler forgot the check, and the handler that forgot is not the one you read.
How to run it
Seed two tenants in every environment, including staging and preview, each holding at least one object of every resource type. Generate one substitution case per endpoint from the API description: take a request tenant A is authorized to make, replace every tenant-scoped identifier with tenant B's, and send it as A. That covers path parameters, body fields, query filters, batch arrays, and nested references inside a payload: the last two are where the checks usually go missing.
Assert on three things, in the order they catch leaks:
- Body. No field belonging to B's object appears in the response.
- Status. 403 and 404 are both acceptable, but the same endpoint must not answer 404 for an ID that does not exist and 403 for a real ID belonging to B. That difference is an existence oracle, and it is enough to enumerate another tenant's object IDs.
- Timing. A response that returns in 3 ms for B's real ID and 40 ms for a random one leaks existence even when the status codes match.
Run the suite in the pipeline and fail the build on any of the three.
Why the specificity matters
A new bulk endpoint shipped without a substitution case. It returned 30 records belonging to another tenant when identifiers were mixed, and the gap survived 21 days until a customer recognized a name they had never seen.
| Endpoint set | Cases | Failures found |
|---|---|---|
| single-object reads | 62 | 0 |
| bulk operations | 18 | 1 |
| exports | 9 | 2 |
What the suite does not cover
Substitution cases test the request path. Four leaks live outside it and need their own cases:
- Second-order references. An object A stores that points at B's ID, resolved later by a background job or an export. The request that created it was clean; the job that reads it is not.
- Caches keyed without the tenant. A shared cache entry written for A and served to B never touches the authorization check at all.
- Admin and support paths, and service accounts. A privileged caller acting on behalf of a tenant is the confused deputy, and it is not covered by sending requests as A.
- Async work. The request returns 202 and the leak happens in the worker, after the test has already asserted on the response.
For production, log the tenant on the request and the tenant on every row returned. A mismatch is an alert, not a metric: it means the suite has a hole.
Isolation is settled when the substitution suite runs across every endpoint and returns zero responses containing another tenant's data.
Curated: · Written: · Reviewed:
Q&A-61How do you review a policy change before it reaches production?
First 90 seconds
Assume policy is code (Rego, Cedar, IAM JSON, or RBAC YAML) versioned in git, with the policy decision point logging every decision (principal, action, resource, context). Under those assumptions the reviewable artifact is not the diff text; it is the change in decisions. So the core of the review is a decision diff: replay a recorded sample of real decision inputs against the old and new policy, classify each request as newly allowed, newly denied, or unchanged, and gate the merge on the newly allowed set. Text review still happens, but as a secondary check on top of that.
Producing the decision diff
Pull a sample of decision inputs from the PDP logs: a day or a week, stratified so rare principals and actions are not drowned out. Evaluate each input against both policy versions. The output is a table like this (illustrative figures):
| Category | Requests | Handling |
|---|---|---|
| newly allowed | 41,000 | block merge, escalate to policy owner |
| newly denied | 260 | review; may be intended tightening or a broken path |
| unchanged | 1,240,000 | sampled, no action |
Newly allowed is the dangerous direction: it is silent privilege expansion. Newly denied is usually intended, but it can break a live path, so I check whether any of those principals hit the endpoint in the last 24h.
The replay harness is small:
# pseudocode; the engine exposes evaluate(policy, input) -> allow/deny
for inp in sample:
old = evaluate(old_policy, inp)
new = evaluate(new_policy, inp)
if old != new:
bucket[old, new].append(inp)
The rest of the review
- Static checks in CI: lint the policy, reject wildcard actions or resources, flag any change to a deny rule.
- Unit tests: policy-as-code engines (OPA, Cedar) ship test frameworks; every new rule lands with allow and deny cases.
- Approval: a policy owner signs off, scaled to the size of the newly allowed set. A handful of requests is a reviewer's judgement call, tens of thousands needs the owner and a written justification.
- Rollout: deploy the policy bundle as data, canary it to one service or region, keep the previous bundle pinned for one-command rollback.
Trade-offs, and when text review is enough
Replay costs: capturing and storing decision inputs is real volume, replaying a week of traffic takes minutes to hours, and the sample goes stale. A change merged today is replayed against last week's principals, so a service principal created yesterday will not appear. For a purely additive change with no wildcards and a small blast radius, adding a read-only role for a new team, text review plus unit tests is proportionate; the decision diff buys little.
Failure modes
- A "behaviour-preserving" refactor that granted a wildcard action to 3 service principals. The text diff was 12 tidy lines; the decision diff showed 41,000 newly allowed requests in the sample. Text review passed it.
- Non-determinism: if the policy reads external data (group membership, resource tags), replay must pin that data at a fixed point, or old and new disagree for reasons unrelated to the change.
- Sample blind spots: principals or actions absent from the sample show as unchanged. I would report coverage (what fraction of distinct (principal, action) pairs the sample touches) rather than trust a clean diff.
What would change my answer
If the engine has no decision logging, replay is unavailable and I would fall back to unit tests plus a canary with deny-by-default and close monitoring.
Curated: · Written: · Reviewed:
Q&A-62What does an authorization log need to be useful six months later?
The 90-second answer
A useful authorization log answers "why was this allowed?" for a decision made six months ago, when both the policy and the attributes have since changed. That means the record has to be replayable: from the log alone you can re-run the decision and get the same outcome. Outcome-only logging (user, resource, allow/deny) cannot do this, because the two inputs that actually decide the answer, the policy in force and the attribute values read, are gone.
The minimum record: subject identity, action, resource identity, the decision, the policy bundle version (content-addressed), the identifier of the rule that fired, the attribute values the decision read, the engine version, and a timestamp with timezone plus a correlation id.
The record (illustrative shape, not a vendor schema):
{
"ts": "2025-03-11T14:02:19.442Z",
"decision_id": "01HQ8Z...",
"trace_id": "b7f1c9...",
"subject": {"id": "user:8412",
"attrs": {"dept": "finance", "clearance": "L3", "region": "eu"}},
"action": "read",
"resource": {"id": "doc:99231",
"attrs": {"classification": "confidential", "owner": "user:1022"}},
"decision": "deny",
"policy": {"bundle": "authz", "version": "sha256:9c1f...",
"rule": "finance-confidential-read"},
"engine": "pdp-2.4.1",
"context": {"authn": "oidc", "session": "s-77ab", "ip": "203.0.113.7"}
}
Why each field earns its place
| Field | Why replay needs it | What breaks without it |
|---|---|---|
| Policy bundle version (content hash) | Pins the exact ruleset | A reused version number points at different content after an in-place edit |
| Deciding rule id | Shows which statement fired | You know the outcome, not the reason |
| Attribute values read | Replay uses values as of decision time | Replay reads today's attributes and returns a different answer |
| Engine version | Semantics shift between releases | Same inputs, different result |
| Timestamp + timezone | Ordering and retention windows | Skew makes the sequence unreconstructable |
| Correlation id | Joins the decision to the request, session, token | You cannot tie a decision to the call that triggered it |
Trade-offs
Attribute capture is the expensive part. The always-on core (ids, action, decision, policy version, rule id) is small; full attribute snapshots are large. Sampling snapshots keeps cost down but leaves unsampled decisions unreplayable, so sample and always capture full attributes for privileged or sensitive resources.
Retention has to be matched. Keeping decisions for seven years while pruning policy bundles after 90 days makes the old decisions unreplayable; retain bundles at least as long as the decisions referencing them.
Attribute values can be personal data. Values only ever compared for equality can be hashed or tokenized; range and set-membership checks need the raw value to replay. Decide per attribute.
Illustrative arithmetic: at 200M decisions/day, a ~300 B core record is ~60 GB/day. A full ~1 KB snapshot is ~200 GB/day. Core plus a 5% full snapshot averages ~335 B, ~67 GB/day, roughly 24 TB/year versus 73 TB/year for full capture: a ~3x saving, not 20x, because the core dominates.
Failure modes and how you catch them
- Log holds user/resource/outcome only. Caught by the replay test below, which fails on the first record.
- Policy edited in place under a stable version number. Caught by content-addressing bundles and alerting when a version's hash changes.
- Attribute references logged instead of values. Caught by replaying a decision whose subject's attributes have since changed; the outcome differs.
- Retention mismatch between decisions and bundles. Caught by a config assertion that bundle retention is at least decision retention.
- Clock skew across PDP replicas. Caught by NTP monitoring plus a monotonic sequence number per node.
The test that settles it
Pick decisions from a past month, re-evaluate each from the log alone, and compare outcomes. A non-zero mismatch rate on decisions whose inputs have not been pruned means the log is not fit for purpose.
A decision you cannot replay is an opinion.
Curated: · Written: · Reviewed:
Q&A-65A team needs an exception to a policy for a migration. How do you grant it?
What I'd say first
An exception is not a policy edit. It's a second, narrower grant that expires on its own, so the default answer to "can we have an exception" is yes: with a scope, an owner, a compensating control, and a date. Before granting anything I ask three things: what is the smallest permission that unblocks the migration, how long does the work actually take, and who is accountable when it lapses. If the first answer is "admin" or the second is "until we're done", the exception is the wrong tool and I'd push back rather than grant it.
When an exception is the wrong tool
| Option | Use when | Cost |
|---|---|---|
| Change the policy | The policy is wrong for everyone, not just this team | Review and rollout; slowest |
| Break-glass | Unplanned incident, no time to scope | Loud, alarmed, post-hoc review |
| Redesign the ask | Permission is broad or open-ended | Engineering time now, no standing risk |
| Time-bound exception | Planned, bounded, one principal and resource | Register and renewal overhead |
A migration is planned work, so break-glass is the wrong fit: break-glass exists for the 3 a.m. page, and spending it on scheduled work burns the alarm's credibility. If the team needs the permission permanently, that's a policy change with a review, not an exception.
How I'd implement it
Enforce the expiry in the engine, not in a calendar reminder. Google Cloud IAM Conditions take a request.time expression; Azure PIM eligible assignments cap activation duration (default 8 hours) and require justification and approval; AWS has no native expiry on a policy, so you grant a role with MaxSessionDuration and hand out sts:AssumeRole with --duration-seconds, or attach a permission boundary the exception role cannot exceed.
{
"title": "migration-write-window",
"expression": "request.time < timestamp('2025-03-14T00:00:00Z')"
}
Kubernetes RBAC has no expiry field at all: a RoleBinding is permanent until someone deletes it. For a cluster migration I'd issue short-lived OIDC credentials instead of a binding, so the grant dies with the token.
Scope it to one principal, one resource, one verb set: not roles/storage.admin on the project, but roles/storage.objectCreator on the single staging bucket. The owner is a named person in the register, not a team alias. Compensating control: the exception role's actions go to a separate log sink with an alert on any write outside the migration prefix.
Renewal: one renewal, requiring a fresh justification and the same approver. A second request means the migration plan is wrong, not the policy.
Failure mode and detection
The pattern I've watched: a broad write grant opened for ten working days, renewed by email four times, still live 15 months later, on a system whose owner had left. Nothing detected it because nothing was looking. Detection is a scheduled query over the register: every active exception must have an expiry in the future, an owner still present in the directory, and a renewal count under the cap. Anything else pages the owner's manager.
An exception without an expiry is the new policy.
Curated: · Written: · Reviewed:
Q&A-68What is your revocation service level, and how do you know you meet it?
The judgement in end-to-end access-removal latency is which identifier is durable and which is only a convenient claim.
Revocation latency runs from the source-of-truth event to the first denied action at the target, not to the ticket closing. Every hop adds to it (connector schedule, cache, session lifetime, token expiry) and the number is credible only when it is measured on real targets.
Concretely, keep a synthetic identity attempting a benign privileged action at each tier on a schedule, trigger a termination event, record the timestamp of the first denial per target, publish the distribution rather than a mean, and treat targets with no measurement as unmeasured rather than compliant.
The reason for that specificity is a failure I have seen: A stated one-hour objective was reported as met from ticket timestamps. Measured first denial at the third-party expense tool arrived 4 days and 6 hours later, and 2 of 9 tiers had no measurement at all.
Measured minutes to first denial.
| Target | Claimed | Measured |
|---|---|---|
| directory | 5 | 6 |
| core finance application | 60 | 74 |
| expense tool | 60 | 6,120 |
| data warehouse | 60 | never measured |
I would not consider it settled without evidence: run the synthetic termination monthly and publish the minutes to first denial for every target.
Measure at the target, not at the ticket.
Curated: · Written: · Reviewed:
Q&A-74We have a password vault. Do we have privileged access management?
The 90-second answer
A vault is one component of PAM, not PAM. If the only thing the system does is store a credential and hand it to someone authorised to read it, you have secrets management. PAM is the governance around the session that uses the credential: who requested it, who approved it, what ran inside it, and what state the credential is left in. So the honest answer is usually "we have vaulting; whether we have PAM depends on whether sessions are brokered."
What to check
| Question an auditor asks | Vault only | Vault + brokered sessions |
|---|---|---|
| Who held the credential at 14:03? | yes (retrieval log) | yes (session log) |
| Who approved that access? | no | yes |
| What commands ran? | no | yes, if the proxy records |
| Was the credential rotated after? | only if rotation is automated | yes, on session end |
| Can the operator keep a copy? | yes | no |
A brokered session, traced
- Operator authenticates to the broker with SSO + MFA; the broker checks group membership.
- Operator requests role
prod-db-admin, scope 30 minutes, with a ticket reference. - An approver (or a policy) grants it; the broker mints a short-lived SSH certificate: principal
prod-db-admin, TTL 30 minutes, source address bound to the broker. - The operator's
sshgoes through the proxy; the terminal stream and every command are written to object storage. - At TTL expiry the certificate is dead. The target account's password was never on the operator's machine, so there is nothing to rotate on the laptop.
Contrast with vault checkout: the password is returned to the operator's shell, where it lives in shell history, process memory and possibly the clipboard, and stays valid until the next rotation.
Failure modes worth naming
- Break-glass outside the broker. The one shared local admin password that never rotates. Detect by enumerating accounts with a direct network path to production and no broker in that path.
- The vault admin is the new privileged tier. Whoever holds unseal keys or root policy can read everything. Detect by counting humans with unseal/rekey or root policy.
- Recording gaps. RDP clipboard, file transfer and out-of-band channels are not captured by a command logger. Detect by replaying a session and trying to move a file out.
- Approval fatigue. Blanket standing approvals are checkout with extra steps. Detect via median approval latency and the share of requests auto-approved.
- Rotation breaking things. Rotating a service account a cron job uses with a static password. Detect via rotation failures and post-rotation auth errors.
Why it matters, in numbers (illustrative): 2,100 credentials readable by 90 people on a 90-day rotation means one compromised operator laptop exposes roughly 40 credentials for up to 90 days. About 3,600 credential-days of exposure. Brokered, the same compromise yields one 30-minute session and a credential rotated at the end: roughly 0.02 credential-days.
Where vault-only is correct. Non-human secrets (CI tokens, application config, a service's database password) do not need session brokering. They need dynamic, short-TTL credentials issued per process. PAM is about humans holding privileged sessions.
Storing the key is not governing the door.
Curated: · Written: · Reviewed:
Q&A-69Why remove standing administrative rights if the people holding them are trusted?
Trust is the wrong variable. A standing right is exercised by whoever holds the session, not by the person you vetted at hire, so the question isn't whether the admin is trustworthy, it's what happens the moment their credential isn't. Removing standing rights converts a stolen credential from immediate privileged access into a request that has to clear an approval, and it leaves a record of every privileged interval. That's the case for removal. It is not free, and the costs are where most rollouts quietly fail.
Mechanism. A standing role is a permanent grant bound to an identity. Phishing, a stolen session cookie, a token lifted off a laptop, a leaked CI secret: any of these inherits the role with no gate in front of it. Just-in-time access replaces the grant with a request carrying a justification, an approver, a maximum duration, and automatic removal. The audit trail changes shape: from "this account has been a domain admin since 2019" to "elevated 14:02–14:47, ticket OPS-3312, approved by j.okafor." The everyday account stays unprivileged; the role is attached only for the window.
What JIT costs.
- Approval latency. An approver in the loop adds minutes. At 03:00 during a page, that's the difference between a five-minute fix and a forty-minute one. Measure it: if p95 approval time exceeds the cost of the outage it's blocking, that role needs a pre-approved or break-glass path.
- Approver bottleneck. One approver per timezone is a single point of failure, and a queue that auto-approves after N minutes is standing rights with extra steps.
- Emergency paths. Break-glass accounts must exist, sit outside the JIT system, have credentials vaulted or split, alert on every use, and be rotated afterwards. Break-glass that is never tested is a standing right with a nicer name.
- Automation. Service accounts, scheduled jobs and CI runners cannot click approve. They keep standing rights by necessity: scoped, non-interactive, bound to a workload identity or short-lived token, and monitored.
- Hoarding. If activation is painful, engineers keep a second privileged account or a cached token, and you have moved the standing right somewhere you cannot see it.
When standing rights are acceptable. When there is no interactive principal (service or automation), when no human approval can arrive inside the required latency, when the blast radius is bounded and monitored, and when the credential is hardware-bound or short-lived. Each such assignment belongs on a named exception list with an owner and a review date.
Failure modes, and how you'd catch them. Rubber-stamp approvals: look at the approval-time distribution per approver. A median under ten seconds means the gate is decorative. Elevation windows left open: alert on any role still active past its maximum duration. Break-glass drift: alert on use and on the account's last-rotation date. And the JIT system itself becomes a crown jewel: it needs its own privileged-access story, or you have just relocated the problem.
Illustrative figures from a representative environment, before and after:
| Role | Standing holders before | After |
|---|---|---|
| domain administrator | 34 | 2 break-glass |
| cloud tenant owner | 12 | 0 |
| database owner | 46 | 3 |
What the change buys, traced on a hypothetical compromise:
| t | with standing rights | with JIT |
|---|---|---|
| 0:00 | attacker phishes an admin workstation, steals the session token | same |
| 0:04 | token replayed; no MFA prompt, session already authenticated | token replayed; session carries no privileged role |
| 0:09 | two service accounts created with domain-wide delegation | elevation request filed, lands in approval queue, alerts |
| 0:18 | scheduled task registered to re-run the payload; no elevation event exists to review | request still pending; nothing privileged has executed |
I would not call it settled without evidence: count the accounts holding a privileged role outside an active elevation window, and require that list to match the documented exceptions exactly. Trust the person; do not leave the rights lying on the desk, but budget for the approval latency and the emergency path, or the desk just moves.
Curated: · Written: · Reviewed:
Q&A-71Why should an administrator have two accounts?
Core answer
Yes. An administrator should hold two identities because the account that reads mail and browses the web is the one most likely to be compromised, and it must not be the account that can change the directory. The point is not the protocol; it is shrinking the blast radius of a stolen credential and making privileged activity legible.
The daily account keeps a mailbox, a browser, and normal endpoint access. The admin account has no mailbox, no general browsing, its own strong authenticator, and sign-in restricted by conditional policy to hardened workstations. Everything that identity does was meant to be privileged, so its logs are signal.
Mechanism
Two effects. Exposure: phishing, malicious documents, and browser exploits land on the daily identity, which holds no directory rights. Attribution: privileged actions come from a single-purpose identity, so an alert on it means something.
Costs and alternatives
The cost is real: two credentials, two MFA registrations, context switching, helpdesk resets, and per-admin licensing in some suites. The failure that erases the benefit is admins staying signed into the privileged account all day. Separation without enforcement is theatre, if you cannot restrict the admin account's sign-in surface, the second account mostly adds friction.
Just-in-time elevation is the main alternative and often the better one: Entra ID PIM, AWS IAM Identity Center permission sets, or sudo with time-bound rules grant rights for minutes instead of standing. There is no long-lived privileged credential to steal. The strongest posture combines both: a separate admin identity that holds no standing rights and elevates on demand with approval.
When it does not apply: a single-admin shop with no directory tiering and no conditional-access capability gets little from a second account; spend the effort on MFA and tested backups instead. Whatever you choose, keep one break-glass account excluded from conditional policy, stored offline, monitored, and tested: otherwise a policy mistake locks you out during the incident.
Failure modes
- Password reuse between the two accounts collapses the separation.
- Privileged account left signed in on a daily workstation.
- Break-glass account never tested, found broken mid-incident.
- MFA fatigue or push bombing on the admin identity.
Artefacts
| Property | Daily account | Admin account |
|---|---|---|
| Mailbox | yes | none |
| General browsing | yes | blocked |
| Sign-in surfaces | all managed endpoints | hardened workstations only |
| Standing directory rights | none | none: elevate JIT |
| Authenticator | password + MFA | phishing-resistant (FIDO2) |
Hypothetical trace, one shared account vs two:
- Admin opens a malicious attachment on the daily identity.
- Code runs as that user.
- Shared account: the process already holds directory admin rights, so the attacker creates a new admin user and persists.
- Two accounts: the process is a standard user; directory writes are denied, and the attacker must separately compromise a phishing-resistant identity that cannot sign in from that endpoint.
Hypothetical arithmetic: if a click compromises the daily identity in 1 of 100 attempts, a shared admin account turns that into full directory control once per 100 clicks. Separation reduces the same click to a standard-user foothold and forces a second, harder compromise.
Curated: · Written: · Reviewed:
Q&A-70A team needs to restart services in production. What role do you give them?
I would answer just-enough administration scope by separating authentication, authorization, and lifecycle into three control planes.
Privilege should be scoped to the operations the job requires rather than to the container those operations live in. Handing over the whole administrative role because it happens to contain the needed action is how a restart permission becomes a permission to change authentication.
Concretely, identify the specific operations, assemble a role from those alone, scope it to the resource group or organisational unit involved, expose it through a constrained endpoint permitting the named commands, and review the operation list rather than the role name.
The reason for that specificity is a failure I have seen: An operations team was given a full administrative role to perform restarts. A scripted mistake used an unrelated capability of the same role, deleted 6 service principals, and broke authentication for 3 applications for 5 hours.
Role composition.
| Approach | Operations granted | Blast radius |
|---|---|---|
| full administrative role | 214 | the whole tenant |
| scoped custom role | 4 | 1 resource group |
| constrained endpoint | 4 commands | 9 named hosts |
I would not consider it settled without evidence: attempt an operation outside the named list with the scoped role and require it to be refused.
Grant the verb, not the container it arrived in.
Curated: · Written: · Reviewed:
Q&A-72Who approves a privileged elevation request?
The approver is the owner of the target the elevation reaches, the person accountable for that system or role, not the requester's manager by default. For the top tier (directory/domain admin, cloud root, production data write), two approvers from different reporting lines, one of them the security team's delegate. Break-glass is the exception: security on-call approves, and the request is reviewed retroactively inside a fixed window.
That is the answer; the rest is which model to pick and where each one breaks.
| Model | Who signs | Good when | Fails when |
|---|---|---|---|
| Manager-only | Requester's manager | Low tier, time-boxed, non-production | Manager has no visibility into the target; approves on trust |
| Role owner | Owner of the target system/role | Default for application admin and data tiers | Owner is the requester, or reports to them |
| Security gate | Security team delegate | Top tier, cross-cutting roles, break-glass | Becomes a rubber stamp if the queue is deep and SLAs are short |
| Two-person rule | Owner + security, different lines | Directory tier, root, production write | Slow for incidents; needs a break-glass path or people route around it |
Lighter approval is acceptable when the elevation is already covered by a change record the owner approved, when the grant is read-only and expires with the session, or during a declared incident where the on-call approver signs and the review happens after. It is not acceptable for standing membership in a privileged role: that is a permanent grant and needs the full path.
A request flow, tier 2 (application administration), single approver:
- Engineer on Payments opens a request for role
prod-db-admin, duration 4h, linked to a change ticket. - Policy lookup resolves the role owner: the Payments DB platform lead. Tier 2, one approver, must not be a report of the requester.
- Request routes to the owner's queue, not to the requester's manager, who has no authority over that database.
- Owner approves. Grant issued with an expiry four hours out; the audit record stores requester ID, approver ID, role, ticket, and expiry.
- If the requester were the owner, or the owner reported to the requester, policy refuses the route and escalates to the security delegate.
Failure modes and how each is detected:
- Self-approval. Requester sits in the approver group for the role they want. Detect at request time with a policy check, and in audit by joining approver-group membership against requester ID.
- Reciprocal pairs. A approves B's elevation, B approves A's. Illustrative shape: A approves B on 12 March, B approves A on 19 March, repeated. Detect by building a directed graph of approver→requester edges over 90 days and flagging 2-cycles with more than one edge.
- Approver-group bloat. Membership grows until the group is everyone senior. Detect with quarterly recertification and a membership-count alert per role.
- Break-glass without retro approval. Detect by alerting when a break-glass grant has no recorded approver within the review window.
Verification is cheap and worth stating: in a test tenant, attempt self-approval and a reciprocal approval and require both to be refused before calling the control live.
An approval the requester can produce is a formality, not a control.
Curated: · Written: · Reviewed:
Q&A-73Your privileged access tool checks out a password for four hours. What would you change?
What I'd say first
The four-hour checkout is the wrong primitive, not the wrong number. A checked-out password is a copyable secret, and its lifetime is a policy setting rather than a fact: once it's in a clipboard, a shell history, or a notes app, the only thing that ends its usefulness is rotation actually working on the target. So the change is to hand over a session, not a secret, issue a credential bound to the session that expires on its own, and treat the four-hour window as a symptom of that.
Assumptions I'd state up front: the tool brokers access to a fleet of targets (Linux hosts, databases, network gear), the operator is a human with an interactive session, and there's an internal CA or secrets engine I can extend. Any target that can't do short-lived credentials gets a proxy in front of it, not a password.
The change
- Replace password checkout with short-lived certificates or tokens. For SSH, sign a certificate per session with the principal and validity window encoded; the operator's private key stays in their agent, and the certificate is worthless once it expires.
ssh-keygen -s /ca/user_ca -I alice-8f3c -n alice -V +20m alice.pub
# sshd: TrustedUserCAKeys /etc/ssh/user_ca.pub
-
Where a password is unavoidable (legacy appliances, mainframes), cut the TTL to minutes and make it single-use: rotate on checkout and again on check-in, and verify the rotation.
-
Never display a credential the operator can copy. The broker injects it into the session (a proxy, or a one-shot credential written to a pipe) so there's nothing to paste.
-
Bind the credential to the session: source address, target, and a short window, so a leaked certificate can't be replayed from somewhere else.
Why rotation is the fragile part
The failure I'd design against: a four-hour checkout pasted into a notes app, still working six weeks later because rotation silently failed on 40 of 380 accounts and the failure only showed up in a log nobody read. Rotation is a distributed operation: the broker rotates the credential, and every consumer of that credential has to pick up the new one. Any consumer that caches it (a service account, a cron job, a config file) either breaks the rotation or keeps the old secret alive. Certificates sidestep this: there's no shared secret to rotate, and expiry is enforced by the target at authentication time.
Comparison
| Model | Copyable | Usable after session | Depends on rotation |
|---|---|---|---|
| 4-hour password checkout | yes | yes, if rotation fails | yes, per account |
| password, minutes, single-use, verified rotation | yes | no, if rotation verified | yes, verified |
| 20-minute session certificate | cert yes, key no | no | no |
Failure modes and detection
- Broker or CA compromise: the broker now mints credentials for everything. Keep the signing key in an HSM, keep the CA key offline, and log every issuance.
- Rotation drift: run a reconciliation job that authenticates with the old credential and asserts failure; alert on any success. That's the check that would have caught the 40 accounts.
- Clock skew: a 20-minute certificate fails if the target's clock is off. NTP is a dependency; monitor it.
- Break-glass: keep one audited, alerted emergency account, or you'll get a shared root password in a wiki.
Verification
I wouldn't call it done without evidence: issue a certificate for a session, confirm it authenticates, wait past expiry, and confirm the same certificate is rejected at the target. Then force a password-path rotation to fail and confirm the reconciliation job alerts.
Curated: · Written: · Reviewed:
Q&A-75A legacy system has one administrative account that four people use. What do you do before you can retire it?
The short answer. Before you can retire that account, every action taken through it has to be attributable to a named person. So the work is to make the shared credential unusable by a human while you build the per-user path that replaces it, and to run those two tracks at the same time, not one after the other.
Assumptions I'd state up front. The four users are all legitimately authorized to administer this system. The system itself has no native per-user account model (no LDAP bind, no SAML or OIDC) which is usually why the shared account exists in the first place. If either assumption is wrong, the plan changes.
Track 1: take the credential out of human hands. Move it into a broker. A PAM vault (CyberArk, BeyondTrust), Teleport, or HashiCorp Vault's credential engine. The vault holds the password. A person authenticates to the vault as themselves, requests a session, and the vault injects the credential into the connection. Rotate the credential after every session, so a copy someone kept is dead within minutes. Record every session and bind it to the requesting identity.
That last point is the whole trick: attribution now comes from the vault's session record, not from the target system's logs. The target system will still only ever say "admin", but you no longer need it to say anything else.
Track 2: build the per-user path. Either native accounts on the target, or a federation shim in front of it. Give it a date and a named owner, or it never ships.
| Control | Shared account today | After brokering |
|---|---|---|
| Who holds the credential | four people | the vault |
| What the target system sees | admin | admin |
| Who the audit trail names | nobody | the vault session's requester |
| Credential lifetime | until someone changes it | one session |
Why not just add four accounts? If the system supports it, do that instead: it's strictly better and brokering is unnecessary complexity. Brokering is the answer when the app can't do per-user auth and you can't replace it this quarter. The cost is a new single point of failure: if the vault is down, nobody administers the system. That needs a documented break-glass path, and break-glass use needs to page someone.
Failure modes to watch for.
- The vault login becomes the new shared account. Detect it by alerting on two concurrent sessions from one identity.
- Someone keeps a copy of the credential and bypasses the vault. Detect it by alerting on any direct authentication to the target that has no matching vault session.
- Automation embeds the credential. Inventory service accounts before you rotate, or you'll break a nightly job at 2 a.m.
How you know it's done. Pull the target system's auth log for the last 30 days. Every session must map to a vault session with a named user. Any session that doesn't is a gap, and the retirement date slips until that count is zero.
Illustrative example: a shared admin login on a manufacturing control system, used by four engineers and two vendors. A configuration change stopped a line for nine hours; the investigation couldn't say who made it, and the corrective action was a reminder to follow policy. Brokering wouldn't have prevented the change: it would have named the person who made it.
Curated: · Written: · Reviewed:
Q&A-76You record privileged sessions. What must never end up in the recording?
What must never be in the recording
Six categories, ordered by how often they actually leak:
- Credentials typed at an echo-off prompt.
sudo,ssh,mysql -p,psql, anopensslpassphrase. The tty disables echo, so a screen recording shows nothing, but a recorder that captures stdin captures the characters anyway. - Secrets on the command line.
export AWS_SECRET_ACCESS_KEY=…,curl -H "Authorization: Bearer …",docker login -p,psql "postgres://user:pw@host/db". - Secrets printed by a command.
cat ~/.aws/credentials,kubectl get secret -o yaml,env,cat /etc/app/config.yml. - Session material that is itself a credential. Cookies, Kerberos tickets, SSH private keys, kubeconfigs, TOTP codes. A 30-second TOTP code recorded next to a session cookie is replayable for the rest of that window.
- Third-party personal data. Another tenant's records in a shared admin console, card PANs, health data. The admin is authorised for the action, not for every record that scrolled past.
- The archive's own keys. Never record the session in which someone retrieves the decryption key for the archive.
Why the filter has to run at capture
A nightly scrub leaves the secret on disk for the scrub interval, up to 24 hours, inside an archive that other people can read. Capture-time masking makes that window zero. The cost is that you must know the pattern in advance, and the masked region is gone for forensics too. Run both: mask at capture, scan the assembled recording afterwards as a backstop.
| Must never be recorded | How it gets in | Control |
|---|---|---|
| Password at an echo-off prompt | stdin capture, not screen | prompt-aware masking: on Password:, mask input until newline |
| Secret on the command line | shell stdin | regex on the input stream at capture |
| Secret printed by a command | command output | command denylist plus output scanning |
| Cookie, ticket, private key | pasted or displayed | high-entropy detection plus format match |
| Another tenant's PII | shared console screen | per-tenant session isolation; OCR scan as backstop |
| Archive decryption key | key retrieval on camera | retrieve out-of-band, never inside a recorded session |
# Python 3.12 — applied to the stdin byte stream before it reaches the cast file
import re
MASK = b"[REDACTED]"
PATTERNS = [
re.compile(rb"(?i)(password|passwd|secret|token|api[_-]?key)[A-Z_]*\s*[=:]\s*\S+"),
re.compile(rb"AKIA[0-9A-Z]{16}"),
re.compile(rb"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
re.compile(rb"(?i)authorization:\s*bearer\s+\S+"),
]
def scrub(chunk: bytes) -> bytes:
for p in PATTERNS:
chunk = p.sub(MASK, chunk)
return chunk
Trace on b"export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n": pattern 1 matches SECRET_ACCESS_KEY=wJalr…EXAMPLEKEY (secret + [A-Z_]* eats _ACCESS_KEY, then =, then \S+ runs to the newline) and substitutes, giving b"export [REDACTED]\n". Patterns 2–4 find nothing. Note what this does not catch: hunter2\n typed at a prompt matches no pattern at all, which is exactly why prompt-aware masking is a separate control rather than a regex.
Failure modes and how you see them
- Chunk-boundary split. A secret straddling two
read()calls escapes per-chunk scrubbing. Detect by scanning the assembled recording, not just the stream. - Masked in the cast, plaintext in a side file. Recorders that keep a raw capture "for debugging" defeat the whole control. Detect by auditing what the agent writes, not what it claims to write.
- Prompt text changes. A new tool prints
Passphrase for key:, the prompt list misses it, and the input is unmasked. Detect with a canary: type a known fake secret at every prompt in a monthly test session and assert zero hits in the stored recording. - Retention outliving the credential. A recording of a password that was rotated last week is inert; one from a session where the credential is still valid is live. Detect by joining archive age against credential rotation dates.
Hypothetical sizing: 150 admins × 90 min/day × 250 days is 56,250 hours a year; a gzip'd text cast at roughly 1 MB/hour is about 56 GB/year, so 180-day retention is ~28 GB. Storage is not the constraint: the constraint is that every extra day of retention is another day a missed secret stays readable.
A recording of an administrator is an administrator's credential.
Curated: · Written: · Reviewed:
Q&A-77Your identity provider is unavailable. How does anyone get in to fix it?
I would treat break-glass independent of the failed IdP as a claim about effective access that has to survive a disablement.
A break-glass path that authenticates through the system it exists to rescue is not a path. It has to be independent of the federation, of the multi-factor service, and of the network controls that could be the thing failing.
Concretely, keep a small number of non-federated accounts with long random credentials split between sealed physical copies, exclude them from conditional policies that depend on the provider, alert on any sign-in to several channels at once, exercise the path on a schedule, and rotate after every use and every test.
The reason for that specificity is a failure I have seen: Both emergency accounts were federated to the provider that failed. The tenant was unreachable for 5 hours and 40 minutes while support restored a trust relationship, and the runbook had not been exercised in 2 years.
Break-glass account properties.
| Property | Required | Found in review |
|---|---|---|
| not federated | yes | 0 of 2 |
| excluded from provider-dependent policy | yes | 0 of 2 |
| exercised in the last quarter | yes | last test 2 years ago |
| alerts on use | yes | 1 channel only |
I would not consider it settled without evidence: sign in with a break-glass account while the identity provider is blocked and confirm the path works and alerts.
The rescue path cannot depend on the thing being rescued.
Curated: · Written: · Reviewed:
Q&A-78Who administers the privileged access system itself?
Short answer: someone who does not use it. In practice a platform-engineering group of 2–4 people owns the PAM platform itself (CyberArk, BeyondTrust, Teleport, HashiCorp Vault), while policy authorship, approval and audit review sit with different groups. If you cannot staff that split, the fallback is two-person control on the dangerous settings plus an audit sink the platform admins cannot write to.
Assumptions I would state before answering: self-hosted or SaaS, how many privileged accounts the platform brokers, whether you are in PCI DSS / SOX / FedRAMP scope, and whether a break-glass path exists that does not go through the platform. The answer changes with each.
Why it is a separate duty. The platform has a control plane and a data plane. Whoever edits the policy engine can grant themselves any account it brokers; whoever edits the recording settings can erase the evidence of having done so. That makes the platform administrator a superuser over superusers, and NIST SP 800-53 AC-5 (separation of duties) and AC-6 (least privilege) both push the same way, as does PCI DSS Req 10's requirement that audit records be protected from modification.
Credible alternatives, and what decides between them:
| Model | Cost | Deciding criteria | Weakness |
|---|---|---|---|
| Dedicated admin workstation + separate IdP for platform admins | hardware, ~1 FTE | >500 brokered accounts, regulated scope | the PAW becomes the new single point |
| Two-person control on policy and recording changes | process only | 50–500 accounts, small team | rubber-stamping if the same pair always approves |
| SaaS vendor administers, customer holds keys | licence premium | no in-house platform skills | vendor support account is standing access |
| Same team administers and uses | free | <50 accounts, no regulated data | survivable only with an external log sink and monthly review |
Deciding criteria, in order: blast radius (how many accounts the platform brokers), regulatory scope, headcount, and whether a break-glass path exists outside the platform. When lighter controls suffice: under roughly 50 privileged accounts, one cloud provider, no regulated data. A two-person rule on recording and approval-rule changes, logs shipped to a store the admins cannot write to, and a quarterly review. Skip the dedicated workstation; it buys little at that size.
Failure mode, and how you detect it. A platform administrator disables recording for one target group, uses the account, and re-enables it. Trace it:
- 14:02. Admin disables session recording for the
prod-dbtarget group. A config-change event is emitted. - 14:05–14:31. Admin checks out the shared DBA account and runs three statements. This target normally produces about 40 session events per hour.
- 14:33. Recording re-enabled.
- The platform's own log shows the config change but no session events for those 26 minutes, and the admin can trim that log.
- An external collector polls the platform's audit API every 5 minutes into append-only storage. It sees the config change on the 14:05 poll, then zero session events where 40/hr × 26/60 ≈ 17 were expected.
- Alert: "target with a baseline above 10 events/hour produced 0 for 15 minutes."
The collector's export credential must be read-only, and the collector must alert on export failure: otherwise disabling the export is the same attack one layer up.
Verification. Try to change recording settings with a single platform-administrator account; a second approver should be required. Then confirm the external sink shows the config change and that the platform's own administrators cannot delete it. Until both hold, the separation is on paper. The system that governs privilege needs its own separation.
Curated: · Written: · Reviewed:
Q&A-79A local administrators group on a server holds members the broker never granted. How did that happen, and what fixes it?
Core answer (first 90 seconds)
Assume the broker is meant to be the only writer to privileged local groups. Then the question splits in two: what other write path put the member there, and why didn't the broker's removal stick? The broker's record is a claim; the local group's membership is the fact. An entry the broker never issued came from a path it doesn't control, or from a removal it recorded but never applied.
How it happens: the four mechanisms I'd check first
- Direct local administration. An operator RDPs in and runs
net localgroup Administrators /add, or follows a break-glass runbook. No broker call is made, so no grant exists. - A second provisioning path. GPO Restricted Groups, Group Policy Preferences Local Users and Groups, an SCCM/Intune script, or a config-management playbook writes the same group. Worse, it re-adds the member after the broker removes it, so the drift reappears.
- Domain group nesting. The broker grants a domain group; someone adds a user to that group in AD directly. The broker has a grant for the group and none for the user, but the user is an effective local administrator.
- Removal that never applied. The broker marks a grant expired, but the agent was offline, the host was rebuilt from an image, or the removal job errored. The record says expired; the target still holds the member. Deleted accounts leave the same shape: an orphaned SID that no longer resolves to a name.
What fixes it
Reconcile against the target, not the broker. Enumerate effective privileged membership on each host, compare it to the grants the broker believes are active, and drive the unmatched set to empty. Then close the write paths: disable direct local administration where the platform allows, put local administrator passwords under LAPS so interactive admin isn't a standing path, and make the broker the only writer. Make removal idempotent and verified: the agent re-reads the group after removing and alerts if the member is still present.
A reconciliation across 610 servers found 143 local administrator entries the broker had never issued. 38 belonged to people who had changed teams, and the oldest dated back 3 years.
| Source of the entry | Count | Action |
|---|---|---|
| broker-issued and active | 402 | keep |
| broker-issued but expired | 27 | remove |
| never brokered | 143 | remove or convert |
How I'd call it settled
Compare target-side privileged membership against broker-issued grants and require the unmatched set to be empty, then re-run it a week later, because a second provisioning path will re-add what you removed. The target's membership is the fact; the broker's record is a claim.
Curated: · Written: · Reviewed:
Q&A-80A service needs to call another service. Why not give it an API key?
The 90-second answer
Ask two things before choosing. Can the callee verify who is calling, not just what secret they hold? And is the caller's runtime something you control and can attest? If both are yes, use workload identity. If the callee is a third party that only accepts a static key, an API key is the only option and the work moves to scoping and rotation.
An API key is a bearer secret: whoever holds the bytes is the caller. It has no expiry unless you build one, no binding to a workload, and no way to tell a legitimate call from a copy. Workload identity replaces the shared secret with a credential the platform issues to an attested runtime: a Kubernetes bound service account token, an AWS IRSA web-identity token, a SPIFFE SVID, so the credential and the workload expire together.
Mechanism
The caller presents evidence about its runtime (a projected service account token, an instance identity document, a node attestation) to an issuer. The issuer returns a short-lived credential scoped to an audience. The callee validates the signature, the audience and the expiry, and authorises on the subject claim. Nothing durable is stored on the caller, so there is nothing to copy out of an image or a config file.
When an API key is the right answer
- Third-party SaaS. Stripe, Twilio, Datadog accept a static key and nothing else. You cannot federate. Mitigate: scope the key to the minimum permission, keep it in a secrets manager, rotate on a schedule, restrict egress.
- Callers outside your trust boundary. A partner hitting your public API has no runtime you can attest. An API key plus per-key quotas and rate limits is the correct tool.
- Small blast radius, high ops cost. Two internal services, one key, rotated every 90 days by automation, stored in a secrets manager: the exposure window is bounded and the control plane you would need to run may cost more than the risk it removes.
What workload identity costs
- A control plane. An OIDC issuer, SPIRE server, or cloud IAM trust configuration, with its own HA, upgrades and trust-bundle distribution.
- A dependency on the issuer. If it is unreachable, new pods cannot get credentials. Cache tokens and refresh at ~80% of TTL; a thundering herd at expiry is a real outage.
- Trust-bundle rotation. When the issuer's signing key rotates, verifiers with a stale JWKS cache reject valid tokens. Overlap the windows.
- Harder debugging. There is no key to grep. You correlate the subject claim in logs.
- Clock skew. JWT
exp/nbfvalidation needs NTP; a few minutes of skew produces intermittent 401s.
Failure modes and detection
| API key | Workload identity | |
|---|---|---|
| Binding | none | attested runtime |
| Lifetime | until rotated | minutes to hours |
| Revocation | rotate everywhere | TTL bounds it |
| Blast radius | every holder | one workload |
| Ops cost | rotation + inventory | issuer + trust bundle |
Hypothetical: a key baked into a container image is pushed to a public registry. It stays valid until someone rotates it: with a 90-day rotation policy, up to 90 days of exposure, and the calls look identical to legitimate traffic. The same leak with a one-hour token is worth at most an hour.
Detect it by alerting on credential age exceeding the maximum lifetime, on issuer error rate, and on token-exchange p99.
What I'd want before calling it settled
An inventory of credentials with no expiry, and a date by which that count reaches zero. A credential that never expires cannot be lost safely.
Curated: · Written: · Reviewed:
Q&A-81The broker holds credentials for every target. What stops a request from naming the wrong one?
Assume the broker is a long-running service holding standing credentials (or able to mint them) for many targets, and that approvals are issued before the session starts. What stops a request naming the wrong target is that the target is fixed at approval time and the request cannot move it: the grant carries the target's stable identifier, the broker resolves the session against that identifier, and any target the request supplies is used only to find the grant, never to select the credential.
Mechanism. At approval, the approver or policy engine writes a grant record: {grant_id, principal, target_id, credential_ref, expiry}. target_id is a stable, non-user-controlled identifier (cloud instance ID, host UUID, resource ARN, SSH host-certificate principal) not a display name or hostname. At session start the broker loads the grant by ID, resolves target_id to a live endpoint, verifies that endpoint's host key or certificate against the one pinned in the grant, and only then injects the credential. If the request names a different target, or the resolved endpoint's identity differs from the grant, the broker refuses and mints nothing.
Alternatives, and when each is right.
| Design | Target chosen by | Cost | Right call when |
|---|---|---|---|
| Grant-embedded target | approver, at approval | one grant per target; group approvals need membership re-checked at session time | standing credentials, human approvals, bounded target count |
| Request-named target + policy check | requester, checked at session time | policy must run at the same point as injection; a check-then-use gap reopens the hole | dynamic fleets where pre-approving each target is impractical |
| No standing broker credentials (SSH CA, Vault dynamic secrets, cloud STS) | minting authority, scoped at mint time | every target must trust the minting authority; revocation is TTL-based | you can change target-side trust and blast radius must be minutes, not days |
| Per-target broker silo | deployment boundary | operational sprawl, N brokers | regulatory isolation between target classes |
Deciding criteria: can you change target-side trust, is the fleet dynamic, how long does a credential live, and what blast radius can you tolerate. Grant-embedded targets are the default because they need no target-side change; the cost is approval latency and stale grants.
Failure modes.
- Name-based matching. Hostname aliases, CNAMEs,
/etc/hosts, or two hosts sharing a display name. Detect: log the resolvedtarget_idnext togrant_id; alert on any resolution that falls back to a name. - TOCTOU on identifier reuse. An IP or instance ID recycled between approval and session. Detect: pin the host key or certificate fingerprint in the grant and verify at connect. A recycled ID fails the fingerprint check.
- Group grants. Approval for "prod-db group", membership changes before use. Detect: re-evaluate membership at session time, log the delta, refuse if the target joined after approval.
- Check-after-inject. Broker injects the credential, then verifies. Detect: order the code so verification precedes minting, and log the injection event with the verified fingerprint.
Trace.
Grant G-4471: principal=alice, target_id=i-0a3f (db-test-01), expiry 15:00
Request: POST /sessions {grant_id: G-4471, target: "db-prod-01"}
Broker: load G-4471 -> target_id=i-0a3f
resolve request target "db-prod-01" -> i-9c21
i-9c21 != i-0a3f -> 403, no credential minted
Log: grant_id=G-4471 requested=i-9c21 resolved=i-0a3f decision=deny
Had the request omitted target entirely, the broker still resolves i-0a3f from the grant and connects there: the request can neither widen nor redirect the grant.
To call it settled I'd want two checks: approve a grant for one target, attempt a session naming another, and see the broker refuse with no credential minted; then swap the target's host key and see the fingerprint check fail.
Curated: · Written: · Reviewed:
Q&A-82A vendor engineer needs privileged access for a support call. How do you set that up?
The short version
Assume the vendor is a third party under an MSA, the support call is scoped to named systems, and you have an IdP that can federate. The answer: never issue a shared credential. Create a named identity for the specific engineer, tie its lifetime to the ticket, broker the session through a recorder, and let it expire without anyone remembering to revoke it.
Mechanism
Three planes, each with its own control:
- Authentication: the vendor engineer authenticates as themselves. Best case, federate to the vendor's IdP (SAML/OIDC) so you inherit their joiner/mover/leaver process; if they can't federate, create a guest identity in your IdP and require a phishing-resistant second factor you control (FIDO2/WebAuthn), not SMS.
- Authorization: access is granted per ticket, not per contract. An access package or entitlement with an expiry equal to the ticket window, scoped to the named hosts or roles. Just-in-time elevation, not standing membership in a privileged group.
- Lifecycle: expiry is enforced by the system, not by a human. Ticket close or a hard cap (30 days is a common one) triggers deprovisioning; SCIM deprovisioning removes the account when the vendor's IdP disables the user.
For SSH specifically, don't hand out keys. Issue short-lived certificates from your own CA:
ssh-keygen -s /etc/ssh/user_ca \
-I "jdoe@vendor.example" \
-n ssh-prod-db01 \
-O no-port-forwarding -O no-agent-forwarding \
-V +8h vendor_jdoe.pub
That cert is valid 8 hours, only for principal ssh-prod-db01, and cannot be reused tomorrow. The CA logs every issuance; the cert serial is the audit key.
Trade-offs
- Federating to the vendor's IdP is less admin work but you inherit their MFA strength and their compromise. If the vendor is small or their IdP is weak, terminate auth at your IdP instead.
- Standing access is faster during an incident but is exactly what produces a multi-year shared account. JIT costs a few minutes of approval latency; that is the price of attribution.
- Full session recording (terminal I/O) is the strongest evidence but generates real storage and can capture secrets typed into the session: scope retention and restrict who can read recordings.
Failure modes
- Orphaned sponsorship: ticket closes, nobody revokes. Detect with a weekly reconciliation of active sponsorships against open tickets; alert on any identity with no open ticket.
- Recording the vendor can erase: the recorder writes to a bucket the vendor's role can delete. Detect by attempting a delete as the vendor role; it must fail. Use object lock / WORM storage.
- Break-glass gap: the broker is down during an outage, so someone re-enables the shared account. Keep one documented emergency path with a sealed credential and mandatory post-incident review.
Evidence it's working
| Attribute | Shared account | Sponsored identity | Federated JIT |
|---|---|---|---|
| Named individual | no | yes | yes |
| Expiry | none | ticket close or 30 days | hours |
| Session recording | no | yes | yes |
| Revocation trigger | manual | ticket close | IdP disable |
Arithmetic on the failure I've seen: one shared login used by 9 engineers over 3 years is roughly 27 engineer-years of unattributable access. Sponsored identities with a 5-day median ticket window cut that to about 5 engineer-days per engineer, and every one of those days maps to a name.
Curated: · Written: · Reviewed:
Q&A-84What is your policy for the cloud root account?
What I'd say first
Root (or tenant-owner) is the one credential that can undo every other control, including the audit trail that would show it being used. My policy: no programmatic credentials on it, a hardware authenticator required, its password in a break-glass vault rather than in anyone's head, an alert to security and an exec channel on every use, the tasks that genuinely need it written down, and each use reviewed within one working day. Anything people reach for root to do that an IAM principal could do instead gets a break-glass role, not root.
The alternatives, and when each wins
Three ways to shrink the problem, and they stack:
- Break-glass IAM role. Trust policy limited to a named group, MFA required, one-hour max session, CloudTrail alarm on
AssumeRole. Covers almost everything root gets misused for: creating admin users, changing IAM, billing. Deciding criterion: if an IAM principal can do it, root never touches it. - Organizations delegation plus SCPs. An SCP denying root actions in member accounts makes those roots inert; the management account root is the only live one, and SCPs don't apply to it. That turns N roots to protect into 1. Worth it once you have more than a couple of accounts and an all-features organization.
- Custody ceremony. Split password, safe-held key, two-person rule. Reserved for the residual root-only tasks: closing the account, changing the support plan, changing the root email, recovering from a policy that denies all access.
When the ceremony isn't worth it: a single-account startup with no second person on call. Two custodians who can't both be reached at 3am is a recovery risk, not a control. There, a hardware key, alerting and a sealed credential in a vault is the right level. The split-custody ceremony earns its friction only when the blast radius is big enough that slow recovery is cheaper than fast compromise. The inverse also holds: if some accounts sit outside your organization and SCPs can't reach them, the ceremony is the only control left and you pay for it.
Failure mode I've seen
Root retained a long-lived access key from an early automation script. It was used from an unfamiliar network to create an administrative user, and the trail configuration was changed in the same session, so reconstruction depended on a second account's logs and took 6 days. The control that would have caught it is an alert on root sign-in and on root key creation, not a weekly log review.
| Item | Before | After |
|---|---|---|
| programmatic keys | 1, four years old | 0 |
| hardware authenticator | none | yes, held in a safe |
| uses in the last year | 11 | 2, both reviewed |
How I'd know it's done
aws iam get-account-summary should show AccountAccessKeysPresent: 0 and AccountMFAEnabled: 1; the credential report carries the same fields per account. Then trigger a benign root sign-in in a test account and confirm the alert lands in minutes, because an alert path nobody has fired is a guess.
Curated: · Written: · Reviewed:
Q&A-104Walk me through break-glass access: how you provision it, time-box it, monitor it, and audit it after use, and where it has to differ from your normal just-in-time elevation.
Break-glass is the access path that still works when the control plane you would normally use to grant access is the thing that is broken, or when the incident cannot wait for an approver. Two properties define it: it must not depend on the systems it exists to recover, and every use is a security event, not a transaction.
Provisioning. Pre-created, never created at the moment of need: an account minted during an incident is an account nobody reviewed. For the identity provider itself that means a local emergency account in the IdP's own directory, with the credential in a sealed, serialised envelope in a physical safe, and the second factor on a FIDO2 key in a different safe held by a different person, so no single engineer can use it alone. For a cloud root account: hardware key in a safe, root password split 3-of-5 with Shamir, and no recovery email that routes through the IdP. The account is disabled by default and enabling it is the break-glass act; the credential rotates after every use and the new one is resealed before the incident closes. What you must not have is a standing admin account with a shared password in a wiki.
Time-boxing. A fixed window, 60 minutes typical, enforced by a timer that lives outside the account's own control: a scheduled job on a separate management host, or the vault's lease TTL. "We'll remember to disable it" is not a control. Session TTL inside the window is short, 15 minutes, with re-authentication per target.
Monitoring. The alert goes out of band: a page to the security on-call's phone, not a SIEM dashboard the incident may already have taken down, and not through the IdP being repaired. Alert on seal break, on enable, on authentication, on privilege escalation, and on window expiry without a disable.
Audit after use. Within 24 hours: written justification tied to an incident number, review by an engineer who was not involved, session recording reviewed, credential rotated, and a diff of everything that changed in the environment during the window. The review's real question is whether break-glass was necessary or the JIT path was merely inconvenient.
| JIT elevation | Break-glass | |
|---|---|---|
| Approver | named, in the loop, before access | none before; review after |
| Identity source | IdP, SSO, MFA | local credential, independent of IdP |
| Duration | 1–4 h, renewable with re-approval | 30–60 min, hard cap, no renewal |
| Frequency | daily | near zero; each use is an incident |
| Detection | ticket and approval record | out-of-band page at enable |
| Credential | short-lived cert or token | sealed secret, rotated after use |
A worked trace: IdP replication breaks at 02:14. On-call breaks seal #3 at 02:21, logs it on paper and calls the security on-call. Enables breakglass-idp-admin at 02:23; a cron on the management host will disable it at 03:23. Authenticates at 02:26 with the local credential plus the key from the second safe. Replication fixed 02:51, account disabled manually 02:53, credential rotated 02:55, new envelope sealed as #7. Review at 09:00 the next morning.
Failure modes worth naming. A break-glass account whose MFA is enforced by the IdP it is meant to recover: useless in exactly the scenario it exists for. A timer implemented as a calendar reminder, so the account stays enabled for weeks. Alerts routed to the same SIEM an attacker already holds. And the common one: break-glass used monthly because JIT approval takes four hours. Five accounts used once a quarter is 20 uses a year; 20 uses a quarter means the approval path is the defect, and normalising break-glass is how you lose the signal that tells you so.
Curated: · Written: · Reviewed: —
Q&A-94Why should you not key application accounts on the distinguished name?
I would answer DN versus immutable directory object ID by separating authentication, authorization, and lifecycle into three control planes.
A distinguished name encodes the object's position in the tree, so it changes whenever the object moves between organisational units or is renamed. The immutable object identifier is the value that survives the move.
Concretely, store the immutable identifier as the join key, keep the distinguished name as a display and search convenience refreshed on each synchronisation, handle rename and move events by updating the name against the stable key, and refuse a synchronisation that would create a new account because a name changed.
The reason for that specificity is a failure I have seen: A restructure moved 1,900 people between organisational units. Applications keyed on distinguished name treated every one as a new person, creating duplicates and orphaning 12 years of records, and reconciliation took 4 weeks.
Identifier behaviour during the restructure.
| Identifier | Changed for 1,900 people | Safe as a join key |
|---|---|---|
| distinguished name | yes, all of them | no |
| account name | 40 changes | no |
| immutable object identifier | 0 | yes |
I would not consider it settled without evidence: move a test object between organisational units and confirm the application still resolves the same account.
Position changes; identity should not.
Curated: · Written: · Reviewed:
Q&A-93A sign-in form passes the username straight into a directory search filter. What is the risk?
The risk is LDAP filter injection. The username is interpolated into a filter string, so the directory parses it as query syntax rather than as a value. Two outcomes matter at sign-in: an always-true filter becomes an authentication bypass, and a wildcard becomes an enumeration oracle.
The bug is concatenation. A filter is a small grammar: parentheses group, &/|/! are operators, * is a wildcard, = separates attribute from value. Interpolate the username and the attacker supplies the operators.
Take the template (&(uid=%s)(password=%s)):
| Username input | Filter sent | Entries returned |
|---|---|---|
alice | (&(uid=alice)(password=…)) | 1 |
* | (&(uid=*)(password=…)) | every entry |
*)(|(uid=* | (&(uid=*)(|(uid=*)(password=…)) | every entry: always true |
The third row is the bypass: the injected )(|(uid=* closes the uid clause, opens an OR, and the whole filter reduces to "any entry", so the search returns the first person in the directory and the app binds as them.
Escaping the value per RFC 4515 fixes it. Only \, *, (, ), and NUL need escaping inside a value: &, |, =, ~, <, > are structural, and the template's parentheses already delimit the value.
from ldap.filter import escape_filter_chars # python-ldap
user = request.form["username"]
flt = "(&(uid={})(password={}))".format(
escape_filter_chars(user), escape_filter_chars(pw))
escape_filter_chars("*") → \2a; escape_filter_chars("*)(|(uid=*") → \2a\29\28|\28uid=\2a. Both now match a literal string, so the directory returns 0 entries.
Choosing between the fixes. Escaping is the minimum and works against any directory, but it has to be right at every call site: one missed concatenation is the whole vulnerability. A structured builder (Spring LDAP's LdapQueryBuilder/Filters, or LdapEncoder.filterEncode) removes the chance of a missed site in languages that have one. The strongest option is to skip the filter entirely: if the DN is derivable (uid=<user>,ou=people,dc=example,dc=com), bind with it directly and never search. No filter, no injection surface. That fails when the DN isn't derivable (aliases, users spread across OUs), and you still have to escape the DN itself per RFC 4514. A pre-auth bind with a service account does not remove the injection; it only changes whose read rights the wildcard runs under.
Failure modes and detection. Enumeration: * returns every entry, and if the app distinguishes "no such user" from "wrong password" the attacker harvests usernames. In a worked example, 5,600 accounts at 2.3 requests/second takes about 40 minutes. Blind extraction: the attacker binary-searches attribute values from the login success signal. DoS: an unindexed (uid=*) forces a full scan and pins directory CPU. Empty-password bind: if the app binds with the user's DN and an empty password, some servers treat it as an unauthenticated bind that succeeds. Reject empty passwords in the app rather than trusting the directory's default. Detect by logging the constructed filter (or a hash) and alerting on metacharacters in the username field, and by alerting when a sign-in search returns more than one entry.
What I'd verify. Submit *, *)(|(uid=*, and a trailing backslash in the username field and confirm the directory receives a literal search returning 0 entries, not a wildcard.
Curated: · Written: · Reviewed:
Q&A-95How do you answer who can reach this share when groups are nested?
First, pin the definition: "who can reach this share" means the effective membership of the access group. The transitive closure over the nesting edges, not the group's direct members. Two assumptions decide the answer: nesting can cross ownership boundaries, and membership changes while the share is live. Under those, direct members are a fragment, and the real design question is where you compute the closure.
Resolve on read. Walk the graph at check time: a recursive CTE over group_members indexed on (group_id, member_id), or an in-process BFS over a cached adjacency list. Always fresh, no invalidation logic, no write amplification. The cost is per check: a four-level closure over 23 groups is cheap for one check, but a share listing that resolves every group and every member multiplies out, and a hot path doing this per request will show it.
Materialise the closure. Keep a group_closure(ancestor, descendant) table, updated on every membership write. Reads become one indexed lookup. The cost moves to writes: adding a member to a group at depth d touches every ancestor, so write amplification is proportional to ancestor count, and a bulk import into a widely-nested group can rewrite thousands of rows.
Caching is the middle path and inherits the same problem: a cache keyed by group id must be invalidated for every ancestor when a descendant changes. The same ancestor walk you were trying to avoid.
When not to materialise: if nesting depth is capped at two or three and the access groups are small, resolve-on-read is simpler and always correct; materialising buys you a staleness window and an invalidation bug. Materialise when the closure is read orders of magnitude more than it is written and you can state the staleness bound out loud.
The failure this guards against: a share granted to one group was reachable through four levels of nesting by 2,700 people, not the 60 the owner believed. The path ran through a self-service group anyone could join, and it surfaced only during a data-loss investigation.
| Depth resolved | Groups included | People reachable |
|---|---|---|
| direct only | 1 | 60 |
| two levels | 7 | 480 |
| full closure at four levels | 23 | 2,700 |
(Figures from that incident, rounded.)
Controls that make the answer defensible: cap nesting depth, alert when a group used for access gains a nested group with a different owner, detect cycles, and expose an effective-members view to the resource owner so the 60-versus-2,700 gap is visible before an incident rather than after.
I would not call it settled without evidence: compute the transitive membership of the access group, diff it against the owner's stated expectation, and log the path that produced each unexpected member.
Curated: · Written: · Reviewed:
Q&A-96A user presents a valid service ticket. Are they authorised?
No, not on the ticket alone. A valid service ticket answers authentication: the KDC vouched for the principal to that service. It does not answer authorization. Whether the principal may read that file, call that API, or assume that role is a separate decision the service makes.
Two things make the distinction bite. First, the service still has to run its own access check: the ticket carries the caller's identity to the service, and the service decides what that identity may do. Second, the authorization data the ticket carries is a snapshot. In Active Directory the PAC, which holds the group SIDs, is built when the TGT is issued and copied into service tickets. Group membership added or removed after that point is not in the ticket, and the service keeps honouring the old set until a new TGT is obtained.
I've watched that fail. Someone was removed from a privileged group, and the removal did not take effect while their tickets lived. They kept reaching a file service for the remaining 10 hours of the service ticket, and every access was logged as authorized. The revocation window is the ticket lifetime, not the directory write.
| Artifact | Default lifetime (AD) | Stale group data window |
|---|---|---|
| Service ticket | 10 hours | bounded by the TGT it was copied from |
| TGT | 10 hours, renewable up to 7 days | up to 7 days |
| Proposed policy | 4 hours, no renewal | up to 4 hours |
So: have the service evaluate its own ACL against the presented identity; keep ticket lifetimes short enough that stale group data cannot outlive the removal objective; and force reauthentication for privileged operations rather than trusting the PAC.
Don't call it settled by reading the directory row. Remove the group membership, then hit the file service once a minute and record the elapsed time until it first refuses. That number is the revocation window you actually have.
Curated: · Written: · Reviewed:
Q&A-97A web tier needs to reach a database as the user. Which delegation do you configure?
Configure constrained delegation, and prefer the resource-based form. Unconstrained is the wrong answer here.
A web tier calling SQL Server as the user is the textbook double hop: the browser authenticates to the web tier, and the web tier then needs a new ticket to the database that still carries the user's identity. "Constrained" is the right family; the follow-up an interviewer wants is which of the two constrained flavours, and which object you configure it on.
Which one, decided by who owns the target. Classic Kerberos constrained delegation (KCD) is configured on the web tier's account: you list the target service principal names it may delegate to in msDS-AllowedToDelegateTo. It works only when both services sit in the same domain, and the web tier's owners decide where it can reach. Resource-based constrained delegation (RBCD) inverts that: you configure it on the database's computer object, in msDS-AllowedToActOnBehalfOfOtherIdentity, naming the web tier's SID. The resource owner grants the right, no Domain Admin is needed, and it works across domains. On a site where the DBA team owns DB01 and the web team owns WEB01, RBCD is the clean split. RBCD arrived with Windows Server 2012; on anything older you are stuck with KCD and its same-domain limit.
# RBCD: on the database, allow the web tier's identity to act on behalf of users
Set-ADComputer -Identity DB01 -PrincipalsAllowedToDelegateToAccount (Get-ADComputer WEB01)
# Classic KCD alternative: on the web tier, allow it to reach SQL's SPN
Set-ADComputer -Identity WEB01 -AllowedToDelegateTo 'MSSQLSvc/db01.example.com:1433'
Two traps in those two lines. -AllowedToDelegateTo overwrites the whole SPN list rather than appending, so read the existing value first. And the principal you configure is whoever the web app actually runs as: if the IIS app pool runs under a domain service account, that service account is where the delegation belongs, not the computer account. Configure the computer account while the pool runs as a user and the KDC never grants the ticket; SQL quietly sees the app pool identity instead of the user.
Protocol transition, and when you need it. If users reach the web tier by something that is not Kerberos - a form login, a client certificate - the web tier never receives a TGT and must first call S4U2Self ([MS-SFU]) to mint a service ticket to itself on the user's behalf. That requires the web tier's account to carry TRUSTED_TO_AUTH_FOR_DELEGATION (userAccountControl 0x1000000, the "Use any authentication protocol" checkbox). It also makes the web tier strictly more powerful, able to impersonate any user who authenticates to it, so enable it only when the row genuinely needs it.
A trace of the exchange. Alice logs into WEB01 and gets a service ticket for HTTP/web01. To reach SQL, WEB01 calls S4U2Proxy, presents that ticket to the KDC, and asks for MSSQLSvc/db01.example.com:1433. The KDC checks DB01's msDS-AllowedToActOnBehalfOfOtherIdentity for WEB01's SID; finding it, the KDC issues a service ticket bearing Alice's identity, which WEB01 presents to DB01.
Failure modes and how to see them. The commonest is a missing or duplicated SQL SPN: the KDC cannot issue, the client falls back to NTLM, and the DB runs as the app pool account. Catch it in SQL with SELECT auth_scheme, client_net_address FROM sys.dm_exec_sessions - you want KERBEROS, not NTLM. A user in Protected Users, or one flagged "Account is sensitive and cannot be delegated", gets no delegatable ticket at all, so delegation fails without any delegation-specific error; check group membership before blaming the config. Classic KCD across a domain boundary simply does not work - that is the case RBCD exists for.
| Mode | Configured on | Reach | Across domains |
|---|---|---|---|
| Unconstrained (0x80000) | web tier | any service, via the user's TGT | yes |
| KCD | web tier (msDS-AllowedToDelegateTo) | the SPNs listed | no |
| RBCD | database (msDS-AllowedToActOnBehalfOfOtherIdentity) | each target decides | yes |
Never unconstrained: it caches the user's TGT, so one compromised web host yields the whole domain. And if the database does not actually enforce per-user permissions, do not delegate at all - have the web tier connect to SQL as a single service identity and pass the user as data.
Curated: · Written: · Reviewed:
Q&A-98You disabled an Active Directory account. How long can they still reach file shares?
Disabling an account stops the KDC from issuing new tickets. It does not reach into the client's ticket cache, and it does not tear down SMB sessions that are already open. So the honest answer is: the user keeps reaching file shares until the service tickets already in their cache expire (up to the domain's maximum service ticket lifetime, 10 hours by default) and possibly longer if an SMB session to the file server is already established.
Mechanism. The KDC checks the account's state on every request, not just at logon. A disabled account gets KDC_ERR_CLIENT_REVOKED back from both an AS-REQ (new TGT) and a TGS-REQ (new service ticket). That is the part people get wrong: a cached TGT is not a skeleton key that keeps minting service tickets after disablement. The TGS request is refused, so the user cannot obtain a ticket for a share they have not already touched.
What survives is what was already issued. Service tickets are bearer tokens: the file server validates them locally against the krbtgt key and does not call back to the DC to ask whether the account is still enabled. A service ticket in the cache stays usable until its own expiry.
Default Kerberos policy in an AD domain:
| Setting | Default |
|---|---|
| Maximum lifetime for user ticket (TGT) | 10 hours |
| Maximum lifetime for service ticket | 10 hours (600 min) |
| Maximum lifetime for user ticket renewal | 7 days |
A service ticket is issued with the lesser of the maximum service ticket lifetime and the remaining TGT lifetime, so it never outlives the TGT that produced it. That bounds the cached-ticket window at the remaining TGT lifetime, and at 10 hours in the worst case: a ticket issued moments before you disabled the account.
Worked example (hypothetical, default policy). User logs on at 08:00; TGT valid until 18:00. They open \\fileserver\share at 08:05 and the service ticket is issued with expiry 18:00. At 16:00 you disable the account. At 16:01 they try \\otherserver\share: the client has no cached ticket, sends a TGS-REQ, and the KDC refuses it. No access. At 16:02 they reopen \\fileserver\share: the cached service ticket is still valid, so the file server grants it. Access to that share continues until 18:00. If they already held an open SMB session to \\fileserver, that session can continue past 18:00, because the session was authenticated once and is not re-validated against the ticket.
What actually cuts it off. Disable plus a password reset is the standard pair, but be precise about what the reset buys you: it changes the account's long-term key so the old password cannot be used to log on again if the account is re-enabled. It does not invalidate tickets already in the cache. To stop access now, terminate the sessions: force logoff on the client, or close the SMB sessions on the file server (Close-SmbSession / Get-SmbSession to find them). Shortening the maximum ticket lifetimes for privileged accounts shrinks the window for everyone, at the cost of more KDC traffic.
Detection and verification. On the client, klist shows cached tickets and their expiry. On the file server, Get-SmbSession lists active sessions; Security log 4624 (logon) and 4769 (service ticket) events show what was granted and when. To confirm the window in your own domain, disable a test account and attempt a share access every minute, recording the first failure.
The answer: up to the remaining lifetime of the cached service tickets, at most 10 hours by default, and longer still if an SMB session is already open. Disable stops issuance; expiry stops use; only session teardown stops it immediately.
Curated: · Written: · Reviewed:
Q&A-99You want to require directory signing and channel binding. How do you get there safely?
What a strong candidate says first
Signing and channel binding are two settings with two different defaults, and they break different clients. Run them as two migrations, not one toggle.
| Setting | Registry value on each DC | Default | Enforce | Rejects |
|---|---|---|---|---|
| LDAP signing | HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\LDAPServerIntegrity | 1 (Negotiate signing) | 2 (Require signing) | SASL binds that don't request integrity; simple binds not protected by TLS |
| Channel binding | HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\LdapEnforceChannelBinding | 0 (Never) | 1 (When supported) → 2 (Always) | Binds whose auth token isn't tied to the TLS channel: this is what kills NTLM/Kerberos relay to LDAP |
Both are also exposed as security policy: Domain controller: LDAP server signing requirements (None / Require signing) and Domain controller: LDAP server channel binding token requirements (Never / When supported / Always). The signing policy offers only None and Require signing: configuring None writes 0, which is weaker than the registry default of 1. Leave it Not Configured until you're ready to write 2.
Why staged enforcement is safe
The DCs already log the evidence. At the defaults, the Directory Service log writes event 2889 (signing) and 2887 (channel binding) once every 24 hours with counts of the non-compliant binds seen. Those are counts, not attribution. To get source addresses, set HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics value 16 LDAP Interface Events to 2; that logs the client IP of each unsigned bind.
So: turn on IP logging, collect a week, build an owner per source, migrate in waves, enforce on one DC as a canary, then the fleet, and gate on zero non-compliant binds for a full week.
What actually breaks
- Simple bind over cleartext 389. Require signing rejects it. This is the big one: appliances, printers with LDAP address books, badge systems, NAS, and Java JNDI apps that never set
java.naming.security.protocol. - Clients that don't implement channel binding. "When supported" (1) only rejects clients that could have used CBT, so it's the safe first step; "Always" (2) rejects everything without it, including some third-party LDAP libraries.
- Channel binding needs TLS on the connection (636, or StartTLS on 389). A client on cleartext fails channel binding before signing even matters.
Windows clients default to LDAPClientIntegrity = 1 (Negotiate signing), so they sign when the server requires it. The breakage is non-Windows and simple-bind clients, which is exactly why the inventory, not the setting, is the project.
The rollback trap
Enforce by writing the registry value, or a GPO that sets only that value, not by applying a hardening baseline. A baseline revert takes unrelated settings with it, which is how fleet-wide enforcement becomes a multi-hour outage plus regressions you didn't ask for.
Verification
Not settled until events 2889 and 2887 report zero non-compliant binds for seven consecutive days, and you've confirmed the DCs' own authentication and replication still work.
Curated: · Written: · Reviewed:
Q&A-100Your forest is compromised at domain controller level. What does recovery look like?
First 90 seconds
Assume the worst case: the attacker held Domain Admin or equivalent on at least one DC, so every secret in the directory is attacker-known. Including krbtgt, the account whose key signs every Kerberos TGT. That makes the directory database itself untrusted, and with it every credential inside it: domain admins, service accounts, machine accounts, and each DC's local DSRM password.
The answer is not restore-and-clean. It is: rebuild the forest from media that predates the compromise, on a network with no route to production, reset every credential, and reconnect only after validation. A DC restored from a post-compromise backup restores the attacker with it.
Clarify before you commit: is the compromise confirmed at Tier 0 or suspected? Do you have a backup you can prove predates it? Single domain or forest? Is there a separate clean environment to rebuild in?
Why the krbtgt double reset is the load-bearing step
krbtgt holds two keys, current and previous. One reset moves the compromised key to "previous", and the KDC still accepts tickets encrypted with the previous key, so a golden ticket forged before the incident keeps working. The second reset discards it. You wait longer than the maximum ticket lifetime (10 hours by default in the Default Domain Policy) between the two resets so legitimate tickets issued under the old key expire first and users aren't broken.
Sequence
| Step | Isolated | Typical elapsed |
|---|---|---|
| Restore one DC in the forest root domain from a pre-compromise backup | yes | day 1 |
| Seize all five FSMO roles to it; mark SYSVOL authoritative | yes | day 1 |
Reset krbtgt, first pass | yes | day 1 |
Reset krbtgt, second pass, after > 10 h | yes | day 2 |
| Reset Tier 0 credentials: domain admins, service accounts, DSRM on every DC | yes | day 2–3 |
| Rebuild remaining DCs by promotion, not restore; clean old metadata | yes | day 3–4 |
| Validate, then reconnect | no | day 5 |
# Seize all five FSMO roles to the recovered DC (AD module, Windows Server 2016+)
Move-ADDirectoryServerOperationMasterRole -Identity "REC-DC01" `
-OperationMasterRole SchemaMaster, DomainNamingMaster, PDCEmulator, RIDMaster, InfrastructureMaster -Force
# krbtgt pass 1
Set-ADAccountPassword -Identity krbtgt -Reset -NewPassword (Read-Host -AsSecureString)
# wait longer than the maximum ticket lifetime (10 h default), then pass 2
Set-ADAccountPassword -Identity krbtgt -Reset -NewPassword (Read-Host -AsSecureString)
Microsoft's AD Forest Recovery guide ships Reset-KrbtgtKeys.ps1, which does the two-pass reset with the wait built in.
The failure mode that costs days
Reconnecting before the second reset. Illustrative numbers: a team restored, reset krbtgt once, and reconnected 20 minutes later; a TGT forged before the incident still validated, the attacker re-established, and the second attempt added 4 days to a 9-day outage. Detect it by requesting a TGT with a pre-incident forged ticket after the second reset and confirming it fails, and by watching for 4768/4769 anomalies against the new key.
Persistence to sweep before reconnect
AdminSDHolder ACLs, SID history, rogue GPOs, scheduled tasks, DSRM backdoors, and any DC you didn't rebuild. In a multi-domain forest, recover the forest root domain first, then each child domain, and re-create trusts.
If no backup predates the compromise
You build a new forest and migrate users, groups and data into it. That is why you verify a backup predates the compromise before you rely on it.
Curated: · Written: · Reviewed: