Encryption fundamentals for production systems

Cryptography supplies specific properties under explicit assumptions; it does not make data, users, or systems "secure" by itself. Start with the asset, threat actor, data lifecycle, and the properties you actually need — confidentiality, integrity, authenticity, freshness, availability — then minimize sensitive data before choosing an established protocol and library. Authorization, isolation, audit, retention and incident response remain necessary even when every byte is encrypted.

This is a frequent senior/staff interview topic because it separates candidates who have shipped encrypted systems from those who have read about them. Interviewers probe for the difference.

Three goals, and which primitive delivers which

Confidentiality means only key holders can read the data: symmetric ciphers (AES, ChaCha20) and asymmetric encryption or key agreement deliver it. Integrity means tampering is detectable: MACs (HMAC), AEAD tags, and signatures deliver it. Authenticity means you can attribute the data to a holder of a specific secret or private key: signatures, MACs, and certificate-validated TLS deliver it, with different trust properties — a MAC proves "one of us sent this," a signature proves "this specific key signed it" to anyone.

A weak answer treats encryption as one blob that "makes things secure." A strong answer names the property first, then the primitive, then the operational cost. When an interviewer asks "should we encrypt this field?" they are usually testing whether you ask "against whom?" before answering.

Symmetric, asymmetric, and why real systems are always hybrid

Symmetric encryption uses one shared secret for both directions and is fast enough for bulk data — AES-GCM on modern x86 hardware with AES-NI runs at multiple GB/s. Its cost is key distribution: every party needs the secret, over a channel that is already secure. Asymmetric cryptography solves distribution — publish the public key, keep the private one — but is thousands of times slower per operation, so it never encrypts bulk data directly.

Real systems are therefore always hybrid: asymmetric operations authenticate peers and agree on a short-lived traffic secret, then symmetric AEAD protects the records. TLS 1.3 is exactly this. Ordinary server TLS authenticates the server, not the client; mutual TLS adds client certificates but still needs application-level authorization — a valid client cert is identity, not permission.

Interview follow-up to expect: "why not just use RSA for everything?" The answer is throughput and ciphertext expansion, plus the fact that raw RSA has its own padding-oracle history (PKCS#1 v1.5 Bleichenbacher attacks) — you use it only inside a vetted protocol.

Modes of operation: where AES actually fails

AES is a block cipher; a bare block cipher is not a construction. The mode is where systems break:

• ECB encrypts each block independently, so identical plaintext blocks produce identical ciphertext blocks. The classic demonstration is an encrypted image where the penguin is still visible. Never use ECB for more than one block.
• CBC chains blocks with an unpredictable IV and hides patterns, but provides no integrity: an attacker can flip ciphertext bits and predictably flip plaintext bits, and padding schemes have a long history of padding-oracle decryption attacks (as in POODLE-era SSLv3 and many application-layer bugs). CBC without a separate MAC is not acceptable.
• GCM / AEAD modes (AES-GCM, ChaCha20-Poly1305) provide confidentiality and integrity in one construction and authenticate unencrypted context — record type, tenant ID, row ID — as associated data, so a ciphertext cannot be replayed onto a different record. Use these through a high-level library API, not by assembling primitives yourself.

Nonce discipline is part of the AEAD contract. Reusing a nonce with the same GCM key does not merely weaken encryption: it reveals the XOR of the two plaintexts and can expose the authentication key, forging tags thereafter. A random 96-bit nonce is safe only while message counts under one key stay far below the birthday bound; a strictly monotonic counter is safer where the system can guarantee it never repeats. Store nonce, algorithm version, key reference, ciphertext and tag in a parseable format, and reject authentication failure without releasing plaintext.

A weak answer names AES and stops. A strong answer says "AES-GCM with a per-record nonce and tenant ID as AAD" and can explain what breaks when the nonce repeats.

Hashing, MACs, and signatures

A hash maps any input to a fixed-size digest and is one-way — it is not encryption, and "decrypting a hash" is a category error worth catching in an interview. Collision resistance (finding any two inputs with the same digest) and preimage resistance are separate properties; SHA-256 provides both today, MD5 and SHA-1 do not provide collision resistance and must not be used for security.

A bare hash detects accidental change only if the expected digest arrives through a trusted channel; an attacker who can replace both file and digest defeats it. The three integrity constructions differ in key distribution:

• HMAC — keyed integrity between holders of a shared secret. Fast, symmetric, proves "one of us." Right choice for internal token verification.
• Digital signature — public verification, private signing. Slower, but anyone can verify and it gives non-repudiation in the sense that possession of the private key is what produced the signature. Note the limit: a signature proves the data validated under a key, not that the signer was authorized or the data benign.
• JWT signing — the same tradeoff in practice. RS256/ES256 let any party verify a token with the public key, which is why they dominate distributed systems; HS256 (HMAC) requires every verifier to hold the signing secret, which both widens the blast radius and creates a key-distribution problem. ES256 gives shorter signatures than RS256 at comparable strength.

If asked about combining signing and encryption, there is no universally safe order — each composition has documented failure modes, and the honest answer names the failure you are defending against:

• Sign-then-encrypt (sign the plaintext, then encrypt both) is what S/MIME does, and Davis's "Defective Sign & Encrypt" paper showed its traps: the signature is hidden inside the ciphertext, so a recipient can decrypt and re-encrypt the signed plaintext for a third party, who sees a valid signature from the original sender and can be misled about who intended the message for whom.
• Encrypt-then-sign (encrypt, then sign the ciphertext) makes the signature visible to anyone, leaking the sender's identity even when the payload is confidential; and a recipient cannot always tell whether the signature was meant for them or is a forwarded artifact.
• Encrypt-then-MAC / AEAD is the composition with the strongest formal backing — NIST SP 800-38C-style AEAD and the TLS 1.3 record layer authenticate the ciphertext itself, so tampering is rejected before any decryption is attempted. But a MAC is symmetric: it gives integrity, not public verifiability or non-repudiation.

The practical takeaway for an interview: don't recite an order as a rule. Say what properties you need (who verifies? is the sender's identity secret? is non-repudiation required?), then pick the composition — and where possible use a vetted AEAD construction or protocol rather than composing signature and encryption yourself.

Password hashing is a different problem

Passwords are low-entropy inputs verified offline against a stolen database, so password verification needs a salted, adaptive, memory-hard password-hashing function — Argon2id, or bcrypt/scrypt where they are already deployed — not a fast general hash. A unique salt per password prevents identical passwords sharing a hash and defeats precomputed rainbow tables. The slowness is the feature: SHA-256 on a GPU rig runs billions of guesses per second; Argon2id with tuned memory cost runs thousands. An optional pepper is a separately managed defense-in-depth secret, and it creates a rotation and recovery obligation.

The classic weak answer is "we hash passwords with SHA-256 so they're secure." The strong answer names the function, the cost parameters, and why memory hardness matters (it prices out GPU/ASIC attacks).

Randomness underpins all of this: keys, nonces where randomness is specified, reset tokens and session IDs need a cryptographically secure generator. Timestamps, bare counters, UUID versions of unknown quality, and general-purpose PRNGs are not substitutes. Key derivation functions such as HKDF derive purpose-separated keys from input keying material and context; they do not turn low-entropy passwords into strong keys — that is what Argon2 is for.

Key and data lifecycle

Key management commonly determines real security. The algorithm is rarely where it fails: a key generated by the application, stored beside the ciphertext, and never rotated provides no protection against the compromise that is actually likely — access to the running system, not cryptanalysis.

• Generate keys with approved randomness inside a managed KMS/HSM where practical, so key material never leaves the boundary.
• Use envelope encryption: data-encryption keys wrap the data, a key-encryption key wraps them, so rotating the outer key does not require re-encrypting everything.
• Record owner, purpose, algorithm, scope, version, and lifecycle states (creation, activation, rotation, expiration, recovery, destruction). Version every ciphertext with the key that made it, or rotation is impossible.
• Rotation needs a transition plan: new writes use the new key, old keys stay decrypt-only while idempotent migration runs, and every backup, replica and queue is verified. Compromise rotation additionally requires containment, exposure analysis, trust-root replacement, and possibly re-encryption — not merely generating a new key.
• Decide in advance what revocation means: destroying a key makes the data unrecoverable, which is either the intended control or an outage, and the difference must be a decision rather than a discovery.
• Recovery is rehearsed under dual control; destruction happens only after inventory proves no retained data depends on the key.

Choose the encryption layer by threat. Disk/database encryption protects lost media and snapshots but not data reachable by a compromised running application. Application or field encryption narrows administrator and storage threats but complicates queries, caching, search, rotation and recovery. TLS protects a channel between validated peers and stops at the endpoint — a service that terminates TLS at a load balancer and forwards plaintext internally is encrypted in transit only up to that hop. State where each boundary is, verify certificate validation is actually on rather than left disabled from an old debugging session, and maintain a cryptographic inventory so algorithms can be migrated without data loss.

Test the things that actually break: golden interoperability fixtures, tampering, wrong key/audience/context, nonce policy, malformed formats, rotation, rollback, backup restore, dependency outage, and performance under load (a KMS call per record read will show up in your p99).

Worked example: AES-GCM nonce reuse is not a small leak

Two 1-block record under the same key and nonce: ciphertext is plaintext XOR keystream. An attacker who sees both ciphertexts recovers the XOR of the two plaintexts with no key.

record	plaintext (hex)	nonce	XOR of the pair
invoice 812	10 00	n	—
invoice 813	00 10	n (reused)	10 10

10 00 XOR 00 10 = 10 10. That is the entire confidentiality failure: the key never left the HSM and the tag still verifies each record individually. A random 96-bit nonce only delays the same failure until a collision:

messages under one key	collision chance (order of)	action
2^20 (~1 million)	2^-57	keep going
2^32 (~4 billion)	2^-33	rotate the data key
2^48	~1/2	already too late

Bind tenant and row ID as additional authenticated data so a ciphertext that verifies for invoice 812 cannot be replayed onto invoice 813 even with a unique nonce.

What interviewers probe, and the weak versions of each answer

• "Walk me through encrypting a new field at rest." They want: threat model first, KMS-managed key, envelope encryption, AEAD with AAD binding the record context, versioned ciphertext, rotation plan, and the query/search tradeoff. Weak: "use AES-256" with no key management.
• "What happens if a nonce repeats?" They want the XOR-of-plaintexts mechanism and, for GCM, authentication-key exposure. Weak: "it's bad practice."
• "Hashing vs. encryption?" They want one-wayness, and password hashing as its own problem with salts and slow KDFs. Weak: conflating the two or proposing SHA-256 for passwords.
• "HMAC vs. signature for our JWTs?" They want verifier key distribution as the deciding factor. Weak: picking by performance alone.
• "TLS is on — is the data protected?" They want the termination-boundary answer and the mTLS/authorization distinction. Weak: "yes, it's encrypted."

Likely follow-ups: key rotation without downtime, what to do when a key is compromised, sign/encrypt composition ordering (answer it as a tradeoff, not a rule), and why you would not build your own crypto. The consistent thread interviewers are scoring: you reason from the adversary and the failure mode, not from the algorithm name.