VPNs, Tunnels, and IPsec — Concept Guide

Review status: rewritten from reviewer feedback; framing sections new, technical core carried forward.

What a tunnel actually is, and the three jobs it gets confused with

A tunnel wraps a packet in an outer header so the underlay carries it somewhere it couldn't go on its own. GRE carries IP (or nearly anything) across an IP network; VXLAN carries Ethernet frames across an IP underlay; IPsec tunnel mode carries a whole inner IP packet inside a new outer one. The underlay is the network doing the carrying; the overlay is the virtual network the tunnel creates.

Interviewers separate three jobs that candidates blur:

• Reachability — encapsulation alone. GRE and VXLAN give you an overlay with zero confidentiality, peer authentication, or integrity. Anyone on the underlay path can read and modify inner packets.
• Confidentiality — encryption of the inner packet (ESP with an encryption suite).
• Integrity and authenticity — proof the inner packet wasn't altered and came from the authenticated peer (ESP with an AEAD or MAC; AH historically).

A weak answer treats "VPN" as one thing that provides all three by default. The strong answer states which guarantees a given design has and where they end. If someone says "GRE tunnel for security," that's a red flag — GRE is reachability only, and encryption has to be layered on (commonly IPsec over GRE, or MACsec if the underlay is a single L2 domain you control).

Topologies and the policy-based vs route-based decision

Site-to-site VPNs connect fixed gateways and subnets; remote-access VPNs connect a roaming user or device to a gateway, and add identity, address assignment, DNS, posture, and session lifecycle on top. Clientless (browser/SSL-portal) VPNs proxy specific applications instead of tunneling — less exposure, far less capability.

The bigger interview question is how protection is attached to traffic:

• Policy-based (Cisco crypto maps, "interesting traffic" ACLs): packets matching a selector are encrypted toward the peer. Simple for two subnets, painful beyond that — every subnet pair needs a matching selector on both ends, and dynamic routing doesn't work because the routing table doesn't know about the tunnel.
• Route-based (VTI, tunnel interfaces): protection attaches to a logical interface. You route into it like any interface, so OSPF/BGP over the tunnel works, multiple subnets come free, and one tunnel serves many peers' traffic via routing.

"Why does route-based scale better?" is a standard follow-up. The answer: routing owns the traffic decision and the SA owns the protection, so adding a subnet is a routing change, not a renegotiated selector matrix. Expect the follow-up "what happens with overlapping tenant prefixes?" — you need VRFs, NAT, or readdressing, because two tenants using 10.0.0.0/16 inside one routing table will collide.

IPsec in the order a tunnel is actually built

Phase 1 — IKE SA. Peers authenticate each other (pre-shared key or certificate), run Diffie-Hellman to establish a shared secret, and derive an IKE SA that protects subsequent negotiation. This is control-plane security; no user traffic flows yet.

Phase 2 — Child SA / IPsec SA. Inside the IKE SA, peers negotiate traffic selectors (the proxy ACL — which source/destination ranges get protected), the ESP suite, and lifetimes. This produces the data-plane SAs, one per direction.

Rekey. SAs expire on time or byte limits and must be rekeyed without dropping traffic — overlapping old/new SAs exist for this, and botched rekey is a classic cause of "tunnel up, traffic blackholed for 30 seconds every hour."

IKEv1 vs IKEv2 comes up constantly. IKEv1 main mode is 6 messages; aggressive mode is 3 but exposes the peer identity in cleartext and should not be used with PSKs. IKEv2 is the default everywhere now: a standard 4-message exchange (one round trip), built-in NAT traversal detection, MOBIKE for endpoint address changes, and EAP support so remote-access can authenticate users against AAA while the gateway still authenticates with a certificate. If a design specifies IKEv1 aggressive mode with a group PSK, that's a finding.

A weak answer recites "phase 1 and phase 2" without saying what each phase produces. Say it: phase 1 produces an authenticated, encrypted control channel; phase 2 produces unidirectional data SAs bound to traffic selectors.

ESP vs AH, tunnel vs transport

AH authenticates the packet including immutable parts of the outer IP header; ESP authenticates and optionally encrypts the payload. AH is effectively dead for two reasons: it breaks through NAT (the authenticated outer header fields change), and it provides no confidentiality. Everything modern is ESP.

Transport mode protects the upper-layer payload and keeps the original IP header — used mostly for host-to-host. Tunnel mode protects the entire inner packet inside a new outer IP header — this is what site-to-site gateways use, because the inner hosts never know the tunnel exists and the gateways do all the work.

Follow-up to expect: "why does AH break under NAT?" Because the source address in the IP header is part of the authenticated data, and NAT rewrites it. ESP's integrity coverage doesn't include the outer header, so NAT can translate the outer addresses without breaking the integrity check — but NAT still can't handle ESP in the common case, for a different reason: ESP is IP protocol 50 with no ports, so a NAT device has nothing to multiplex sessions on and typically drops it. That is what NAT traversal solves (RFC 3948): during IKE, peers detect NAT in the path and re-encapsulate ESP in UDP port 4500, giving the NAT device a normal UDP flow with ports to track and a mapping to keep alive.

Crypto you must be able to defend

Current sane defaults: AES-GCM (or AES-CBC plus a SHA-2 HMAC) — never DES or 3DES; SHA-256 or better — never MD5 or SHA-1; DH group 14 (2048-bit) as a floor, 19/20 (ECDH, 256/384-bit) preferred. PFS comes from ephemeral DH at Child SA creation — if a design reuses phase 1 keys for all Child SAs without fresh DH, one key compromise decrypts captured historical traffic. State that trade-off explicitly; "we have PFS" without knowing where the ephemeral exchange happens is a weak answer.

NAT-T: as above — ESP in UDP 4500 once NAT is detected. Keepalives keep the NAT mapping from expiring — but they cost bandwidth per idle peer and prove nothing about peer or application health.

Proposal overlap must be deliberate: both peers need a matching entry in their proposals, and "make everything match so it comes up" is how obsolete algorithms survive for years. An interviewer will ask what you'd do about a legacy peer that only supports 3DES — the answer is a scoped, documented exception with an expiry date, not a global downgrade.

MTU, fragmentation, and the black hole nobody tests

Every tunnel eats bytes: outer IP (20) + UDP for NAT-T (8) + ESP header/trailer and padding, typically 50–70+ bytes total on top of the inner packet. Double encapsulation (IPsec over GRE, VXLAN over IPsec) stacks that. If the inner packet exceeds the path MTU, it fragments, gets dropped, or silently disappears when the ICMP "fragmentation needed" message is filtered by an underlay firewall — the classic "small pings work, large file transfers hang" symptom.

Mitigations, in order of preference: allow Path MTU Discovery (don't filter ICMP type 3 code 4), set a conservative tunnel interface MTU, and clamp TCP MSS on the tunnel interface — noting that MSS clamping only helps TCP; UDP and ICMP still need PMTUD to work. Test with minimum and maximum payloads before production, because this never shows up in a lab with 1500-byte-clean links.

Failure modes, rekey, and failover

Tunnels fail in ways the status light doesn't show. "SA up" doesn't prove the intended subnet pair matches policy, or that return traffic takes the same protection. IKE can be up while data paths are broken, and vice versa.

Rekey and failover are state transitions, and transitions are where tunnels break:

• Overlapping old/new SAs prevent gaps but can drop traffic on sequence-window or duplicate-state issues.
• HA gateway pairs need a model for synchronizing IKE/Child SA state, anti-replay windows, NAT mappings, and virtual addresses. Without it, failover means establishing a brand-new tunnel — and existing TCP flows through it reset.
• Asymmetric routing can send the return path outside the protection policy entirely.

The test list an interviewer wants to hear: planned rekey under load, simultaneous rekey from both sides, certificate expiry, node crash, link flap, failback, and clock skew (certificates and replay windows both care about time).

Remote access: full tunnel vs split tunnel

Full tunnel sends all client traffic through the organization: consistent inspection and DLP, at the cost of bandwidth, latency, and privacy. Split tunnel sends only selected traffic: less load and better latency, but the endpoint becomes dual-homed — local-network attacks, exfiltration past the org's controls, and DNS leaks if the client's resolver doesn't follow the policy. The decision is threat-specific, not a default. A weak answer picks one universally; a strong answer names the threat model that decides it.

Whatever the choice, the kill switch / always-on policy must define behavior for captive portals, sleep/wake, reconnect, trusted-network detection, and failure — including how a user recovers when the policy locks them out.

Security ends at termination, and so does the interview

Decrypted traffic inside the VPN gateway still needs least-privilege routing, firewalling, and segmentation — a tunnel is an identity boundary, not a license to flatten the network. Protect keys (HSM or OS keystore, not config files in git), rotate and revoke credentials, rate-limit IKE handshakes, and never log pre-shared keys or full payloads; log identity, selectors, SA transitions, bytes, and auth/replay failures instead.

End-to-end validation is the closer interviewers look for: name and route selection → policy/selector match → IKE identity and proposal → SA installation → inner encryption → outer tuple and path → remote decapsulation → firewall/route to destination → and the full return path. A green tunnel icon is an observation, not evidence. If you can walk a packet through that chain in both directions and name where each guarantee starts and stops, you've answered the question most candidates haven't.