Skip to content
Tech Interview Prep

Top 100 Network Engineer Interview Questions and Answers

The questions most likely to come up in your Network Engineer interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.

Curated: · Written: · Reviewed:

Reviewed 100Review pending 8

108 questions

Q&A-107Walk me through carving 10.20.30.0/24 into subnets for a 100-host floor, a 50-host floor, and two /30 point-to-point links. Where does the address plan break as you grow?

First thing I do is convert the prefix to a mask and a block size, because block size gives you network, broadcast and usable range without memorising formulas. Block size = 256 minus the last non-255 octet of the mask. /25 → 255.255.255.128 → block 128. /26 → 255.255.255.192 → block 64. /30 → 255.255.255.252 → block 4.

Then: network = floor(host octet / block size) × block size. Broadcast = network + block size − 1. Usable = network+1 to broadcast−1. For /31 (RFC 3021) there is no broadcast and two usable addresses on point-to-point links; /32 is a single host.

Worked arithmetic on a non-boundary address: 172.16.45.200/26. Block 64. 200 / 64 = 3.125, floor 3, 3 × 64 = 192. Network 172.16.45.192. Broadcast 172.16.45.255. Usable .193–.254, 62 hosts.

VLSM means you do not have to use the same mask on every subnet. Carve the largest requirement first, then the next, from what remains. That is what lets a /24 hold a 100-host floor, a 50-host floor and two /30s without waste.

Carving 10.20.30.0/24:

RequirementPrefixNetworkBroadcastUsable
100 hosts/2510.20.30.010.20.30.127.1–.126
50 hosts/2610.20.30.12810.20.30.191.129–.190
P2P link/3010.20.30.19210.20.30.195.193–.194
P2P link/3010.20.30.19610.20.30.199.197–.198
spare10.20.30.20010.20.30.25556 addresses

100 hosts needs 7 host bits (2^7 − 2 = 126), so /25. 50 needs 6 (2^6 − 2 = 62), so /26. /30 gives 2 usable. Largest first is the rule: start with the /30s and you fragment the space so the /25 no longer fits.

Summarisation: all four subnets sit inside 10.20.30.0/24, so you advertise one route instead of four. In OSPF an ABR does this with area range; in BGP, aggregate-address 10.20.30.0/24 summary-only. Why: fewer prefixes in the RIB and TCAM, smaller updates, faster convergence, and a single flap does not ripple. The trade-off: the summary stays up when a component subnet is down, so traffic to 10.20.30.193 black-holes at the summarising router unless you have a more specific route or a discard route (a static to Null0 on the aggregator) to prevent loops.

Practical judgement: plan for growth, not just current host count. A 10-host floor gets a /28 (14 usable) if growth is flat, a /27 (30 usable) if you expect to double. Do not hand out /24s to 10-host VLANs: you burn 244 addresses and make summarisation harder. Leave gaps between blocks so you can grow a subnet in place without renumbering.

Overlap mistakes: two subnets that overlap, e.g. 10.20.30.0/25 and 10.20.30.64/26, cause duplicate addresses, ARP conflicts and asymmetric routing. Detection: duplicate IP alerts, an ARP table showing the same MAC on two addresses, or a route table where two prefixes cover the same range. Prevention: keep an IPAM and validate the plan before you deploy.

How routes get advertised: connected subnets are redistributed into the IGP, or summarised at the ABR/ASBR. Without summarisation each /30 is a separate route: a /24 carved into 60 /30s is 60 routes. That is the reason summarisation exists.

Curated: · Written: · Reviewed: —

Q&A-102Walk me through how you use the OSI model to troubleshoot a live outage, when do you go bottom-up, when top-down, and give me a concrete example?

The OSI model is a bisection tool, not a checklist. Nobody walks 1 through 7 on a live outage: you pick the layer that halves the remaining hypothesis space and let the symptom pick the direction.

Direction rule: broad scope, bottom-up. Whole site dark, nothing resolves, nobody can print: start at L1 and climb, because the fault is probably shared infrastructure. Narrow scope, top-down. One app, one URL, one user: start at L7 and descend, because the shared layers are demonstrably fine for everyone else. The common case is neither: start at L3 with a ping, because it's the cheapest split. It separates "packets can't get there" from "packets get there and the application can't talk."

SymptomStartFirst command
Whole site darkL1show interfaces status, then `show interfaces
One VLAN silent, neighbours fineL2show mac address-table, show spanning-tree
Ping to a subnet failsL3show ip route, show ip arp
Ping works, TCP port doesn'tL4nc -vz 10.8.4.20 443, firewall counters
Handshake completes, app stallsL5-7 + MTUping -M do -s 1472, capture

Concrete example. 60 users in a branch report the ERP at 10.8.4.20:443 is down. Internet and O365 work.

L1/L2, 30 seconds: show interfaces status on the access stack. All up, no CRC, no input errors. show mac address-table count normal, STP root is the distribution pair where it belongs. Clean.

L3: ping 10.30.0.1 (gateway) 1 ms, 0% loss. ping 10.8.4.20 0% loss, 42 ms. IP forwarding end to end works. That's the bisection: the fault is above L3, or in something ping doesn't exercise.

L4: nc -vz 10.8.4.20 443. SYN out, no SYN-ACK, timeout. But a capture on the branch router shows the handshake completing: SYN, SYN-ACK, ACK, then silence. So L4 is fine and the stall is in the TLS exchange.

That's the tell. Small packets work, large ones don't:

$ ping -M do -s 1472 10.8.4.20   # 1500-byte frame, DF set
100% packet loss
$ ping -M do -s 1372 10.8.4.20   # 1400-byte frame
3 received, 0% loss

Path MTU is 1400: the branch rides an IPsec tunnel with 100 bytes of overhead. The server's TCP MSS is 1460 because it only sees its own 1500-byte LAN, so its TLS Certificate flight goes out as 1500-byte frames, hits the tunnel, needs fragmentation, has DF set, and dies. The ICMP type 3 code 4 that would have told the server is filtered by the data-centre edge ACL. Classic PMTUD black hole: the failure is size-dependent, not layer-dependent, which is why the symptom looked like L7.

Fix: ip tcp adjust-mss 1360 on both tunnel interfaces (1400 − 40), and permit ICMP type 3 code 4 on the edge. Verified by re-running the DF ping and watching the TLS handshake complete.

Where the frame misleads you:

  • A successful ping is not proof the path works. It's one DSCP, one ECMP hash, one packet size. The ERP traffic may take a different member.
  • The layer where the error surfaces is rarely the layer that's broken. A TLS stall was an MTU problem; a "DNS outage" is often an L3 reachability problem to the resolver.
  • "Link is up" is L1-ish and means almost nothing. show interface counters (CRC, input errors, late collisions) are the L1 evidence.
  • OSI has no layer for NAT, stateful firewalls, load balancers or policy. A stateful device that drops a return packet looks like an L4 listener problem. Name the box, not the layer.

When not to bother: if streaming telemetry or a traceroute has already localised the fault to one hop, walking layers is theatre. Use the model when you don't yet know where to look.

Curated: · Written: · Reviewed: —

Q&A-101An application team asks whether to use TCP or UDP. What does each actually guarantee, and how do you decide?

Core answer (first 90 seconds). The question is not "TCP is reliable, UDP is fast." It is: what does the application need the transport to guarantee, and what can it do itself? TCP gives you a reliable, ordered byte stream over a connection, with flow control and congestion control in the kernel. UDP gives you datagrams with a checksum and port multiplexing: nothing else. The decision: if the app needs every byte in order and cannot implement retransmission, ordering, and congestion control itself, use TCP. If the app can tolerate loss, or needs to own reliability for a reason (latency, multicast, NAT traversal, userspace congestion control) use UDP.

What each actually guarantees.

PropertyTCPUDP
DeliveryRetransmits until ACKed or connection breaksBest effort
OrderingIn-order byte streamDatagram boundaries preserved, order not
DuplicationSuppressedPossible
Flow controlReceiver window (rwnd)None
Congestion controlKernel (CUBIC default on Linux, BBR optional)None unless the app adds it
Connection3-way handshake, state on both endsStateless
ChecksumMandatoryOptional in IPv4, mandatory in IPv6
Head-of-line blockingYes, per connectionNo

TCP does not guarantee delivery. It guarantees that what is delivered is complete and in order, or the connection resets. That distinction matters when you debug.

Where each is used. TCP: HTTP/1.1 and HTTP/2, SSH, SMTP, IMAP, BGP (179), LDAP, most databases. UDP: DNS (53), DHCP (67/68), NTP (123), SNMP traps (162), RTP for voice and video, VXLAN (4789), WireGuard (51820), IPsec NAT-T (4500), TFTP (69), syslog (514), and QUIC (443/UDP), which carries HTTP/3.

How to decide. Four questions, in order:

  1. Can the app tolerate loss? Voice and video can, within a jitter buffer. File transfer and config push cannot.
  2. Who retransmits? If the kernel, TCP. If the app, UDP: QUIC does this in userspace with per-stream ordering, so one lost packet does not stall the whole connection.
  3. Does it need congestion control? Bulk senders do. A DNS resolver sending 40-byte queries does not.
  4. Will the path pass it? UDP 443 and TCP 443 traverse most firewalls. UDP 500/4500 for IPsec. Some enterprise networks block all UDP except DNS, which is why QUIC falls back to TCP.

Why it bites: the lossy long-RTT path. TCP throughput on a path with loss p and RTT is roughly MSS × 1.22 / (RTT × √p) (Mathis et al.). With MSS 1460 bytes, RTT 100 ms, and 1% loss: 1460 × 1.22 / (0.1 × 0.1) ≈ 178 kB/s ≈ 1.4 Mbps. A 1 Gbps link with 1% loss and 100 ms RTT gives a single TCP flow about 1.4 Mbps. That is why UDP-based protocols with their own congestion control (QUIC, RTP/RTCP) exist: they recover faster and avoid head-of-line blocking.

Failure modes. TCP: head-of-line blocking stalls every stream on one connection; SYN floods and TIME_WAIT exhaust state; middleboxes drop idle connections. Detect with retransmission rate, duplicate ACKs, zero-window probes, RST counts. UDP: no congestion control means a misbehaving sender can congest-collapse a link; amplification attacks (DNS, NTP, memcached) abuse spoofed source addresses; NAT mappings expire. Detect with loss and jitter counters, and by checking whether the app implements its own backoff.

Curated: · Written: · Reviewed: —

Q&A-103A host's ARP table maps a gateway IP to a MAC that isn't the gateway's. What does ARP actually solve, what do duplicate and stale entries break, and how do you diagnose ARP-related reachability failures?

ARP exists because IPv4 forwarding hands a packet to Ethernet with a next-hop IP, but Ethernet needs a destination MAC. ARP resolves that: broadcast "who has 10.0.0.5?", unicast reply "10.0.0.5 is at aa:bb:cc:dd:ee:ff", cached in the ARP table with an aging timer. On Cisco IOS the default is 4 hours (14400 s). Linux keeps a neighbour REACHABLE for base_reachable_time_ms (30 s default, randomized), then STALE, and only re-probes on use.

What breaks:

  • Stale entry. The MAC changed (NIC replaced, VM migrated, HSRP/VRRP failover, firewall cluster) but the cache still points at the old MAC. Frames are switched to the old port. If that port is down they are dropped; if another device now owns that MAC they are delivered to the wrong host. Symptom: ping to the IP fails or is intermittent while the ARP entry looks "complete".
  • Duplicate entry. Two hosts answer for the same IP, either by misconfiguration (a static address inside a DHCP pool) or ARP spoofing. The table flips between two MACs. Symptom: connections reset, intermittent loss, sometimes traffic to a rogue host. On Linux, ARP flux, multiple interfaces on the same subnet all answering, is a common self-inflicted version; arp_ignore=1 and arp_announce=2 fix it.

Diagnosis:

  1. On the failing host: ip neigh show 10.0.0.5 (Linux) or show ip arp 10.0.0.5 (IOS). Note the MAC and state. FAILED/INCOMPLETE means no reply at all: a different problem (L2 path, VLAN, or the target is down).
  2. Compare that MAC against the switch's MAC address table: show mac address-table address aa:bb:cc:dd:ee:ff. If it is on an unexpected port, you have found the wrong device or a duplicate.
  3. Capture ARP: tcpdump -i eth0 -n arp or debug arp (sparingly). Look for two different MACs replying to the same IP, or gratuitous ARPs.
  4. Check for duplicate IP: arping -D -I eth0 10.0.0.5 sends a probe and reports if anyone answers. On the switch, DHCP snooping plus Dynamic ARP Inspection logs will show denied ARP if you have enabled them.
  5. Clear and re-test: ip neigh flush 10.0.0.5 or clear ip arp 10.0.0.5. If it comes back wrong, the source is still there.

A worked trace: host A pings 10.0.0.5, gets no reply. ip neigh shows 10.0.0.5 dev eth0 lladdr 00:11:22:33:44:55 STALE. show mac address-table address 0011.2233.4455 puts that MAC on Gi1/0/7, but 10.0.0.5 is supposed to be on Gi1/0/12. Gi1/0/7 is a laptop with a static IP in the DHCP range. Fix the static, clear ARP, traffic returns.

Do not forget proxy ARP: a router answering ARP for a host on another subnet can mask a wrong mask. And gratuitous ARP is what makes failover work, if it is filtered (some ACLs drop broadcast ARP), the stale entry persists until the timer expires, which on IOS is up to 4 hours.

Curated: · Written: · Reviewed: —

Q&A-64The interface is administratively up with 0 pps and no light. What have you not got?

You have no optical receive. The interface is administratively up because someone configured it; it has no carrier because the receiver is seeing no light. The 0 pps is not a second symptom: it is the consequence. No photons means no signal detect, no line protocol, no frames to count.

Mechanism

A pluggable optic's receive path is a photodiode, a transimpedance amplifier, and a clock/data recovery block. The receiver asserts signal detect only when incoming optical power sits above its sensitivity floor and below its overload point. Below the floor, there is nothing to recover a clock from, so the MAC never sees carrier, line protocol stays down, and the packet counters stay at zero. Admin up is a software flag in the config; carrier is a physical measurement. They are independent, which is exactly why this state exists.

One thing to be precise about: "no light" is the receive side. The local laser is usually still transmitting. So you cannot conclude the far end is fine from your own port alone.

What the numbers look like

Typical IEEE 802.3 optical budgets, so you know what "dark" and "too hot" mean:

StandardFiber / λTx rangeRx sensitivityRx overload
1000BASE-SXMMF / 850 nm−9.5 to −4 dBm−17 dBm−3 dBm
1000BASE-LXSMF / 1310 nm−9.5 to −3 dBm−19 dBm−3 dBm
10GBASE-SRMMF / 850 nm−7.3 to −1 dBm−9.9 dBm−1 dBm
10GBASE-LRSMF / 1310 nm−8.2 to +0.5 dBm−14.4 dBm+0.5 dBm

A reading at or below the sensitivity floor is your "no light." Many platforms report a floor value (commonly around −30 dBm) or N/A rather than a true number, because the meter cannot measure below it.

show interface Te1/0/1
  Te1/0/1 is up, line protocol is down (notconnect)
  0 packets input, 0 packets output
show interface Te1/0/1 transceiver
  Tx Power  -2.1 dBm
  Rx Power  N/A        <- below measurement floor

Deciding what to do next

Read both Tx and Rx power before touching anything else, then branch:

  • Tx in range, Rx dark → the fault is on the path or the far end. Check the far-end Tx power, inspect and clean both endfaces, verify the patch panel and fiber continuity.
  • Tx also out of range → suspect the local optic or port. Move the optic to a known-good port.
  • Rx above overload → too much power; add an attenuator. This also produces link-down or flapping, not just darkness.
  • Loopback test → loop the port (physical plug or config loopback). If Rx now reads your own Tx, the local port and optic are good and the fault is in the path.

Common causes of a dark Rx: a dirty or damaged connector (a fingerprint on an SMF endface can cost several dB), the wrong optic type (an 850 nm SR optic on a single-mode path will not propagate), a far-end port that is shut or has Tx disabled, a macrobend, or a broken strand.

Do not start IGP or protocol debugging on an interface with no carrier. There is no frame path to debug; you are chasing a physical-layer fault with a control-plane tool.

Curated: · Written: · Reviewed:

Q&A-54The router shows incomplete ARP for the next hop. What has not happened, and what do packets do?

Core answer. Incomplete ARP means the router resolved the next hop at L3 but not at L2. The route is in the FIB, so the router knows the next-hop IP; it sent a who-has broadcast for that IP and no is-at reply came back. What has not happened is the ARP reply: the neighbour's MAC is unknown, so no L2 rewrite can be built. Packets to that next hop are queued briefly, the request is retried, and then they are dropped with host-unreachable returned to the sender. IP reachability of the prefix does not populate ARP; only a who-has/is-at exchange does.

Mechanism. On Linux the neighbour entry walks INCOMPLETE → FAILED. While INCOMPLETE the kernel holds a small per-neighbour queue, retransmits the request on a timer, and after the retry budget expires marks the entry FAILED and returns EHOSTUNREACH. For forwarded traffic that means the packet is dropped and an ICMP host-unreachable is generated, subject to icmp_ratelimit, so the original sender may see a silent drop rather than an error.

sysctl (net.ipv4.neigh.default)DefaultEffect
mcast_solicit3ARP requests sent before FAILED
retrans_time_ms1000Gap between requests
unres_qlen3Packets queued per unresolved neighbour
unres_qlen_bytes65536Byte cap on that queue
gc_stale_time60Seconds before a stale entry is re-probed

So an unresolved next hop costs roughly 3 seconds and 3 queued packets before the drop becomes permanent.

$ ip neigh show 10.0.0.1
10.0.0.1 dev eth0  INCOMPLETE
$ tcpdump -ni eth0 arp
12:00:01.001 ARP, Request who-has 10.0.0.1 tell 10.0.0.254
12:00:02.002 ARP, Request who-has 10.0.0.1 tell 10.0.0.254
12:00:03.003 ARP, Request who-has 10.0.0.1 tell 10.0.0.254
# no is-at reply arrives
$ ip neigh show 10.0.0.1
10.0.0.1 dev eth0  FAILED

Requests leaving with no replies returning is the signature of an L2 filter (a VLAN ACL, port security, or a host that is not actually on the segment) not of a routing-protocol problem.

Alternatives, and when they change the call.

  • Static neighbour entry (ip neigh add 10.0.0.1 lladdr ... nud permanent): correct when the peer legitimately never answers ARP. It breaks on any MAC change (NIC swap, VRRP/HSRP failover, VM migration) so use it only for a stable, known peer.
  • Proxy ARP: if the next hop sits on a different L2 segment, an intermediate router must answer on its behalf. Check proxy_arp on that router; if it is off, resolution fails even though the L3 path is fine.
  • A different path: if the next hop is simply gone, withdrawing the route or letting the IGP reconverge onto an alternate next hop is the right move, not repairing L2.

The deciding question is whether the next hop is supposed to be on this segment. If yes, fix L2. If the route points at a decommissioned next hop, remove the route. If the peer cannot ARP by design, pin a static entry.

Failure modes and detection. A filtered ARP shows who-has requests with no is-at in tcpdump and an INCOMPLETE entry whose counters climb. A wrong VLAN or a host that moved shows the same, but the request is never seen by the intended peer. Queue overflow, many unresolved neighbours at once, drops packets immediately instead of queueing, so latency does not rise before the failure. Confirm the fix by requiring a complete entry with a MAC before blaming the routing protocol.

Curated: · Written: · Reviewed:

Q&A-98The pool has 254 addresses and 0 free. New hosts get what?

Nothing. The client gets no DHCPOFFER, retransmits, and eventually self-assigns a link-local address.

Said aloud in the first minute: the pool is a fixed set of leases, and a full pool means the server has no address to put in the yiaddr field, so it stays silent. DHCP has no "I'm out" message: DHCPNAK exists only to reject a client that asked for the wrong address. The failure is a timeout, not an error.

Mechanism. The Discover arrives (broadcast, or unicast to the server with giaddr set by the relay). The server picks the subnet from the receiving interface or giaddr, walks the pool for a free lease, finds none, and drops the packet. ISC dhcpd logs DHCPDISCOVER from <mac> via eth0: no free leases; Kea logs a pool-exhausted event (DHCP4_POOLS_EXHAUSTED); Windows DHCP logs a scope-level event. Existing clients keep working because RENEW/REBIND are unicast to the server that already holds their binding, which is why "DHCP is up" looks true.

Trace (tcpdump on the server, hypothetical client 3c:22:fb:...):

12:04:11.201 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 3c:22:fb:...
12:04:15.318 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 3c:22:fb:...
12:04:23.402 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 3c:22:fb:...

Three Discovers, no Offer. The 4 s then 8 s gaps are the RFC 2131 §4.1 retransmission backoff (1, 2, 4, 8, 16, 32, 64 s). Once the retransmit sequence is exhausted the client falls back to 169.254.0.0/16 (APIPA) on Windows and macOS. That is the evidence to collect: a Discover with no matching Offer, plus the server's no-free-leases log line.

Why it filled: arithmetic. A /24 gives 254 usable. With the Windows default 8-day lease, steady-state capacity is 254 / 8 ≈ 31 new leases per day. If 200 devices a day join (guest Wi-Fi, BYOD), demand is 200 × 8 = 1600 leases. The pool fills in days and stays full.

Options, and when each fits:

OptionWhat it buysWhen it fits / cost
Shorten lease (8 d → 1 h)254 × 24 = 6096 leases/dayhigh-churn guest; more renew traffic
Split scope 80/20second server serves 20%you already run two servers (Microsoft's classic split)
Superscope / shared-networksecond subnet on the same L2can't renumber; needs routing for the new subnet
Grow the subnet (/24 → /23)510 usableplanned growth; renumbering and ACL churn
Reclaimfrees leases held by gone deviceschurn is the cause, not growth

Defaults to know. ISC dhcpd's shipped example sets default-lease-time 600 and max-lease-time 7200; Kea's default valid-lifetime is 7200 s; Windows Server defaults to 8 days; Cisco IOS pools default to 1 day. There is no vendor default for a utilisation alert: teams typically warn at 80–85% of the pool.

Failure modes. Silent exhaustion, masked by renewals. Reservations and static assignments inside the dynamic range shrink the usable count. A relay pointing at a subnet whose pool is full while another subnet has space, because the server only offers from the subnet giaddr selects. Detect with free-address count per pool, a Discover-without-Offer probe, and an alert on leases-in-use.

Curated: · Written: · Reviewed:

Q&A-55IPv4 works and IPv6 neighbour solicitation is filtered. Why is that not "ARP for v6"?

The 90-second answer. Because NDP is not a link-layer protocol with its own EtherType: it is ICMPv6 (IPv6 next header 58) carried inside IPv6 packets. ARP is EtherType 0x0806, broadcast to ff:ff:ff:ff:ff:ff, and a switch ACL written as "permit ARP" never inspects an ICMPv6 packet at all. Three things follow. NS goes to a solicited-node multicast group (ff02::1:ff00:0/104 plus the low 24 bits of the target), mapped to MAC 33:33:ff:xx:xx:xx, so it reaches only hosts that joined that group. NDP carries TLVs (source link-layer address, MTU, prefix information, nonce) and covers router/prefix/parameter discovery, duplicate address detection and redirect: ARP does address resolution and nothing else. And NS/NA/RA must arrive with hop limit 255 (RFC 4861), so a device that rewrites hop limit breaks NDP even when it permits the type.

ARPNDP
EncapsulationEtherType 0x0806ICMPv6, next header 58, types 133–137
Destinationbroadcast ff:ff:ff:ff:ff:ffsolicited-node multicast ff02::1:ffXX:XXXX → 33:33:ff:xx:xx:xx
Scopeaddress resolutionresolution, DAD, router/prefix/parameter discovery, redirect
Payloadfixed fieldsTLVs, extensible
HardeningnoneSEND (RFC 3971), RA Guard (RFC 6105), ND inspection

What to do, and when. The filter is either a bug or a policy, and that decides the fix.

  1. Permit ICMPv6 133–137 (at minimum 135/136) on the segment. Right when the filter is a mistake and you own the ACL. Cost is a change window; the risk is that if the filter was deliberate you have just reopened rogue-RA exposure.
  2. Keep the filter, add RA Guard or ND inspection per port. Right when the filter is a security policy against rogue RAs. Needs a managed switch that can express the policy per port.
  3. Static neighbour entries: ip -6 neigh add 2001:db8::7 lladdr 52:54:00:aa:bb:cc dev eth0 nud permanent. Right for a handful of fixed peers (an upstream router, a jump host) during an incident. It does not scale, breaks on a NIC swap, skips DAD, and a permanent entry hides the underlying failure from monitoring.
  4. Proxy NDP on the router: net.ipv6.conf.all.proxy_ndp=1 plus ip -6 neigh add proxy .... Right when you control the router but not the switch filter and need to bridge two segments.
  5. Disable IPv6: net.ipv6.conf.all.disable_ipv6=1. Only when nothing depends on IPv6; it removes the symptom and the capability together.

Failure modes and how they show up. NS dropped but NA permitted: entries resolve only for peers that learned your MAC some other way, so ip -6 neigh shows a mix of REACHABLE and INCOMPLETE/FAILED. RA dropped: hosts keep link-local only, ip -6 route has no default, IPv4 keeps working. DAD blocked: the address stays tentative in ip -6 addr and the host will not source from it. Hop limit rewritten: the kernel drops NS/NA silently. Permitted in the ACL, still broken.

Capture both sides. ip6[40] is the ICMPv6 type byte, since the IPv6 header is 40 bytes.

tcpdump -ni eth0 arp
tcpdump -ni eth0 'icmp6 and (ip6[40] == 135 or ip6[40] == 136)'

Hypothetical output from a host whose NS is filtered in both directions:

$ tcpdump -ni eth0 arp
12:04:11.882 ARP, Request who-has 10.0.0.7 tell 10.0.0.5, length 28
12:04:11.883 ARP, Reply 10.0.0.7 is-at 52:54:00:aa:bb:cc, length 46
$ tcpdump -ni eth0 'icmp6 and (ip6[40] == 135 or ip6[40] == 136)'
0 packets captured
$ ip -6 neigh show dev eth0
2001:db8::7 dev eth0  FAILED

The entry reads INCOMPLETE while the probe is in flight and FAILED once retransmits exhaust: RFC 4861 sets MAX_MULTICAST_SOLICIT to 3 with a 1000 ms RETRANS_TIMER, so roughly three seconds of silence before the kernel gives up. That is the signature: IPv4 adjacency complete, IPv6 adjacency FAILED, and no NS on the wire.

Curated: · Written: · Reviewed:

Q&A-52You offer DHCPv6 addresses and no Router Advertisements. How do hosts get a default gateway?

They don't, not from DHCPv6. There is no standardized DHCPv6 option that installs a default route. Default routers reach a host only through Router Advertisements (RFC 4861), so if you suppress RAs and hand out addresses with DHCPv6, hosts end up with a global address and no way off-link. The fix is to send RAs; DHCPv6 and RA are not alternatives, they cover different things.

Why it works that way. An RA is an ICMPv6 message from a router, sourced from its link-local address with hop limit 255. Its Router Lifetime field is what makes the sender a default router: a non-zero lifetime installs default via fe80::…, and a lifetime of 0 explicitly says "not a default router." The Prefix Information Option is a separate, optional part of the same message. That matters here: you can advertise yourself as the default router with no PIO at all, keep the M and O flags set so hosts know to fetch addresses and other config from DHCPv6, and let DHCPv6 do the addressing. RFC 4861's default AdvDefaultLifetime is 3 × MaxRtrAdvInterval, i.e. 1800 s with the default 600 s interval.

DHCPv6 (RFC 8415) carries addresses, and options like DNS resolvers (RFC 3646), but nothing that populates the routing table's default entry. RFC 4191's Route Information Option, which can carry more-specific routes, also travels inside an RA: it does not give you a DHCPv6 path to a gateway.

Two things that bite people. First, Windows will not perform stateful DHCPv6 at all unless it has seen an RA with the M flag set, so a DHCPv6-only network leaves Windows hosts with link-local addresses only: you get a different failure than the one you were expecting. Linux and systemd-networkd can be configured to run DHCPv6 with IPv6AcceptRA=no, which is exactly how you produce the address-without-gateway state. Second, hosts silently drop RAs whose source is not link-local or whose hop limit is not 255, so an RA that is routed or rewritten by a middlebox looks like no RA at all.

Detection. On the host, ip -6 route show default is empty while ip -6 addr shows a global address; an off-link connect fails with ENETUNREACH while IPv4 keeps working. On the wire, tcpdump -i eth0 icmp6 shows no router advertisements. Illustrative Linux output, DHCPv6-only versus after enabling RA:

# DHCPv6 only, no RA
$ ip -6 route show
2001:db8:1::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
$ ping -6 2001:db8:2::1
connect: Network is unreachable

# after the router starts advertising, lifetime 1800
$ ip -6 route show default
default via fe80::1 dev eth0 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium

If you genuinely cannot send RAs, the only remaining option is a static default route on every host, which does not scale past a lab. Everything else standardized for default routing on IPv6 is RA.

Curated: · Written: · Reviewed:

Q&A-53AAAA is faster to resolve than A, and IPv6 is a black hole. What does the client do?

Core answer

The client resolves both A and AAAA, sorts the results with IPv6 first, and starts TCP connections to them in a staggered race rather than one at a time. If the IPv6 connect hangs (SYN sent, nothing back) the client does not wait for the kernel's TCP connect timeout; it starts the IPv4 attempt after the Connection Attempt Delay (250 ms by default in RFC 8305) and uses whichever connection completes first. The user sees roughly one RTT plus that delay, not a two-minute stall. How fast AAAA resolved is irrelevant: the client picks the family by preference order, not by which answer arrived first.

Mechanism

RFC 8305 (Happy Eyeballs v2) defines two timers:

  • Resolution Delay, 50 ms default: after the first address family's answer arrives, the client waits this long for the other family before starting connection attempts, so a slow AAAA is not skipped.
  • Connection Attempt Delay, 250 ms default: the gap between starting one connection attempt and starting the next, with address families interleaved (IPv6, IPv4, IPv6, IPv4...).

The race exists because a black-holed IPv6 path produces no signal at all: no SYN-ACK, no RST, no ICMP. Without the race, the only feedback is the kernel's SYN retransmission budget, which is far longer than a user will wait.

Trace (labelled example: Linux client, tcp_syn_retries=6)

tEvent
0 msA and AAAA queries sent
15 msAAAA answer: 2001:db8::10
30 msA answer: 192.0.2.10
65 msResolution Delay (50 ms from first answer) expires; TCP SYN to 2001:db8::10
315 msConnection Attempt Delay (250 ms) expires, IPv6 not established; TCP SYN to 192.0.2.10
335 msIPv4 SYN-ACK; connection used, IPv6 attempt abandoned
~130 sKernel would have given up on the IPv6 SYN here (tcp_syn_retries=6)

The IPv6 SYN is dropped silently, so the client never learns the path is dead: only that it is slow.

Failure modes

  • No Happy Eyeballs: an application that takes the first getaddrinfo result and calls connect() waits the full SYN budget. About 130 s on Linux, about 21 s on Windows. That is the IPv6 black-hole outage.
  • HE present but fallback disabled or misconfigured (older runtimes, some HTTP clients): the same stall.
  • RST instead of silent drop: the IPv6 attempt fails immediately and IPv4 starts at once, so the 250 ms cost never appears. The delay only bites on silent filtering.
  • Connect succeeds but data stalls (MTU black hole): HE does not help, because the handshake completed.

Alternatives and when to use them

  • Fix the IPv6 path (routing, firewall, ACL): the only fix that keeps IPv6.
  • Tune the Connection Attempt Delay: lower (100 ms) fails over faster but opens more spurious IPv4 connections when IPv6 is merely slow; higher reduces races but lengthens the worst case.
  • Deprioritise IPv6 in the RFC 6724 policy table (gai.conf, precedence): makes IPv4 preferred everywhere, forfeiting working IPv6.
  • Publish A only: removes the race, loses IPv6 entirely.

Application-level HE is common: Go's net.Dialer FallbackDelay defaults to 300 ms, curl's --happy-eyeballs-timeout-ms to 200 ms.

Curated: · Written: · Reviewed:

Q&A-50A MAC has a reservation for 10.20.20.8. Why can another device still use that address?

A DHCP reservation is a server-side offer policy, not access control. The server promises 10.20.20.8 to a client identifier (the chaddr MAC, or option 61 client-id when the client sends one) and nothing in the DHCP exchange proves the device presenting that identifier is the device you meant. Any host that can put the reserved MAC or client-id on the wire gets the address, and a host configured with a static IP never asks the server at all.

Assumptions: one flat L2 segment, no 802.1X on the access port, DHCP snooping not enforcing bindings.

Why the server can't tell

DHCP has no authentication. The server matches a reservation against the client identifier in the request; ISC dhcpd uses option 61 when the client sends one and falls back to chaddr otherwise. Both are attacker-controlled fields, and a device doesn't even need to change its own MAC: setting option 61 to the printer's MAC is enough. The server also doesn't check whether the address is already in use unless conflict detection is enabled (Windows DHCP: conflict detection attempts, default 0; ISC dhcpd: ping-check, off by default). So it will ACK a reservation for an address another host is already using.

Trace

Printer 00:1b:44:11:3a:a4 is offline. Attacker on Gi1/0/7 runs:

ip link set eth0 address 00:1b:44:11:3a:a4
dhclient -v eth0
  1. DHCPDISCOVER broadcast, chaddr 00:1b:44:11:3a:a4, option 61 = 01:00:1b:44:11:3a:a4.
  2. Server matches the reservation, DHCPOFFER 10.20.20.8.
  3. DHCPREQUEST, DHCPACK 10.20.20.8, lease 86400 s.
  4. Attacker sends a gratuitous ARP for 10.20.20.8; switches and hosts overwrite their ARP entries and traffic for the printer's address now lands on Gi1/0/7.

The printer returning later either conflicts (static IP) or gets NAKed or duplicated (DHCP).

What actually makes the address exclusive

ControlStopsDoesn't stop
802.1X on the access portunauthenticated device joiningauthenticated device spoofing another MAC
Port security, static secure MAC bound to the printer's portthe cloned MAC appearing on another porta clone on the printer's own port
DHCP snoopingrogue DHCP servers; a client using an address it wasn't leaseda spoofed client-id getting the lease on its own port
IP source guard + dynamic ARP inspectionsource IP or ARP not matching the snooping bindinga lease the attacker legitimately obtained

DHCP snooping alone is the common mistake: it binds 10.20.20.8 to the attacker's port, because that is where the ACK was snooped.

Detection

  • The switch MAC table shows 00:1b:44:11:3a:a4 learned on two ports; port-security violation or MAC-flap syslog.
  • Dynamic ARP inspection drops, or arpwatch reporting a changed ethernet address for 10.20.20.8.
  • Hosts log an address-conflict event when the printer comes back.

Reservations are still the right tool for printers, cameras and anything you want stable for firewall rules: just don't treat one as a security boundary.

Curated: · Written: · Reviewed:

Q&A-51Two access circuits share the same relayed subnet but require different address policies. How can the server distinguish them?

giaddr can't separate them. giaddr is the relay's address on the client-facing subnet, and both circuits terminate on the same relayed subnet, so the server sees the same giaddr for both and selects the same scope. The lever that distinguishes them is DHCP option 82 (RFC 3046), the relay agent information option: the relay inserts a circuit-id (sub-option 1) and/or remote-id (sub-option 2) before forwarding the Discover, and the server matches that value to a class that selects the pool and gateway policy.

Mechanism. The relay receives the client's broadcast Discover, sets giaddr to its client-facing interface (10.20.0.1 for both circuits here), adds option 82 with a per-circuit identifier, typically switch port plus VLAN, and unicasts to the server. The server first picks the subnet from giaddr, then evaluates class expressions against option 82. The class decides which pool, gateway and options the client gets. giaddr selects the subnet; option 82 selects within it.

Worked example. Circuit A = Gi1/0/1 VLAN 100, circuit B = Gi1/0/2 VLAN 200, both 10.20.0.0/24, relay giaddr 10.20.0.1.

Kea 2.4 client classes:

"client-classes": [
  { "name": "circuit-A",
    "test": "substring(option[82].circuit-id, 0, 8) == 'Gi1/0/1:'" },
  { "name": "circuit-B",
    "test": "substring(option[82].circuit-id, 0, 8) == 'Gi1/0/2:'" }
]

Pools: circuit-A → gateway 10.20.0.253, circuit-B → gateway 10.20.0.254.

Trace a Discover from circuit B:

  1. Relay sets giaddr=10.20.0.1, option 82 circuit-id="Gi1/0/2:200".
  2. Server maps giaddr → subnet 10.20.0.0/24 (shared).
  3. circuit-A test: substring = "Gi1/0/2:" ≠ "Gi1/0/1:" → false.
  4. circuit-B test: true → pool B, gateway 10.20.0.254.
  5. Offer carries 10.20.0.254. Request/Ack complete.

If the relay omits option 82, both tests are false, the client falls to the default class, and circuit B gets circuit A's gateway 10.20.0.253: DORA still completes, so nothing looks broken.

Alternatives. If the two circuits can be given separate relayed subnets (separate relay interfaces, or VLANs with distinct giaddr), do that: subnet selection is simpler and needs no option 82. Separate relays per circuit work the same way. Option 82 is the right tool only when the circuits genuinely share one subnet and one giaddr. It's the wrong tool when the relay doesn't insert it, when you can't trust the relay, or when the policy difference is really a subnet difference.

Trust. Strip or ignore option 82 arriving from clients; accept it only from known relay addresses. Otherwise a client forges circuit-id and picks its own gateway.

Detection. Capture the Discover at the server and check both giaddr and option 82 presence/value; a completed DORA with the wrong gateway is the symptom.

Curated: · Written: · Reviewed:

Q&A-104A client on a remote subnet gets no address. Walk me through the DORA exchange, where the relay agent sits in it, and how you'd tell a relay fault from a scope fault.

DHCP is four messages, UDP 67 server-side and 68 client-side.

DISCOVER: the client has no address, so it sends from 0.0.0.0:68 to 255.255.255.255:67, broadcast. OFFER: the server proposes an address, unicast to the client's MAC (or broadcast if the client set the broadcast flag, which it does when it cannot yet receive unicast). REQUEST: the client broadcasts the offer it accepted so other servers withdraw theirs, still from 0.0.0.0. ACK: the server commits the lease with lease time, T1 at 50% and T2 at 87.5% of it, and options.

The relay agent is what lets this cross a router. The client's broadcast dies at the first hop, so the router interface on the client's subnet runs a relay: on Cisco, ip helper-address 10.0.0.10 on the SVI. It receives the broadcast, writes its own interface address into the giaddr field, and unicasts to the server on UDP 67. The server selects the scope by matching giaddr against a configured subnet, allocates from it, and sends the reply back to giaddr on UDP 67. The relay then delivers to the client.

That split gives you two clean failure domains.

Relay broken: the server never sees a usable request. Either no DISCOVER arrives, or it arrives with giaddr 0.0.0.0: the relay is not configured on that interface, or the packet entered on the wrong SVI. Check the helper address on the client-facing interface, interface state, an ACL permitting UDP 67/68, and a route from the relay to the server.

Scope broken: the server sees the request but cannot answer. A DISCOVER with a correct giaddr arrives and no OFFER leaves. Causes: no scope whose subnet contains giaddr; the pool is exhausted or all addresses are excluded; the scope is disabled; or the server has no route back to giaddr. On ISC dhcpd, look for "no subnet declaration" or "no free leases" in the log.

Trace of a working exchange, client on 10.20.30.0/24, relay 10.20.30.1, server 10.0.0.10:

#DirectionSrcDstNote
1Client to Relay0.0.0.0:68255.255.255.255:67DISCOVER, giaddr 0
2Relay to Server10.20.30.1:6710.0.0.10:67giaddr set to 10.20.30.1
3Server to Relay10.0.0.10:6710.20.30.1:67OFFER, yiaddr 10.20.30.50
4Relay to Client10.20.30.1:67255.255.255.255:68broadcast flag set

Worked failure: same client, but the scope is declared as 10.20.31.0/24. The server receives DISCOVER with giaddr 10.20.30.1, finds no matching scope, and drops it. The client retransmits DISCOVER with exponential backoff (roughly 1, 2, 4, 8, 16, 32 seconds with randomization) and after about a minute falls back to 169.254.0.0/16 APIPA. That APIPA address is the tell: the client never got an ACK, not that it got a bad one.

Other failure modes worth naming: a one-way route to giaddr means the server sends the OFFER and the relay never receives it; a rogue DHCP server on the segment hands out addresses from the wrong scope, which DHCP snooping on the access switch detects and blocks; and a scope mask that does not match the interface mask puts giaddr outside the declared subnet, so the server finds no match even though the scope looks right.

Do not start by rebooting the client or clearing the lease: you lose the evidence. Capture on the client subnet and at the server, compare giaddr, and the relay-versus-scope question answers itself.

Curated: · Written: · Reviewed: —

Q&A-82OSPF offers 10.0.0.0/16 at AD 110 and BGP offers 10.4.1.0/24 at AD 20. Who wins for 10.4.1.9, and who would win if they were the same length?

Short answer. For 10.4.1.9 the BGP route wins: 10.4.1.0/24 is a longer match than 10.0.0.0/16, and longest-prefix match is evaluated before administrative distance. If both protocols offered the same prefix: say both carried 10.4.1.0/24: BGP would win on AD 20 versus OSPF's 110.

Why. AD is a tiebreaker inside the routing table, applied only between routes for the identical prefix (same network and mask). It never compares a /24 against a /16. Both routes install into the RIB; the forwarding lookup then picks the longest match. On Cisco, show ip route 10.4.1.0 shows both entries with the /24 marked as the one used for forwarding; show ip route 10.4.1.9 shows only the /24.

Route in RIBADInstalled?Used for 10.4.1.9
10.4.1.0/24 via BGP20yesyes: longest match
10.0.0.0/16 via OSPF110yesno

Same-length case: if OSPF also carried 10.4.1.0/24, the BGP route at AD 20 wins and the OSPF route is not installed. Withdraw the BGP route and OSPF takes over: that's the failover you actually get.

When AD does decide. Same prefix, different sources: eBGP (20) beats OSPF (110) beats iBGP (200) on Cisco. Juniper uses route preference with different numbers (OSPF internal 10, eBGP 170) so the same topology can select differently across vendors. Floating statics (AD 250) and backdoor links are the classic cases where you tune AD deliberately: an eBGP route at 20 will beat an OSPF path at 110 to the same prefix, which is why backdoor links need the OSPF route's AD raised or the prefix filtered.

Where redistribution and filtering change the picture. Redistribution rewrites the AD of the route as it crosses protocols: OSPF routes redistributed into BGP arrive as BGP routes (AD 20), and BGP routes redistributed into OSPF arrive as OSPF external (AD 110 on Cisco). If you redistribute the /16 into BGP, you now have two BGP routes of different lengths: still longest match, still the /24. If you filter the /24 at the BGP boundary, the /16 wins by policy, not by AD. That distinction matters when you're debugging: "the /24 disappeared" is a filter or a session flap, not an AD change.

Failure mode. An operator sees OSPF AD 110 and BGP AD 20, concludes the /16 must be losing, and deletes the BGP /24 to "clean up." Traffic to 10.4.1.9 then follows the /16 into a black hole until the /24 is restored. Detection: show ip route 10.4.1.9 before and after the change, plus a traceroute to the host. The route that was winning is the one you removed.

Curated: · Written: · Reviewed:

Q&A-1A routing table shows 10.0.0.0/8 via OSPF and 10.4.0.0/16 via BGP. Where does 10.4.1.7 go, and why is administrative distance the wrong first answer?

Short answer. 10.4.1.7 goes to the BGP next hop for 10.4.0.0/16. Forwarding is longest-prefix-match: the /16 matches 16 bits, the /8 matches 8, and the longer match wins before any preference is consulted. Administrative distance is the wrong first answer because AD only ranks routes to the same prefix: these two routes describe different destinations, so AD never enters the decision.

The lookup, step by step.

StepCandidatePrefix lengthMatches 10.4.1.7?Still in play
110.0.0.0/8 via OSPF (AD 110)8yesyes
210.4.0.0/16 via BGP (AD 20)16yesyes
3keep the longest match16—10.4.0.0/16 wins

Step 3 is the whole answer. The /8 is a candidate only until a longer match appears; once the /16 is in the table, the /8 is not a fallback, not a backup, and not a tie to be broken.

Where AD is the deciding criterion. AD breaks ties between routes to the same prefix from different protocols. If both OSPF and BGP advertised 10.4.0.0/16, the prefix lengths would be equal and AD would pick the winner:

Same-prefix tie: 10.4.0.0/16 from bothCisco ADInstalled
eBGP20yes
OSPF110no
iBGP200no: OSPF at 110 wins

That table is Cisco IOS/IOS-XE defaults, and the ordering is not portable. Juniper calls the same knob preference and ranks OSPF internal (10) ahead of eBGP (170): the opposite of Cisco. So "lower AD wins" is a vendor convention, not a forwarding law, which is a second reason it is a bad first answer.

Why AD is the wrong first answer, precisely. Two reasons. First, it is a tie-break, and there is no tie here. Second, AD is a control-plane decision made once, at route-install time, when the RIB chooses which route to hand to the FIB. Longest-prefix-match is a forwarding-plane decision made per packet against the FIB, after the loser has already been discarded. Diagnosing a forwarding question with a RIB concept means reasoning about a route that is no longer in the forwarding path.

Failure mode. A stale more-specific wins LPM even when the less-specific is healthy. If the BGP session for 10.4.0.0/16 flaps and the route is left installed with an unresolvable next hop, 10.4.1.7 keeps matching the /16 and blackholes while 10.0.0.0/8 sits in the table looking like the obvious path. Worked example, figures hypothetical: 18 minutes at 2,000 pps is 2.16M packets into a dead adjacency. Detection is a forwarding-table lookup, not a RIB lookup: on Cisco, show ip route 10.4.1.7 prints the longest match and show ip cef 10.4.1.7 shows the adjacency actually used; on Juniper, show route 10.4.1.7 and show route forwarding-table destination 10.4.1.7. If the FIB entry shows a next hop that never resolves, the /16 is the problem, and withdrawing the /8 would change nothing.

Curated: · Written: · Reviewed:

Q&A-83A /32 static to a dead next hop remains installed, and 0.0.0.0/0 is healthy. Where do packets to that /32 go?

Core answer. The packets go to the dead next hop and are dropped. The /32 is still in the forwarding table, longest-prefix match picks it over 0.0.0.0/0, and the default is never consulted. Nothing in the lookup considers whether the next hop is alive.

Why. Forwarding is a FIB lookup keyed on the destination address alone. The FIB is populated from the RIB, and a static route is installed by configuration, not by reachability. Longest-prefix match is a pure function of prefix length: /32 beats /0 regardless of next-hop state. So the packet is handed to the adjacency for the dead next hop.

What happens on the wire. If the next hop is on a directly connected subnet, the router needs ARP (IPv4) or ND (IPv6) for it. Resolution fails, packets are buffered while ARP retries, then dropped. The router typically returns ICMP host unreachable, or drops silently if ICMP is rate-limited or disabled. Either way the host's traffic dies and the healthy default carries none of it.

Platform note: on IOS, a static pointing at an interface (ip route 10.1.1.1 255.255.255.255 Gi0/0) is installed unconditionally and stays. A static pointing at a next-hop IP needs ARP/ND resolution and is withdrawn when resolution fails, so the stuck-route case is the interface-static form, or a platform that does not gate install on resolution.

Fixing it: the options differ in detection time and what they require.

OptionHow it removes the /32DetectionNotes
IP SLA + trackip route 10.1.1.1 255.255.255.255 Gi0/0 track 1; object down withdraws the routeSLA probe interval, default 60 sNeeds SLA + track config; convergence bounded by the probe
BFDSub-second next-hop failure signal~50 ms × multiplierNeeds BFD on both ends; static-with-BFD support varies by platform
Floating staticBackup /32, AD 250, different next hopOnly if the primary is withdrawnUseless if the primary never leaves the RIB
Dynamic routingProtocol withdraws on next-hop lossProtocol hello/hold timersDefault then used; AD 110 OSPF, 20 eBGP

AD reference: connected 0, static 1, eBGP 20, EIGRP 90, OSPF 110, RIP 120; floating statics are conventionally 250.

Failure modes to name. A floating static that never installs because the primary is interface-static. An SLA probe that stays up because it pings a different address than the one that is dead. A stale ARP entry (IOS default timeout 4 hours) keeping a dead adjacency. Test by failing the actual next hop and confirming the /32 leaves the RIB or a tracked backup takes over, not by pinging the host.

Curated: · Written: · Reviewed:

Q&A-23You advertise 10.8.0.0/16 while 10.8.4.0/24 is down. Where do packets for 10.8.4.10 go?

The 60-second answer. The /24 is gone from the routing table, so 10.8.4.10 matches only 10.8.0.0/16 by longest-prefix match. Packets are forwarded toward whoever originated the aggregate. At that router the packet hits a discard route for the summary and dies there, or, if no discard exists, it is re-forwarded on the same /16 and loops until TTL expires. The fix is a discard route for the summary, plus withdrawing the aggregate when its last component dies.

Why the /16 is the only match. Longest-prefix match picks the most specific route in the RIB. With 10.8.4.0/24 withdrawn, the candidate set for 10.8.4.10 is just 10.8.0.0/16. The packet does not "know" the /24 is down; it follows the only route that covers the address. That route points at the aggregator (an OSPF ABR, a BGP speaker running aggregate-address, or a static summary), so the packet is delivered to the router that made the reachability promise it can no longer keep.

Trace, hop by hop.

  1. Host sends to 10.8.4.10, TTL 64.
  2. Edge R1 RIB: 10.8.0.0/16 via ABR. No /24. Forward to ABR.
  3. ABR RIB, discard present: 10.8.0.0/16 -> Null0. Packet dropped at the ABR. TTL 63.
  4. ABR RIB, discard absent: 10.8.0.0/16 via peer. Forward to peer, which forwards back: loop until TTL hits 0.
RouterRoute for 10.8.4.10Action
Edge R110.8.0.0/16 via ABRforward to ABR
ABR, discard present10.8.0.0/16 -> Null0drop
ABR, discard absent10.8.0.0/16 via peerforward, loop to TTL 0

Where the discard comes from. OSPF ABRs install a Null0 route for an area range they originate, and Cisco BGP aggregate-address installs a discard for the aggregate. That is what stops the loop: the aggregator is the one router guaranteed to have the summary, so it is the right place to drop. A static summary with no discard is the classic hole: the aggregate is advertised but nothing owns the drop.

Trade-offs of the three fixes.

  • Discard route for the summary. One line, kills the loop, keeps the aggregate stable. Cost: it hides the failure. Traffic is silently dropped with no per-prefix signal. Right when the aggregate must stay up for policy or table-size reasons.
  • Leak more-specifics. Reachability stays accurate and the failure is visible per prefix. Cost: table and LSA space, and it defeats the point of aggregating. Right when components are few and churn matters more than table size.
  • Withdraw the aggregate when the last component dies. Most honest: no promise you can't keep. Cost: reconvergence and flap for the whole /16, and impossible if the aggregate is a hard requirement.

Detection. Watch the Null0 counters on the aggregator: a nonzero rate means the summary is absorbing traffic for a dead component. Alert on "aggregate advertised with zero live components" rather than waiting for a user report.

Failure mode to name. Aggregate advertised, component withdrawn, no discard: packets loop between routers that each hold only the /16, and the symptom is TTL-expired ICMPs, not unreachables. That distinction is what tells you it is a summary hole and not a plain outage.

Curated: · Written: · Reviewed:

Q&A-2show ip route lists a prefix. Why is that not proof the packet will be forwarded there?

First 90 seconds. Before answering I'd pin down three things: which table the command read, which VRF the packet uses, and which platform. show ip route prints the RIB: the control plane's selected routes. Forwarding consults the FIB, the structure the ASIC or forwarding process actually indexes. A RIB row is a candidate; a FIB entry is a decision. They diverge whenever the install path fails, and they also diverge when the packet simply doesn't match the prefix you looked at.

How a route reaches the FIB. Best-path selection picks one route per prefix and installs it into the FIB. The FIB then resolves the next hop recursively, resolves the adjacency (ARP/ND for Ethernet, a label for MPLS), and programs the result into hardware. Each step can fail independently, and the RIB keeps the route regardless.

Where it breaks.

FailureRIBFIBCheck
Recursive next hop resolves via a route not installedpresentabsentshow ip cef <prefix> internal
Adjacency incomplete (ARP/ND unresolved)presentpresent, points to punt/dropshow adjacency <intf> detail
Hardware table full (TCAM/adjacency)presentpartially installedshow ip cef summary vs show ip route summary
Line card out of sync / OIRpresentstale or missing on that cardshow platform ... per slot
More specific prefix winspresenta different entry is usedlongest-prefix match
VRF/table mismatchpresent in another tableabsent in this oneshow ip route vrf <name>
PBR / uRPF / egress ACLpresentpresentdrop counters, show ip policy

Two of these are worth separating. A missing FIB entry is a programming fault. A present FIB entry that points at an incomplete adjacency is a resolution fault: the route is programmed, the packet still drops. And a FIB hit is necessary but not sufficient: uRPF, an egress ACL, or a policer can drop the packet after the lookup succeeds.

Worked example (hypothetical figures). A line card accepts 14,112 BGP prefixes into the RIB and programs 12,004 into the FIB; the remaining 2,108 black-hole for 3 h 20 min while every BGP session stays Established. The RIB count and the FIB count are the tell: show ip route summary and show ip cef summary disagree by 2,108.

What I'd do. Issue a hardware exact-route lookup for the failing 5-tuple (show ip cef exact-route <src> <dst> on IOS/IOS-XE, show route forwarding-table destination <prefix> on Junos) and require a programmed adjacency, not just a RIB row. Then check the adjacency rewrite and the drop counters on the ingress and egress interfaces.

Curated: · Written: · Reviewed:

Q&A-9The neighbour is FULL. Why are the prefixes you expected still missing?

FULL is a statement about one adjacency's link-state database, not about your prefix. Two routers can sit FULL for days while exchanging none of the LSAs you care about, because FULL only says the two databases match, and they can match on a database that never contained your prefix.

So the real question is where the prefix died. Three places, ranked by how often I've seen them:

  1. Never originated. The prefix was never put into an LSA at all: redistribution statement missing, network statement with the wrong mask, a route-map denying it, or the static/connected route isn't in the originator's RIB to begin with. This is the most common cause by a wide margin, and it's exactly the one FULL tells you nothing about.

  2. Originated, then filtered. The LSA exists on the originator but doesn't reach you: a stub or NSSA area dropping Type-5, area range summarization swallowing the specific, an area filter-list on the ABR, distribute-list in on your own router, or the prefix living in a different area than you assumed.

  3. In the LSDB, lost locally. The LSA is in your database but the route never lands in the RIB: administrative distance losing to another protocol, next hop unreachable, a Type-5 forwarding address that won't resolve, or a distribute-list/prefix-suppression on the receiving side. Then RIB→FIB: recursion failure, or the prefix in the RIB but not programmed.

One command splits these apart. On the router that should have the route, ask the LSDB directly:

show ip ospf database external 10.20.0.0
show ip ospf database summary 10.20.0.0
show ip route 10.20.0.0
show ip cef 10.20.0.0

Absent from the database → hypothesis 1 or 2; go look at the originator. Present in the database but absent from show ip route → hypothesis 3, and the neighbour state is irrelevant. Present in the RIB but not in CEF → forwarding plane.

A trace from an incident I worked: two core routers, OSPF area 0, adjacency FULL for days, 64 static prefixes redistributed into OSPF. On the receiving router:

show ip ospf neighbor 10.0.0.2   -> FULL/DR, up 4d02h
show ip ospf database external   -> 3 Type-5 LSAs, none of the 64
show ip route 10.30.0.0/24       -> % Network not in table

The 64 statics were in the originator's RIB, so redistribution should have produced 64 Type-5s. It produced 3. The redistribution route-map had a deny 10 with no matching permit, so 61 were dropped at the point of origination. show ip ospf database self-originate on the originator showed the same 3. Nothing about the adjacency was wrong: the LSDB was perfectly synchronized on a set of LSAs that didn't include the prefixes. It took 5 hours to find because every dashboard checked show ip ospf neighbor and went green.

When the neighbour state does matter: if the adjacency is not FULL, you get no LSAs from that neighbour at all, and the answer is trivially "fix the adjacency first." It also matters in multi-area topologies. The adjacency you're staring at may not lie on the path between the originator and you. A FULL adjacency to an ABR in area 1 says nothing about whether area 0 originated the prefix. Check the adjacency toward the originator, not the one that's convenient.

What I'd want before calling it fixed: the specific LSA (type, link-state ID, advertising router) for the missing prefix, and a FIB entry. Neighbour state is not evidence about prefix presence.

Curated: · Written: · Reviewed:

Q&A-10You have three OSPF areas and no contiguous backbone. What paths are illegal?

Three path classes are illegal in OSPF, and all three fall out of one rule: inter-area routes must traverse area 0.

1. Area-to-area shortcuts. An ABR attached to area 1 and area 2 but not area 0 will not relay prefixes between them. RFC 2328 §12.4.3 is explicit: an ABR advertises only intra-area routes from non-backbone areas into the backbone, and only backbone intra-area plus inter-area routes into non-backbone areas. So a router in area 2 never learns area 1's prefixes through that ABR, even though the adjacency sits FULL.

2. A disjoint backbone joined through a non-backbone area. Two area-0 fragments connected only via area 1: area 1 is not permitted to carry backbone inter-area routes, so each fragment keeps its own type-3 set and the halves never merge into one backbone.

3. A virtual link whose transit area is a stub. RFC 2328 §15 forbids it; the link never comes up. The transit area must be a regular area with a path between the two ABR endpoints.

Mechanism. An ABR originates type-3 summary LSAs in one direction only: non-backbone intra-area routes go into area 0, and area-0 intra-area plus inter-area routes go into non-backbone areas. It never reads type-3 LSAs from one non-backbone area back into another. That asymmetry is the loop prevention: the inter-area topology is a tree rooted at area 0, so no inter-area path can loop. Break the tree and the routes simply do not exist; nothing is misrouted.

Trace it: R is an ABR on area 1 and area 2, no area-0 interface. R originates type-3 LSAs for area 1's prefixes into area 0, but R has no area-0 interface, so they go nowhere. R does not originate area 1's prefixes into area 2, because area 2 is not the backbone. A router in area 2 therefore has no type-3 route to area 1. Intra-area routes inside each area are unaffected.

PathWhy illegalWhat you observe
area 1 ↔ area 2 via an ABR with no area 0ABR won't relay type-3 between non-backbone areasFULL adjacencies, no inter-area routes
area-0 fragment A ↔ fragment B via area 1transit area won't carry backbone inter-area routeseach fragment keeps its own type-3 set
virtual link through a stub areaRFC 2328 §15 forbids stub transitlink never comes up

Virtual link vs redesign. A virtual link is an unnumbered point-to-point adjacency between two ABRs through a transit area, making them logically adjacent in area 0. It repairs a partition without new hardware, but the transit area now carries area-0 traffic, the link dies if the transit path breaks, and it hides the real topology problem. Redesign, a physical backbone link or renumbering an area into area 0, is the correct fix when you control the topology. Reach for the virtual link only when you cannot add a link and the transit area is regular, not stub.

Detection. After a backbone-link failure, check for FULL adjacencies with missing type-3 routes: that combination means a shortcut or a partition, not a link problem. Fail the backbone path in a change window and require inter-area prefixes to stay installed through the remaining area-0 ABR.

Curated: · Written: · Reviewed:

Q&A-18One ping across an eight-next-hop ECMP succeeds. What have you not measured?

The 90-second answer

One ping is one flow, and ECMP forwards per flow, not per packet. A successful ping tells you that exactly one of the eight members forwarded one five-tuple in each direction. You have not measured the other seven members, the hash distribution across the flow space, the reverse path for any other flow, packet sizes above the ping payload, or what happens when a member fails and traffic rehashes. A green ping is a sample of size one.

Why one ping is one bucket

ECMP picks a member by hashing fields from the packet header (commonly source and destination IP, protocol, and L4 ports) and taking the result modulo the number of members. Every packet in the same flow gets the same result, which is the point: reordering is avoided. A single ICMP echo is one such flow. Worse, ICMP has no ports, so many platforms hash it on source/destination IP alone, or punt it to the control plane; the path the ping takes may not be the path a TCP flow between the same hosts would take.

What is unmeasured

  • Members 2–8: their forwarding state, loss, latency, jitter, MTU, and queue behaviour.
  • Hash uniformity: whether the hash spreads flows evenly or polarizes onto a subset.
  • Reverse path: the reply traverses the reverse ECMP group for that one tuple only.
  • Packet size: default ping payload is 56 bytes; a member with a smaller MTU blackholes larger packets silently.
  • Load: no congestion, no microbursts, no queue drops.
  • Failure and rehash: what happens to the other flows when a member goes down.
  • Liveness vs forwarding: BFD or link state can be up while the data plane drops (uRPF, ACL, MTU mismatch).
DimensionCovered by one ping
Which member1 of 8
Directionforward and reverse, for that one tuple only
Packet size56-byte payload only
Loadnone
Member failure / rehashno
Hash uniformityno

How to measure it

Probe each member with a distinct flow. Vary the tuple so the hash lands on different members, and confirm which member each flow used with a traceroute that preserves the flow identifier (paris-traceroute) or by reading per-member counters.

# Python 3.11, scapy 2.5.0
from scapy.all import IP, TCP, sr1

dst = "203.0.113.10"          # VIP behind the 8-way ECMP group
for sport in range(20000, 20008):
    pkt = IP(dst=dst)/TCP(sport=sport, dport=443, flags="S")
    r = sr1(pkt, timeout=1, verbose=0)
    print(sport, "reply" if r else "no reply")

Each source port is a distinct flow. A member that blackholes shows as no reply for the ports that hash to it, while the others still answer. Vary the destination port too, since some platforms hash on both. Require every member to be hit by at least one probe, and check that per-member counters move.

Failure modes

  • Silent blackhole member: link and BFD up, data plane drops. Only per-member flow probes catch it.
  • Polarization: two tiers use the same hash input, so a flow that picks member 3 at tier 1 always picks member 3 at tier 2; effective capacity collapses. Detected by comparing per-member counters across tiers.
  • Rehash oversubscription: losing a member moves its flows onto survivors; if the group was already near capacity, the survivors now drop. Detected by watching per-member utilisation during a planned member drain.

When one ping is enough

If you only need to know the VIP is reachable and the control plane has the route, one ping is fine. It is not a data-plane validation of an ECMP group.

Curated: · Written: · Reviewed:

Q&A-85The RIB holds 518,000 prefixes and the TCAM holds 512,000. What happens to the extra 6,000?

The 6,000 prefixes stay in the RIB and never make it into the hardware forwarding table. BGP stays Established and the routing table still shows all 518,000, so the control plane looks perfectly healthy while those 6,000 destinations are unreachable. What happens to their packets depends on the platform: most drop them in hardware, a silent black hole with no ICMP and no per-packet log, and some punt them to the CPU, which forwards them but at a fraction of the rate and with a control-plane load that a modest packet rate can turn into a problem. Which 6,000 lose is not deterministic either; it is whichever the install order or the platform's priority scheme leaves out, so you cannot predict which customers go dark.

The mechanism is the split between the two tables. The RIB lives in DRAM and is software; the FIB lives in TCAM and is what the forwarding ASIC actually reads. BGP computes a best path, the RIB manager tries to program it into the FIB, and when the FIB region of TCAM is full that install fails. The route stays in the RIB. Some platforms raise a FIB/TCAM-full condition in the log; others just stop installing, which is why comparing the counts matters more than reading the log.

If you are designing against this, the options and when each applies:

  • Filter at the edge: prefix-lists or a maximum-prefix limit on the session. Use it when you control the peer and can bound what it sends; it caps the RIB before the FIB ever fills.
  • Summarization: aggregate contiguous space into fewer entries. Use it when you own the address space and the aggregate is stable; the cost is losing the more-specifics, so a single member withdrawal can black-hole the whole aggregate.
  • Default route: 0.0.0.0/0 as one FIB entry. Use it when you do not need the full table; the cost is losing path diversity and the risk of forwarding into a loop.
  • Repartition TCAM: on platforms that support a hardware profile or SDM template, shift capacity from the ACL region to the FIB region. Use it when the platform allows it and you need FIB space more than ACL space; it is a fixed trade, not free capacity.

The failure modes to watch: silent black holes, because nothing logs per packet; CPU punt, which works until the control plane saturates; and churn, where a withdrawal followed by an announcement cannot reinstall, so a prefix that was working goes dark without any configuration change.

TableWhere it livesEntriesForwarded
RIBDRAM (software)518,000all 518,000
FIBTCAM (hardware)512,000512,000
Overflowneither6,0000

The overflow is 6,000 of 518,000, about 1.2% of the table. If those are /24s, that is 6,000 × 256 = 1,536,000 addresses: roughly 1.5 million: that no packet reaches.

To verify, compare the RIB count against the hardware FIB count and require them equal for the prefixes you expect to forward. On IOS that is show ip route summary against show ip cef summary; on Nexus, show platform tcam utilization. The exact command varies by platform, but the invariant does not: if the two counts differ, the difference is the set of destinations that are not being forwarded.

Curated: · Written: · Reviewed:

Q&A-84Forwarding is at 0 percent drop and the routing process is at 94 percent CPU. What fails next?

The next failure is the routing adjacencies, not the forwarding plane. The routing process at 94% CPU cannot service hello, keepalive, or BFD timers. BFD drops first if it is running, then OSPF/IS-IS, then BGP. Once an adjacency drops, routes are withdrawn, the FIB changes, and traffic that was being forwarded can blackhole or shift to a backup path.

Assumptions: the 94% is in a routing process (ospf_router, bgp_router, isis_router), not a punt path like IP Input; the forwarding ASIC is still switching; and CoPP is either not dropping or is dropping protocol packets because the CPU is busy.

Mechanism: control-plane protocol packets are punted to the CPU. If the scheduler starves the routing process, hellos are not sent or processed. CoPP may drop them before they reach the CPU. The data plane keeps forwarding from the FIB, so interface drop counters stay flat. That is why 0% drop is a lagging indicator.

Failure order with default timers:

ProtocolDefault hello/keepaliveDefault dead/holdTypical first failure
BFD300 ms interval (common)3× = 900 msfirst
OSPF10 s40 ssecond
IS-IS10 s30 ssecond
BGP60 s180 sthird

BFD intervals are configured, not universal; 300 ms × 3 is a common aggressive setting. OSPF and IS-IS defaults are for broadcast networks. BGP defaults are keepalive 60 s, hold 180 s.

Trade-offs:

  • CoPP tuning: raise the protocol policer or reclassify traffic. Right when CPU has headroom and drops are from a too-tight policer. Wrong here: at 94% CPU, admitting more punted packets can push the CPU to 100% and make the flap worse. On IOS-XE the protocol class is usually system-cpp-police-protocol; on IOS-XR it is control-plane-protocol. Read the configured rate and drop counter for that class with show policy-map control-plane before changing rates.
  • Protocol timers: increase hello/hold, e.g. OSPF 10/40 to 30/120, or BGP hold 180 to 300. Right for transient CPU spikes that you expect to clear. Downside: slower failure detection and longer reconvergence, and it masks a sustained CPU problem.
  • Graceful restart / NSR: right for a planned routing process restart or switchover. It does not help if hellos are already being dropped by CPU starvation.
  • Reduce control-plane load: summarization, stub areas, BGP max-prefix, route dampening. Right for a sustained 94% CPU.

Detection: show processes cpu sorted 5sec, show policy-map control-plane, show ip ospf neighbor, show bgp summary, show bfd neighbors. Logs: %BFD-6-SESSION_STATE_DOWN, %OSPF-5-ADJCHG, %BGP-5-ADJCHANGE. Interface error counters should stay flat; if they are moving, you have a separate link problem.

Check BFD first. It will tell you the control plane is starving before OSPF or BGP logs do.

Curated: · Written: · Reviewed:

Q&A-17OSPF and BGP redistribute into each other at two border routers. What can the prefixes do?

What a strong candidate says first

With mutual redistribution at two border routers and no tags or filters, a prefix can be re-originated back and forth between the two protocols. It does not multiply into an unbounded number of copies, the same prefix stays one prefix, but three things go wrong. The two border routers can disagree about the best path and the route oscillates. The prefix can be preferred via the protocol you did not intend, giving asymmetric or suboptimal forwarding. And if the redistribution is unfiltered, the whole BGP table leaks into OSPF and OSPF externals leak into BGP, inflating both databases.

Why it happens

Redistribution is re-origination. BGP→OSPF at R1 creates a Type-2 external LSA with R1 as advertising router and a seed metric (20 by default on Cisco IOS). OSPF→BGP at R2 creates a BGP route with R2's AS prepended and origin incomplete. Neither carries the other protocol's identity, so nothing stops the prefix being handed back.

Whether the loop reaches the RIB depends on administrative distance (Cisco defaults):

SourceAD
eBGP20
OSPF110
iBGP200

An eBGP-learned prefix (AD 20) always beats its OSPF copy (110), so it is protected. An iBGP-learned prefix (AD 200) loses to the OSPF copy (110), so the OSPF copy wins and gets re-originated: that is where the loop bites. OSPF Type-2 externals do not increment their metric across the domain, so the redistributed copy looks equally good at both borders. That is what makes the path flip.

Trace (AS 65000, R1 and R2 iBGP peers, both ASBRs, mutual redistribution, no tags; 10.0.0.0/24 originated by a route reflector):

  1. R1 learns 10.0.0.0/24 via iBGP, AD 200, installs.
  2. R1 redistributes BGP→OSPF: E2 LSA, advertising router R1, metric 20.
  3. R2 receives the E2 LSA. OSPF AD 110 beats iBGP AD 200, so R2 installs the OSPF copy.
  4. R2 redistributes OSPF→BGP: re-originates 10.0.0.0/24, AS_PATH 65000, origin incomplete, MED = OSPF cost.
  5. R1 receives it via iBGP, AD 200: a tie with its route-reflector copy. Best-path (lower router ID) picks R2's copy; R1's next hop for the prefix is now R2.
  6. R1 redistributes again with a metric derived from R2's copy. R2's E2 metric changes, SPF runs, R2 re-redistributes with a new MED, R1 sees a change: the cycle repeats.

The prefix count stays at one; the databases churn and the forwarding path flips.

Failure modes and how to detect them

  • Oscillation: SPF runs and BGP updates climb. Detect with show ip ospf statistics (SPF run count) and BGP update counters.
  • Table inflation: unfiltered redistribution dumps the BGP table into OSPF. Detect by counting show ip ospf database external before and after.
  • Asymmetric forwarding: R1 and R2 pick different copies. Detect by comparing show ip route for the prefix on both.
  • Blackhole: the re-originated copy points at a next hop that has gone away.

What to do

Tag on export (OSPF tag field, BGP community), deny tagged routes on the reverse redistribution, prefer a single redistribution point, and set explicit seed metrics. Verify by tracing one prefix's path attributes and LSA advertising-router across both borders and confirming the tag stops re-entry.

Worked example (hypothetical figures)

Two ABRs, mutual redistribution, no tags, unfiltered. 2,400 BGP prefixes leaked into OSPF; the re-origination then oscillated.

MetricBeforeDuring the 50 minutes
OSPF external prefixes1202,520
SPF runs per minute0.19
redistribution points with tags0 of 20 of 2

Shutting one redistribution stopped it.

Curated: · Written: · Reviewed:

Q&A-22BFD is set to 50 ms times three. The fibre is fine. Why is OSPF still bouncing?

Core answer

50 ms × 3 is a 150 ms detection window. Clean fibre rules out the physical layer, but on most platforms BFD is punted to the control plane, not forwarded in hardware. So the real question is whether the box can deliver and process a BFD packet every 50 ms, three times in a row, under current load. When OSPF bounces with clean optics, the answer is usually no, and the cause is CoPP, CPU scheduling, or queue drops, not the fibre.

Mechanism

BFD async mode: each side transmits at the negotiated interval; the peer declares the session down after missing multiplier consecutive packets. Detection time = interval × multiplier = 150 ms. That 150 ms is the budget for the whole punt path: line card → CPU queue → BFD process → timer. One packet dropped or delayed past 50 ms eats into it; three in a row kills the session. OSPF then tears down because BFD is its fast-failure signal, so OSPF looks like the victim when it is the messenger.

Arithmetic that usually explains it

50 ms interval = 20 packets/s per direction per session. 100 sessions on a box = 2,000 pps punted to the CPU. If the CoPP policer for the BFD class is set below that, you drop BFD silently while user traffic keeps flowing.

SessionsIntervalPunt rate (one direction)
150 ms20 pps
10050 ms2,000 pps
50050 ms10,000 pps

Culprits, in the order I would check

  1. CoPP. BFD is UDP 3784 (single-hop) / 3785 (multi-hop), TTL 255. Compare the policer rate for that class against the punt rate above.
  2. CPU scheduling. A starved BFD process (route churn, an unrelated high-CPU event, a process restart) slips past 150 ms.
  3. Queue drops / microbursts. Optics clean, but the interface output queue drops BFD during a burst. Look at output drops, not CRC.
  4. Echo mode. If echo is enabled, the remote must loop packets back; a broken return path or unsupported echo flaps the session even with a healthy forward path.
  5. Timer or auth mismatch. One side 50×3, the other 300×3 or different auth: the session never stabilises.

Diagnostics

show bfd neighbors details
show policy-map control-plane
show processes cpu sorted | include BFD
show interface <intf> | include drops
show logging | include BFD

Read the BFD session history for up/down timestamps, the CoPP drop counter for the BFD class, CPU during the flap window, and interface output drops.

Config

interface GigabitEthernet0/0/0
 bfd interval 50 min_rx 50 multiplier 3
!
router ospf 1
 bfd all-interfaces

Trade-off

150 ms is aggressive. If the platform cannot guarantee punt delivery at that rate, relax to 150×3 (450 ms) or move BFD to hardware offload where the line card supports it. Do not touch the fibre: it is not the problem.

Curated: · Written: · Reviewed:

Q&A-12An implementation still accepts and advertises all eBGP routes by default. What does RFC 8212 say you should do instead?

Core answer (first 90 seconds)

RFC 8212, Default External BGP (eBGP) Route Propagation Behavior without Policies (BCP 194, July 2017), says the default must be reject, not allow. An eBGP speaker MUST NOT advertise a route to an external neighbor unless an export policy explicitly permits it, and MUST NOT accept a route from an external neighbor unless an import policy explicitly permits it. No policy configured means no routes in either direction, not "all routes".

Two things it does not say. It does not prescribe what the policy contains; it only moves the default. And it applies to eBGP only: iBGP sessions and route reflection are out of scope, because the trust model inside an AS is different. Assume here a session between two different ASes on a stack that still defaults to allow.

Why the old default existed

BGP predates policy engines. The first implementations exchanged whatever they had, and when import/export filters were added they defaulted to permit so existing deployments didn't break. That default survived for two decades. RFC 8212 flips it: the safe behavior is what you get with no configuration, and the dangerous behavior requires you to type it.

What to configure instead

Per eBGP session, per address family:

  • Import: prefix filter (or RPKI/ROA validation), AS-path filter (reject your own AS, reject bogons and private ASNs), and a max-prefix limit as a backstop.
  • Export: only prefixes you originate plus customer routes; tag with communities; never re-export transit routes to another transit.

Max-prefix is a backstop, not a substitute for policy: it caps the damage, it does not decide what is legitimate.

Implicit allow (pre-8212 default)RFC 8212 default
No import policyinstalls every prefix the peer sendsinstalls 0
No export policyadvertises entire Loc-RIBadvertises 0
Failure modefull-table leak / transit leaksession up, no routes

Worked trace

Session to 192.0.2.1 (AS 65002) comes up with no import policy.

Implicit allow:

  • t=0: session Established.
  • t≈0.1s: first UPDATEs arrive; every prefix goes into Loc-RIB.
  • t≈30s: PfxRcd ≈ 900,000 (the global IPv4 table passed 900k prefixes in 2023). Those prefixes are now eligible for your other peers.
  • If this is a lab or edge link, traffic for those destinations is now attracted to it.

RFC 8212 default:

  • t=0: session Established.
  • t≈0.1s: UPDATEs arrive, no import policy matches, nothing installed. PfxRcd stays 0, PfxSnt stays 0.
  • The session is up but carries no routes: the correct, visible failure.

Detection

show bgp summary PfxRcd/PfxSnt per neighbor; max-prefix syslog warnings; prefix-count alarms. A neighbor whose PfxRcd is far above what you expect is the signal.

Implementation note

Not every stack defaults this way yet. FRR exposes bgp ebgp-requires-policy; check your release's default. On stacks that still default to allow, configure the reject explicitly and treat "no policy on an eBGP session" as a config error in review.

Curated: · Written: · Reviewed:

Q&A-13You need your AS's outbound traffic to prefer one exit. Why is MED the wrong first knob, and what does local preference actually decide?

Core answer (first 90 seconds)

Local preference is the knob for outbound exit selection; MED is a request to a neighbour about its inbound selection. They sit at different points in the BGP decision process, and MED is compared after local preference and AS path length, so it cannot override either.

Assumptions: AS 65001 with two transit exits, Cisco IOS-style decision order, default bgp always-compare-med off.

Where each attribute sits

StepAttributeRule
1Weighthighest; local to one router, Cisco-only
2Local preferencehighest; AS-wide via iBGP
3Locally originatednetwork/redistribute/aggregate
4AS path lengthshortest
5OriginIGP < EGP < incomplete
6MEDlowest; only among paths from the same neighbour AS by default
7eBGP over iBGP—
8IGP metric to next hoplowest

Local preference is step 2, before AS path length. MED is step 6, after it. That ordering is the whole answer: a higher local preference beats a shorter AS path, and a lower MED is never consulted if the AS paths differ.

What local preference decides

Local preference is a well-known discretionary attribute, default 100, higher wins. It is propagated inside the AS over iBGP and is not sent to eBGP peers. Set it on inbound eBGP sessions at the edge; every iBGP speaker in the AS then agrees on the exit.

route-map LP-PRIMARY permit 10
 set local-preference 200
route-map LP-BACKUP permit 10
 set local-preference 100
router bgp 65001
 neighbor 203.0.113.1 route-map LP-PRIMARY in
 neighbor 198.51.100.1 route-map LP-BACKUP in

Why MED is the wrong first knob

MED is optional non-transitive: it is sent to eBGP peers, not propagated beyond the receiving AS, and by default compared only among paths learned from the same neighbouring AS. Setting MED on your outbound announcements tells the neighbour which of your entry points you prefer: it does not change your own exit. Setting MED on routes you receive puts it at step 6, behind local preference and AS path length.

Worked trace

Prefix 10.0.0.0/8, two candidates on an internal router:

  • A via 203.0.113.1: LP 200, AS path 65001 174 3356, MED 0
  • B via 198.51.100.1: LP 100, AS path 65001 174, MED 0

Step 2: LP 200 > 100 → A wins. B's shorter AS path is never reached. That is local preference doing its job.

Now the failure: primary tagged MED 50 outbound, LP left equal at 100.

  • A via 203.0.113.1: LP 100, AS path 65001 174 3356, MED 50
  • B via 198.51.100.1: LP 100, AS path 65001 174, MED 0

Step 4: AS path 2 < 3 → B wins. MED at step 6 is never consulted. If the two exits are to different neighbouring ASes, MED is not compared at all by default and the tie falls to IGP metric to next hop (step 8), which is how a team ends up on the backup exit with the primary idle.

Failure modes and detection

  • MED from different neighbours silently ignored: show ip bgp 10.0.0.0/8 marks the best path; the reason line names the deciding attribute. If it names AS path or IGP metric, MED was never in play.
  • Local preference not propagated: a missing iBGP session, or a route reflector that does not reflect the edge route, leaves some routers at LP 100 and others at 200. Compare show ip bgp 10.0.0.0/8 across routers; divergence in the best path is the signal.
  • Provider ignores MED: many ISPs strip or override it, or require a bilateral agreement. Communities are the reliable request; MED is best-effort.

Verification

Do not call it fixed until the decision trace shows local preference as the first differing attribute among internal candidates for the prefix, and every iBGP speaker picks the same exit.

Curated: · Written: · Reviewed:

Q&A-14One path has two AS hops and another has four. Why might the four-hop path still be the right one?

Because AS_PATH length is a tiebreaker in the BGP decision process, not a measure of distance. It counts ASNs, and an ASN is not a hop, a cable, or a millisecond. A four-AS path can be a short direct peering; a two-AS path can be a transit provider hauling the traffic across an ocean and a congested exchange.

Two mechanisms decide this.

LOCAL_PREF is compared first. In the common Cisco ordering the comparison runs weight, LOCAL_PREF, locally originated, shortest AS_PATH, origin type, MED, eBGP over iBGP, IGP metric to next hop, then tiebreakers down to router ID. If you set LOCAL_PREF 200 on the four-hop peering and leave the transit at the default 100, the AS_PATH comparison never happens: the four-hop path wins outright. That is the normal way to say "this path is better despite looking longer":

route-map PREFER-PEER permit 10
 match ip address prefix-list VOICE
 set local-preference 200

When LOCAL_PREF does tie, AS_PATH length is a crude proxy. It counts ASNs, not routers, and it is blind to everything inside an AS. A tier-1 ASN can span three continents; a regional ISP's four ASNs can sit in one metro. The IGP metric to the next hop is compared later, so BGP will happily pick a two-AS path whose next hop is 40 ms away over a four-AS path whose next hop is 2 ms away. AS path prepending exists precisely because operators make paths look longer on purpose to steer inbound traffic: a four-hop path may be the one the far side wants you to use.

Direction matters too. You control outbound selection with LOCAL_PREF; inbound is influenced by MED, communities, and prepending. The path you install outbound is not necessarily the path the reply takes, so a short outbound AS_PATH can still pair with a long, lossy return path.

Failure mode. Outbound policy preferred a two-AS transit over a four-AS peering. The transit added 91 ms versus 12 ms on the peering, and a voice cluster stayed pinned to the long path for three days because the attribute looked shorter. Detection is per-next-hop measurement, not topology: hash-pinned probes so the same flow hits the same next hop, show ip bgp <prefix> to read the chosen next hop and its LOCAL_PREF, and mtr to each candidate next hop. Worked example, figures illustrative:

PathAS hopsRTTDays selected
transit291 ms3
peering412 ms0 until policy change

When the two-hop path is right. If the four ASNs are a hairpin through a distant IX while the transit lands in the same metro as your edge, the two-hop path is correct and the four-hop path is the trap. The rule is the same either way: install from measured RTT and loss on the forwarding path you actually use, and encode that decision in LOCAL_PREF or communities rather than trusting AS hop count. AS hops are not milliseconds.

Curated: · Written: · Reviewed:

Q&A-11The eBGP session has been Established for two hours. Why might you still be advertising nothing useful?

Established means the TCP session came up and both routers agreed on the OPEN: capabilities, hold time, and the AFI/SAFI pairs each side offered. It says nothing about any prefix. A session can sit Established for hours with an empty Adj-RIB-Out, and the number most people check is the wrong one: the PfxRcd column in show bgp ipv4 unicast summary counts prefixes you received. It says nothing about what you sent.

Four gates have to be open at once for a prefix to leave:

  1. The address family is activated on both ends. The session is per-neighbor; the AFI/SAFI set is negotiated in the OPEN. If the family was negotiated but you never activated it locally, the session is up and carries nothing in that family. If it was never negotiated, the neighbor does not appear in that family's summary at all, and you need a session reset to add it. On IOS, IPv4 unicast is activated by default for a new neighbor unless no bgp default ipv4-unicast is set; IOS-XR and FRR require the activation explicitly.
  2. The prefix is in your Loc-RIB. network 203.0.113.0 mask 255.255.255.0 advertises nothing unless an exact /24 exists in the routing table. A /25 learned from a downstream customer does not match it.
  3. Outbound policy permits it. An implicit deny at the end of a route-map, a prefix-list whose le/ge excludes the length you meant, or a missing neighbor ... route-map X out all produce silence.
  4. The prefix is eligible and best in your Loc-RIB. A route whose next hop does not resolve in your IGP is not installed, so it is never advertised. And BGP advertises only the best path per prefix: if a competing path wins, that is what leaves, not the one you meant.
GateCheckFailure signature
AF activatedshow bgp ipv4 unicast summaryneighbor absent from that family's summary
Prefix in Loc-RIBshow route 203.0.113.0/24no exact match
Outbound policyshow bgp ipv4 unicast neighbors 192.0.2.1 advertised-routesempty, or fewer prefixes than the policy intends
Eligible and bestshow bgp ipv4 unicast 203.0.113.0/24not in Loc-RIB, or best path is not the one you meant

A peering I worked on stayed Established for two hours with IPv4 unicast never activated on our side. Adj-RIB-Out was empty and 1,100 customer prefixes sat in the Loc-RIB the whole time. Nothing in the session log looked wrong, because nothing was.

Default export versus explicit policy is a real choice, not a hygiene question. Advertising everything in the family is correct for a full-table transit peer, and it fails open: a new prefix is advertised the moment it appears. Explicit outbound policy fails closed: a new customer prefix is invisible until someone adds it to the prefix-list. That is the right default when you are a transit for someone else, when you peer at an IXP and must not leak, or when the peer is a route server. Pick per peer and write down which you picked, because the failure modes are opposite: default export leaks, explicit policy goes quiet.

A quiet session is sometimes the design. A remote-triggered blackhole peer should carry only /32s tagged with a blackhole community, so zero prefixes in the steady state is correct. A receive-only peer, or a backup path held down by a community until failover, is the same. For those, the check is not "is Adj-RIB-Out non-zero" but "does it match the intended set", which may legitimately be empty.

Other failure modes worth ruling out:

  • Stale Adj-RIB-Out. advertised-routes recomputes from current policy, so after a policy change without a route refresh it can show prefixes the peer never received. Compare against the peer's Adj-RIB-In, or send clear bgp ipv4 unicast 192.0.2.1 soft out.
  • Aggregation. aggregate-address ... summary-only suppresses the more-specifics; if the aggregate itself never forms because no contributing route carries the right attributes, you advertise neither.
  • Peer-side filtering. Your Adj-RIB-Out is non-zero and their Adj-RIB-In is empty: that is their inbound policy or max-prefix limit, not yours. Check with them before re-rolling your export policy.
  • ORF. If the peer negotiates outbound route filtering, their prefix list prunes your Adj-RIB-Out before it is sent.

One next-hop caveat, since it is easy to get backwards: eBGP rewrites the next hop to the local interface address by default, so the peer can normally reach what you send. The exception is a preserved next hop (third-party next hop on a shared IXP subnet, or next-hop-unchanged on a route reflector) where the next hop you carry may not be reachable by the peer. That is the case to check on the peer's side.

The check that settles it is a non-zero, policy-expected count in Adj-RIB-Out for the specific address family, confirmed against the peer's view, not the session state.

Curated: · Written: · Reviewed:

Q&A-15A prefix is RPKI invalid. Do you drop it, and what happens if you still forward it?

Yes: make invalid ineligible at import. Not-found is a different state and should stay.

RFC 6811 gives a route three outcomes against the ROA set: valid (a ROA matches both the origin AS and a prefix length within maxLength), invalid (a covering ROA exists but the origin AS differs, or the prefix is longer than maxLength allows), and not found (no covering ROA). Invalid is a positive contradiction of a signed authorization. Not found is missing data. Collapsing the two into one bucket is the usual mistake.

So the policy is: reject invalid at import, keep not found, and alert on the invalid count so a stale ROA doesn't quietly blackhole a legitimate more-specific.

What forwarding it costs you. An invalid route is usually a more specific, so longest match wins and it takes the traffic. The valid covering route sits in the RIB and receives nothing. Nothing looks broken: the session is up, the path resolves, packets flow. If the invalid route is a hijack, you deliver your users' traffic to the hijacker for as long as it lives, and at forwarding time you cannot tell a hijack from a legitimate route whose ROA went stale. The second direction matters too: if you originate an invalid route, every neighbor running origin validation drops it and your prefix goes dark there.

The failure I've seen: invalids accepted to "avoid risk." One hijacked /24 stayed in the FIB for 6 hours and took 19,000 packets, while the valid covering /22 got none.

PrefixRPKIInstalledPackets
203.0.112.0/22validno: less specific0
203.0.113.0/24invalidyes, 6 hours19,000

Alternatives. Some operators set invalid to a low local-pref instead of dropping, trading hijack exposure for outage resistance. That is defensible only with the invalid count monitored and a documented threshold for switching to drop. Dropping outright is the safer default.

Verify it, don't assume it. Inject a test invalid prefix in a lab VRF and require it to be discarded before RIB installation, not merely deprioritized. In production, graph the invalid count per peer; a step change is either a hijack or a ROA that needs fixing, and you want to know which within minutes.

Curated: · Written: · Reviewed:

Q&A-16A customer re-advertises your transit table to another provider. What did they just become, and how do you limit the blast?

They became your transit provider, and you became theirs, for free.

You buy transit from upstream U. Your customer C takes the routes you learned from U and announces them to a third network B. B now reaches those prefixes along B → C → you → U. Two role changes happen at once:

  • C is now carrying traffic toward you for prefixes you pay U to reach. Feeding you traffic is what an upstream does. Your customer is now your upstream.
  • You are now providing free transit to B. B gets your paid transit table with no contract, and you pay U for the traffic it generates.

The blast is not "a few prefixes leaked." It is B's entire demand for your transit table arriving on a customer link sized for the customer's own traffic.

Assumptions: C is a customer (you sell them transit), U is your upstream, B is a third party C announces to, and the C session has no role-based export policy.

Limiting the blast: layered, cheapest first:

ControlCatchesBlind toCost
Max-prefix on the C sessionVolume leaks (C suddenly sends 950k prefixes)A 2-prefix more-specific leakOne line
Prefix filters (IRR objects / RPKI ROA)C announcing U's prefixes, which no ROA or IRR object names C forC announcing a prefix it is authorized for to the wrong partyMaintained IRR/ROA data
AS-path filterReject from C any path containing your own ASN (loop) or your upstream's ASNA leak whose path does not traverse youLocal, no cooperation
RFC 9234 BGP roles + OTC attributeTag routes learned from customers with Only-To-Customer; reject any OTC route arriving from a peer or upstreamNothing prefix-specific: it is role-basedBoth sides must implement it

Deciding criteria: if you can do only one thing, do prefix filters plus max-prefix. Both are local and need no cooperation. Add OTC where the peer supports it, because it is the only control that survives a customer who is legitimately authorized for the prefix.

Worked example (figures illustrative). You are AS64500, transit from AS3356. Customer AS65010 normally sends you 12 prefixes and 400 Mbps inbound. At 14:02 it announces your full transit table (950,012 prefixes) to AS65020. Your session with C accepts them; max-prefix was unset.

Trace: AS65020 installs 950,012 routes with AS_PATH 65020 65010 64500 3356. Its traffic to those prefixes now arrives on your C link. Inbound on that 10 Gbps link goes 400 Mbps → 3.4 Gbps within 90 seconds, and your egress to AS3356 rises by the same 3 Gbps, billed to you. Detection: prefix count accepted from C jumps 12 → 950,012, and AS_PATHs from C now contain 3356, your own upstream. An AS-path filter rejecting paths containing 3356 from a customer, or a max-prefix of 1,000, drops the session at 14:02.

Where a filter would not help: if C holds a valid ROA for the prefix and announces it to B, RPKI and IRR both pass it. The prefix is legitimate, the audience is wrong. Only role-based export (OTC) or B's own import policy stops that.

Curated: · Written: · Reviewed:

Q&A-20You enable strict uRPF on an edge with two ISPs. Whose legitimate inbound packets die?

The judgement in uRPF strict versus loose is which state the packet uses, not which state the operator intended.

Strict uRPF requires the source to be reachable via the ingress interface. Loose uRPF requires the source to be reachable via any interface. Asymmetric multihoming fails strict and usually passes loose.

Concretely, use loose or feasible-paths uRPF on asymmetric edges, reserve strict for single-homed access where spoofing risk dominates, and test inbound from each provider with a source that returns via the other.

The reason for that specificity is a failure I have seen: Strict uRPF on both ISP interfaces dropped 31 percent of inbound customer replies for 4 hours because return routing preferred the other ISP, while spoofed packets were not the traffic that failed.

Two ISPs, strict mode.

ModeInbound droppedHoursCause
strict31 percent4source not via ingress
loose (after change)0 percent of legitimate—any-interface reachability

I would not consider it settled without evidence: send a packet whose source prefix is installed only on the other ISP interface and require strict to drop it and loose to accept it.

Strict uRPF assumes symmetric routing.

Curated: · Written: · Reviewed:

Q&A-3The VLAN is saturated and traceroute looks clean. How do you tell a Layer 2 loop from an IP forwarding loop?

The 90-second answer

TTL is the discriminator. Ethernet has no hop limit, so a Layer 2 loop is self-sustaining: the frame circulates forever and every pass floods it again, so the broadcast domain saturates and no ICMP Time Exceeded is ever generated. IP has a hop limit, so a forwarding loop is finite per packet: TTL hits zero, the router emits ICMP Time Exceeded (Type 11, Code 0), and traceroute shows the same two or three hops repeating as TTL climbs. A storm with zero TTL-expired messages is Layer 2; repeating L3 hops with a climbing Time Exceeded counter is a routing loop.

Why they look different

A Layer 2 loop lives below IP. The looping frames are broadcast, multicast, or unknown-unicast: the switch floods them out every port in the VLAN, including back toward the source of the loop. Nothing decrements, so the only thing that stops it is STP blocking a port or a human pulling a cable. Because the traffic never gets routed, the gateway is still one hop away and traceroute can look perfectly clean while the VLAN is at line rate.

An IP loop lives above the link layer. Each router decrements TTL by one; at zero it drops the packet and sends ICMP Time Exceeded back to the source. The looped packets are unicast to one destination, so they don't flood the broadcast domain: VLAN utilisation can stay low while the looped path itself is saturated.

What to measure

  • show mac address-table / MAC move counters: the same MAC learned on two ports, flapping, is the signature of a Layer 2 loop.
  • Interface counters: broadcast + multicast + unknown-unicast pps per VLAN. A storm shows all three climbing together.
  • show spanning-tree topology-change counters and root-bridge changes.
  • Switch CPU: punted control-plane traffic (STP BPDUs, ARP) spikes when a loop floods.
  • On the IP side: show ip traffic ICMP Time Exceeded counter, and traceroute output.
ObservationLayer 2 loopIP forwarding loop
TTL-expired ICMP0climbing, thousands/min
tracerouteclean, gateway 1 hopsame 2-3 hops repeating
Traffic typebroadcast/multicast/unknown-unicastunicast to one dest
MAC movesflappingnone
Stops when source stopsnoyes

The decisive test

Stop the source. A Layer 2 loop keeps storming with no host generating traffic: the frames circulate on their own. An IP loop drains within TTL hops once the source stops. If you can't stop the source, capture on a suspect link: a Layer 2 loop shows the same frame with the same source MAC and unchanged TTL repeating; an IP loop shows the same 5-tuple with TTL decrementing on each pass.

Failure modes and gotchas

  • "Traceroute is clean" is not proof of no Layer 2 loop. If the storm has starved the switch control plane, traceroute may fail outright rather than look clean; and if the loop is confined to a VLAN whose hosts don't route, traceroute never touches it.
  • A routing loop can hide if the loop is between two routers that both have a route to the destination and the source is inside the loop: you'll see Time Exceeded from both, not one.
  • STP misconfiguration (BPDU filter on an access port, or a loop where STP is disabled) is the usual root cause of the L2 case; mutual route redistribution or a static route pointing back is the usual root cause of the L3 case.

Curated: · Written: · Reviewed:

Q&A-4Spanning tree elected a closet switch as root. What traffic path did you just accept?

The short answer

Assuming one STP instance for the VLAN (RSTP, or a single PVST+/MSTP instance) and a campus with a distribution pair above access closets: the closet is now the root bridge, so it is the reference point for every path cost in that VLAN. The tree is rooted there, which means every path between two points in the VLAN bends toward the closet. Traffic between two hosts that both sit above the closet (two servers on the distribution pair, or hosts in two different closets) now hairpins through the closet and back, and the closet's uplink becomes the transit for every flow that crosses the root.

What that costs, traced

VLAN 20. D1 and D2 are the distribution pair, joined by a 10 Gbps link. A1 is a new access switch dual-homed to both over 1 Gbps. Nobody set priorities, so all three sit at the default 32768 and the lowest MAC wins: A1.

With A1 as root:

  • A1 is root; all its ports are designated.
  • D1's root port is its uplink to A1, cost 4 (1 Gbps, 802.1D short costs).
  • D2's root port is its uplink to A1, cost 4.
  • On the D1–D2 link both advertise cost 4; the tie breaks on bridge ID, so D1's port is designated and D2's blocks. The 10 Gbps link sits idle.

Traffic from a host on D1 to a host on D2 now goes D1 → A1 → D2: two 1 Gbps hops instead of one 10 Gbps hop.

With D1 as root instead: D2's root port is the direct 10 Gbps link (cost 2), A1's root port is its uplink to D1, and the D1–D2 flow is one hop at 10 Gbps.

D1 root (intended)A1 root (accepted)
D1→D2 path1 hop, 10 Gbps2 hops, 1 Gbps
D1–D2 10 Gbps linkforwardingblocked
Bandwidth ceiling10 Gbps1 Gbps

Serialization makes the ceiling concrete: a 1500-byte frame is 12,000 bits. 12 µs on a 1 Gbps link, 1.2 µs on 10 Gbps. Under load the hairpin also queues behind every other flow in the VLAN that now shares the closet uplink.

When a non-core root is fine

The rule is not "the root must be a core switch." It is "the root must be the most central, highest-bandwidth, most stable device for that instance."

  • Per-VLAN roots. Rooting VLAN 10 on D1 and VLAN 20 on D2 is normal and desirable: it load-shares the uplinks. "Core pair owns root" is per VLAN, not one global root. PVST+ and MSTP elect a root per VLAN/instance; RSTP elects one for the whole topology.
  • Stub sites. A branch with a single uplink is a line, not a loop. Root placement changes nothing about the path, so a local root is harmless.
  • Flat or single-switch networks. If there is one switch, it is the root and that is correct.
  • Leaf-spine with STP only at the access layer. The root is the leaf pair, not a "core."

Failure modes and detection

  • The closet uplink saturates: every inter-switch flow in the VLAN shares it.
  • A topology change at the closet sends TCNs that flush MAC tables VLAN-wide, causing flooding.
  • A closet reboot or uplink flap reconverges the whole VLAN.

Check with show spanning-tree root (root ID, root path cost, root port per VLAN) and show spanning-tree vlan 20: the closet prints "This bridge is the root." On the distribution pair, root path cost should be 0 or small. Fix by setting spanning-tree vlan 20 root primary and root secondary on the intended pair (24576 / 28672), or explicit priorities.

Curated: · Written: · Reviewed:

Q&A-6Untagged frames on a trunk are being learned in different VLANs at each end. What is leaking?

A native VLAN mismatch. The untagged frames are leaking between the two VLANs each end calls native, so the two broadcast domains merge into one. Nothing in the frame carries the VLAN, so each switch re-derives it from local config and they disagree.

Assumptions: an 802.1Q trunk with both ends in trunk mode, and untagged traffic belonging to the native VLAN on each side (VLAN 1 by default unless it was changed).

Why the tag disappears

802.1Q has no tag for the native VLAN. On ingress, an untagged frame is assigned to the port's native VLAN (its PVID). On egress, frames in the native VLAN are sent untagged. The tag is never carried, so the receiving end re-derives the VLAN from its own PVID. If switch A's native is 10 and switch B's is 20, a frame A sends untagged for VLAN 10 lands in VLAN 20 on B.

What leaks is both planes:

  • Data plane: broadcasts, unknown unicast and multicast from each side's native VLAN.
  • Control plane: protocols sent untagged on the native VLAN. CDP, DTP, VTP, LLDP, PAgP/LACP, and STP/RSTP BPDUs.

Worked trace (hypothetical)

Host A sits in VLAN 10 on switch A; switch A's trunk native is 10, switch B's is 20.

StepWhereFrame stateVLAN
1Host A sends an ARP broadcastuntagged10
2Switch A trunk ingress, PVID 10untagged, assigned10
3Switch A floods out the trunk, VLAN 10 is nativeegress untagged10
4Switch B trunk ingress, PVID 20untagged, assigned20
5Switch B floods to its VLAN 20 access portsuntagged20

Host A's ARP now appears in VLAN 20. A reply from a VLAN 20 host travels the same path in reverse and is learned in VLAN 10 on switch A. Both VLANs are one domain.

Failure modes

  • STP merge. BPDUs are untagged, so the two VLANs' spanning-tree instances become one. You get a single root bridge for two VLANs, ports blocking where you did not expect, and a loop if the merged topology has no blocking point.
  • VTP propagation. A VTP advertisement crosses and rewrites the VLAN database on the far side.
  • Wrong subnet. DHCP replies from the other VLAN's pool reach hosts; duplicate addresses and ARP conflicts follow.
  • Security boundary gone. Hosts in VLAN 10 are directly reachable from VLAN 20, defeating whatever the segmentation was for.

Detecting it

show interfaces <int> trunk prints the native VLAN per port. show cdp neighbors detail prints the neighbor's advertised native VLAN, so a mismatch is visible from either end. show spanning-tree showing an unexpected root, and DHCP snooping or dynamic ARP inspection logs, are secondary signals. To confirm rather than infer, send a known untagged test frame and require it to be learned under the same VLAN ID on both switches.

Fixing it

Set the native VLAN explicitly and identically on both ends. Better, move it off VLAN 1 to a dedicated unused VLAN (999 is common) and prune that VLAN from the allowed list on access ports. Where the platform supports it, tag the native VLAN so no frame crosses untagged: on Cisco IOS that is the global vlan dot1q tag native. Leaving the native VLAN at its default is what makes this failure common, because a default on one end and a configured value on the other is a mismatch nobody typed.

Curated: · Written: · Reviewed:

Q&A-5Both ends of a trunk are up and in trunk mode. Why can a VLAN still be silent across it?

The 60-second answer

An up/up trunk proves only that the link and line protocol are up. A VLAN crosses it only when three independent gates are open on both ends:

  1. The VLAN exists in the switch's VLAN database.
  2. The VLAN is in the trunk's allowed list.
  3. The VLAN is in STP forwarding state on that trunk port.

Close any one on either end and the VLAN is silent while the trunk stays up and other VLANs keep passing. show interfaces trunk prints exactly these three lists, so it tells you which gate is shut before you change anything.

Why the gates are independent

  • Allowed list. switchport trunk allowed vlan defaults to all VLANs (1–4094) on IOS, but any explicit allowed vlan statement replaces that default. A one-sided switchport trunk allowed vlan remove 47 on the far end is invisible from the near end.
  • VLAN database. A VLAN that isn't defined can't be active. On IOS, extended-range VLANs (1006–4094) can't be created while the switch is a VTP server or client in VTP v1/v2: the switch must be transparent or VTP off (VTPv3 lifts this). A VLAN 1100 that exists in your config may not exist on the far switch.
  • STP state. PVST+ runs an instance per VLAN; MSTP maps VLANs to instances. If the two ends disagree on MST region name, revision, or the VLAN-to-instance map, they are in different regions, the trunk becomes a region boundary, and per-VLAN forwarding can differ from what the physical link suggests. An unmapped VLAN falls into instance 0 (CIST).
  • VTP pruning. With pruning enabled, a VTP server/client drops a VLAN from a trunk when no downstream switch has an active port in it. Pruning covers VLANs 2–1001 only and is off by default; transparent switches don't prune. A pruned VLAN leaves the trunk's forwarding list with no config change on the trunk itself.

The artefact that answers it

show interfaces trunk on each end:

Port    Mode  Encapsulation  Status      Native vlan
Gi0/1   on    802.1q         trunking    1

Port    Vlans allowed on trunk
Gi0/1   1-4094

Port    Vlans allowed and active in management domain
Gi0/1   1-10,47

Port    Vlans in spanning tree forwarding state and not pruned
Gi0/1   1-10

VLAN 47 is allowed and active but missing from the third list, so it is either pruned or not forwarding. show vtp status (pruning mode) and show spanning-tree vlan 47 (per-VLAN state) separate the two. If 47 were missing from the second list, the allowed list or the VLAN database is the problem.

Worked example (illustrative numbers)

Two switches, one LAG trunk, VLANs 10 and 47. VLAN 10 carries fine; VLAN 47 is silent.

GateVLAN 10VLAN 47
In VLAN database, both endsyesyes
In trunk allowed list, both endsyesno: pruned on far end
STP forwarding on trunk, both endsyesn/a (pruned)
Endpoints affected01,840
Time to notice—7 h

The near end's show interfaces trunk lists 47 as allowed; the far end's does not. That asymmetry is the whole bug: the trunk is up, VLAN 10 passes, and no interface counter moves for VLAN 47.

Failure modes and how you catch them

  • Allowed-list mismatch: compare show interfaces trunk on both ends; the "allowed on trunk" lists must match.
  • VLAN not in database: show vlan id 47 on both ends; check VTP mode for extended VLANs.
  • STP blocking or MST region mismatch: show spanning-tree vlan 47 on both ends; compare region name/revision/map with show spanning-tree mst configuration.
  • VTP pruning: show vtp status for pruning mode, show vtp counters for prune events.
  • Native VLAN mismatch: untagged frames land in the wrong VLAN; CDP logs "Native VLAN mismatch" and show interfaces trunk shows different native VLANs per end.

What I would not do

I wouldn't call it fixed because the trunk is up, or because a ping from the switch's own SVI succeeds: that SVI is local to the switch and never crosses the trunk. I'd capture a tagged 802.1Q frame for VLAN 47 on the trunk (SPAN or a tap) and confirm the same VLAN is forwarding on both member switches. Until a tagged frame for that VLAN crosses the link in both directions, an up trunk is not evidence the VLAN is carried.

Curated: · Written: · Reviewed:

Q&A-7A MAC address oscillates between two ports. What plane is sick?

The 90-second answer

The flap shows up in the data plane: the MAC address table is a forwarding-plane structure, and the unknown-unicast flooding it triggers is forwarding-plane behaviour. But the fault is usually in the control plane, STP/RSTP/MSTP failing to keep a redundant path blocked, or outside the switch entirely, in a host that has put one MAC on two ports. So: symptom in the data plane, root cause in the control plane or on the host. If STP is converged and both ports are access ports to the same server, the switch's control plane is healthy and you are looking at a NIC-teaming or hypervisor problem.

Why the planes separate like that

PlaneOwnsWhat a flap looks like here
ControlSTP/RSTP/MSTP BPDUs, LACP, routing protocolsTopology-change notifications, a BPDU on a port that should be blocked, LACP not bundling
DataMAC table, forwarding, floodingSame MAC learned on two ports, unknown-unicast flood, table churn
ManagementCounters, syslog, telemetryMACFLAP syslog, move counters, CPU spent processing notifications

Mechanism

A switch learns (source MAC, VLAN) → ingress port. When the same MAC arrives on a different port, the entry is rewritten and the old binding dropped. Two ports seeing frames from one MAC means either the frames genuinely arrive on both (a loop, or a host with two live NICs sharing a MAC) or the topology is changing faster than the table can settle.

Cisco IOS logs it as %SW_MATM-4-MACFLAP_NOTIF: Host 0011.2233.4455 in vlan 10 is flapping between port Gi1/0/12 and port Gi1/0/18. That message is the data-plane symptom; the real question is which plane produced the two live paths.

Ranked causes

  1. L2 loop with STP not blocking: control plane. Check show spanning-tree for a topology-change count that keeps climbing and a port that should be blocking but is forwarding. A broadcast storm follows.
  2. Dual-homed host, mis-bundled: host/outside. Two access ports, one MAC, no LACP. The common one in virtualised racks.
  3. Active-active MLAG/vPC misconfigured: control plane (peer-link or ISL down, or the pair never formed). Both switches forward for the MAC.
  4. VM MAC spoofing or a live migration leaving a stale entry: host.

Worked example (figures illustrative)

A dual-homed server with a mis-bundled bond presented one MAC on Gi1/0/12 and Gi1/0/18. The switch logged 8,400 MAC moves over 22 minutes (4,200 per port), unknown-unicast flooded VLAN 10, and the first ticket was filed as a gateway ARP fault: wrong plane. Fixing the bond so LACP ran on both sides dropped the move count to zero and the flood stopped.

Failure modes and detection

  • Unknown-unicast flooding saturating access links: watch interface broadcast and unknown-unicast counters.
  • MAC table churn plus CPU spent on MACFLAP notification processing: show processes cpu on older platforms.
  • STP topology changes every few seconds: show spanning-tree, "number of topology changes".
  • The silent case: if the two ports sit on different switches and the MAC moves slowly, you may see it only in the move counters, never in syslog.

What I would do

Capture the move rate, VLAN, and both ports; trace both ports through trunks, LAGs, and hypervisors; check STP state on each. Contain by shutting one port only if the domain is already saturating. Declare it fixed only when the MAC sits on one port with zero moves across a full polling interval.

Curated: · Written: · Reviewed:

Q&A-8The port-channel is up with four collecting-distributing members. Why is a quarter of the traffic still black-holing?

Assuming a standard 802.3ad LACP bundle with four members and a flow hash over the 5-tuple (source/destination IP, source/destination port, protocol), up/up only proves the control plane. LACP PDUs are small, low-rate, and frequently punted to a different queue or path than payload. A member can complete the LACP exchange while its data path drops or black-holes traffic. With four members and a flow hash, roughly 25% of flows land on the broken member, so about a quarter of your sessions fail while the bundle looks healthy.

Confirm it per member, not per bundle. Compare actor/partner state, key, port state, speed, MTU and VLAN on all four members. Then send hash-diverse five-tuples, vary source and destination IPs and ports to force flows onto each member, and watch each member's TX/RX counters and error counters. A member that shows collecting/distributing but flat TX increments or rising output drops is the culprit. The 25% is statistical: with a handful of flows the split can be lopsided, but with hundreds of sessions it converges on a quarter.

Worked example (hypothetical): four members, 100 TCP sessions, hash spread evenly. Member 4 has a payload fault but LACP stays up.

MemberLACP stateFlows hashedSessions failed
1collecting, distributing250
2collecting, distributing250
3collecting, distributing250
4collecting, distributing2525

The port-channel shows up/up the whole time. That is the signature.

What you do about it: three credible moves, each with a cost:

  1. Shut the member (or remove it from the bundle). The hash re-converges to three members and service for the remaining 75% is restored immediately. Cost: you lose 25% of bundle capacity and you lose the evidence in place. Choose this when black-holing is causing customer-visible failures and you cannot diagnose fast enough.

  2. Keep probing in place. Preserves capacity and lets you capture counters, but 25% of sessions keep failing. Choose this only if the impact is tolerable and you need root cause: for example, to prove a hardware fault before an RMA.

  3. Per-port loop test. Take the suspect member out of the bundle, plug a loopback, and run a per-port test. This isolates physical-layer faults better than traffic probes, but it requires the same outage as option 1 and does not exercise the ASIC forwarding path.

What does not help: changing LACP rate to fast. LACPDUs still traverse the broken data path, so the member stays collecting/distributing. min-links would take the whole bundle down if you drop below the threshold, not what you want with four members.

Failure modes to name: a unidirectional data path, where LACP completes because control frames pass in both directions but payload is dropped one way; a bad optic that passes small frames but drops large ones (frame-size dependent); an ASIC or forwarding-table fault on one member; MACsec replay drops; or a member with a mismatched VLAN/MTU that still completes LACP. Detect all of these with per-member counters plus hash-diverse probes.

I would not call it settled until each member's TX/RX counters move only on collecting-distributing ports and a hash-diverse probe per member succeeds.


Key points a strong answer hits

  • up/up is a control-plane statement; LACP PDUs do not prove the payload path.
  • A flow hash sends ~25% of flows to each of four members, so one broken member black-holes ~25% of sessions.
  • Confirm per member, not per bundle: LACP state, key, VLAN, MTU, then per-member counters under hash-diverse probes.
  • The 25% is statistical, not exact: it converges as flow count rises.
  • Remediation is a trade-off: shut the member (restore service, lose capacity and evidence), probe in place (keep capacity, keep failing), or per-port loop test (isolate physical layer, same outage).

What interviewers look for

  • Separating control plane from data plane without being prompted.
  • Knowing the hash is per-flow, not per-packet, and what that implies for blast radius.
  • A concrete detection procedure rather than "check the logs".
  • Naming the cost of each remediation, not just the fix.
  • Not reaching for LACP fast rate or min-links as the answer.

Weak-answer patterns

  • "Reseat the SFP / replace the cable" with no diagnosis.
  • Blaming LACP timers or the bundle as a whole instead of one member.
  • Claiming the hash is exactly 25% per member regardless of flow count.
  • Suggesting min-links as a fix (it takes the bundle down).
  • Treating up/up as proof the data path works.

Likely follow-ups

  • How do you prove which member is bad without taking the bundle down?
  • What if all four members show equal counters but traffic still fails?
  • How does the distribution change if you switch from L3+L4 hashing to L2 hashing?
  • What does min-links do here, and why isn't it the fix?
  • How would you test one member's data path with a loopback, and what does that miss?

Curated: · Written: · Reviewed:

Q&A-87You enable BPDU guard on an uplink. What happens when the upstream sends a BPDU?

BPDU guard err-disables the port. On an uplink that means the link goes down, not that STP blocks it. The upstream switch is running STP and sends a BPDU every hello interval, 2 seconds by default, so the port hits err-disable within about one hello of the link coming up. If that uplink is the only path to the downstream switch, every host behind it is isolated until someone clears the error by hand.

Assumptions: Cisco-style BPDU guard (spanning-tree bpduguard enable), default hello timer, err-disable recovery left at its default (disabled).

Why it fires on an uplink

Per-interface BPDU guard does not care whether the port is an edge port. Configured with spanning-tree bpduguard enable, it arms on any port, including a trunk to the core. The global form, spanning-tree portfast bpduguard default, only arms PortFast ports, which is why the mistake usually comes from a per-interface line copied onto an uplink.

When a BPDU arrives, the switch logs it and err-disables the port:

%SPANTREE-2-BLOCK_BPDUGUARD: Received BPDU on port GigabitEthernet1/0/24 with BPDU Guard enabled. Disabling port.
%PM-4-ERR_DISABLE: bpduguard error detected on Gi1/0/24, putting Gi1/0/24 in err-disable state

Err-disable is a hard down: no traffic, no BPDUs out, and the port shows err-disabled with reason Bpduguard in show interfaces status err-disabled. Downstream switches see the link drop and reconverge.

The right tool for each direction

! uplink toward an untrusted/core switch
interface GigabitEthernet1/0/24
 switchport mode trunk
 spanning-tree guard root

! access edge
interface range GigabitEthernet1/0/1 - 20
 switchport mode access
 spanning-tree portfast
 spanning-tree bpduguard enable
GuardTriggerActionRecovery
BPDU guardAny BPDU on the porterr-disableManual, or errdisable recovery cause bpduguard (off by default)
Root guardSuperior BPDU on a designated portroot-inconsistent (blocking)Automatic when superior BPDUs stop
Loop guardBPDUs stop arriving on a blocking portloop-inconsistentAutomatic
BPDU filterBPDU on the portDrops it, no err-disablen/a

Root guard is the uplink answer: it blocks the port instead of killing it, and it heals itself when the upstream stops claiming to be root. BPDU guard is for ports where a BPDU means a miswired host or a rogue switch.

Failure modes

  • Isolation. Single uplink err-disabled, downstream switch dark. Detected by show interfaces status err-disabled and the ERR_DISABLE log line; the port reads err-disabled, not blocking, so it is easy to misread as a cabling fault.
  • Flap loop. If you enable errdisable recovery cause bpduguard while the upstream still sends BPDUs, the port comes up, sees a BPDU within 2 seconds, and err-disables again: one cycle per recovery interval (default 300 s). Repeated ERR_DISABLE entries for the same port are the tell.
  • Reconvergence hit. The err-disable is a link-down event, so the rest of the topology reconverges around it. If the alternate path is not ready, expect a brief blackhole or loop rather than a clean failover.

Verifying the behaviour

Send BPDUs and check the state, not the link light. On an edge port, expect err-disabled with reason Bpduguard: that is the feature working. On an uplink, expect the same thing, which is the bug: the port should stay up and, with root guard, go root-inconsistent only if the upstream sends a superior BPDU. show spanning-tree interface Gi1/0/24 and show errdisable recovery confirm which cause is armed and whether recovery is on.

Curated: · Written: · Reviewed:

Q&A-88PortFast is enabled on a trunk to another switch. What window do you open?

The 60-second answer

The window is the STP convergence window that PortFast removes: the listening and learning states, 15 s each, 30 s total by default. PortFast makes the port forward at link-up instead of after that 30 s. On an access port that window exposes one VLAN. On a trunk it exposes every VLAN carried on the trunk at once, because the port is forwarding for all of them from t=0. So the window you open is "all VLANs, immediately, until STP blocks the port."

The window closes when STP receives a BPDU that makes the port non-designated and blocks it. With the default hello timer of 2 s, that is typically within a couple of seconds, but a couple of seconds of a loop across every VLAN is a broadcast storm.

Mechanism

PortFast does not disable STP on the port. It only skips the listening/learning delay. BPDUs are still sent and processed, so the port forwards immediately and STP then converges normally. The danger is the gap between "forwarding" and "blocked."

Trace

Two switches, SW1 and SW2, joined by two trunks: Gi0/1 (PortFast enabled) and Gi0/2 (normal). SW2 is root. Gi0/2 is already up and forwarding; Gi0/1 then comes up.

  • t=0: Gi0/1 link-up. PortFast → forwarding immediately, all VLANs.
  • t=0: SW1 now has two forwarding paths to SW2 → loop.
  • t=0 to ~2 s: broadcast frames loop; MAC table flaps between Gi0/1 and Gi0/2.
  • t≈2 s: SW2's BPDU arrives on Gi0/1; SW1 sees the port as non-designated and blocks it.
  • Loop ends.

Window ≈ one hello interval (2 s), but across every VLAN on the trunk.

PortPortFastForwarding startsVLANs exposed in the window
accessyest=01
trunkyest=0all on the trunk
trunknot≈30 s0 (not forwarding)

Failure modes

  • Broadcast storm across all VLANs on the trunk during the window.
  • MAC address flapping between the two links: visible in show mac address-table as the same MAC moving ports.
  • If BPDU guard is configured on the port, the first BPDU err-disables it; that is the correct outcome, and the err-disable is the detection (show errdisable recovery).
  • If BPDU filter is also enabled, no BPDU is processed, the window never closes, and the loop persists.

When not to use it

Never on a switch-to-switch link. PortFast assumes nothing behind the port bridges. Keep trunks in normal STP; reserve PortFast for true access edges, and pair it with BPDU guard there.

Curated: · Written: · Reviewed:

Q&A-86Storm control drops 1 percent of broadcast and the loop is still there. What did you treat, and what remains?

Storm control treated the flood rate on the port it's configured on. The loop is untouched, because storm control is a policer sitting above the forwarding decision: it counts broadcast, multicast, and, depending on platform and config, unknown-unicast frames, and drops whatever exceeds the configured rate. It never changes where a frame goes. The cycle is still there, so every frame that gets through the limiter still circulates, the MAC table still churns, and the control plane still burns CPU on MAC moves and ARP/ND floods. The 1% drop counter proves the limiter is working. It says nothing about topology.

What remains is the forwarding cycle itself, plus everything downstream of it: a broadcast domain saturated up to the limiter's threshold, MAC flapping across the looped ports, and a control plane busy enough that show mac address-table stops being a reliable way to locate the loop. STP, or pulling the cable, is what removes the cycle. Storm control only caps the blast radius while you look.

The 1% figure is a statement about the limiter, not the topology. It means the offered flood rate is just above the configured threshold. If broadcast storm control is set to 5% on a 1 Gbps port, that's a 50 Mbps cap, and 1% drop means the loop is pushing roughly 50.5 Mbps of broadcast. That's a small loop or a generous threshold: either way it tells you nothing about where the cycle is.

Hypothetical worked example: two access switches mispatched with a pair of cables, 1 Gbps uplinks, broadcast storm control at 5% (50 Mbps). At 1500-byte frames that cap is 50e6 / (1500 × 8) ≈ 4,166 broadcast frames per second. Each frame crossing the loop re-learns its source MAC on the other port, so the MAC-move counter climbs at roughly that rate per lap: about 2.5 million moves over ten minutes. The switch stays reachable, the drop counter reads 1%, and the loop is exactly where it was.

Storm controlSTP
Acts onframe rate on one portforwarding topology
Changeshow many frames passwhich ports forward
Leavesthe cycle—
Evidence it workeddrop counter incrementsport in blocking, TC count stops climbing
Fails whenloop bypasses that port, or threshold too highSTP off per-VLAN, BPDU filter, unidirectional link

The failure modes are worth naming because each one leaves the loop in place while the drop counter looks healthy. Storm control on the uplink only: if the loop is between two access ports on the same switch, the flood never reaches the uplink and the uplink limiter never fires. Broadcast-only storm control: a loop carrying mostly unknown unicast or multicast slips past it. Per-VLAN STP disabled, or BPDU filter on an edge port that is actually part of the loop, means nothing is removing the cycle even though STP looks enabled globally. A threshold set high enough to stop the pages is the same as no limiter at all: the loop just runs faster.

For detection, watch the MAC-move counter and the topology-change counters in show spanning-tree detail, not the storm-control drop counter. Broadcast rate on the looped ports and switch CPU tell you whether the flood is still live.

Close the incident on STP blocking a port or a cable removed, plus a stable MAC table and broadcast rate back at baseline. A drop counter sitting at 1% is not that.

Curated: · Written: · Reviewed:

Q&A-25A Type-2 MAC/IP route is in the EVPN table. Why can the inner frame still die?

A Type-2 route in the EVPN table only proves the control plane learned a MAC/IP-to-VTEP mapping. The inner frame is forwarded by the data plane, which has its own chain of requirements. If any link in that chain is missing, the frame is dropped even though the route is present.

The chain for a frame arriving on an access port:

  1. The access port maps the frame to a bridge domain/VLAN.
  2. That VLAN maps to a VNI. If the mapping is missing or points to a different VNI than the Type-2 route, the frame never enters the MAC-VRF that holds the route.
  3. The MAC lookup in that MAC-VRF finds the remote VTEP next-hop and VNI from the Type-2 route.
  4. The underlay must have a route to that next-hop. If the BGP next-hop is unreachable, some platforms keep the EVPN route in the BGP table but do not install it in forwarding.
  5. The local NVE interface must be up and able to encapsulate. VXLAN adds 50 bytes (outer Ethernet 14, outer IP 20, UDP 8, VXLAN 8). If the underlay MTU is 1500, an inner frame larger than 1450 bytes will be dropped when the DF bit is set.
  6. The remote VTEP must have the VNI configured and up. If not, it drops the packet at decapsulation.
  7. After decapsulation, the remote VTEP looks up the inner destination MAC in its MAC-VRF. If the MAC is unknown and unknown-unicast flooding is disabled, the frame is dropped.

Failure modes and how to detect them:

FailureSymptomCheck
VNI mismatchType-2 for VNI 10100, local VLAN 10 mapped to VNI 10200show nve vni, show vlan vni mapping on both VTEPs
Next-hop unreachableEVPN route present, no NVE peershow ip route <next-hop>, show nve peers
Remote VNI missing or downLocal encap counters increment, remote decap counters stay zeroshow nve vni on remote VTEP
MTUSmall pings pass, large pings failping -M do -s 1472, show interface MTU
Remote MAC lookupDecap counters increment, no deliveryshow mac address-table on remote VTEP, packet capture

Each verification method proves a different part of the path. The BGP EVPN table is fast but only proves the advertisement was received. NVE peer and VNI state proves local readiness. VXLAN encap/decap counters isolate whether the break is local or remote. An end-to-end ping with the DF bit proves the full path including MTU. A packet capture on the remote VTEP proves decapsulation and inner forwarding but requires access. Use them in that order. The deciding criterion is where the counters stop: local encap zero points to local VNI mapping or next-hop resolution; local encap increments but remote decap zero points to remote VNI or underlay; both increment but the ping fails points to MTU or remote MAC lookup.

Example: a Type-2 route for MAC 00:11:22:33:44:55, VNI 10100, next-hop 10.0.0.2 is in the EVPN table. Local VLAN 10 is mapped to VNI 10200. The frame arrives, is looked up in the VNI 10200 MAC-VRF, finds no remote entry, and is flooded or dropped. Local encap counters for VNI 10200 increment; remote decap counters for VNI 10100 stay at zero. show nve vni on both VTEPs shows the mismatch.

Curated: · Written: · Reviewed:

Q&A-27A shared-services prefix appears in a tenant VRF you did not intend to join. What leaked?

Short answer: an unintended import route target on the tenant VRF. The shared-services prefix was exported with an RT the tenant VRF imports, so the route landed in the tenant RIB and then the FIB. The interface-to-VRF binding is fine; the leak is in the import policy.

Assumptions: MPLS L3VPN or EVPN with RT-based import, the shared-services prefix lives in its own VRF, and the tenant VRF has at least one import RT. If you run a single global table with VRFs only for interfaces, the answer changes: see the alternatives below.

Mechanism: a VRF is a RIB plus a classification rule. BGP VPN routes carry an extended community (the RT). A VRF imports every route whose RT list intersects its import RT set. Add one RT to that set and every route carrying it appears in the tenant RIB. From there it is ordinary forwarding: longest-prefix match decides. A leaked /32 beats the tenant's own /24 or default, so traffic that should have stayed inside the tenant goes to shared services.

Three causes, and how to tell them apart:

CauseWhere to lookSignature
Extra import RTconfigured import RTs (show vrf detail on IOS-XE/NX-OS) vs. the intended setForeign prefix present with the shared-services RT in its extended communities
Leaked static / redistributeshow ip route vrf TENANT <prefix> originRoute is static or connected, not BGP; no RT
Global-table leakinterface VRF assignment, ip vrf forwardingPrefix reachable via an interface in the wrong VRF, or a default route pointing out

The first is the one the question describes: a shared-services prefix, not a connected or static one, appearing in a tenant VRF you did not join.

Containment, in order of blast radius:

  • Remove the import RT. One line, immediate, but it also cuts any legitimate shared-services reachability the tenant had.
  • Import filter (route-map on the import, or RT-Constrain). Keeps the RT, denies the specific prefix. Per-VRF, so it does not scale to hundreds of tenants.
  • Export-side fix: stop tagging the shared prefix with the tenant's RT. One change, affects every importer: correct if the export was the mistake.

Failure mode to watch: the leak is silent. Sessions stay up, the tenant's own routes still work, and only a RIB diff or a traceroute from inside the tenant VRF shows the foreign next hop. Check show ip route vrf TENANT for prefixes whose next hop is in another VRF, and diff the tenant RIB against the intended prefix list.

Worked example (hypothetical figures): tenant B's VRF had one extra import RT, 100:200, matching shared services. Over 9 days, 17 shared-services prefixes sat in B's RIB. One was a /32 for a billing API; B's own route to that API was a /24 via its firewall. Longest-prefix match sent 1 query straight to shared services, bypassing the firewall. Detection was a RIB diff, not an alert.

VRFExtra import RTForeign prefixesDaysTraffic that crossed
tenant B100:2001791 query (bypassed firewall)

I would not call it contained until the tenant RIB shows zero foreign prefixes, or the ones that remain are documented and policy-approved.

Curated: · Written: · Reviewed:

Q&A-75You carry tenant traffic in GRE across a shared underlay. Who can read it?

Anyone who can see the underlay packet can read the tenant's inner IP packet. GRE (RFC 2784, updated by RFC 2890) prepends a 4-byte header (plus optional checksum, key and sequence fields) and sets the outer IP protocol to 47. The inner packet, header and payload, is carried verbatim. GRE has no confidentiality transform.

So the reader set is: the underlay operator and anyone with access to underlay devices, anyone with a SPAN or mirror port on the underlay, any host that can capture on an underlay segment, and any transit provider on the path. VRF or VNI separation at the PE keeps tenants apart in the forwarding plane; it does not change the bytes on the wire.

Mechanism

outer IP (proto 47) | GRE hdr (4B, +4B csum, +4B key, +4B seq) | inner IP | inner payload

A capture on the underlay decodes straight through to the inner header:

# tcpdump -i eth0 -n -v proto 47
12:00:01.123456 IP 10.0.0.1 > 10.0.0.2: GREv0, length 88:
    IP 192.168.10.5.44321 > 192.168.20.9.80: Flags [P.], seq 1:49, ack 1, win 502, length 48

The inner 5-tuple and payload are visible. With IPsec the same capture shows ESP(spi=0x...,seq=0x...) and no inner IP.

What to add, and when it is worth it

OptionWhere it protectsOverheadNotes
IPsec transport mode over GREend-to-end between tunnel endpointsESP ~24–50 B + GRE 4–24 Bmature, hardware offload common, IKE key management
IPsec tunnel mode (GRE inside)samesimilarone SA per tunnel
WireGuardend-to-end32 B (16 B header + 16 B auth tag)fixed cipher suite, simple config, UDP only
TLS/DTLS overlayend-to-end at the overlay layerTLS record overheadtraverses middleboxes, per-connection cost
MACsec (802.1AE)hop-by-hop on the link8–16 B SecTAG + 8–16 B ICVprotects the link, not the path; each hop sees plaintext

Encrypt when the underlay is not under your physical control (carrier, shared DC fabric, cloud interconnect), when tenants are mutually hostile, or when a compliance regime requires tenant data encrypted in transit. Skip it when the underlay is single-tenant fibre you control end to end, or when the underlay already provides line-rate encryption and you accept hop-by-hop trust. The cost is MTU: GRE 4 B plus IPsec ESP ~24–50 B means you lower the tunnel MTU and let PMTUD work, or you black-hole large packets.

Failure modes and detection

VRF mistaken for encryption. Detection: SPAN the underlay, filter proto 47, check whether the inner payload is readable. Worked example (hypothetical): a 1 Gbps tenant link averaging 50 Mbps for 30 minutes carries roughly 11 GB of inner traffic; if the capture shows inner HTTP headers, isolation is edge-only.

MTU black hole. Detection: ping -M do -s 1400 across the tunnel; if it fails while smaller sizes pass, PMTUD is blocked somewhere on the path.

Rekey storms. Detection: correlate IKE/ESP rekey logs with packet loss on the tunnel.

GRE is topology, not confidentiality. If the underlay is shared, add IPsec, WireGuard or a TLS overlay, and verify with a capture, not with a VRF config.

Curated: · Written: · Reviewed:

Q&A-29A GRE tunnel is up. What confidentiality did you just claim?

None. A GRE tunnel being up tells you the outer header is reachable and both endpoints agree on the encapsulation. It says nothing about confidentiality. If you told anyone that link was confidential, you claimed something GRE cannot deliver.

GRE (RFC 2784) is a pure encapsulation header: flags/version, protocol type, optional checksum, optional key, optional sequence number. No cipher, no key exchange, no integrity check. IP protocol number 47. Whatever you put inside (IPv4, IPv6, MPLS) is on the wire in the clear.

A capture on the underlay makes it unambiguous:

# tcpdump -ni eth0 -vv 'proto 47'
12:04:31.882 IP 10.0.0.1 > 10.0.0.2: GREv0, proto IP (0x0800), length 88
        IP 192.168.10.5.34512 > 192.168.20.7.80: Flags [P.], seq 1:49, ack 1, win 502, length 48

The inner IP header, the TCP ports, and the 48 bytes of payload are all readable. Encapsulation moved the packet; it did not hide it.

Confidentiality comes from a crypto transform layered on top. Three credible ways to get it, and they are not interchangeable:

OptionWhat it givesOverhead (IPv4, approx)Pick it when
GRE over IPsec ESP, transport mode (RFC 3884)ESP confidentiality + integrity over the GRE packet~50-60 Bsite-to-site where you need multicast or dynamic routing (OSPF, BGP) across the tunnel
WireGuardNoise_IK handshake, ChaCha20-Poly1305, Curve25519, BLAKE2s~60 Bpoint-to-point or hub-spoke with static routes; simple config, roaming clients
DTLS (AnyConnect/OpenConnect-class)DTLS record confidentiality + integrity~55-60 Bremote-access users behind NAT, per-user auth and policy

Transport mode is the usual choice for GRE+IPsec because the outer IP header is reused, saving 20 bytes versus tunnel mode; RFC 3884 exists precisely for dynamic routing over IPsec. WireGuard is the right answer when you control both ends and don't need multicast or a routing protocol on the wire: one crypto suite, no negotiation to misconfigure. DTLS is the right answer when the endpoint is a human's laptop, not a router.

Failure modes worth naming:

  • Policy mismatch. The GRE tunnel is up, but the IPsec policy/ACL doesn't match the inner traffic, so it egresses in the clear. Detect it by watching ESP byte counters stay flat while GRE counters climb, and by capturing the underlay.
  • Keepalive outside the policy. GRE keepalives exempted from the crypto map keep the tunnel "up" while data is dropped or unprotected. Detect it by comparing keepalive counters with ESP counters.
  • MTU. GRE+ESP overhead pushes large packets past the path MTU; small keepalives still pass, so the tunnel looks healthy while TCP sessions stall. Detect it with DF pings of increasing size and by checking the interface MTU.

Verification is the same in every case: read the ESP (or WireGuard/DTLS) counters, capture the underlay, and confirm the inner payload is unreadable. "Tunnel up" is a liveness signal, not a confidentiality claim.

Curated: · Written: · Reviewed:

Q&A-76Three tenants share an underlay, each with a VNI. What stops a curious VTEP from reading VNI 10010?

Short answer: nothing in VXLAN stops it. If a curious VTEP can put a packet on the underlay and reach UDP 4789, it can read the inner frame of VNI 10010 in the clear. What stops it is everything around VXLAN: underlay reachability, control-plane membership, and crypto.

Assumptions: three tenants, one underlay, one VNI each, EVPN as the control plane, and VTEPs are the only things on the underlay that speak 4789.

Mechanism. VXLAN (RFC 7348) is a UDP encapsulation. Outer IP header, outer UDP with destination port 4789, an 8-byte VXLAN header whose low 24 bits are the VNI, then the tenant's original Ethernet frame byte for byte. No encryption, no integrity check, no authentication of the outer header. The VNI is a 24-bit tag (a namespace label, roughly 16 million of them) not a key. A VTEP that receives that packet decapsulates and hands the inner frame to the bridge for that VNI. A tap on the underlay sees the same bytes.

So what actually stops a curious VTEP? Three layers, weakest to strongest:

  1. Underlay reachability. If the underlay is a routed fabric and only authorized VTEP loopbacks can send to UDP 4789, a host that isn't a VTEP can't inject or snoop. This stops outsiders, not a VTEP that's already a member.

  2. Control-plane membership. With EVPN, a VTEP only imports routes whose route target matches its per-VNI config. Tenant A's VTEP is configured with RT 65000:10010 and never imports 65000:10011. That keeps honest VTEPs apart. It is policy, not crypto: a rogue VTEP that advertises the tenant's RT as a Type-3 IMET route gets the BUM traffic and the Type-2 MAC/IP routes.

  3. Crypto. The only thing that stops a curious VTEP that is on the underlay and speaks EVPN. Two options:

    • MACsec (802.1AE) on the underlay links, GCM-AES-128 or -256. Line-rate in hardware, hop-by-hop: each router decrypts and re-encrypts, so it protects links, not the end-to-end path.
    • IPsec, typically transport mode between VTEP loopbacks, ESP with AES-GCM keyed by IKEv2. End-to-end across a routed fabric, but it costs CPU or a crypto offload, adds roughly 50–73 bytes of ESP overhead (watch the MTU: VXLAN already adds 50), and full-mesh key management doesn't scale, so it's usually paired with a group-key scheme or a hub.

Failure modes I'd watch for:

  • Rogue VTEP joins the EVPN: it advertises the tenant's RT and receives traffic. Detect by diffing the NVE peer list and Type-3 IMET membership against the expected VTEP set; alert on any new VTEP IP.
  • RT collision: two tenants configured with the same route target import each other's routes. Detect by auditing RT config per VNI; the symptom is cross-tenant MAC learning.
  • One underlay link without MACsec: the path is cleartext on that hop. Detect by checking MKA session state on every link.
  • 4789 reachable from a management or server segment: detect with an ACL audit and a capture.

The test I'd run to settle it: capture on the underlay, decap the VXLAN header by hand, and confirm the inner Ethernet frame is readable without any key. If it is, VNI 10010 is a tag, not a secret.

Curated: · Written: · Reviewed:

Q&A-78Both sides of a VPN use 10.0.0.0/8. What must you do before interesting traffic can be unique?

You must make the two address spaces disjoint before the tunnel can carry traffic, because both the routing table and the SA's traffic selectors key on the prefix. Two ways: NAT one or both sides into a unique range, or renumber one side. Then the selectors must name the translated prefixes, not the originals.

Why the overlap breaks it

Two mechanisms, both fatal:

  1. Route lookup. Each side has a connected route for 10.0.0.0/8 on its LAN interface. A route to the tunnel for the same /8 can't win longest-prefix match against a connected /8, so a packet to 10.1.1.1 leaves the LAN NIC, not the tunnel. The SA can be up the whole time.
  2. Selector match. IKEv2 traffic selectors (TS_i/TS_r) or IKEv1 proxy IDs are 10.0.0.0/8 on both ends. Even if you force the packet into the tunnel, the far peer sees source 10.x and destination 10.x and treats it as its own local subnet. Return traffic to 10.1.1.1 is delivered to the local 10.1.1.1 or dropped.

NAT, concretely

Cisco ASA 9.x, translating side A's 10/8 to 172.16.0.0/16:

object network SITE-A-LOCAL
 subnet 10.0.0.0 255.0.0.0
object network SITE-A-MAPPED
 subnet 172.16.0.0 255.255.0.0
nat (inside,outside) source static SITE-A-LOCAL SITE-A-MAPPED

! crypto ACL matches the POST-NAT source
access-list VPN-TO-B extended permit ip 172.16.0.0 255.255.0.0 10.0.0.0 255.0.0.0

Side B's crypto ACL is the mirror: permit ip 10.0.0.0 255.0.0.0 172.16.0.0 255.255.0.0. On Palo Alto the equivalent is a source-NAT policy plus proxy-IDs on the IPsec tunnel using the translated range. Only one side needs to NAT; the other then sees the translated range as the peer's network.

Do not confuse this with NAT-T (UDP 4500). NAT-T encapsulates ESP through a NAT device; it does nothing for overlapping prefixes.

NAT vs renumber

NATRenumber
Blast radiusVPN edge config onlyevery host, DHCP/DNS, ACLs, monitoring
Breaksprotocols that embed IPs (FTP, SIP, some H.323), IPsec-in-IPsecnothing protocol-wise, but downtime
Needs control of far sidenoyes
Coststate table, harder to debug asymmetric pathsproject and change window

Renumber when you own both ends and the site is small or can take a window. NAT when the far side is a partner, an acquisition, or a cloud you don't control, or when renumbering is a multi-quarter project.

Failure mode and verification

NAT applied in one direction only: return traffic arrives with the translated source and no reverse mapping, so it's dropped. Detect it with show crypto ipsec sa, encaps/decaps counters stay flat while the SA shows up, or packet-tracer on the ASA. Confirm the fix by tracing 10.1.1.1 to the tunnel interface, not the LAN, and by checking the far side's route for the mapped prefix points at the tunnel.

Curated: · Written: · Reviewed:

Q&A-79Two VRFs use the same RD and different RTs. What collided, and what did not import?

Two VRFs with the same RD and different RTs: the collision is in the VPNv4 NLRI, and the non-import is the RT policy doing its job. The RD is an 8-byte prefix prepended to the IPv4 prefix to form the 12-byte VPN-IPv4 NLRI. If two VRFs both originate 10.0.0.0/24 and both use RD 65000:1, the NLRI is 65000:1:10.0.0.0/24 on both PEs. Identical. BGP treats that as one route, runs best-path selection, and installs only the winner; the loser is hidden. If the IPv4 prefixes differ, the NLRI differ even with the same RD, so no collision. The RT is a BGP extended community used for import. VRF A exports RT 65000:10, VRF B imports only 65000:20, so B's VRF does not import A's route. No leak. But A's prefix can still disappear if B's advertisement won the collision.

The trade-off is RD design. Per-VRF RD, same RD on every PE for that VRF, is simple and works when each prefix is originated from one PE. Per-PE-per-VRF RD, such as ASN:PE-id, guarantees uniqueness even when the same prefix is advertised from multiple PEs, which is what you need for multi-homed sites or anycast. The cost is more RDs to track and losing the one-RD-per-VRF mental model. Overlapping prefixes across tenants are legitimate; that is why VRFs exist. The criterion is simple: any two routes that must coexist in VPNv4 need distinct (RD, prefix). RT is not a uniqueness mechanism.

Failure mode: one tenant's prefix missing from its VRF while the other tenant's is present. Detect it with show bgp vpnv4 unicast rd 65000:1. You see one path for 10.0.0.0/24, not two. Check the other PE's advertisement and the import RT lists to confirm no cross-import. Fix by giving each VRF a unique RD, or per-PE RD if multi-homing.

VRFRDRTPrefixVPNv4 NLRIImported into other VRF?
A65000:165000:1010.0.0.0/2465000:1:10.0.0.0/24no
B65000:165000:2010.0.0.0/2465000:1:10.0.0.0/24 (same)no

If A used 65000:1 and B used 65000:2, the NLRI would differ and both would be in VPNv4; RTs would still keep them from importing into each other's VRF.

Curated: · Written: · Reviewed:

Q&A-77A policy-based VPN misses 17 prefixes that a route-based tunnel would have taken from the routing table. Why?

Direct answer. Policy-based and route-based differ in what decides encryption: a selector list versus a route lookup. In a policy-based tunnel the encryption domain is the ACL / traffic-selector set that was negotiated into the CHILD_SA. A prefix that isn't in that set never enters the SA: it falls through to the routing table and leaves via the default route in cleartext. The 17 prefixes are missing from the selector, so they were never encrypted, and the SA for the listed prefixes stayed up and healthy the whole time, which is why nothing alarmed.

Why the selector is static. IKEv2 negotiates traffic selectors (TS_i/TS_r) when the CHILD_SA is created, and the responder may narrow them (RFC 7296 §2.9). IKEv1 is stricter: proxy IDs must match exactly on both peers or quick mode fails. Either way the selector set is bound to the SA at creation. Adding a prefix means editing the ACL and then tearing down and renegotiating the SA (clear crypto ikev2 sa / clear crypto ipsec sa) before the new selector takes effect, until then the old SA keeps the old set. Route-based has no such binding: the tunnel interface is just an interface, and the RIB decides. Adding a prefix is a route statement; no IKE exchange, no rekey.

Cisco ASA, policy-based:

access-list VPN-ACL extended permit ip 10.1.0.0 255.255.0.0 10.2.0.0 255.255.0.0
crypto map CMAP 10 match address VPN-ACL
crypto map CMAP 10 set peer 203.0.113.2

Route-based (VTI, ASA 9.7(1)+):

interface Tunnel1
 ip address 169.254.1.1 255.255.255.252
 tunnel source interface outside
 tunnel destination 203.0.113.2
route inside 10.2.0.0 255.255.0.0 169.254.1.2

Trace a missed prefix. Packet from 10.1.1.1 to 10.2.5.5 (one of the 17). Policy-based: ACL lookup misses → route lookup → default route out outside → cleartext. Route-based: route lookup → Tunnel1 → encrypted. Same packet, same RIB, different decision point.

Failure modes and detection.

  • Prefix added to the network but not to the ACL → cleartext. Detect: show crypto ipsec sa shows encaps/decaps counters per selector; the 17 prefixes show 0 while the listed ones climb. packet-tracer input inside tcp 10.1.1.1 1024 10.2.5.5 443 shows the ACL miss and the egress interface.
  • IKEv1 proxy-ID mismatch → phase 2 never comes up at all (different symptom: no SA, not a silent leak). Detect: show crypto ikev1 sa / debug crypto ikev1.
  • Responder narrows the selector → only a subset encrypted even though the ACL looks right.
  • Asymmetric selectors (one peer lists the prefix, the other doesn't) → SA up, traffic dropped or cleartext in one direction.

When not to use policy-based. If the prefix set changes, or you need dynamic routing (BGP/OSPF) across the tunnel, route-based is the right shape: it carries the RIB. Policy-based is fine for a fixed, small, symmetric set of subnets where you want the tunnel scoped tightly.

Policy-basedRoute-based
Encryption decided byACL / traffic selectorroute lookup into tunnel interface
New prefixedit ACL + renegotiate SAadd route
Dynamic routing over tunnelnoyes
Selector bound to SAyes (TS / proxy ID)no

Curated: · Written: · Reviewed:

Q&A-24Hosts send 1500-byte inner frames into a VXLAN overlay. The underlay is 1500. What breaks, and where?

Short answer: the encapsulated packet is 1550 bytes on the underlay. A 1500-byte underlay cannot carry it, so it drops at the VTEP's underlay egress interface or at any underlay hop whose MTU is below 1550. The overlay control plane stays up, so the failure is silent for hours.

VXLAN adds 50 bytes of outer headers to the inner frame:

HeaderBytes
Outer Ethernet14
Outer IPv420
UDP8
VXLAN8
Inner frame1500
Total on wire1550

If the outer IP header has DF set, common on VTEPs, the underlay router drops the packet and may send ICMP fragmentation-needed back to the VTEP's outer IP. The VTEP typically does not relay that ICMP to the inner host, so the inner TCP connection blackholes. If DF is clear, the packet fragments, but many underlays drop fragments outright and the extra overhead still hurts. Either way the break is at the underlay egress, not in the overlay.

Symptoms. Small pings work, large file copies fail. A 64-byte inner ping becomes ~114 bytes on the underlay and passes; a 1500-byte inner frame becomes 1550 and drops. EVPN/BGP sessions stay Established because their keepalives are small. This can persist for hours while the control plane looks healthy.

Detection. From an inner host, ping -M do -s 1458 <remote-inner-host> builds a 1500-byte inner Ethernet frame (1458 ICMP payload + 8 ICMP + 20 IP + 14 Ethernet). If it fails while small pings pass, you have an MTU blackhole. On the VTEP, tcpdump the underlay egress and check the encapsulated length, or watch interface drops. Check every ECMP member, VLAN and VRF: one member with MTU 1500 is enough to break large flows.

Fixes. Raise the underlay MTU to at least 1550 (1600 if you need headroom for outer IP options or an IPv6 underlay, which adds 20 more bytes). Or clamp the inner TCP MSS so the encapsulated packet fits 1500: an inner IP MTU of 1436 gives MSS 1396, and 1436 + 50 = 1486 fits with 14 bytes of margin. Clamping MSS to 1450 is not enough: that yields a 1540-byte underlay packet and still drops. If you only clamp MSS, apply it on both ends and on the VTEP if it rewrites MSS.

Verify by sending a 1500-byte inner frame and capturing the encapsulated size on every underlay path.


Key points / rubric

A strong answer must contain:

  1. The 50-byte VXLAN overhead, itemized (14 + 20 + 8 + 8).
  2. 1500 + 50 = 1550 > 1500, so the drop is at the underlay egress.
  3. DF set → ICMP frag-needed to the VTEP's outer IP, not relayed to the inner host → blackhole.
  4. Control plane stays up; only large flows fail.
  5. A concrete detection command (ping -M do -s 1458).
  6. Correct MSS arithmetic: MSS 1450 is wrong; MSS 1396 (inner IP MTU 1436) fits.
  7. One bad ECMP member is enough to break it.

What interviewers look for

  • Can you itemize the overhead from memory, not hand-wave "VXLAN adds some bytes"?
  • Do you separate the overlay failure domain from the underlay failure domain?
  • Do you understand the DF / PMTUD interaction and why the ICMP never reaches the inner host?
  • Do you give a runnable detection command rather than "check the MTU"?
  • Do you get the MSS arithmetic right, including why 1450 is a trap?
  • Do you mention ECMP, VLAN and VRF as places the bad MTU hides?

Weak-answer patterns

  • "VXLAN adds overhead, so lower the MTU" with no numbers.
  • Blaming the overlay, the VTEP CPU, or the EVPN control plane.
  • Claiming fragmentation always saves you.
  • Clamping MSS to 1450 (yields 1540 on the wire, still drops).
  • "Check MTU" with no command and no expected output.
  • Saying the control plane would flap: it does not.

Follow-up questions

  • What changes if the underlay is IPv6? (Outer header grows 20 bytes; budget 1570.)
  • What if you cannot raise the underlay MTU anywhere?
  • How does the VTEP learn the inner MSS to rewrite it?
  • How do you find the single bad ECMP member without taking the fabric down?
  • Does Geneve or VXLAN-GPE change the arithmetic?
  • Would you fragment at the VTEP instead, and what breaks if you do?

Curated: · Written: · Reviewed:

Q&A-26EVPN looks healthy and one underlay ECMP member is lossy. Which users fail?

The users who fail are the ones whose flows hash onto that member, and inside that set, the ones sending full-size frames. It is not "a quarter of users" unless every flow carries large packets.

Start from the packet. VXLAN adds 50 bytes of overhead (outer IPv4 20 + UDP 8 + VXLAN 8 + inner Ethernet 14), so a 1500-byte inner frame needs 1550 on the underlay. RFC 7348 says the outer UDP source port should be derived from a hash of the inner packet fields, which is what gives underlay ECMP anything to spread. If the platform pins that source port instead, every flow between a VTEP pair lands on one member and the blast radius is the whole pair, not 1/N.

Overlay BGP EVPN stays Established throughout. The session is TCP between VTEP loopbacks, it hashes to one member, and it needs only small packets to survive. A member that drops large frames keeps the session up while bulk traffic dies.

Field example: four underlay members, member 4 with a degraded optic. Frame error rate scales with frame length, so large frames drop and small probes pass:

FlowMemberFrame sizeObserved
Backup TCP, full MSS41500 B~12% loss, throughput collapse
VoIP RTP4~200 B~0.1% loss, calls fine
DNS / small API4<200 Bno visible impact
Any flow1–3anyclean
VTEP ping, default size1–364 Bclean: proves nothing about member 4

So the failing population is: flows hashed to member 4 that carry full-size frames. Small-packet flows on the same member look healthy, which is why the ticket says "intermittent" and why a VTEP ping that hashed elsewhere came back clean.

Remediation, and what each costs

OptionEffectCost
Repair the member (optic, FEC, MTU)removes the lossmaintenance window, possibly hardware
Remove member from ECMPrehashes flows1/N capacity; without resilient hashing every flow rehashes, so all users see TCP reordering and retransmits
Resilient hashingonly flows on the drained member moveconfig change, platform support required
Per-packet hashingspreads loss across all flowsreordering; TCP throughput collapses: almost never right
More hash entropy (L4 source port)spreads flows evenly across membersconfig only
Restart BGP EVPNnothingflaps every overlay route

Restarting BGP EVPN is the wrong lever: it does not move a single flow's hash, and it converts a partial outage into a full one.

When hashing changes the answer

  • Pinned VXLAN source port: the whole VTEP pair fails, not 1/N of flows. Fix entropy first.
  • Per-packet hashing: nobody fails hard, everybody degrades by roughly the member's loss ratio divided by member count. With 12% on one of four, about 3% loss on every flow. That is worse for TCP than a clean subset failing.
  • Resilient hashing: draining the member becomes nearly free, which is the configuration you want before the next bad optic.

Confirming it

Probe each member between VTEPs with 1500-byte payloads and a swept source port so every member is covered, and require all members under a stated loss bound. Check interface counters for uncorrected FEC, CRC and input errors: a link can be up, passing BFD, and still dropping frames. BFD on the physical link will not catch this, because the link never went down.

Curated: · Written: · Reviewed:

Q&A-28The IPsec SA is up. Why is the interesting traffic still leaving in the clear?

Short answer: an SA being up means IKE negotiated keys and the two peers agreed on a set of traffic selectors. It says nothing about whether the packet in front of you matches those selectors. If it doesn't match, the policy lookup finds no IPsec policy, the packet falls through to the ordinary forwarding path, and it leaves in the clear while the SA sits there with zero encapsulating bytes.

Assumptions I'd state up front: policy-based IPsec (crypto map ACL or IKEv2 traffic selectors), not route-based; the SA is up in both directions; the cleartext is observed on the outside interface.

How the decision is actually made. For every outbound packet the stack does a policy lookup keyed on the inner five-tuple (source and destination address, protocol, source and destination port) plus, on some platforms, the interface and VRF. Match, and the packet is encrypted under the SA. No match, and it goes through the normal route lookup and out in the clear. The selectors are the admission list; the SA is just the keyed channel behind it. So the two things to compare are the selector set and the real five-tuple.

The mismatches I see most: the ACL permits 10.8.0.0/24 but the host is 10.8.1.0/24; the ACL permits TCP/443 but the app moved to 8443; a NAT rule on the path means the address or port the selector was written against isn't the one the policy lookup sees; an IKEv2 responder narrowed the traffic selectors and the initiator installed the narrowed set; a more specific route sends the flow out an interface that has no crypto map at all.

A worked case. The SA covered 10.8.0.0/24 while the application ran on 10.8.1.0/24 inside 10.8.0.0/16. The SA stayed up with zero matching bytes for two hours.

PrefixIn selectorApp trafficSA encaps bytesWindow
10.8.0.0/24yesno02 h
10.8.1.0/24noyes, 880 MB02 h

880 MB over two hours is 440 MB/h, about 0.98 Mbit/s sustained: a steady backup job, not a burst. Counters should have moved. They didn't, which is the whole diagnosis.

Detection. On IOS/IOS-XE, show crypto ipsec sa prints the local and remote ident per SA and the #pkts encaps/decaps counters. Zero encaps with a live flow is the tell. Then show ip route <dst> to see where the packet actually went, and a capture on the outside interface to confirm it's cleartext. On Linux the same divergence shows up between ip xfrm policy and ip -s xfrm state.

Trade-offs: policy-based vs route-based. Policy-based makes the selector the routing decision, so a selector bug is silent. Traffic keeps flowing, just unprotected. Route-based (VTI, GRE over IPsec, strongSwan with mark-based routing) puts a tunnel interface into the route table: anything routed to it gets encrypted, and a misconfiguration usually surfaces as a black hole (no route, or packets dropped) rather than cleartext. That's the main argument for route-based on new builds: the failure is loud. The cost is that you now need routing to steer traffic, and per-flow policy (encrypt only port 443 to one partner) gets awkward: you end up with PBR or several tunnel interfaces. Policy-based still wins when the peer dictates the selectors, when you need many small host-to-host policies on one interface, or on platforms without VTI support.

Failure modes and how each is caught:

  • Selector mismatch: zero encaps, cleartext on the wire. Catch it by comparing the ident lines to the real five-tuple and watching the encaps counter.
  • Responder narrowed the traffic selectors: the installed policy is narrower than intended. Catch it by reading the ident lines on both peers.
  • A competing route: the flow exits an interface with no crypto map. Catch it with show ip route plus a capture.
  • Asymmetric ACL: outbound encrypted, return traffic dropped or clear. Catch it when decaps stays at zero on one side.

I wouldn't call it fixed until encapsulating bytes rise by roughly the payload size for that specific flow.

Curated: · Written: · Reviewed:

Q&A-30IKE comes up on UDP 500, then the path has a NAT. Why does ESP die until you move to 4500?

Core answer

ESP is IP protocol 50. It carries no source or destination port, and that is the whole reason it dies behind a NAT.

A NAT builds its translation table from the 5-tuple: source address, source port, destination address, destination port, protocol. Remove the ports and there is nothing to key the mapping on. Two hosts behind the same NAT sending ESP to the same gateway both present as (10.0.0.5, 203.0.113.9, proto 50); the NAT cannot tell them apart, so it maps one and drops or misdelivers the other. Return traffic has no port to demultiplex on either. Plenty of NATs don't forward protocol 50 at all: there is no ALG for it.

IKE survives because it runs on UDP 500 and has ports. So the control plane completes, the SAs install, and the data plane silently drops. That's the signature: IKE up, zero ESP.

NAT-Traversal wraps ESP in UDP on 4500. The packet now has ports, the NAT can build a mapping, and two peers behind one NAT get distinct source ports. IKE detects the NAT during IKE_SA_INIT using NAT-D payloads, hashes of each side's address and port, and both peers float to 4500 for the remaining IKE messages and for ESP. On 4500, IKE and ESP share the port; a four-byte non-ESP marker (0x00000000) tells them apart.

Why AH is worse

AH (protocol 51) has the same port problem, plus its ICV covers the immutable fields of the IP header. Any address rewrite invalidates the ICV, so AH cannot traverse a NAT at all. ESP's ICV covers the ESP header, payload and trailer, not the outer IP header, so ESP survives header rewriting in principle; it still can't be mapped. In transport mode there's a second bite: the inner transport checksum's pseudo-header uses the outer addresses, which NAT rewrites, so the receiver decrypts to a bad checksum. Tunnel mode avoids that because the inner header keeps the private addresses.

Alternatives and when to pick them

  • NAT-T on 4500: default. Needs UDP 4500 open end to end.
  • Static 1:1 NAT with a dedicated public address per peer. Works without NAT-T, costs one address per peer, and only when you control the NAT.
  • IPsec-aware NAT/ALG: vendor-specific, breaks with more than one peer behind the NAT, and rewrites SPIs. Avoid.
  • DTLS or TLS VPN on 443, when UDP is blocked outright, not just 4500.

Deciding criteria: how many peers sit behind one NAT, whether you control the NAT, and whether 4500 is permitted.

Failure modes

  • 4500 blocked after 500 allowed: IKE completes, ESP never flows. Detect by comparing IKE SA count against ESP-in-UDP byte counters.
  • NAT mapping expires: keepalives every 20 s (the common default) hold it open; without them the mapping times out and the tunnel goes one-way.
  • Multiple peers behind one NAT without NAT-T: SPI collision, one peer's traffic lands on another's SA.
  • NAT present but not rewriting (stateful firewall): NAT-D reports no NAT, no float, and ESP may still work. Don't assume NAT-T is always on.

Table

PortsNAT mappingICV covers outer IP headerThrough NAT
ESP, proto 50noneimpossiblenodies
AH, proto 51noneimpossibleyesdies
ESP-in-UDP 4500yesworksnoworks

Trace

Peer A behind NAT, peer B public. A sends IKE_SA_INIT from 500 with NAT-D. B replies with its own NAT-D; A's hash doesn't match what B computed, so B knows a NAT is in the path. Both move to 4500. ESP from A is wrapped: outer UDP 4500, src port 4500, dst port 4500, then the non-ESP marker, then ESP. The NAT maps (10.0.0.5:4500 → 203.0.113.9:4500) and B replies to that mapping. Counters on B show ESP-in-UDP bytes incrementing; if they stay at zero while IKE SAs are up, 4500 is being dropped.

Curated: · Written: · Reviewed:

Q&A-34Phase 1 is up. Phase 2 is up. A middlebox allows UDP 500 and drops protocol 50. What still fails?

What fails: the data path. Phase 1 and Phase 2 are both IKE exchanges, and both ride UDP 500. The child SA is created by the CREATE_CHILD_SA exchange inside the IKE SA: no ESP packet is required to bring it up. So a middlebox can pass every IKE packet and drop every ESP packet, and both peers will report the tunnel as established while zero payload crosses it.

ESP is IP protocol 50, not UDP. A rule that permits UDP 500 says nothing about protocol 50. NAT-T does not rescue this: when NAT is detected, ESP is encapsulated in UDP 4500, and UDP 4500 is not in the permit list either. Both the native ESP path and the NAT-T fallback are blocked.

ChannelWire formAllowedCarries
IKE_SA_INIT, IKE_AUTH, CREATE_CHILD_SA, INFORMATIONAL (DPD, rekey)UDP 500yeskey negotiation, SA lifetime, liveness
ESP, nativeIP protocol 50noall tunneled payload
ESP-in-UDP, NAT-TUDP 4500noall tunneled payload when NAT is detected

What still works: IKE rekeying, DPD, SA state, and any traffic whose selector does not match the SPD. What fails: every packet the SPD steers into the tunnel.

Detecting it. On both peers, tcpdump -ni any 'ip proto 50 or udp port 4500' shows nothing while tcpdump -ni any 'udp port 500' shows steady IKE traffic. In strongSwan, ipsec statusall prints the child SA with 0 bytes_i, 0 bytes_o while the IKE SA shows recent activity. That asymmetry (live IKE, dead child counters) is the signature. Vendor dashboards that poll IKE state will not show it.

Why it stays silent. DPD runs as an INFORMATIONAL exchange over the IKE SA, so it never probes the ESP path. Child SA rekeying also runs over IKE, so the SA can rekey on schedule and stay up for the full IKE SA lifetime. A hypothetical worth recognising: four hours of green "VPN up" tiles, zero bytes delivered, no alert, because nothing on the control plane ever touches protocol 50.

Fix. Permit IP protocol 50 end to end in both directions, or force NAT-T and permit UDP 4500 in both directions. In strongSwan, encap=yes on the connection forces ESP-in-UDP even when no NAT is detected, which is the usual workaround when the middlebox is not under your control. The cost is 8 bytes of UDP header per packet and a slightly smaller effective MTU; some middleboxes rate-limit or deprioritise UDP, so measure before committing.

One more check before you call it fixed: if the middlebox is stateful and only ever saw UDP 500, the return path for ESP may be dropped independently of the forward path. Verify with counters on both peers, not one.

Curated: · Written: · Reviewed:

Q&A-35A reordering path drops ESP after a burst. Why does increasing the replay window matter, and what does a window of 64 actually mean?

Assumptions: ESP tunnel mode, one SA per direction, RFC 4303 anti-replay enabled, reorder coming from the underlay (ECMP, QoS, or a path change), and no retransmission at the IPsec layer. ESP is fire-and-forget, so a packet dropped by anti-replay is gone and the transport above it pays for it.

What a window of 64 means. The receiver keeps a 64-bit sliding bitmap anchored to the highest sequence number it has accepted. If the highest is 1000, the window covers 937–1000; the left edge is highest − 63. Four cases:

  • seq above the top → accept, slide the window up, set the bit.
  • seq inside the window, bit clear → accept, set the bit.
  • seq inside the window, bit set → duplicate, drop as replay.
  • seq below the left edge → drop as replay, whether or not it was ever seen.

That last case is the whole problem. A packet at 936 that was merely delayed is indistinguishable from a replay of a packet already processed, so it is dropped. RFC 4303 requires a window of at least 32 and recommends no more than 64; 64 is the common default.

Why reordering matters. Reordering does not change which packets are legitimate, it changes their arrival order. If the path delivers a packet 80 positions late, by the time it lands the highest accepted sequence has moved 80 ahead, the left edge has moved past it, and it is dropped as a replay. ESP does not retransmit, so the loss surfaces as TCP retransmission and a throughput collapse, not as an IPsec error. Widening the window to 128 moves the left edge back and the same 80-deep reorder now lands inside the window.

Worked trace (hypothetical). Two-member ECMP bundle, per-packet hashing, reorder depth up to 80. Window 64.

EventHighest acceptedLeft edgePacketResult
in order10009371001accept, slide to 938–1001
reordered108010171000below edge → replay drop
reordered108010171010below edge → replay drop
after widening to 12810809531000inside window → accept

Trade-offs. Memory is the cheap part: the bitmap is W bits per SA, so 64 bits is 8 bytes and 1024 bits is 128 bytes. A hub with 10,000 SAs goes from 80 KB to 1.25 MB: real but not the reason to hesitate. The real cost is detection range: the window is exactly the set of sequence numbers the receiver will still accept, so a larger window lets an attacker successfully replay older captured packets whose originals never arrived (the bit is clear, so they look new). Enlarging the window trades replay resistance for reorder tolerance.

When not to enlarge. If the reorder comes from per-packet ECMP hashing, fix the hash to be flow-based so a flow stays on one member; widening the window hides a path problem and weakens replay detection. If the drops are integrity (ICV) failures rather than sequence errors, the window is irrelevant.

Detection. On Linux, the XFRM sequence-error counter (XfrmInStateSeqError in /proc/net/xfrm_stat) increments on anti-replay drops; compare it against ICV failures to tell reorder from corruption, and measure reorder depth from a capture or sequence-gap logs before changing the window.

Curated: · Written: · Reviewed:

Q&A-31Small packets through the VPN work. 1500-byte packets vanish. Where is the hole?

The hole is the tunnel's inner MTU, not the SA. Small packets fit inside the encapsulated path MTU; a 1500-byte inner packet plus IPsec headers does not. If that inner packet has DF set and the ICMP "fragmentation needed" that would tell the sender is filtered somewhere on the path, the packet is dropped silently and the sender retransmits the same full-size segment. The SA stays up because IKE/DPD keepalives are tens of bytes.

Why size decides it. Tunnel mode adds headers outside the inner packet. For an IPv4 outer path, ESP tunnel mode, AES-GCM-16:

LayerBytes
Outer IPv420
ESP header (SPI + sequence)8
ESP trailer (pad length + next header)2
ESP ICV16
Total46

A 1500-byte outer path leaves an inner MTU of 1500 - 46 = 1454. GRE over IPsec adds 4 bytes for the GRE header; PPPoE on the outer path takes another 8. A conservative 1400 is common because the outer path is rarely a clean 1500.

Two fixes, and when each is right.

MSS clamping rewrites the MSS option in TCP SYN and SYN-ACK as it crosses the tunnel, so both ends advertise a segment that fits. It is cheap, needs no fragmentation, and is the right default when the traffic is TCP and you control both tunnel endpoints. It does nothing for UDP, ICMP, GRE or QUIC, because those have no MSS option to rewrite: a large UDP datagram (a DNS response, a VXLAN-encapsulated frame, a VoIP packet) still black-holes. It also only helps if the clamp sits on the path the SYN actually traverses, so clamping on one endpoint fixes one direction and leaves the other broken.

Lowering the tunnel interface MTU tells the inner IP stack the real size, so every protocol sees it. The cost is that non-DF inner packets get fragmented after encapsulation, and some paths drop fragments; if DF is set and ICMP is still filtered, you have moved the black hole rather than closed it.

Deciding criteria: TCP-only traffic with both endpoints under your control, clamp MSS. Anything non-TCP, or endpoints you do not control, lower the tunnel MTU. In practice do both, so TCP gets the right number and everything else gets the right MTU.

Failure modes and how to see them.

  • ICMP type 3 code 4 filtered by a firewall: no frag-needed arrives, the sender retransmits the same segment. Detect with tcpdump on the inside interface: repeated identical full-size segments, no ICMP.
  • Clamp applied on the physical interface instead of the tunnel: no effect. Check the running config on the tunnel interface.
  • Clamp on one endpoint only: one direction passes, the other black-holes. Test both directions.
  • Post-encapsulation fragments dropped by a middlebox: fragments leave the outer interface and never reassemble. Look for fragments outside and nothing arriving inside.

Diagnosis. Sweep inner packet sizes with DF set and record the first size that dies; that is your inner MTU. Worked example, IPv4 outer, ESP tunnel mode, AES-GCM-16:

Inner IP packetResult
64pass
1400pass
1454pass
1500drop

Set the tunnel MTU to 1454 and clamp MSS to 1414 (1454 - 40 for the IPv4 and TCP headers), then re-run the sweep in both directions to confirm.

Curated: · Written: · Reviewed:

Q&A-32You exclude RFC1918 from the tunnel to save bandwidth. What inspection did you just skip?

The short answer: every control that sits on the concentrator's path. Excluding a prefix from the tunnel installs a route pointing at the physical interface, so those packets never reach the VPN gateway: no stateful firewall policy, no IPS/IDS, no DLP or egress filtering, no TLS inspection, no DNS-layer security if the client resolves locally, and no flow logs into the SIEM. The only enforcement left is the endpoint's own firewall and EDR, which the user controls.

Why the route decides it. Split tunnel is a route table, not a policy engine. The client gets a more specific route for each excluded prefix via the physical interface, and longest-prefix match wins:

$ ip route get 10.1.2.3
10.1.2.3 via 192.168.1.1 dev wlan0 src 192.168.1.42
$ ip route get 10.1.2.3        # exclusion removed
10.1.2.3 dev tun0 src 10.8.0.6

(Illustrative output.) The concentrator sees no session, so there is nothing to inspect and nothing to log.

RFC1918 is the wrong granularity. Private space is not "the corporate network". It is every network that happens to use private addressing: the hotel LAN, the home router at 192.168.1.0/24, the coffee shop, a co-tenant VM in a cloud VPC, and any attacker who can put a host on the local segment. Excluding 10.0.0.0/8 tells the client to send traffic to all of that in the clear, and the client cannot tell your 10.20.0.0/16 from someone else's.

Control on the concentrator pathAfter excluding RFC1918Where it lives now
Stateful firewall / ACLnot appliedhost firewall
IPS/IDSno session to inspectendpoint EDR
DLP / egress filteringbypassedendpoint DLP
DNS-layer securitybypassed if the resolver is localnone, unless the client uses corporate DoH/DoT
Flow logs / NetFlow to SIEMno recordendpoint telemetry
TLS inspectionnot appliednone

Failure modes and how you see them.

  • Rogue DHCP or a hostile local gateway hands the client a DNS server; internal names resolve to attacker addresses and the client connects without the concentrator ever seeing it. Detect: compare the client's resolver against policy and alert on any non-corporate resolver.
  • Lateral movement to a co-located host on the same private subnet. There is no concentrator log, so network detection is gone; EDR on the endpoint is the only signal.
  • Exfiltration to an attacker-controlled 10.x host on the local segment. Detect: endpoint DLP and per-host egress volume, not NetFlow.

What to do instead. Exclude named destinations you control and can inspect elsewhere (a cloud VPC with its own firewall and flow logs, a specific printer subnet) not an address class. Keep DNS tunneled so split-DNS and DNS-layer security still apply. If the bandwidth saving matters, a cloud-delivered secure web gateway or ZTNA client inspects locally and reports centrally, which is the only way to keep the telemetry you just deleted.

Curated: · Written: · Reviewed:

Q&A-33You point 0.0.0.0/0 at a tunnel whose destination is learned via the default. What happens at the next lookup?

The lookup can't resolve to a usable path. The tunnel destination matches 0.0.0.0/0, and 0.0.0.0/0's next hop is the tunnel interface: the same interface the encapsulated packet has to leave by. The route to the endpoint resolves through the tunnel, so the tunnel can't reach its own endpoint.

Assumptions: a GRE or IPsec tunnel, a routing protocol (OSPF, EIGRP or BGP) running over it, and a default originated into the tunnel. The usual "default over the tunnel" hub-and-spoke design. The endpoint address is reachable only via that default.

What happens next depends on the platform:

  • Cisco IOS/IOS-XE has a recursion guard. When the route to the tunnel destination resolves via the tunnel interface, the router disables the tunnel and logs %TUNNEL-3-RECURSION: Tunnel0 temporarily disabled due to recursive routing. The physical interface stays up; the tunnel's line protocol goes down.
  • Linux has no equivalent guard by default. The route installs, and the outer packet's destination lookup matches the same default via the tunnel device, so the encapsulated packet re-enters the tunnel. You see a loop or a blackhole, not a clean down state.

Trace the resolution on IOS:

show ip route 203.0.113.9        # tunnel destination
  -> matches 0.0.0.0/0
  -> 0.0.0.0/0 next hop = Tunnel0
  -> endpoint resolves via Tunnel0  => recursion

The fix is to make the endpoint resolve via the underlay. Several ways, with different costs:

ApproachHowCost
Static host route for the endpoint/32 to the physical next hop, AD below the learned routeManual; breaks when the underlay next hop changes
Underlay/overlay VRF splitOuter packets looked up in a transport VRF with no tunnel routeVRF config and route leaking; harder to troubleshoot
Don't put the default in the tunnelAdvertise only specific prefixes over the tunnelLoses the default-over-tunnel design
Rely on the platform guardLet IOS disable the tunnelDetects the failure, doesn't fix it

The static /32 is the common answer, but it's the one that rots: change the underlay next hop and the tunnel drops again. The VRF split is the durable one when you control both ends.

Detection: show ip route <endpoint> showing the tunnel interface as the outgoing interface, the %TUNNEL-3-RECURSION syslog, and tunnel line protocol flapping in step with IGP reconvergence. If the tunnel comes up, carries traffic, then drops whenever the IGP re-advertises the default, that's recursion, not a crypto or MTU problem.

Curated: · Written: · Reviewed:

Q&A-80The VPN drops and the kill switch is off. Where does the next packet go?

Assumptions: an always-on full-tunnel client (WireGuard or OpenVPN) on a laptop, tun-style interface, kill switch disabled, Linux or macOS. "The VPN drops" means the tunnel interface goes down and the client withdraws its routes, not the case where the interface stays up and the peer is merely unreachable.

The short answer: the next packet goes out the physical interface in the clear, to the local default gateway. The client's intent does not matter; the routing table decides, and once the tunnel's routes are gone the kernel falls back to the physical default route. Source address selection picks the LAN address, the home router NATs it, and the packet reaches the internet unencrypted. The kill switch is the only thing that would have stopped it, and it is off.

Why, mechanically. While the tunnel is up, the client installs routes that send everything through it:

$ ip route
default via 10.8.0.1 dev wg0
0.0.0.0/1 via 10.8.0.1 dev wg0
128.0.0.0/1 via 10.8.0.1 dev wg0
203.0.113.7 via 192.168.1.1 dev wlan0   # the VPN server itself
192.168.1.0/24 dev wlan0

The two /1 routes are the standard trick to beat an existing default route without deleting it. When the interface goes down, the kernel removes every route on it:

$ ip route
default via 192.168.1.1 dev wlan0
192.168.1.0/24 dev wlan0

Now trace a SYN to 10.0.0.5:443. Route lookup matches the physical default, next hop 192.168.1.1 on wlan0. Source selection picks 192.168.1.42. The frame goes to the home router, which NATs it to the public address and forwards it. The destination server sees the home IP, not the VPN exit IP. That is the leak, and it is the routing table working exactly as configured.

One nuance worth saying out loud. Existing connections usually break rather than leak: a socket bound to the tunnel address 10.8.0.2 fails to send once that address is removed from the down interface. The leak is mostly new flows, which is why the client icon showing "reconnecting" is misleading. The browser has already opened fresh connections that took the physical default.

What the kill switch does. It is a firewall rule, not a routing rule: drop outbound packets on the physical interface except the tunnel's outer encapsulation (UDP 51820 to 203.0.113.7, or ESP/UDP 4500 for IKEv2). With it on, the same SYN is dropped at the firewall and the connection times out. With it off, it leaves.

Kill switchNew flow to 10.0.0.5Existing flow bound to 10.8.0.2
offout wlan0 in the clearsend fails, connection breaks
ondropped at firewallsend fails, connection breaks

The trade-off, honestly. Fail-closed is not free. It breaks captive portals: you cannot reach the hotel login page because everything is dropped until the tunnel is up, and the tunnel cannot come up until you log in. It breaks local printing, NAS access, Chromecast and AirDrop. That is why some deployments choose fail-open split tunnel deliberately: route only the corporate subnets through the tunnel and leave the rest on the physical default. That is a design decision, not a leak. The leak is when you intended full tunnel and silently got the physical default.

Failure modes and how to catch them.

  • Silent leak: capture the physical interface for inner destinations. tcpdump -i wlan0 -n 'not port 51820 and not host 203.0.113.7': anything there is leaked. Expect zero bytes.
  • Fail-closed by accident: the interface stays up, the peer is unreachable, packets go into the tunnel and are dropped. No leak, but a black hole. Distinguish by checking whether the route still points at wg0.
  • DNS leak: if the resolver was 10.8.0.1, queries now route to the LAN gateway and die, or the client falls back to DHCP DNS and queries go to the ISP in the clear.
  • IPv6: a v4-only tunnel with a live v6 default route leaks v6 in the clear even with a v4 kill switch. Check ip -6 route.

Test it, don't assume it. Kill the tunnel peer, then capture the physical interface for inner destinations. Zero bytes except the tunnel outer is the pass condition.

Curated: · Written: · Reviewed:

Q&A-21VRRP fails over in three seconds. Why are still-open firewall sessions dead?

The 60-second answer

VRRP is a control-plane protocol: it elects a master and moves ownership of a virtual IP and its virtual MAC. It carries no connection state. A firewall's sessions live in a separate state table: NAT translations, TCP state, sequence/ack windows, timeouts. When the backup becomes master it owns the VIP but has an empty table, so the first packet of every established flow arrives out of state and gets dropped. The address failed over; the flows did not.

Why the timer is a red herring

Master_Down_Interval = 3 × Advertisement_Interval + Skew_Time, where Skew_Time = (256 − Priority)/256 seconds (RFC 5798). At the default 1 s advertisement and priority 100 that is 3 × 1 + 0.61 ≈ 3.6 s; you only get a clean 3.0 s at priority 255. Either way the number describes how fast the VIP moves, not how fast sessions survive. Quoting it as a session-preservation SLA is the mistake.

What actually breaks

  • TCP: the new master sees a mid-stream segment with no conntrack entry. A stateful firewall drops it as out-of-state: often silently, so the client hangs until its retransmit timeout instead of getting a RST.
  • NAT: the translation (inside local:port ↔ inside global:port) existed only on the old master. Even if the packet is forwarded, it may be re-NAT'd to a different global address, and return traffic lands on the wrong translation.
  • Asymmetric routing: if the pair is not symmetric, return packets can hit the wrong node.

Worked example (hypothetical figures)

ObjectOld masterNew master at +3.6 sUser-visible
VIP + virtual MACownedowned (gratuitous ARP sent)ICMP to VIP recovers in ~3.6 s
NAT translations12,6000—
Established TCP flows12,6000clients hang, then reset
Application recovery——~12 min of retries and rebuild

The 3.6 s is the VIP. The 12 min is the application. They are different clocks.

What to do instead

  1. Sync the state table. conntrackd (Linux), pfsync (pfSense/FreeBSD), or vendor HA links (Palo Alto HA3, Cisco ASA failover) replicate sessions to the standby. Failover then preserves flows and the VIP move is invisible to TCP.
  2. If you cannot sync, make the reset explicit and cheap: short app-level timeouts, idempotent retries, connection pools that re-establish. Budget the reset rather than pretending it does not happen.
  3. Active/active with ECMP or anycast does not fix it by itself: each flow is still pinned to one node's state, and losing that node kills its flows unless state is shared.

How you would detect it

  • Count conntrack entries on the new master right after failover (conntrack -C); 0 means no sync.
  • Watch for out-of-state drops in firewall logs, and for clients that hang rather than reset.
  • Measure recovery at the application layer, not with ping to the VIP.

VRRP moves the address, not the flow table.

Curated: · Written: · Reviewed:

Q&A-81VRRP master is elected and has no default route. What do hosts with a VIP gateway get?

Short answer: hosts get a gateway that answers ARP for the VIP and then drops everything that needs to leave the subnet. The VIP is reachable; the path behind it is not. VRRP does not notice, because the election is decided by priority and preemption, not by whether the master can forward.

Why the election ignores the routing table

A VRRP advertisement (RFC 5798) carries priority, the advert interval, and the address list. It carries no reachability information. As long as the master's VRRP process is alive and its interface is up, it keeps sending adverts every advert_int (1 s default), the backup keeps resetting its master-down timer, and no failover happens. The routing table is a separate process (BGP, OSPF, or a static default) that can lose its default without touching VRRP at all.

Master-down interval is 3 * advert_int + skew, where skew = (256 - priority) / 256 seconds. With priority 100 and a 1 s advert, that is roughly 3.6 s before a backup would even consider taking over, and it never gets there, because the adverts keep arriving.

What hosts actually experience

ARP for the VIP resolves to the master's MAC. ping to the VIP succeeds. ping to anything off-subnet fails: the master has no matching route, so it drops the packet or returns ICMP network unreachable, depending on platform. TCP SYNs retransmit at 1, 2, 4, 8 s; DNS queries to an off-subnet resolver time out; the application reports a hang or "network unreachable". If the master still holds some prefixes but lost the default, you get partial connectivity, which is harder to diagnose than a clean outage.

Fix: make upstream reachability a condition of holding the VIP

Track the default route and decrement priority below the backup's, so the backup preempts.

# keepalived.conf — checked against keepalived 2.2.x
vrrp_script chk_default {
    script   "/bin/sh -c 'ip route show default | grep -q default'"
    interval 2
    weight   -60      # priority 100 -> 40, below the backup's 100
    fall     2
    rise     2
}

vrrp_instance VI_1 {
    state             MASTER
    interface         eth0
    virtual_router_id 51
    priority          100
    advert_int        1
    track_script      { chk_default }
    virtual_ipaddress { 10.0.0.1/24 }
}

Cisco IOS does the same with track 1 ip route 0.0.0.0 0.0.0.0 reachability plus vrrp 1 track 1 decrement 60.

Failure modes worth naming

  • Tracking the interface instead of the route. The uplink stays up while BGP withdraws the default; the track never fires and the black hole persists.
  • No preempt, or a long preempt_delay. Priority drops but the backup refuses to take over, or waits minutes.
  • Both routers lose the default. Failover moves the black hole rather than fixing it. You need a second upstream path or a different next hop.
  • Stale ARP on hosts. After failover, gratuitous ARP updates host caches; if it is suppressed, hosts keep sending to the old master MAC until the entry expires (Linux base_reachable_time is ~30 s, longer with retries).
  • Asymmetric paths. Master reaches the internet via provider A, backup via provider B; failover changes the path and can break MTU or stateful-firewall sessions.

Detection

Probe from a host behind the VIP, not from the router. A check that pings the VIP passes while the outage is live; a check that resolves the VIP and then reaches an off-net target fails correctly.

ObjectStateHost impact
VRRP masterR1, priority 100—
R1 default routewithdrawn by BGP—
ARP for 10.0.0.1resolves to R1 MAC—
ping 10.0.0.1replieslooks healthy
ping 8.8.8.8no reply / ICMP unreachableblack hole

Owning the VIP is not owning a default.

Curated: · Written: · Reviewed:

Q&A-19The SYN leaves through firewall A and the SYN-ACK returns through firewall B. What happens to the session?

The 90-second answer

The SYN-ACK arrives at firewall B, which has no state for that 5-tuple. A stateful firewall forwards only packets that match an existing entry, so B classifies the SYN-ACK as out-of-state and drops it. The client never sees a response, retransmits the SYN (which still hashes to A, where the entry exists) and eventually fails with a connect timeout rather than a RST, because nothing on the path is generating one. The fix is to get both directions of the flow onto one state table: pin the pair with routing, or run the two devices as a cluster that genuinely shares state.

Why it behaves that way

State is keyed on the 5-tuple (source and destination IP, source and destination port, protocol) plus the ingress zone. A creates the entry on the SYN and holds it in SYN_SENT. B never saw the SYN, so it has nothing to match.

Whether the SYN-ACK is actually dropped depends on B's ruleset, and this is the part people get wrong. If B has the usual hardening rule dropping INVALID/out-of-state packets, the SYN-ACK dies. If B's policy is permissive, the SYN-ACK may be forwarded anyway, the handshake completes, and you get a session that works but is inspected by nobody on the return path. That is worse than a clean failure, because it stays hidden until the first asymmetric-sensitive feature (NAT, an ALG, IPS, application identification) breaks it.

PacketSeen atMatching stateResult
SYNAnew entry, SYN_SENTforwarded
SYN-ACKBnonedropped as out-of-state
SYN retransmit at +1 s, +2 s, +4 sAmatches entry, timer refreshedforwarded
SYN-ACK retransmitBnonedropped
client gives up after tcp_syn_retries=6 (~127 s)——ETIMEDOUT

The Linux conntrack SYN_SENT timer defaults to 120 s and each retransmitted SYN refreshes it, so the entry outlives the client's retry window. The server side is fine throughout: it has state, it retransmits its SYN-ACK (tcp_synack_retries defaults to 5), and it eventually closes the half-open connection.

Failure modes and how you detect them

  • ECMP rehash. A link flap or route change alters the path set and the 5-tuple hash lands on the other firewall. Because the hash is stable per flow, every new flow breaks the same way until the topology changes again, not intermittently, consistently.
  • NAT on one side. If A translates the source, the reverse path must return to A. B has no translation entry, so it drops the SYN-ACK even with a permissive ruleset, and the server has been talking to a translated address the whole time.
  • HA failover. A standby that did not sync state drops in-flight flows at the moment of takeover.

Detection is cheap and specific. On Linux, conntrack -S shows the invalid counter climbing while entries stays flat:

conntrack -S | grep -E 'invalid|drop'
conntrack -L -p tcp --dport 443 | grep SYN_SENT

On an ASA you get syslog 106015, Deny TCP (no connection) ... flags SYN ACK. On the client, ss -tan sits in SYN-SENT. A capture on both firewalls shows the SYN on A only and the SYN-ACK on B only: that asymmetry in the captures is the proof.

Alternatives, and when not to use them

  • Pin the flow. ECMP already hashes a flow consistently; the problem is topology, not the hash. Route both directions through the same device or cluster.
  • Session sync. conntrackd, ASA failover, or HA state sharing. This is for failover, not steady-state asymmetry, unless the devices replicate the table in real time, which costs latency and bandwidth on every flow.
  • TCP state bypass. ASA's set connection advanced-options tcp-state-bypass, or the equivalent asymmetric-routing handling elsewhere. It forwards without full stateful inspection. Take it only when you have accepted losing that inspection.
  • Stateless ACLs. Correct if you genuinely do not need state, and the honest answer when the requirement is just filtering.

Don't call this settled from a traceroute in one direction. Capture the SYN and the SYN-ACK at both devices and require them to hit the same state table.

Curated: · Written: · Reviewed:

Q&A-93You commit a routing change with a 10-minute confirm. What happens if you are locked out?

The 90-second answer

At the 10-minute mark the device throws away the config you committed and reloads the previously committed one, then commits that. Routing returns to the pre-change state: the old BGP sessions, the old statics, the old IGP metrics. Your session is not restored: the rollback is a config event, not a session event. You reconnect and you are on the old config.

That distinction is the whole point. If the change cut the path to the box: a bad static, a policy that dropped management traffic, a BGP change that killed the session carrying your SSH: the timer is what brings the box back. You do not need the console; you need to wait ten minutes.

Mechanism

Junos, where this is native, keeps three things: the candidate config you edited, the active config, and the last committed config. commit confirmed 10 activates the candidate and starts a rollback timer. Nothing is confirmed yet. If a plain commit arrives before the timer expires, the new config becomes permanent. If it does not, the system loads the last committed config and commits that.

Default timer is 10 minutes; you can pass 1 to 65,535 minutes.

user@r1# commit confirmed 10
# ... verify: BGP neighbor up, route in RIB, ping across the changed path ...
user@r1# commit          # confirms, cancels the timer

If you never type that second commit, the box rolls back on its own.

Sizing the timer, and the alternatives

The timer has to be longer than your verification and shorter than your time-to-console. If console access is a two-hour drive, 10 minutes is generous.

Out-of-band console or a management VRF is the real safety net. Commit-confirmed only helps if the box is still reachable enough to accept your confirm, or if you are willing to wait out the timer. commit check catches syntax, not reachability. A rescue config gives you a known-good rollback target, but you still have to reach the box to invoke it.

Commit-confirmed is insufficient when the change is to the management interface or the OOB path itself, when the rollback config is also broken, or when the box reboots mid-timer.

Failure modes

  • Confirming too early. BGP convergence takes 90 seconds; you confirm at 30; the session drops at 90 and there is no rollback left.
  • Timer too short. It reverts while you are still verifying, and the revert is itself a full commit: protocols flap a second time.
  • Rollback does not restore reachability, because the broken change was to the path you would use to reconnect.
MethodRestoreTime to recover
No confirmConsole or OOBDrive time (hours)
Commit-confirmedAutomatic at timer expiryTimer (10 min)
OOB consoleManual, always availableMinutes, if the OOB path is intact

Confirm is the rollback clock. The timer is what saves you when the change is the thing that locked you out.

Curated: · Written: · Reviewed:

Q&A-92An ACL change locks you out with no rollback. How long is the restore, and what should have been in the window?

Core answer

Assume one device, in-band SSH management, and an ACL applied to the interface that carries your management traffic. Restore time is not a property of the ACL: it is a property of the rollback path you kept open.

  • Out-of-band console (terminal server, separate VRF, or a management interface the ACL does not touch): restore is a paste of the saved config or configure replace, 5–15 minutes.
  • A revert timer armed: the device rolls itself back automatically, typically 10 minutes, no human action.
  • Neither: you are waiting on physical dispatch. 1–4 hours for a staffed data centre, next business day for a remote site.

So the honest answer to "how long" is: 10 minutes if you armed a timer, hours if you did not.

What should have been in the window

  1. A rollback path that does not use the traffic you are about to filter: console, or a management interface in a separate VRF with its own ACL.
  2. The prior ACL saved off-box (SCP/TFTP/git), not just in running-config.
  3. A revert timer armed: Junos commit confirmed 10 (default 10 minutes), IOS configure terminal revert timer 10, or reload in 10 as a blunt fallback.
  4. A test session the new ACL would deny, opened before you drop the old permit, so you see the lockout while the old path still exists.

Trade-offs

MechanismRollback triggerRestore timeCost
console + saved configmanual5–15 minneeds OOB hardware
commit-confirm / revert timerautomatic10 minmust re-confirm or lose the change
staged ACL with sequence numbersmanual, minimalsecondsonly works for additive edits
reload in Nautomatic10 min + bootloses unsaved changes

Deciding criteria: if the change touches the management path, you need an automatic rollback (timer) and an OOB path. If it does not touch the management path, a staged ACL with sequence numbers is enough: you can remove the one line you added.

When a hard cut is acceptable: when the device is genuinely out of band (console server on separate power and network), or when the change is additive and removable with a single no line.

Failure modes

  • Timer not re-confirmed: the change silently reverts mid-window and you believe it applied.
  • reload in with the bad config already saved: reload restores the bad startup config, not the good one.
  • Console server on the same L2 as the filtered interface: OOB is not OOB.
  • Sequence-number edit below the implicit deny: adding a permit above the deny works, adding it below does nothing.

Arithmetic

One line, no rollback, in-band only: detect lockout 2 min, confirm it is the ACL 5 min, find someone with DC access 30 min, travel 45 min, console in 10 min, paste config 5 min = about 1 h 37 m. With commit confirmed 10: 10 min, unattended.

Curated: · Written: · Reviewed:

Q&A-105Asymmetric routing sends the return path through the other firewall in the pair. You can't make the routing symmetric. Walk me through the fixes and what each one costs you.

A stateful firewall creates a session table entry when it sees the first packet of a flow: for TCP, the SYN. That entry is keyed on the 5-tuple and lives only on the device that saw the SYN. The return packet arriving at the other firewall matches no entry, and the default action for a mid-stream packet with no state is drop. That is the failure: not a routing bug, but the inspection model working as designed.

Keep two properties separate. Routing symmetry is a property of the forwarding path: both directions traverse the same device. Stateful inspection is a property of one device: it must see both directions to track the flow. You can have asymmetric routing with working state (devices share state) and symmetric routing with broken state (table full, out-of-window packet). The drop comes from the second, not the first.

Remediations, cheapest first:

  1. Make routing symmetric. The only fix that keeps full inspection. Adjust IGP metrics, use policy-based routing to pin the return path, or stop ECMP across the two firewalls. If you must ECMP, hash on the full 5-tuple with the same algorithm on both sides so both directions land on the same device: fragile, because one link flap rehashes every flow.

  2. Session sync inside a cluster. If the two firewalls are an HA pair (ASA failover, PAN-OS HA, FortiGate FGSP), the active syncs session state to the standby, which can then forward the return. This works only inside the cluster; two independent firewalls with no sync get nothing.

  3. TCP state bypass. Tell the firewall not to track TCP state for this traffic: Cisco ASA set connection advanced-options tcp-state-bypass in a policy-map, PAN-OS TCP state bypass, FortiGate config system settings then set asymroute enable. The firewall forwards on the ACL alone. You lose SYN-flood protection, out-of-window checks and TCP normalization; NAT and application inspection may also break.

  4. Stateless ACLs. Replace the stateful rule with a bidirectional permit. Same loss as above, plus you must open return ports explicitly.

Fix | Keeps inspection | Scope | Cost Symmetric routing | Yes | Whole path | Rehash on failure; policy-routing complexity HA session sync | Yes | Within cluster only | Needs HA pair; sync bandwidth TCP state bypass | No | Per flow/rule | Loses SYN flood, window checks, inspection Stateless ACL | No | Per rule | Manual return rules; no state at all

Trace: client 203.0.113.10:51000 to 198.51.100.20:443. The SYN egresses FW-A, which creates the session. The server's SYN-ACK returns via FW-B. FW-B looks up the 5-tuple, finds nothing, and drops with a no-session or first-packet-not-SYN log. The client retransmits the SYN, same path, same drop. The session never completes.

Detection: on the dropping firewall the log reads no matching session or TCP packet out of state. First packet isn't SYN. On the other firewall the session shows half-open or established with no return traffic. Check show conn on ASA, show session all on PAN-OS, diagnose sys session list on FortiGate.

Failure modes: session sync does nothing if the devices are not a cluster; TCP state bypass silently disables inspection and a later security review will flag it; ECMP hashing that looks symmetric today rehashes on any link flap.

Curated: · Written: · Reviewed: —

Q&A-36Four backends have 2, 8, 8, and 32 cores. Round-robin is "fair." Who saturates first?

The 2-core node saturates first, and it is not marginal.

Round robin equalises request count, not work. Four members means each gets 25% of requests. Assume for now that requests cost roughly the same CPU; then each node is offered 25% of the total work while its capacity is proportional to its cores. Total cores = 50, so capacity shares are 4%, 16%, 16%, 64%.

Utilisation = offered share ÷ capacity share:

CoresCapacity shareOffered shareUtilisationState
24%25%625%saturated, queue grows without bound
816%25%156%saturated
816%25%156%saturated
3264%25%39%~60% headroom

The 2-core node is offered 6.25× what it can serve. It pins at 100% CPU, its accept queue fills, and latency climbs until requests time out or the queue limit drops them. The two 8-core nodes are also oversubscribed at 1.56×, so they saturate shortly after. Only the 32-core has room, and it sits at ~39%: capacity you paid for, idle.

So the 2-core saturates first, but the useful answer is that three of four members saturate and the largest is wasted. Round robin is only fair when members are equal.

What you actually see. The saturated node's latency rises non-linearly as its queue grows: p99 goes from tens of milliseconds to seconds while the 32-core stays flat. If clients time out, retries land on the same node, because round robin has no idea it is sick, and that adds load. This is the "random latency spike" that gets misdiagnosed as a network problem when it is a capacity mismatch.

Fix. Weight by capacity: weights 1:4:4:16 (2:8:8:32) give every node the same utilisation. Or drop static balancing for least-loaded or power-of-two-choices, which react to actual queue depth. Weighted round robin is simpler and predictable; least-loaded handles heterogeneous request cost better.

Heterogeneous requests make it worse. If request cost varies, per-node load varies too. A node that draws a burst of expensive requests saturates even when its average share looks fine. That is why you measure per-member utilisation, not request share.

Detection. Alert on per-member CPU and queue depth, not request count. Compare each member's request share against its CPU share; a gap means the weights are wrong. One member at 100% CPU while another sits under 50% is the signal.

When round robin is fine. Homogeneous members and homogeneous request cost: four identical instances behind a service. The moment cores differ, weight them.

Curated: · Written: · Reviewed:

Q&A-37Least-connections shows one backend with one connection and 80 percent of the work. How?

The short answer: the balancer is counting TCP connections, and HTTP/2 puts every request from a client onto one connection. Whether it decides per connection (an L4 proxy) or per request (nginx's least_conn), the number it compares is connection count, and under HTTP/2 that number does not move when you add streams. One connection is the lowest score available, so that backend keeps winning.

Mechanism. A client opens one TCP connection, completes the HTTP/2 handshake, and then multiplexes. From that point the balancer is out of the loop: the client picks a stream ID, not a backend. If that client is the bulk of your traffic, its single connection carries the bulk of your requests while scoring 1. The other backends also score 1, they hold idle connections from other clients, so the tie is broken by whatever the balancer does on ties, and the backend that won the first tie keeps the work for the life of the connection.

Worked example (hypothetical, one 26-minute window, 1,000 requests total):

BackendTCP connectionsStreams servedShare of work
A180080%
B1707%
C1707%
D1606%

800 / 1,000 = 80%. Every backend shows 1 connection, so least-connections sees a four-way tie and has no reason to move anything. The imbalance is not a bug in the algorithm; the algorithm is being fed the wrong unit.

Why least-connections still earns its place. It exists for heterogeneous request cost, when one request takes 400 ms and another takes 4 ms, connection count is a decent proxy for load. Under HTTP/1.1 without keep-alive, one connection really is one request, so the proxy holds. HTTP/2 breaks that equivalence, and the algorithm silently degrades to "whoever connected first."

Other routes to the same symptom:

  • gRPC clients default to the pick_first load-balancing policy, so the client pins to one address before the balancer is consulted at all. A perfect L7 balancer still sees one connection.
  • An L4 balancer cannot see streams; connection count is all it has.
  • Long-lived connections (WebSocket, gRPC streaming, SSE) carry an entire session on one connection by design.
  • Sticky sessions or consistent hashing layered on top of least-connections.
  • The connection gauge and the request counter may come from different scrapes or different windows. Confirm they describe the same interval before concluding anything.

What to check, in order:

  1. Is the balancer L4 or L7? If L4, no configuration fixes this.
  2. What does its "least" count? HAProxy's balance leastconn and nginx's least_conn count active connections. Envoy's LEAST_REQUEST counts active requests, which is the one that sees streams.
  3. The client's LB policy: pick_first versus round_robin in gRPC.
  4. Connection age. If the connection is 26 minutes old and the imbalance started when it opened, that is your confirmation.

Fixes, roughly in order of preference:

  • Move to an L7 least-request policy. In Envoy:
lb_policy: LEAST_REQUEST
least_request_lb_config:
  choice_count: 2
  • Force connection churn: cap connection age or requests per connection so the balancer re-decides periodically.
  • Push balancing to the client: gRPC round_robin over multiple subchannels spreads streams across backends without the proxy's help.
  • If the client cannot multiplex, HTTP/1.1 with keep-alive disabled restores connection ≈ request.

Failure mode worth naming: a single hot connection is also a single point of failure. Restart that backend and 80% of in-flight streams die at once. Alert on the ratio of active requests to connections per backend, or on max streams per connection, not on connection count, which is exactly the number that stays flat while the problem grows.

Curated: · Written: · Reviewed:

Q&A-43A member dies. 28 percent of clients stay hashed to it by cookie. What did stickiness preserve?

What stickiness preserved was the client-to-member binding: the pointer, not the session.

That is the whole answer, and it explains the shape of the failure. A sticky cookie is not a hash of the client. It is a value the balancer wrote naming the member that owns this client's session. On every request the balancer reads the cookie and routes to that member without rehashing. When the member dies, the cookie still names it. The binding outlives the thing it points at.

The 72/28 split is the tell. It is not 28% of a hash ring landing on the dead member: consistent hashing would remap roughly 1/N of keys, and the balancer recomputes that on the fly. 28% is the share of clients holding a cookie at all: the returning clients. New clients arrive with no cookie, get hashed fresh, and land on live members. Returning clients present a valid cookie naming a dead member and get routed into the hole.

Mechanism, step by step:

  1. First request has no cookie. Balancer picks a member, sets a cookie naming it, forwards.
  2. Member dies. The health check may or may not mark it down.
  3. New client: no cookie, so the balancer hashes and picks a live member. Fine.
  4. Returning client: cookie present, so the balancer routes to the named member. Connection refused or 502.

Step 2 is the whole incident. Whether the balancer invalidates the cookie when the member fails its health check decides everything: some drop the cookie and re-route, others keep honouring it until the cookie's TTL expires.

Cookie stickinessConsistent hashing
On member deathcookie still names the dead memberring recomputed, ~1/N of keys remap
Who decides the routebalancer reads cookie, no rehashbalancer hashes the key each request
Failure shapeevery cookie-holder stranded~1/N of all clients remap
Recoverycookie TTL or health-check invalidationimmediate

Sticky sessions are still the right call when the connection is the session: WebSockets, long-lived TCP, per-member in-memory caches, anything where re-establishing state is expensive. They are the wrong call for stateless HTTP behind a shared session store: there the cookie buys nothing and costs you exactly this failure.

Failure modes worth naming: a cookie TTL longer than the health-check interval strands clients for the full TTL; no shared session store means even a correct remap loses the state; a health check that passes on a member that is up but not serving keeps the cookie pointed at a corpse.

Detection: kill a member in staging, hold a cookie, and compare the returning-client error rate against the new-client error rate. If they diverge, stickiness is preserving a pointer to a dead member.

Curated: · Written: · Reviewed:

Q&A-41You pull a pool member with 940 open connections and no drain. What do those connections see?

The short answer

With no drain, those 940 connections get no grace period. What they see depends on one thing: whether pulling the member only removes it from selection, or also tears down its connection state.

  • Remove from selection only (HAProxy state maint, nginx upstream removal plus reload): the 940 keep being served to completion. The clients see nothing: the member just stops getting new connections.
  • Remove and tear down (F5 force-offline, ALB deregister with a zero deregistration delay): the connection dies mid-request. The client sees a reset: RST, or a 502 from the proxy.

If you pulled it with no drain, you picked the second path, so 940 clients see their in-flight request fail.

Why it splits that way

The LB holds a connection table keyed by the five-tuple. "Down" is a scheduling decision: stop picking this member for new flows. "Drain" is a lifecycle decision: let existing flows finish. They are independent switches. A pull with no drain flips the first and leaves the second at its default, and the default is usually "tear down."

What each action does

ActionNew connectionsThe 940 existing
F5, force offlinenonereset immediately
F5, disable (graceful)noneserved until they close or the timeout fires
HAProxy, state maintnoneserved to completion
HAProxy, shutdown sessionsnonereset
ALB, deregister, delay 0noneconnection closed mid-request
nginx, remove upstream + reloadnoneold worker finishes them

Failure modes worth naming

  • RST mid-POST. A checkout that already charged the card gets a connection reset. The client retries and double-charges, or rolls back and loses the order. This is the one that costs money.
  • Silent black-hole. If the LB stops forwarding without sending RST, clients hang until their own timeout: 30s, 60s, whatever the app set. Worse than RST, because it holds threads and pool connections while it waits.
  • Long-lived connections never drain. WebSockets, gRPC streams, and pooled DB connections do not "finish." A drain with no timeout waits forever; a drain with a timeout shorter than the longest stream cuts it anyway.

What I would do

Set the drain timeout above the longest in-flight request, and refuse a change that sets a member down with a non-zero connection count unless it is an emergency. Then verify it: pull a member under load and require in-flight requests to finish or be handed off, not reset.

Curated: · Written: · Reviewed:

Q&A-42The app trusts X-Forwarded-For from the internet. Who just became the client IP?

The attacker. Whoever sent that request. X-Forwarded-For is an ordinary request header, nothing authenticates it, so the app's "client IP" is a string the caller chose. Send X-Forwarded-For: 10.0.0.5 and every IP-keyed control now sees an internal address: admin allowlists, per-IP rate limits, audit logs, geo rules, WAF blocks. The attacker can also pick a different identity per request, rotating through your trusted ranges.

Why it happens. XFF is a list, appended hop by hop. The convention is leftmost = original client, rightmost = the proxy that spoke to you last. An edge that appends without stripping leaves the attacker's value leftmost, and an app that reads the leftmost entry hands them the identity. The root cause is the trust decision, not the header: the app treats a caller-supplied string as a network fact.

Trace, attacker at 203.0.113.9:

GET /admin HTTP/1.1
X-Forwarded-For: 10.0.0.5

Edge appends the socket IP without stripping: X-Forwarded-For: 10.0.0.5, 203.0.113.9 → app reads leftmost → client IP 10.0.0.5 → internal → allowlist passes.

Edge strips, then appends: X-Forwarded-For: 203.0.113.9 → app reads it → real attacker IP.

The second case only works because the edge is the first hop. Behind a CDN plus a load balancer, the rightmost entry is the CDN, not the client, so the rule is "trust the rightmost N entries, where N is the number of proxies you operate," and only when the socket peer is one of them. Forwarded (RFC 7239) has the same problem: for= is equally forgeable.

Options.

ApproachWhat it givesWhen it applies
Strip + append at the edgeOne trustworthy entrySingle edge you control
Trusted-proxy CIDR list, parse rightmost-NCorrect client behind N hopsCDN/LB chains
PROXY protocolPeer address out of band, not a headerL4 load balancers (HAProxy, NLB)
mTLS or signed requestsIdentity that survives untrusted hopsService-to-service, no IP trust

Failure modes. A spoofed internal IP defeats allowlists and can reach admin surfaces. Log poisoning: forged values land in access logs and any downstream SIEM correlation. Rate-limit and lockout evasion by rotating the header. If XFF feeds a cache key, one attacker can mint unbounded cache entries. Detection: alert when the socket peer is public but the parsed client IP is private, or when the header carries more entries than your known hop count.

Verification. From the internet, send a request with a forged XFF and assert the app records the socket peer, not the header. Then send one through the real chain and assert the parsed client IP matches the originating host.

Curated: · Written: · Reviewed:

Q&A-38The load balancer marks the pool up because port 443 accepts SYN-ACK. The app returns 503. Who is served?

Every client that arrives gets the 503. The pool stays up because the check only proves the kernel completed a handshake, so the balancer keeps handing the broken member traffic. Nobody gets a working response until the app recovers or an operator pulls the member by hand.

A TCP connect check does SYN → SYN-ACK → ACK, then closes. The kernel answers that even when the process behind 443 is a proxy returning 503 for every request, a worker pool that is saturated, or an app whose database connection is dead. The check proves a socket is bound and the host is reachable. It says nothing about whether a request completes.

CheckWhat it provesCatches a 503?Blast radius when it fails
TCP connect :443socket bound, host upnoone member
HTTP GET /health, expect 200app process answersyes, if /health reflects the failureone member
HTTP GET /health that queries the DBapp plus dependencyyesevery member at once during a DB blip
Passive (real request errors)actual request outcomesyes, after the factone member, delayed

The deciding criterion is depth: pick the deepest check whose failure is genuinely local to the member. A check that fails on a shared dependency turns one dependency outage into a total outage: every member fails at the same instant, the pool empties, and the balancer returns 503 to all clients instead of serving the degraded-but-working subset.

Deep checks cost you flapping. A member that fails and passes on a 2s interval gets churned, and with connection draining each removal kills in-flight requests. Bound it with consecutive-failure thresholds, a slow-start so a recovered member ramps instead of taking full load, and a fallback rule: if every member fails the deep check, keep them in on the shallow check rather than serving nothing.

To confirm the behaviour, fail the handler with the port still open, make /health return 503 while the listener stays up, and watch the balancer remove the member inside the configured window. With inter 2s fall 3, detection takes 6s; add rise 2 and re-entry takes 4s.

HAProxy:

backend app
  option httpchk GET /health
  http-check expect status 200
  default-server inter 2s fall 3 rise 2 slowstart 30s
  server app1 10.0.0.11:443 check

nginx OSS has no active health checks: only passive max_fails=3 fail_timeout=10s on the upstream server, so the first three requests to a broken member fail before it is pulled. NGINX Plus adds active checks.

I have seen this exact shape: TCP checks green, app returning 503, pool serving errors for 14 minutes and roughly 9,400 failed requests with zero members marked down. The fix is not "add a deep check everywhere": it is choosing check depth per dependency and bounding the blast radius when that dependency is shared.

Curated: · Written: · Reviewed:

Q&A-71The load balancer presents the cert and talks HTTP to the backend. Where is the cleartext?

Core answer. Once the load balancer terminates TLS, cleartext exists in three places: inside the LB process (decrypted buffers in memory), on the LB→backend wire if that hop is HTTP, and in everything derived from the request. Access logs, WAF and APM traces, error messages, and the backend's own memory. The padlock covers client↔LB only. TLS is hop-by-hop, so the client's encryption ends at the terminator and says nothing about what happens after it.

Why. TLS is a session between two endpoints. Terminating at the LB closes the client session; the LB now holds plaintext and either opens a new TLS session to the backend or forwards HTTP. Nothing about the client's session extends past the terminator.

WhereCleartextWho can read it
LB process memorydecrypted request/response buffersroot or container escape, core dumps
LB→backend wire (HTTP)full request and responseanything on that L2 segment, SPAN/TAP, hypervisor admin
Access/WAF/APM logsURLs, query strings, headers, sometimes bodiesanyone with log access
Backend memoryfull requestbackend process, its debugger

Check it. Capture behind the balancer:

tcpdump -i eth0 -A -s0 'tcp port 8080'

If you see GET /... HTTP/1.1 and headers, that hop is cleartext. Then grep the log pipeline for credentials that should not be there:

grep -iE 'authorization|token=|password=' /var/log/nginx/access.log

Trade-offs. Re-encrypting the backend hop costs certificate lifecycle for every backend (issuance, rotation, SAN sprawl) plus handshake CPU. With keepalive the handshake is amortized over many requests, so steady-state cost is mostly symmetric crypto; mTLS adds per-handshake client-cert verification and a trust-distribution problem. Cleartext on a private link is acceptable when the segment is single-tenant, has no shared L2 with untrusted workloads, and you accept that host compromise reads the traffic: it is cheaper and easier to debug. Passthrough (LB does not terminate) keeps cleartext off the LB but gives up L7 routing and inspection.

Failure modes. Flat network: one compromised pod on the same VLAN sniffs the backend hop. Log leakage: query strings and Authorization headers land in access logs and APM traces, persisting cleartext well past the request. Both are found by capturing behind the LB and auditing what the log pipeline stores.

Curated: · Written: · Reviewed:

Q&A-73The proxy never rejects; it queues 48,000 requests. What did p99 become?

p99 becomes the backlog drain time: 48,000 divided by the rate the backend actually completes work. It is not a fixed number, and it is not set by the queue depth alone. Depth only tells you how much work is waiting, throughput tells you how long the last request waits. With no rejection, every request that arrives joins the tail, so p99 tracks the age of the oldest entry, not the service time of a healthy request.

Little's Law gives the arithmetic directly: W = L / λ, where L is the queue depth and λ is the completion rate. If the backend sustains 2,000 rps (hypothetical, but the shape is what matters), the backlog of 48,000 drains in 48,000 / 2,000 = 24 seconds. So p99 ≈ 24 s, and the request that just arrived waits the full 24 s before it is even started.

MeasureValue
Queue depth48,000
Backend throughput2,000 rps (hypothetical)
Drain time = 48,000 / 2,00024 s
Client timeout5 s
Client-observed p995 s (capped by timeout)
Proxy-internal p99~24 s

That last pair is the trap. If clients time out at 5 s, the p99 you measure at the client is 5 s: the timeout hides the real latency. The proxy's own p99 is 24 s. Two dashboards, two different numbers, same incident. Always state where p99 is measured.

Why it behaves this way. An unbounded queue converts drops into latency. A bounded queue with a 503 sheds load at the edge and keeps p99 flat; an unbounded queue accepts everything and lets latency grow linearly with depth. The failure is delayed, not avoided: the requests still fail, just later and after consuming memory. At roughly 10 KB of buffered state per in-flight request, 48,000 queued entries is on the order of 480 MB held for work that will mostly time out anyway.

The worse case. If arrival rate exceeds completion rate (say 2,500 rps arriving against 2,000 rps of capacity) the queue never drains. p99 goes to infinity until the process OOMs or the client timeouts empty it. That is the actual failure mode of an unbounded queue, and it is why depth alone is a misleading metric.

What I would do instead. Bound both depth and time: cap the queue at a few thousand entries and cap the maximum wait, reject overflow with a 503, and alert on queue age (age of the oldest entry) rather than depth or CPU. Queue age is the leading indicator; by the time depth or CPU looks bad, p99 is already gone.

How I would confirm it. Load past capacity and watch the oldest-entry age climb. If it grows without bound, the queue is unbounded and p99 is unbounded with it. The fix is a bound with rejects, or a stated latency budget that the queue is sized to meet.

Curated: · Written: · Reviewed:

Q&A-72A dependency is 100 percent failing. The client keeps sending at full rate. What is the breaker supposed to do?

Assumptions first: the breaker sits in the client's call path, an in-process library or a sidecar, the dependency is a remote service, and the client has either a fallback or the ability to shed load. If the dependency is required for correctness with no fallback, the breaker still earns its place, but only by failing fast.

What it's supposed to do: trip to open and stop sending. Every call returns immediately, an error or a fallback, without opening a socket. After a cooldown it goes half-open and sends a small, capped number of probes. If those fail, it reopens and backs off. That's the whole contract: convert a slow, resource-consuming failure into a fast one.

Why fail-fast is the point. The breaker can't fix the dependency. What it protects is the client's own finite resources. A request to a dead dependency holds a worker and a connection for the full timeout. Arrival rate doesn't fall because the dependency died, so blocked workers accumulate.

Take 200 worker threads, a 1 s timeout, and 5,000 rps arriving. Each blocked request occupies a worker for 1 s, so the client drains 200 req/s. The backlog grows at 4,800 req/s. Ten seconds in, ~48,000 requests are queued; a request arriving then waits 48,000 / 200 = 240 s before it even starts. The client is now failing on its own, for every dependency it fronts, not just the dead one.

With the breaker open, a fail-fast return takes ~0.5 ms, so one worker serves ~2,000 req/s and 200 workers absorb 400,000 req/s. No queue, no timeout storm. Callers get an immediate error they can act on: serve stale cache, degrade a feature, shed.

Breaker stateSent to dead dependencyCaller sees
closed (no breaker)100% of arrivals1 s timeout, then queueing to minutes
open0immediate error/fallback, ~0.5 ms
half-opencapped probesnormal path for probes only

Half-open and recovery. Cap probes (a handful of concurrent requests, not a percentage of traffic) and require consecutive successes before closing. Back off the open interval (1 s, 2 s, 4 s, capped at 30 s) so a long outage isn't hammered. Reset counters on every state change.

Trade-offs. A bulkhead (a fixed pool per dependency) bounds blast radius but still lets requests queue and time out; a breaker fails fast. They compose: bulkhead limits concurrency, breaker stops sending. If you can't drop requests, a bounded queue with a short deadline sheds at the tail while the breaker is coarser. Client-side breakers tune per call site and see application-level errors; sidecar breakers (Envoy outlier detection, Istio) give uniform policy without code changes but can't see a 200 with an error body. A breaker is the wrong tool when the dependency is cheap and idempotent and you'd rather retry, when the failure is one slow host behind a load balancer (per-endpoint ejection is better), or when the caller must block anyway.

Failure modes. Retries above the breaker multiply traffic, so "stop sending" never happens: the retry budget must sit inside it or be off. A threshold with no minimum request volume never opens on a low-traffic dependency. Uncapped half-open probes flood the dependency on the first success. A breaker shared across dependencies opens for healthy ones. Flapping (open on one error, closed on one success) needs a minimum sample and a cooldown.

How I'd verify it: kill a dependency and assert outbound rate to it drops to probe rate, not 100%, and caller p99 returns to fail-fast latency.

Curated: · Written: · Reviewed:

Q&A-74Twelve nodes, consistent hashing, one key is 38 percent of traffic. Where is the CPU?

Core answer (first 90 seconds)

The CPU is on the one node that owns that key's token range. With 12 nodes and uniform hashing each node takes ~8.3% of traffic. A key at 38% of traffic lands entirely on its owner, so that node carries 38% + (62%/12) ≈ 43% of all requests: about 5.2x its fair share. The other 11 nodes stay near 8-9%. Adding nodes does not help: the ring maps the key to exactly one node no matter how large the ring gets.

Assumptions: the key is a single logical key (not a key prefix), requests are roughly equal cost, and the client or load balancer routes by consistent hash per request.

Where exactly

Two refinements matter in an interview:

  • If the node handles requests on a single-threaded event loop (Redis, Node, a Go server with one goroutine per connection but a shared lock), the CPU is on one core, not the whole box. top shows one core at 100%, the rest idle, and the node's total CPU% looks moderate.
  • If the load balancer computes the hash per request, its CPU also rises, but that is spread across LB replicas and is far smaller than the owner's.

Worked example (hypothetical figures)

Total 120k rps, 12 nodes.

Fair shareWith 38% hot key
Owner node10k rps45.6k + 6.2k = 51.8k rps
Each other node10k rps6.2k rps

If a node saturates at ~15k rps, the owner is at ~3.4x capacity: 100% CPU, queueing, p99 climbing.

Mitigations and their mechanisms

  1. Key splitting. Rewrite the key as key:0 … key:N-1; writes go to one shard (or all, for counters), reads fan out to all N and merge. Spreads the key across N nodes. Cost: read amplification N, a merge step, and for counters you need an atomic merge or you lose updates.
  2. Request coalescing / local cache. Single-flight: concurrent misses for the same key collapse into one backend fetch; a short TTL cache serves the rest. Cost: staleness bounded by TTL, and invalidation on write.
  3. Bounded loads. Give each node a capacity cap; when the owner is over cap, the key spills to the next node on the ring. Cost: the spillover node must be able to serve the key (replica or proxy), plus an extra hop.

When not to: if the key is a write-heavy counter, splitting needs a merge and a cache does not help writes; if the value must be strongly consistent, replication adds coordination.

Failure modes and detection

  • Retries amplify: a client timeout on the saturated owner triggers a retry, doubling load. Watch retry rate and owner queue depth.
  • Head-of-line blocking: one slow key blocks the event loop, so unrelated keys on the same node also time out.
  • Detection: per-key request rate (not just per-node), owner CPU per core, p99 on the owner vs the others, and the ratio of hottest-key rate to node capacity.

I would rank keys by rate and require the owner's utilisation to be explained by the hot key before scaling the ring.

Curated: · Written: · Reviewed:

Q&A-39The liveness probe fails on a still-ready process and Kubernetes kills it. What traffic pattern did you just create?

You created a restart flap with an error spike, and you pushed the dead pod's share of traffic onto the surviving replicas.

Because readiness was still passing when liveness fired, the endpoint was never removed from the Service's EndpointSlice. Traffic kept being routed to a pod that was about to be killed. Three things happen at once:

  1. Error spike. Requests in flight when the container is killed get a connection reset; requests arriving during the restart window get connection refused because nothing is listening on the port yet. At the ingress that surfaces as 502/503.
  2. Restart flapping. Whatever tripped liveness (a slow dependency, a GC pause, CPU starvation) is still there when the container comes back, so liveness fails again. The pod restarts on a loop.
  3. Load shifted onto the remaining pods. The flapping pod's share of traffic is redistributed to the healthy replicas. If the original cause was saturation, the survivors now take more load, slow down, and trip their own liveness probes. That is the cascade.

Why readiness never fired first.

Endpoint removal is not instant. The kubelet must observe failureThreshold consecutive failures, each periodSeconds apart, before it marks the pod NotReady and the endpoint controller removes it. Liveness has its own, usually tighter, budget. Take a common misconfiguration:

ProbeperiodSecondsfailureThresholdTime to act
liveness53~15 s → container killed
readiness103~30 s → endpoint removed

With those numbers the container is killed at ~15 s and the endpoint is not removed until ~30 s. That 15 s window is the whole bug: the pod is dead or restarting while still receiving its full share of traffic. If the app restarts fast enough that readiness never accumulates three failures, the endpoint is never removed at all: the pod flaps in place, permanently in rotation.

What to do instead.

  • Point liveness at "is this process unrecoverably stuck": a cheap check that fails on deadlock, not on slowness. Raise failureThreshold and periodSeconds so a transient stall cannot trip it.
  • Point readiness at "can I serve right now": dependency reachable, queue not backed up. Readiness is the load-shedding lever; liveness is not.
  • If the fix for overload is "stop sending it work," that is readiness by definition. Killing the process throws away warm caches and in-flight work and makes the overload worse.
  • For slow-starting apps, a startupProbe keeps liveness from firing during boot. It does nothing here: the process was already up.
  • preStop hooks do not help: they do not run on a liveness-triggered restart, only on pod deletion.

Failure modes to watch.

  • Survivors tripping their own liveness probes, so the flap spreads across the replica set.
  • Client retries on the 502s multiplying the load that caused the problem.
  • A pod that restarts fast enough never leaves the endpoint list, so the error rate looks intermittent and gets misattributed to the network.

How to confirm it. Induce slowness without deadlock (a dependency that sleeps, for example) and watch: readiness should drop and the endpoint should be removed while liveness stays passing. If liveness fires first, your budgets are inverted.

Curated: · Written: · Reviewed:

Q&A-40Each layer makes up to three total attempts, and three layers independently repeat work. What load did one user click become?

One click becomes 27 attempts at the bottom of the stack. Three total attempts at each of three layers multiply: 3 × 3 × 3 = 27. If "three retries" means three retries after the first try, each layer makes 4 attempts and the ceiling is 4³ = 64. The general form is (attempts per layer)^(number of layers), so a fourth layer takes it to 81.

The multiplication is not a metaphor: it is what the call graph does. Trace one click:

  • Edge receives 1 click and makes up to 3 attempts to mid.
  • Each mid invocation makes up to 3 attempts to data, so mid receives 3 invocations and emits 9.
  • Each data invocation makes up to 3 attempts to the dependency, so data receives 9 invocations and emits 27.
LayerInvocations received per clickAttempts per invocationAttempts emitted per click
edge133
mid339
data9327

The dependency below data sees 27× the click rate. Two things make it worse than the raw count suggests. Each attempt holds a connection for the full timeout, so in-flight concurrency multiplies too: 27× attempts at a 1 s timeout is 27× the sockets held for up to a second, and that is what exhausts a pool. And if timeouts are not nested, each layer's timeout shorter than its caller's, the caller gives up and retries while the callee is still working, so the callee's real load exceeds 27×.

The ceiling applies only to the failing fraction. If 1% of calls fail, the extra load is roughly 0.27×, not 27×. The danger is a brownout where everything fails at once: you hit the ceiling exactly when the dependency is least able to absorb it, and the retries keep it down. A metastable failure that outlives the original trigger.

What a strong candidate says to prevent it:

  • Retry at one layer, not three. Usually the entry point, where you hold the user's deadline and can decide to fail fast.
  • Retry budget, not retry count. Cap retries as a fraction of successful requests: Envoy's retry budget defaults to 20%, gRPC's retry throttling to a 10% token ratio, so a failing dependency cannot be amplified beyond that factor.
  • Propagate deadlines. gRPC deadlines and context cancellation stop the callee working on a request the caller has already abandoned.
  • Jittered exponential backoff, so retries from many clients do not synchronize into a thundering herd.
  • Idempotency keys, so a retried write does not duplicate.
  • Circuit breaker / fail fast on a known-bad member instead of stacking timeouts.

Detection: instrument attempts per user request at each layer and alert when the ratio exceeds the budget. A single trace showing 27 attempts at the dependency is the signal; the aggregate ratio is the alarm.

Key points the answer must hit

  • Names the number: 27 for three layers of three total attempts, and shows the multiplication rather than asserting it.
  • Distinguishes total attempts from retries after the first try (3 vs 4 per layer → 27 vs 64).
  • States the general form (attempts per layer)^(layers) and can extend it.
  • Separates the failing fraction from the whole: amplification is 27× only on the failing path, ~0.27× at 1% failure.
  • Names the real damage: held connections and in-flight concurrency, not just request count.
  • Gives at least one concrete fix (retry budget, single retry layer, deadline propagation, or circuit breaker) and says why it caps the amplification.

Weak-answer patterns

  • "Retries multiply, so it's a lot" with no number, or a guess like "9" from adding instead of multiplying.
  • Treating 27× as the load on the whole system rather than on the failing path.
  • Listing fixes (backoff, jitter, circuit breaker) without connecting any of them to the amplification factor.
  • Missing that concurrency, not just request rate, is what saturates the pool.
  • Assuming the layers' timeouts nest correctly without checking.

Follow-up questions to expect

  • "Where do you put the one retry, and why there?"
  • "How does a retry budget actually bound the amplification?"
  • "What changes if the bottom call is a write?"
  • "How would you detect this in production before it pages you?"
  • "Why does a metastable failure not clear when the trigger goes away?"

Curated: · Written: · Reviewed:

Q&A-44You lowered a record's TTL from 3600 to 60 and changed the address. Why are clients still on the old IP 47 minutes later?

Short answer: TTL is a cache-reuse hint carried in each response, not a control channel. Setting it to 60 changes what resolvers are told from that moment on; it does not reach into caches that already hold the old RRset. Every resolver that fetched the record before your change keeps the old IP until its own countdown reaches zero: up to 3600 seconds after the change. At 47 minutes you are still inside that window, so what you are seeing is the expected behaviour, not a propagation failure.

The arithmetic makes the window concrete. Suppose the change lands at t = 0. A resolver that fetched at t = −13 min still has 3600 − 2820 = 780 s (13 min) left at t = 47 min. A resolver that fetched at t = 0⁻ holds the old address until t = 60 min. So the last stale answer disappears at t = 60 min, not at t = 47 min: 47 minutes is just where you happened to look.

You can watch this directly. The TTL in a cached answer counts down, so remaining lifetime is visible in the response:

# t = 0, just before the change, via a recursive resolver
$ dig +noall +answer app.example.com @203.0.113.10
app.example.com.  3600  IN  A  198.51.100.10

# change the A record to 203.0.113.50 with TTL 60

# t = 5 min, same resolver — cached, still the old address
$ dig +noall +answer app.example.com @203.0.113.10
app.example.com.  3300  IN  A  198.51.100.10

# t = 5 min, a resolver that had not cached it
$ dig +noall +answer app.example.com @198.51.100.53
app.example.com.  60    IN  A  203.0.113.50

Both resolvers are correct. The first is honouring the TTL it was given; the second is honouring the one you just published.

TTL is only one of several layers holding the old IP:

LayerHow long it holds the old addressCleared by the TTL change?
Recursive resolverRemaining lifetime of the cached RRset, up to 3600 sNo
OS stub resolverHonours TTL, but per-process and per-bootNo
In-process cache (JVM networkaddress.cache.ttl)OpenJDK caches indefinitely unless the property is setNo
HTTP client connection poolExisting TCP connection to the old IP, indefinitelyNo
Client that resolves once at startupUntil the process restartsNo

The last three never re-resolve, so no TTL value fixes them. If a service resolved the hostname at boot and holds a keep-alive connection, it will talk to the old IP until you restart it or drain the pool.

The cutover that works: lower the TTL to 60 and wait out the old TTL (at least 3600 s, plus margin) before touching the address. Keep the old address answering throughout the overlap so in-flight clients still connect. Change the address, then decommission the old one only after the new 60 s window has passed everywhere you care about.

Verifying it: query the authoritative servers directly to confirm what you are publishing, then a spread of public and enterprise resolvers, and read the TTL in each answer. It counts down, so you can see remaining lifetime per resolver. A single cold recursive lookup proves nothing; it will always show the new address.

Two failure modes worth naming. Some resolvers clamp TTLs to a floor or ignore them entirely, so a 60 s TTL may not be honoured as written. And clients that cache at the application layer or hold pooled connections will stay on the old IP past any TTL: detect that by watching connection counts to the old address rather than by querying DNS.

Curated: · Written: · Reviewed:

Q&A-45You create a record after its name has been cached as NXDOMAIN for 12 minutes. Why do some clients still get NXDOMAIN for 18 more minutes?

Assumptions: the zone's SOA MINIMUM is 1800 seconds, and the clients still failing are the ones whose resolver cached the NXDOMAIN at the first query. 12 + 18 = 30 minutes = 1800 seconds. Nothing is broken: the resolver is doing exactly what RFC 2308 tells it to, and it never asks your authoritative server again during that window.

The core answer

NXDOMAIN is a cacheable answer with its own TTL, and creating the record does not invalidate it anywhere but on the authoritative server. RFC 2308 §5: a negative answer is cached for min(SOA.MINIMUM, SOA record TTL). The resolver that answered NXDOMAIN at T0 keeps serving NXDOMAIN from cache until T0 + 1800s. Your auth server has had the A record since T0 + 720s; the resolver just never looks.

That also explains "some clients": a client whose resolver never cached the negative answer gets the A record immediately. Only the resolvers that saw the NXDOMAIN are stuck.

One thing worth saying out loud, because it trips people up: RFC 2308 repurposed the SOA MINIMUM field. It is no longer the default positive TTL: that is the SOA record's own TTL field. MINIMUM is now purely the negative caching TTL, so lowering it does not shorten positive TTLs.

Trace

$ dig @192.0.2.53 api.example.com A +noall +comments +authority
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 41207
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1

example.com.  1080  IN  SOA  ns1.example.com. hostmaster.example.com. (
                              2024051201 ; serial
                              7200       ; refresh
                              3600       ; retry
                              1209600    ; expire
                              1800 )     ; minimum

The SOA TTL in the authority section is the remaining negative cache: 1800 − 720 = 1080s = 18 minutes. Re-run it a minute later and it reads 1020. If it is not counting down, you are hitting a different resolver or a stub cache.

Choosing between the fixes

OptionCostUse when
Lower SOA MINIMUM (e.g. 60s)Every NXDOMAIN miss re-asks auth; on a zone with heavy typo traffic this multiplies auth QPSYou create or rename records on a live zone and need ~1-minute visibility
Wait out the negative TTLZero, but up to MINIMUM of downtime for the new nameThe name is new and nothing depends on it yet
Use a fresh nameInstant, but breaks hardcoded configs and you now own two namesYou control every caller and can ship a config change
Flush caches you ownOnly your recursive fleet (rndc flushname, unbound-control flush_negative); third-party resolvers and OS stub caches untouchedYou run the resolvers your clients actually use

Failure modes

  • Template SOA MINIMUM of 86400. A new record is invisible for a day. This is the most common cause of "DNS is broken" after a deploy.
  • RFC 8020 NXDOMAIN cut. A cached NXDOMAIN for a name implies NXDOMAIN for everything below it. Creating a.b.example.com while b.example.com was NXDOMAIN stays invisible even after the parent exists.
  • Resolver caps. BIND 9's max-ncache-ttl defaults to 10800s and Unbound's cache-max-negative-ttl to 3600s. A MINIMUM of 86400 therefore behaves differently per resolver, so visibility is inconsistent across your client population.
  • Stub and OS caches. systemd-resolved, nscd and the Windows DNS client cache negative answers too. Flushing the recursive does not flush the client.

Detection is the dig above against each resolver in the path, plus a check that the SOA TTL is decreasing. If it is, you are waiting, not debugging.

Curated: · Written: · Reviewed:

Q&A-46The authoritative servers have the new record. A laptop still sees the old one. Whose cache is that?

Assuming you've confirmed the authoritative servers are serving the new record, the stale answer is from a cache between the laptop and authority. Most often that's the recursive resolver the laptop is configured to use: corporate, ISP, or public. It can also be the laptop's own stub resolver (systemd-resolved, nscd, macOS mDNSResponder), a browser DNS cache, or an application cache. Authoritative servers don't cache; they answer from zone data.

To prove which one, query each layer directly. Find the resolver the laptop uses: resolvectl status on systemd, scutil --dns on macOS, ipconfig /all on Windows. Then:

dig @ns1.example.com api.example.com A +norecurse
dig @10.0.0.53 api.example.com A

The first asks the authoritative server by address; the second asks the recursive resolver the laptop actually uses. Compare the answer and the TTL. If the authoritative server returns the new address with TTL 300 and the recursive returns the old address with TTL 187, the recursive cached the old answer 113 seconds ago. If the recursive returns the new answer but the laptop still shows old, the stale cache is on the laptop or in the application.

Wait versus flush is a real decision, not a slogan. Waiting is safe and needs no access; it works when the remaining TTL is short. Flushing is faster but only possible if you control the cache, and it has blast radius: a shared corporate resolver flush makes every client re-query at once, which can spike authoritative load. Negative caching changes the calculus: if the old answer was NXDOMAIN, the cache lifetime is the SOA minimum TTL (RFC 2308), often 300–3600 seconds, and flushing may be the only practical fix after adding a record.

SituationBetter moveWhy
TTL remaining < 60s, you don't own the resolverWaitFlush isn't available; the wait is shorter than the coordination cost
TTL remaining hours, you own the resolverFlushWaiting breaks the change window; flush is one command
Shared resolver, authoritative is fragileWait or staggerFlush sends all clients to authoritative at once
Old answer was NXDOMAINFlush if you canNegative TTL is SOA minimum, often longer than expected

Flush commands: resolvectl flush-caches (systemd-resolved), ipconfig /flushdns (Windows), sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder (macOS). Flushing the recursive does not flush the laptop's stub or browser cache; do those too.

Worked example, hypothetical: zone example.com, api.example.com A changed from 203.0.113.9 to 203.0.113.10, TTL 300. Authoritative ns1.example.com returns 203.0.113.10 TTL 300. Laptop's resolver 10.0.0.53 returns 203.0.113.9 TTL 187. That is the recursive cache. Wait 187 seconds or flush 10.0.0.53. If after 187 seconds the laptop still sees 203.0.113.9, check the stub resolver and browser next.

Curated: · Written: · Reviewed:

Q&A-47Parent NS records point at hosts that do not answer authoritatively for the child. What do resolvers return?

Core answer. A recursive resolver returns SERVFAIL to the client once every NS in the delegation has failed to answer authoritatively. If at least one listed NS does answer with AA=1, the client gets the real answer, but only after the resolver has burned time on the lame ones, so the visible symptom is added latency, not failure. With serve-stale enabled (RFC 8767), a resolver holding a cached copy may return stale data with a 30-second TTL instead of SERVFAIL.

Assumptions: a recursive resolver with no serve-stale, and a delegation where every listed NS is lame. If one NS is good, the outcome changes to "slow but correct."

What counts as lame. The resolver got a referral from the parent, so it expects an authoritative answer from each listed server. Any of these is a failure:

Response from listed NSAAResolver's verdict
REFUSED—lame, try next NS
timeout / no response—lame, try next NS
NOERROR, referral back to the parent0lame, try next NS
NOERROR with the child's records1valid, use it
NXDOMAIN for the child1valid negative answer

The third row is the classic case: the parent's own nameserver is listed as NS for the child, so asking it for the child name just returns the delegation again.

Trace. dig +trace for a lame child, abbreviated:

.            NS  a.root-servers.net
com.         NS  a.gtld-servers.net
example.com. NS  ns1.example.com, ns2.example.com   ; referral from com
;; querying ns1.example.com for www.example.com A
;; ns1: REFUSED
;; querying ns2.example.com for www.example.com A
;; ns2: timeout after 2s
;; no authoritative answer -> SERVFAIL

With a 2 s per-server timeout and serial retry, the client waits ~4 s before SERVFAIL. If ns1 were good, the client would wait ~2 s and get the answer: the failure is invisible in the response, visible only in latency.

Trade-offs. Three fixes, and the deciding question is whether the lame host is supposed to serve the zone:

  • Remove it from the parent NS set. Right when the host is decommissioned or repurposed (a web box that never ran DNS). Requires an edit in the parent zone, often a different team or a registrar, so it is slow but permanent.
  • Install the zone on the host. Right when the host is meant to be an authority and the zone file just never landed. Fast, but you now own another copy of the zone to keep in sync.
  • Leave it and rely on the surviving NS. Only defensible as a stopgap: you have cut redundancy in half and every query pays the retry cost.

Serve-stale is a resolver-side mitigation, not a fix: it trades correctness (clients see data past its TTL) for availability, and it hides the broken delegation from the monitoring that should catch it.

Detection. Query every NS in the parent delegation for the child's SOA and require AA=1. Any NS that returns REFUSED, times out, or answers with AA=0 is lame. Run it per-NS, not against the zone as a whole: a single good NS masks the rest.

Curated: · Written: · Reviewed:

Q&A-48A name has an A record and you add a CNAME at the same owner. What must happen?

Core answer (first 60–120 seconds)

The zone is invalid, and a conforming authority must refuse to load it. A CNAME asserts that the owner name is an alias and that all data lives at the target, so any other record type at that owner contradicts it. RFC 1034 §3.6.2 and RFC 2181 §10.1 forbid the combination. The only records allowed to sit beside a CNAME are the DNSSEC ones that sign or prove it: RRSIG, NSEC, NSEC3 (RFC 4035 §2.5, RFC 5155).

So what must happen is: the server rejects the zone (or the API call), and you fix it by giving the CNAME its own owner name. If you need alias behaviour at the apex, where SOA and NS must live, you use a provider's ALIAS/ANAME or CNAME flattening, which synthesizes A/AAAA at answer time and never writes a CNAME into the zone.

Why there is no precedence rule

It is tempting to think the CNAME "wins" and the A is shadowed. Nothing in the protocol says that. The two records are simply an illegal set at one node, and the spec's remedy is rejection, not arbitration. A resolver that receives both has no defined behaviour to fall back on.

What actually happens in practice

  • Strict authoritative servers reject at load. BIND 9.18's zone parser fails with a CNAME and other data error and keeps serving the previous version of the zone (or SERVFAILs if there is none).
  • Managed DNS APIs usually reject the write outright, so the bad state never reaches a zone file. That is the good path.
  • Lenient or older servers may load it anyway. When answering, most of them return the CNAME and drop the A, but that is an implementation choice, not a guarantee.

Worked trace (hypothetical, three authorities)

Zone example.com, owner www.example.com currently holds A 203.0.113.10. Someone adds CNAME www.example.com -> lb.example.net and pushes to three authorities.

AuthorityBehaviourServes for www
ns1 (lenient)loads the pairCNAME -> lb.example.net
ns2 (BIND 9.18)load fails, old zone retainedA 203.0.113.10
ns3 (BIND 9.18)load fails, old zone retainedA 203.0.113.10

A resolver that asks ns1 gets an alias and follows it; one that asks ns2 or ns3 gets the address directly. Same name, two different answers, decided by which authority answered first and then pinned by TTL. That is the real damage: not a clean error, but a partial rollout that looks like it worked from whichever vantage point you tested.

Detection

Query every authority for both types and compare:

for ns in ns1 ns2 ns3; do
  dig +short @$ns A     www.example.com
  dig +short @$ns CNAME www.example.com
done

Exactly one of those two commands should return data on every authority, and it should be the same one everywhere. Run named-checkzone (or the provider's zone validator) in CI before publish so the illegal pair never reaches a server.

Alternatives when you genuinely need an alias

  • Put the CNAME on a fresh owner name (www2, app) that holds nothing else.
  • At the apex, use ALIAS/ANAME or CNAME flattening if your provider offers it, and confirm in writing that it synthesizes records rather than storing a CNAME.
  • If you need both an address and a service at one name, that is two names, not one.

Failure modes to name

  • Split answers across authorities after a partial load: detected by the per-authority query loop above, not by a single dig from your laptop.
  • Silent API acceptance by a provider that stores the pair and serves CNAME-only: detected by querying the authoritative servers directly rather than a recursive resolver, whose cache hides the origin.
  • A zone that fails to load entirely, leaving the previous version live and your change invisible: detected in the server's load log, which is why the validator belongs in CI rather than in a post-deploy check.

Curated: · Written: · Reviewed:

Q&A-49The browser uses DNS over HTTPS. Does that sign the zone?

No. DoH encrypts the query between the browser and whichever resolver it picked; it does not sign anything in the zone. Signing is DNSSEC's job, and it happens at the zone operator and its parent, not on the browser's transport.

State the assumption that matters: "the zone" means the authoritative data for the name, and the DoH resolver is a third party the browser chose (Cloudflare, Google, or an enterprise resolver). If that resolver is one you run and it validates DNSSEC, you inherit DNSSEC's guarantees through it, but that is the resolver's validation, not something DoH did.

Mechanism. DoH (RFC 8484) carries a wire-format DNS message inside an HTTPS request. TLS authenticates the resolver's certificate and encrypts the path. Nothing in the payload is signed by the zone. DNSSEC (RFC 4033–4035) works the other way: the zone operator signs each RRset with an RRSIG, publishes the matching DNSKEY, and the parent publishes a DS that hashes that key. A validating resolver walks DS → DNSKEY → RRSIG up to the root trust anchor and sets the AD bit on the answer.

So a DoH resolver can hand back a forged A record and the browser cannot tell, because the encryption only proves the answer came from the resolver, not from the zone.

Browsers. Current Chrome and Firefox do not run a DNSSEC validator themselves. At most they surface the resolver's AD bit, which is only as trustworthy as the resolver.

Failure modes.

  • Unsigned zone over DoH: looks private, is unauthenticated. A validating resolver returns the answer with AD=0 (insecure delegation) rather than SERVFAIL, so nothing screams.
  • DoH resolver that does not validate: AD=0 on everything, indistinguishable from an unsigned zone.
  • An on-path attacker cannot read the query, but can still be the resolver if it controls the DoH endpoint or the browser's config.

Check it. DS at the parent and RRSIG at the zone, for a signed zone:

$ dig +short DS yourzone.example @a.gtld-servers.net
12345 13 2 <hash>
$ dig +dnssec A yourzone.example @1.1.1.1 | grep -E 'flags:|RRSIG'
;; flags: qr rd ra ad; ...
yourzone.example. 300 IN RRSIG A 13 2 300 (...)

AD set means the resolver validated. AD absent on a signed zone means validation failed or the resolver does not validate.

The same query over DoH, with no DNSSEC check of your own:

$ curl -s -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=yourzone.example&type=A' | jq '{AD, Answer}'
{ "AD": true, "Answer": [ ... ] }

The AD field is the resolver's claim. DoH gave you confidentiality; that boolean is the only authentication signal, and it is the resolver's word.

PropertyDoHDNSSEC
Protectsquery path (stub↔resolver)zone data (RRsets)
Who signsnobody; TLS cert on the resolverzone operator + parent DS
Detects a lying resolvernoyes, if the client validates
Detects an on-path observeryesno: data is signed, not hidden
Browser validates itn/ano; trusts the resolver's AD

If you need to trust the answer rather than the resolver, DNSSEC is the only thing that gives it to you, and the browser will not do it for you.

Curated: · Written: · Reviewed:

Q&A-70You drain a PoP in DNS. TTL is 3600. How long do clients keep steering there?

Core answer (first 60–120 seconds)

Up to one hour. TTL 3600 means a resolver that cached your geo answer keeps steering to that PoP for at most 3600 seconds after it last resolved. If you withdraw the PoP from the geo map at t=0, the traffic you still see is the set of resolvers whose cached answer has not yet expired. The worst case is a resolver that resolved one second before the change: it keeps sending you traffic for another 3599 seconds. The best case is a resolver whose cache expired at t=0: it stops immediately. So the answer is not a single number: it is the distribution of remaining TTLs across the resolvers that send you traffic.

Mechanism

Geo-DNS steering is an A/AAAA answer selected by the authoritative server based on the resolver's IP. That answer is cached by the recursive resolver, not by the end client. The unit of caching is the resolver. A single large resolver (a public resolver, a corporate resolver, an ISP resolver) can hold one cached answer and serve thousands of clients from it. So the drain tail is weighted by query volume per resolver, not by client count.

If resolutions were uniformly spread over the TTL window, the fraction of resolvers still holding the old answer at time t after the change is (3600 − t) / 3600. That gives a linear decay:

Time after geo changeFraction of resolvers still steering (uniform assumption)Resolvers out of 10,000
0 s100%10,000
600 s (10 min)83.3%8,333
1800 s (30 min)50%5,000
3540 s (59 min)1.7%167
3600 s0%0

Real traffic is not uniform. High-traffic resolvers resolve more often, so they expire sooner; the tail is dominated by infrequent resolvers. The observed curve is usually convex: a fast initial drop, then a long thin tail that can run the full hour.

Trade-offs

Lowering TTL ahead of time is the standard fix. Set TTL to 60, wait 3600 seconds for the old TTL to drain, then withdraw. The cost is query load: a 3600→60 change multiplies authoritative query rate for that name by 60. If the name gets 10,000 queries per second at TTL 3600, it gets 600,000 queries per second at TTL 60: that is a real capacity problem for the authoritative tier and for resolvers. Some resolvers also enforce a minimum TTL floor, so a 60-second TTL may not be honored everywhere.

The alternative is to drain at the load balancer while DNS still points at the PoP. Keep the PoP in the geo map, but stop accepting new connections at the LB: return 503 or redirect to another PoP. This is immediate for new connections. It does not help existing long-lived connections: HTTP/2, WebSocket, and gRPC streams stay until they close, so you still need a connection-draining grace period. It also breaks non-HTTP protocols that cannot follow a redirect.

If the PoP is anycast, withdrawing the BGP route is faster than any DNS TTL: traffic reroutes in seconds. But the question is about DNS steering, so that is a different mechanism.

Failure modes

  • Withdrawing at t=0 and assuming the PoP is drained: you keep getting traffic for up to 3600 seconds.
  • Removing the record instead of changing it: resolvers cache NXDOMAIN for the SOA minimum TTL, which can be longer than 3600.
  • Resolver TTL clamping: some resolvers enforce a minimum, so your lowered TTL is not honored.
  • Long-lived connections: DNS expiry does not close TCP/QUIC connections.
  • Client-side pinning: some clients cache IPs or use connection pools that outlive DNS TTL.

Detection

Measure application requests at the drained PoP per minute, plot against time since the geo change, and compare to the expected remaining-TTL curve. Also measure resolver-level query volume to see which resolvers still send traffic. If the curve is flatter than expected, you have a resolver TTL floor or a long-lived connection problem.

Curated: · Written: · Reviewed:

Q&A-95Four anycast DNS PoPs share an address. One is overloaded and still advertised. What do some clients get?

Direct answer: Clients whose BGP best path points at the overloaded PoP get SERVFAIL or query timeouts. Clients steered to the other three PoPs are unaffected. Anycast carries no DNS-level health signal, so the routing system keeps feeding traffic to a PoP that is up but saturated.

Why clients land on the sick PoP. All four PoPs announce the same service prefix, say 203.0.113.0/24. A resolver's query goes to whichever PoP BGP selects for its source. BGP picks on attributes, not load or latency: highest local-preference, then shortest AS-path, then lowest MED, then IGP cost to the next hop. None of those change when a nameserver's CPU pins at 100%, so the PoP keeps winning its share of the internet.

Two things make it worse:

  • More-specifics. If one PoP announces a /25 inside the /24, longest-prefix match wins for every router that sees it, regardless of health. A more-specific is a traffic magnet.
  • Convergence. Even after you withdraw, the withdrawal propagates hop by hop. With route reflectors and multiple transit providers, expect seconds to tens of seconds; BGP route-flap dampening can hold the prefix down longer.

Why withdrawal fixes it. Withdraw the prefix from the routing domain and BGP reconverges to the next-best path: another PoP. That is the only lever anycast gives you, because the health signal is the advertisement.

What withdrawal costs. It is binary and instant. All of that PoP's traffic moves to the remaining three at once. If each was running at 60% of capacity, three PoPs now carry 133% of the original load: cascading overload. Drain by depreferring first (AS-path prepend or lower local-pref) if you can, then withdraw.

Alternatives, and when they are better:

ApproachMechanismWhen to use
Withdraw prefixBGP reconvergesPoP is dead or must leave rotation
AS-path prepend / lower local-prefDeprefer, shift some trafficGraceful drain, PoP degraded not dead
Health-checked delegationPer-PoP NS records, resolver picksUnicast, not anycast: does not apply to a shared address
Unicast fallbackRoute to a known-good PoPSmall footprint, few PoPs

Failure modes and detection. The trap is a health check that tests liveness, not capacity: a TCP connect or a single trivial query passes while the PoP drops real traffic. Detect per-PoP, not globally: SERVFAIL rate per PoP, query rate per PoP (a drop means traffic left), response latency p99, and BGP session state. A global success average hides one bad PoP out of four: 25% of clients failing moves the average by a few points.

Worked example (hypothetical): four PoPs, one saturated and still advertised. 25% of clients are steered to it and get SERVFAIL. The global success average falls from 100% to about 75%, which still looks "mostly fine" on a dashboard. Per-PoP SERVFAIL goes 0% to 100% on the sick PoP.

What I would verify: fail one PoP's nameserver and confirm its prefix is withdrawn from the routing domain within your convergence budget, and that the remaining PoPs hold under the shifted load.

Curated: · Written: · Reviewed:

Q&A-56DF is set, a hop has MTU 1400, and ICMP destination unreachable fragmentation-needed is filtered. What size dies?

Anything with DF set that is larger than 1400 bytes dies at that hop. A 1500-byte DF packet is dropped there, the sender is never told, and it keeps sending 1500-byte packets that keep dying. Packets of 1400 bytes or less pass. The boundary is inclusive: 1400 fits, 1401 does not.

Why

The router receives a 1500-byte packet with DF set, egress MTU 1400. It cannot fragment (DF forbids it) and cannot forward (too big). Correct behaviour is to drop it and return ICMP type 3 code 4, fragmentation needed, carrying the next-hop MTU of 1400, or ICMPv6 type 2, packet too big, on IPv6. The sender caches that and shrinks. Filter that ICMP and the signal never arrives, so the sender's path MTU stays pinned at 1500 and every full-size packet is lost. That is a PMTUD black hole.

Packet sizeDFResult at the 1400 hop
1400setforwarded
1401setdropped, ICMP 3/4 suppressed
1500setdropped, ICMP 3/4 suppressed
1500clearfragmented to 1400, forwarded

Why it looks like a partial outage

Small packets survive, so the symptoms mislead. The TCP handshake completes (SYN and SYN-ACK are tiny), small requests work, traceroute works because its probes are small, and then bulk transfer stalls: the sender retransmits 1500-byte segments, gets nothing back, and the connection hangs until it times out. The application sees a slow or frozen transfer, not an error. Whether DF is set depends on the sender: TCP sets it by default on Linux (IP_PMTUDISC_WANT), and QUIC sets it on every packet, so a QUIC service behind this hop fails outright while a UDP service that leaves DF clear fragments and keeps working.

Confirming it

Probe the boundary directly with DF set. From a host with a 1500-byte interface:

$ ping -M do -s 1472 10.0.0.1     # 1472 + 28 = 1500 bytes on the wire
# ICMP permitted: "Frag needed and DF set (mtu = 1400)"
# ICMP filtered:  no reply at all, 100% packet loss, no message
$ ping -M do -s 1372 10.0.0.1     # 1400 bytes
# replies normally

The silent timeout on the first command with a clean reply on the second is the black hole. Confirm on the router with tcpdump -ni any icmp while the large ping runs: with the filter in place you see the drop and no ICMP 3/4 leaving.

Fixes, in order of preference

  1. Permit ICMP type 3 code 4 (and ICMPv6 type 2) through the filter, at least from routers on the path. This restores PMTUD for every protocol at once.
  2. Clamp MSS on the tunnel or interface to 1360: 1400 minus 20 bytes IP and 20 bytes TCP. This fixes TCP only; QUIC and other DF-setting UDP protocols still break.
  3. Raise the hop MTU to 1500 so the mismatch disappears.

MSS clamping is the usual stopgap because it needs no change to the ICMP policy, but it is a workaround for TCP, not a repair of PMTUD.

Curated: · Written: · Reviewed:

Q&A-57You allow echo and block all other ICMP. Which breakage did you keep?

The breakage you kept is path MTU discovery. Echo still answers, so ping is green, and every connection that only ever sends small packets works. Anything that needs a segment larger than the smallest MTU on the path stalls after the handshake.

Mechanism: a router that receives a packet with DF set and a size larger than the next hop's MTU drops it and returns ICMP type 3 code 4 (fragmentation needed, DF set) naming the next-hop MTU. The sender shrinks its segment and retries. Drop that message and the sender never learns; it retransmits the same too-big segment with exponential backoff until the connection times out. That is the PMTUD black hole (RFC 2923).

What else you kept:

  • ICMP type 11 (time exceeded): traceroute stops at the first hop that would have replied. You lose the only tool that shows you where the path breaks.
  • ICMP type 3 code 3 (port unreachable): UDP-based traceroute and any UDP service that relies on the error to fail fast now hangs instead.
  • On IPv6, everything. NDP is ICMPv6: router solicitation/advertisement (133/134) and neighbor solicitation/advertisement (135/136). Block those and hosts cannot resolve link-layer addresses or learn a default route. ICMPv6 type 2 (packet too big) is the v6 PMTUD message. RFC 4890 is the filtering reference; the short version is that ICMPv6 cannot be filtered the way ICMPv4 can.
ICMPPurposeBlocked →
type 3 code 4PMTUD (v4)large TCP stalls, ping fine
type 3 code 3port unreachableUDP fail-fast lost, traceroute hangs
type 11time exceededtraceroute dead
ICMPv6 133–136NDPIPv6 host cannot resolve neighbors
ICMPv6 type 2PMTUD (v6)same black hole, on v6

Why people block it, and what to do instead. The reason to block ICMP type 3 is that it is forgeable. A spoofed type 3 code 4 makes a sender shrink MSS and can degrade throughput; a spoofed destination-unreachable can tear down an established connection (RFC 5927). So the answer is not "allow all ICMP":

  • Permit type 3 code 4 and type 11, and rate-limit the rest with a token bucket rather than blanket-deny.
  • Permit type 3 code 4 only from the routers on the path, your own hops, and drop it from elsewhere. That kills the spoofing vector without killing PMTUD.
  • MSS clamping as belt-and-braces: ip tcp adjust-mss 1400 on the tunnel interface, or iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu. The sender then never emits a too-big segment, so PMTUD is not needed on that path. It is a workaround, not a substitute: it covers TCP only, and only the paths you configure it on.

Failure mode and detection. The signature is a connection that completes its handshake, moves a few KB, then hangs: SSH login works, scp of a large file stalls. Confirm with ping -M do -s 1472 <host> (1472 + 8 ICMP + 20 IP = 1500). If that fails while ping -s 1400 succeeds, you have a black hole. tracepath prints the MTU per hop. ss -ti shows the retransmit counter climbing. On the router, watch the output-drop and ICMP-unreachable counters.

I would not call it fixed without lab evidence: block type 3 code 4 on one hop, run a large DF transfer, and require it to fail while echo succeeds. Then re-enable and require it to pass.

Ping is not the ICMP you needed.

Curated: · Written: · Reviewed:

Q&A-96One side is 9000 and the other is 1500, and ICMP too-big is filtered. Which frames die?

Frames larger than 1500 bytes with the DF bit set die. Everything at or below 1500 bytes gets through.

A 9000-byte frame arriving at a 1500-MTU hop with DF set can't be forwarded and can't be fragmented, so the router drops it. The only thing that would tell the sender is an ICMP "fragmentation needed" (type 3, code 4), or ICMPv6 "packet too big" (type 2), and that's filtered, so the sender never learns. It retransmits the same 9000-byte segment, the router drops it again, and the connection stalls. That's a PMTUD black hole.

Mechanism, in order:

  1. The sender's TCP advertises MSS 8960 (9000 − 20 IP − 20 TCP) in the SYN.
  2. The peer sends segments up to 8960 bytes with DF set: Linux sets DF on TCP by default (net.ipv4.ip_no_pmtu_disc=0).
  3. The 1500 hop can't forward them, emits ICMP too-big, and the filter eats it.
  4. PMTUD never lowers the path MTU, so the sender retransmits at 8960 until the application times out.

Small frames are untouched: a 64-byte ping, a 1460-byte segment, the handshake itself. All under 1500, all forwarded. That's why "ping works, NFS hangs" is the signature.

IPv6 is worse: routers never fragment, so every IPv6 packet over the path MTU behaves as if DF were set. Filtering ICMPv6 type 2 breaks IPv6 PMTUD unconditionally.

FrameSizeDFAt the 1500 hop
ICMP echo64 B—forwarded
TCP segment1460 Bsetforwarded
TCP segment8960 Bsetdropped, no ICMP back
UDP datagram8000 Bsetdropped
UDP datagram8000 Bclearfragmented (IPv4 only)

Remedies, and when each applies:

  • Permit ICMP type 3 code 4 (and ICMPv6 type 2) through the filter. The correct fix, and the only one that keeps PMTUD working for every protocol. Costs you an ICMP allowance in the policy, and the return path has to be open too.
  • Clamp MSS on the 9000 side. iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu, or Cisco ip tcp adjust-mss 1452. Fixes TCP only: UDP and any other DF-set datagram still die.
  • Lower the 9000 side's MTU to 1500. Blunt but complete; you give up jumbo on that interface entirely.

Detection: ping -M do -s 8972 <host> on Linux (or ping -D -s 8972), tracepath, mtr --mtu. A 1472-byte ping succeeding while an 8972-byte ping fails localizes the hop.

Curated: · Written: · Reviewed:

Q&A-99You clamp MSS on the VPN but not on the internet edge. Which sessions still black-hole on PMTUD failure?

The sessions that still black-hole are the ones whose SYN never crosses the clamp point, plus the ones the clamp can't help. Concretely: every TCP session that egresses the internet edge (no clamp there), any VPN session whose SYN or SYN-ACK doesn't traverse the clamped interface, VPN sessions where the clamp value is still above the real bottleneck MTU, and all non-TCP traffic. MSS clamping is a TCP option rewrite and does nothing for UDP.

Why the clamp is what stops the black hole

PMTUD works like this: a sender with a 1500-byte MTU emits a 1500-byte IP packet with DF set. A router on a 1400-byte link can't forward it, so it drops it and returns ICMP type 3 code 4 ("fragmentation needed"). The sender lowers its segment size and retries. If that ICMP is filtered (very common, firewalls drop ICMP wholesale or rate-limit it) the sender never learns. It retransmits the same 1500-byte packet with exponential backoff until the connection times out. The handshake and small requests pass fine, so the session looks up while large transfers hang. That is the black hole.

MSS clamping sidesteps PMTUD: the rewrite point edits the MSS option in the SYN (and SYN-ACK) as it passes, so both sides advertise a segment size that fits the reduced path and never send an oversized packet. It is a stateless rewrite on an interface, so it only touches packets that actually traverse that interface. Clamp on the tunnel and the internet-edge SYNs go out untouched, still advertising 1460.

Which sessions survive

SessionClamp appliedICMP 3/4 reaches senderResult
Internet TCP, large transfernonoblack hole
Internet TCP, large transfernoyesworks: PMTUD lowers MSS
VPN TCP, clamped both directionsyesn/aworks
VPN TCP, clamp egress-onlyhalfnoblack hole on return-path data
VPN TCP, clamp 1400 but a hop is 1280yes, too highnoblack hole
UDP / QUIC large datagramn/a (TCP-only)noblack hole unless the app does DPLPMTUD

The trace

Host at 1500 MTU, MSS 1460, no clamp on the internet edge, a 1400-byte hop in the path, ICMP filtered:

14:02:11.001 IP 10.0.0.5.51234 > 203.0.113.9.443: Flags [S],  mss 1460
14:02:11.024 IP 203.0.113.9.443 > 10.0.0.5.51234: Flags [S.], mss 1460
14:02:11.050 IP 10.0.0.5.51234 > 203.0.113.9.443: Flags [.], length 0
14:02:11.051 IP 10.0.0.5.51234 > 203.0.113.9.443: Flags [P.], length 1460
14:02:11.301 IP 10.0.0.5.51234 > 203.0.113.9.443: Flags [P.], length 1460  # retransmit
14:02:11.801 IP 10.0.0.5.51234 > 203.0.113.9.443: Flags [P.], length 1460  # retransmit
14:02:12.801 IP 10.0.0.5.51234 > 203.0.113.9.443: Flags [P.], length 1460  # retransmit

Same 1460-byte segment retransmitted with doubling RTO, no ICMP type 3 code 4 anywhere in the capture. That is the signature. Detect it with ss -ti (retrans counters climbing), nstat -az TcpRetransSegs, and tracepath to find where the MTU drops; confirm the path MTU with ping -M do -s 1372 (1400 minus 28 bytes of IP+ICMP headers).

Alternatives and when clamping is wrong

  • Clamp everywhere. Works, but blunt: it hides the real path MTU and caps throughput on paths that are actually 1500.
  • Clamp only at the PMTUD-failing hops. Targeted, but you have to know where they are.
  • Raise the tunnel MTU if the underlay supports 1500: removes the need entirely.
  • TCP MTU probing (net.ipv4.tcp_mtu_probing=1, RFC 4821 PLPMTUD): the sender probes downward when PMTUD fails, so no clamp is needed. QUIC does this natively.
  • Fix the ICMP filter. If the root cause is dropped type 3 code 4, clamping masks it and you'll hit the next broken path.

Clamping is the wrong tool for UDP and for paths where the real problem is ICMP filtering. And clamp both directions: an egress-only clamp leaves the return-path data unclamped.

Curated: · Written: · Reviewed:

Q&A-60Wireshark on the sender shows 64 KB TCP segments. The network MTU is 1500. What was on the wire?

What was on the wire: ordinary 1500-byte Ethernet frames, each carrying a 1460-byte TCP payload. Nothing 64 KB ever crossed the link. The 64 KB "segments" are a capture artifact.

Mechanism

With TSO (and its software twin GSO), the TCP stack builds one large skb (up to the GSO maximum, 64 KB) and hands it to the NIC driver. The NIC splits it into MTU-sized frames, filling in per-frame IP ID, TCP sequence numbers and checksums as it goes. Wireshark on the sending host captures at the AF_PACKET layer, above that split, so it sees the pre-segmentation buffer. The 64 KB is the size of the buffer the stack passed down, not a frame.

Arithmetic

65536 / 1460 ≈ 45. One 64 KB TSO buffer becomes ~45 Ethernet frames on the wire, each 1514 bytes on the wire (14 Ethernet + 20 IP + 20 TCP + 1460 payload).

Capture points

Capture pointWhat you seeWhy
Sender host (AF_PACKET)64 KB pre-segmentation buffersTSO/GSO artifact
TAP / SPAN port1514-byte frames, 1460-byte payloadground truth
Receiver host1514-byte frames, unless GRO/LRO is onGRO/LRO re-coalesce

Getting ground truth: trade-offs

  • TAP or SPAN port. Sees real frames, no change to the sender. Costs a mirror port, and a busy switch SPAN can drop under load.
  • Disable TSO/GSO on the capture host (ethtool -K eth0 tso off gso off). Host capture then matches the wire. Costs CPU, the stack segments per-frame instead of the NIC, and it changes the behavior you are measuring, so it can hide a genuine TSO bug and skew throughput numbers.
  • Capture on the receiver. No sender change, but disable GRO/LRO there (ethtool -K eth0 gro off lro off) or you get the same illusion from the other end.

Failure modes

  • Reading a host capture as wire truth and hunting jumbo frames or an MSS/MTU mismatch that does not exist.
  • GRO/LRO on the receive side coalescing frames into large "segments": same illusion, opposite end.
  • Virtualized paths: veth/tun with GSO. Capture inside the guest shows 64 KB; capture on the host bridge shows 1500.
  • Checksum offload: with TSO the NIC computes checksums, so the host capture may flag "checksum incorrect". Another artifact, not corruption.

Detection

Compare frame lengths for the same flow captured at the host and at a TAP. In Wireshark, look at "Frame length" versus "TCP segment length": if segment length exceeds 1460, you are above the segmentation point. Confirm with ethtool -k for tso/gso/gro/lro.

Curated: · Written: · Reviewed:

Q&A-66The receiver window is 2 segments and cwnd is large. Which control is limiting, and what is not "the network"?

Flow control is the limiter. The receiver's advertised window (rwnd) is 2 segments, and a sender may keep at most min(cwnd, rwnd) bytes in flight. With cwnd large, rwnd wins: in-flight data is capped at 2×MSS. "The network" (the path, its RTT, its loss, its bandwidth) is not the constraint. The receiver is.

Mechanism. rwnd is the free space in the receiver's socket receive buffer, carried in the 16-bit window field of every ACK (scaled by the window-scale option). The sender stops sending when unacked bytes reach rwnd. The throughput ceiling is rwnd / RTT, and it is independent of cwnd. Congestion control only binds when cwnd is the smaller of the two.

Arithmetic. 2 × 1460 = 2920 bytes. At 40 ms RTT: 2920 / 0.040 = 73 kB/s ≈ 0.58 Mbps. That is the ~0.6 Mbps you would measure on a healthy 40 ms path with a 2-segment window. Nothing about the path explains the number; the window does.

Why rwnd is 2 segments.

  • The application is not reading fast enough, so the buffer stays full and the kernel advertises only the small space that frees up between reads.
  • SO_RCVBUF is small, or receive-buffer autotuning is off, so the buffer itself is tiny.
  • Window scaling was not negotiated, capping the window at 64 KB. Not the cause here, but a common one.

How to tell which control binds. Read the window field in the ACKs from a capture, and compare it against the sender's cwnd. If the advertised window is the small one, you are flow-control-limited and path tuning is wasted work. If cwnd is the small one, you are congestion-limited and path or congestion-control tuning is the right move.

LimiterSignalCeilingFix
Flow controlrwnd small, cwnd largerwnd / RTTReceiver buffer or application
Congestion controlcwnd small, rwnd largecwnd / RTTPath, loss, or CC algorithm

Receiver-side fix. Raise the receive buffer and make sure scaling is on:

int sz = 256 * 1024;
setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &sz, sizeof sz);

The kernel doubles the value you pass and caps it at net.core.rmem_max, so set that sysctl first if you need more than the default cap. Then confirm the advertised window actually grew in the next capture.

Application-side fix. If the app blocks on processing between reads, the buffer drains slowly and rwnd stays small no matter how large the buffer is. Fix the consumer (read continuously, read in larger chunks) not the socket.

Failure modes to name. A zero-window state: the receiver advertises 0 and the sender falls back to persist probes every RTO, so throughput drops to near zero while the path looks fine. Silly-window syndrome: the receiver advertises tiny increments and the sender dribbles one small segment at a time. Both appear in a capture as small or zero window fields, not as loss or retransmits.

So: flow control is limiting, the receiver, its buffer or its application, is the bottleneck, and the network is not.

Curated: · Written: · Reviewed:

Q&A-68UDP keepalives are 45 seconds and the NAT mapping idle timer is 30. What happens to the flow?

The flow breaks. The NAT mapping expires 30 seconds after the last packet in either direction. The 45-second keepalive arrives 15 seconds too late, and when it does arrive the NAT builds a fresh mapping: normally on a different external port. Return traffic from the server is still addressed to the old external port, so the NAT has no translation for it and drops it. The keepalive does not resume the flow; it opens a new one the server does not know about.

What actually happens, second by second

Assume the client is behind a NAT, the server is public, and the last media packet crosses at t=0 on external port 40000.

tEventResult
0Last packet refreshes the mappingMapping 10.0.0.5:5004 → 203.0.113.7:40000
0–30Server sends to 203.0.113.7:40000Translated to the client, media flows
30Idle timer fires, mapping removedServer's packets to :40000 now dropped
30–45Server keeps sending to :40000One-way silence, no error to either side
45Client keepalive leavesNew mapping, external port 40001
45+Server still sends to :40000Still dropped; server sees a packet from a new source port

Two things make this a break rather than a pause. First, the mapping is gone for 15 seconds, so any return traffic in that window is dropped silently: UDP has no reset to tell either side. Second, recreation usually picks a new external port. Port allocation is implementation-defined: many NATs hand out the next free port rather than reusing the old one, and you cannot rely on reuse. Even if the port is reused, the server's session state is keyed to the 5-tuple, and a 15-second gap is enough for many media servers to tear the session down.

Why the keepalive interval is the lever you actually have

The NAT idle timer is set by whoever runs the NAT: carrier-grade NAT, a cloud NAT gateway, a firewall. You rarely control it, and the value varies: 30 s is common for UDP, but 60 s and 120 s exist. RFC 4787 recommends a UDP mapping timeout of at least 2 minutes, but that is a recommendation and plenty of deployments ignore it. The keepalive interval is set by your client, so that is the side to change.

Set the keepalive comfortably below the shortest timer you expect to meet: 15–25 s against a 30 s timer gives margin for jitter and for the timer being shorter than documented. The cost is real: on mobile, each keepalive wakes the radio, so a 15 s interval is three times the wakeups and battery drain of 45 s, plus more packets and more server load. That is the trade-off: shorter keepalives buy reliability at the cost of battery and bandwidth, and the right value is the longest interval that still stays under the timer.

Detecting it

Symptom is one-way audio or a dropped call roughly 30 seconds into silence, with no error on either side. Confirm by capturing at both ends and comparing the external port before and after the gap: if it changed, the mapping was recreated. A direct test is to silence a UDP flow for longer than the timer and check that the mapping is gone before the next keepalive.

Curated: · Written: · Reviewed:

Q&A-58You see SYNs out and no SYN-ACK. How do you decide whether the return path or the listener is at fault?

SYNs leaving the client with no SYN-ACK returning means exactly one of three things: the SYN never reached the listener, the listener got it and didn't answer, or the listener answered and the SYN-ACK died on the return path. A capture on the client alone proves only that the SYN left and nothing came back: it cannot separate those three. So the first move is a second observation point, not a guess.

Which second point you pick is a cost trade-off. Packet capture at both ends is authoritative: it shows where the SYN is last seen and whether a SYN-ACK was generated. It costs host access, synchronized clocks, and capture buffers sized for the retry rate. Conntrack or a state table on the stateful midpoint (firewall, load balancer) is cheaper, no host access, and shows whether the device tracked the flow and what state it holds, but it shows the device's decision, not what actually crossed the wire, and NAT rewrites hide the real endpoints. Socket statistics on the server (ss -lnt, ss -s, /proc/net/tcp) are cheapest of all and show listener state, SYN backlog and accept-queue depth, but say nothing about the return path.

ApproachProvesBlind spot
tcpdump on client + serverwhere SYN last seen; whether SYN-ACK generatedneeds host access, clock sync, buffer sizing
conntrack / state table on midpointwhether flow tracked, its statedevice view, not wire truth; NAT hides endpoints
ss / netstat on serverlistener, SYN backlog, accept queuenothing about return path

Decision procedure: capture simultaneously at client and server.

  • SYN present at server, no SYN-ACK generated → listener side. Check ss -lnt for the socket, then the SYN backlog (net.ipv4.tcp_max_syn_backlog) and accept queue (net.core.somaxconn). A full accept queue with tcp_abort_on_overflow=0 silently drops the final ACK; a full SYN backlog drops SYNs before the listener ever sees them.
  • SYN-ACK generated at server, absent at client → return path. Check conntrack on the midpoint for the flow state, the return ACL, and asymmetric routing where the reply leaves by a different interface than the SYN arrived.
  • SYN never arrives at server → forward path: routing, an inbound ACL, or a midpoint dropping it.

Worked example (hypothetical figures): client capture shows 8,400 SYNs, 0 SYN-ACKs. Server capture over the same window shows 8,400 SYNs and 8,400 SYN-ACKs. Conntrack on the midpoint shows the flows still in NEW state with no reply seen. That pins it to the return path, a return ACL dropping SYN-ACKs, not the application. Without the server capture you would have spent the incident on the listener.

Failure modes to name: conntrack table full drops new flows silently (nf_conntrack: table full, dropping packet in dmesg); SYN cookies engaging under backlog pressure change the sequence numbers and can look like a broken handshake; asymmetric routing makes the return path invisible to a capture on the forward path.

I don't call it settled until I can show the SYN-ACK leaving the server and dying on the return path, or the SYN never arriving. Embryonic connections on one side are not evidence of a closed listener.

Curated: · Written: · Reviewed:

Q&A-59The client receives a RST. Why might the server log show no such RST?

Short answer: the server's log is an application log, and a RST is a kernel event. The app only ever learns about a RST if it is blocked in a syscall on that socket, and then only as ECONNRESET. Everything else is invisible to it. There are three families of explanation, and they split on whether a RST ever left the server host at all.

1. The server never sent it. A middlebox on the path (NAT gateway, stateful firewall, load balancer, or an off-path injector) generated the RST. The server's kernel has no record because no packet left it. Common triggers: conntrack idle timeout (AWS NAT Gateway drops idle TCP at 350 s), a firewall rule change, or a load balancer resetting both sides after a health-check failure.

2. The server's kernel sent it and the application never saw it. Linux emits RST without involving the process in several cases:

  • A segment arrives for a 4-tuple with no socket: after close(), after TIME_WAIT expiry, after the process exited. tcp_v4_send_reset fires; nothing is logged.
  • A SYN arrives on a port with no listener.
  • close() with SO_LINGER set to {1, 0}: abortive close, RST instead of FIN.
  • Data arrives on a socket the app has already shut down for reading; Linux resets it (TcpExtTCPAbortOnData).

If the app already closed the fd, there is nothing left to log. The RST is a consequence of the close, not a separate event.

3. The application saw it and didn't log it. read() returned ECONNRESET, the handler logged "connection closed" at INFO, or the line is at DEBUG and DEBUG is off, or the process restarted before flushing. Also: you may be reading the wrong host. Behind a load balancer or in a pool, the RST came from a different backend than the one whose log you opened.

How to tell them apart. The kernel keeps counters for exactly this. On Linux:

nstat -az | grep -Ei 'rst|abort'
# TcpOutRsts             segments sent with RST
# TcpExtTCPAbortOnClose  early user close
# TcpExtTCPAbortOnData   unexpected data on a shut-down socket
# TcpExtTCPAbortOnLinger SO_LINGER abort
# TcpExtTCPAbortOnMemory out of memory
# TcpExtTCPAbortOnTimeout

If TcpOutRsts is flat across the incident window, the server host did not send the RST: look at the middlebox. If it increments, the kernel sent it; match the timing against ss -tan state and the app's close() log. Note the kernel does not log RSTs by default either; you need tcpdump or an iptables LOG/NFLOG rule to see them.

Capture on both sides:

tcpdump -i any -nn 'tcp[tcpflags] & tcp-rst != 0'

A RST present at the client and absent at the server's NIC was injected on the path. A RST present at both, with a sequence number valid for the connection, came from the server's kernel.

Client seesServer app logLikely sourceConfirm with
RST after idlenothingNAT/firewall idle timeoutconntrack -L, NAT timeout config
RST mid-transfer"connection closed"kernel abort (no socket, SO_LINGER 0)TcpOutRsts, TcpExtTCPAbortOnClose
RST, app logged ECONNRESET"read failed"server kernel, app saw itapp log level, ss state
RST, no server-side capturenothingmiddlebox / off-pathtcpdump at both ends

What to do: don't page the app team on a RST alone. Check TcpOutRsts on the server host first: it is a one-command answer to "did we send it". Then capture at the server NIC and at the client. Only if both show the RST with a valid sequence number is it the server's kernel, and only then does the app's close/read path matter.

Curated: · Written: · Reviewed:

Q&A-62You see one TCP retransmission. Which hop dropped the packet?

Core answer: you can't name the hop from one retransmission. A retransmission is a sender-side event. A segment wasn't ACKed within the retransmission timer (RTO), or enough duplicate ACKs arrived to trigger fast retransmit. It tells you the ACK didn't come back in time, not where the bytes went. The drop could be on the forward path (the data segment), the reverse path (the ACK), or there may be no drop at all: a delayed ACK, a stalled receiver, or a reordered segment. To localize, you need a second observation point and a comparison.

Assumptions I'd state up front: one capture or counter at one vantage point (say the sender), one flow, timestamps I trust. Under those, the honest answer is "unknown hop."

What the retransmission type does tell you. The two triggers behave differently:

  • Fast retransmit (three duplicate ACKs, RFC 5681): the receiver got later segments but not this one, so the gap is on the forward path between sender and receiver. The dup ACKs are the receiver telling you what it's missing.
  • RTO expiry with no duplicate ACKs: the sender heard nothing. That's consistent with a forward-path loss, a reverse-path loss (the ACK died), or a delayed ACK / slow receiver. The sender cannot separate these.

Worked trace. Two flows, same single retransmission, different locations.

Scenario A: forward-path loss:

Sender → Receiver   seq 1000 (500 B)   [dropped in transit]
Sender → Receiver   seq 1500
Sender → Receiver   seq 2000
Receiver → Sender   ACK 1000   (dup #1)
Receiver → Sender   ACK 1000   (dup #2)
Receiver → Sender   ACK 1000   (dup #3)
Sender → Receiver   seq 1000   (fast retransmit)

The receiver's dup ACKs prove seq 1000 never arrived, so the loss is on the forward path between the two capture points. With taps at hops 1..N, the drop is between the last tap that saw seq 1000 and the first that didn't.

Scenario B: reverse-path loss:

Sender → Receiver   seq 1000
Receiver → Sender   ACK 1000   [dropped in transit]
   ... RTO expires (hypothetical 200 ms) ...
Sender → Receiver   seq 1000   (RTO retransmit)
Receiver → Sender   ACK 1000

Here the sender sees no dup ACKs: just a timer firing. A sender-side capture cannot tell this from "the receiver was slow to ACK." Only a receiver-side capture, or a tap near the receiver, shows the ACK leaving and not arriving.

How to localize in practice: bisection. Put a capture or a counter at each candidate hop, or as many as you can. Find the last point where the segment or ACK appears and the first where it doesn't; the drop is between them. If you can only instrument one end, you can bound the loss to "forward or reverse" but not to a hop: say so rather than guessing.

Failure modes and gotchas.

  • One retransmission is normal. TCP is built to recover from it; a single event on a long flow is not a signal. Look at the retransmission rate over the flow before chasing a hop: a rate near zero is healthy.
  • Reordering looks like loss. A segment that arrives late triggers dup ACKs and a spurious fast retransmit; the "missing" segment was never dropped. Check whether the retransmitted segment arrives twice at the receiver.
  • Delayed ACK. The receiver may hold an ACK waiting for data to piggyback; RFC 1122 allows up to 500 ms. A short RTO can fire on an ACK that was never lost.

A retransmit is a symptom at the sender, not a coordinate on the path. Name a hop only when a second observation point agrees.

Curated: · Written: · Reviewed:

Q&A-61UDP traceroute dies at hop 6 while the TCP application works. What did traceroute actually test?

What traceroute actually tested

Nothing about your TCP application. Classic traceroute sent UDP datagrams to destination port 33434 (incremented per probe) with TTL 1, 2, 3… and waited for ICMP. It tested exactly two things: whether each hop decremented TTL to zero and emitted ICMP Time Exceeded (type 11, code 0) back to your source address, and whether the destination emitted ICMP Port Unreachable (type 3, code 3) because nothing listened on 33434. TCP/443 was never exercised: different protocol, different port, different five-tuple, and on a stateful firewall a different policy.

"Dies at hop 6" is one of three things, and the output alone cannot distinguish them:

  1. The UDP probe was dropped at hop 6 by an ACL, firewall, or uRPF check. No TTL decrement, no ICMP: silence.
  2. The probe arrived, TTL hit zero, the router generated Time Exceeded, and that ICMP was filtered or rate-limited on the return path.
  3. The ICMP was generated but left an interface with no route back to you (asymmetric routing), so it never arrived.

Cases 1 and 2 are byte-identical in the output. That ambiguity is the whole problem.

Read the hops after 6

If hop 7, 8, 9 answer, the probe passed hop 6: hop 6 forwarded it and suppressed its own ICMP. That is a reply-policy problem, not a forwarding hole. If everything from hop 6 onward is silent, the probe itself stopped there. This one observation splits the diagnosis in half before you touch anything.

Rate limiting looks like a filter

A Linux router rate-limits ICMP generation: icmp_ratelimit defaults to 1000 ms and icmp_ratemask to 6168, which has bit 11 set. Time Exceeded is capped at roughly one per second toward a given destination. A hop that answers probe 1 but not probes 2 and 3 prints 6 a * *. Intermittent asterisks mean rate limiting; a clean stop at one hop with answers beyond it means reply filtering.

ECMP can put hop 6 off the app's path

UDP 33434 and TCP 443 hash differently across equal-cost links. Hop 6 may not be a device your application traffic ever crosses. paris-traceroute (flow-id based) or several runs with -q 5 will show whether the path is stable.

What to run instead

traceroute -T -p 443 <dst> (Linux traceroute, TCP SYN probes), mtr --tcp --port 443 <dst>, or tcptraceroute <dst> 443. These carry the application's five-tuple, so they hit the same policy and the same ECMP bucket. Confirm with a capture on both ends: a SYN leaving and a SYN-ACK returning proves the path; a SYN leaving with nothing back localises the drop.

Worked exchange, source 10.0.0.5 → 203.0.113.10:

TTL 1  -> 10.0.0.1      ICMP 11/0 from 10.0.0.1
TTL 2  -> 10.0.0.2      ICMP 11/0 from 10.0.0.2
...
TTL 5  -> 198.51.100.1  ICMP 11/0 from 198.51.100.1
TTL 6  -> 198.51.100.2  (nothing)     <- probe dropped, or reply dropped
TTL 7  -> 203.0.113.1   ICMP 11/0 from 203.0.113.1  <- probe passed hop 6

The TTL 7 line is the evidence: hop 6 forwarded the packet and chose not to answer. No routing change is warranted.

ProbeTestsDoes not test
UDP 33434+TTL decrement and ICMP Time Exceeded at each hop; Port Unreachable at the destinationTCP/443, stateful policy, TCP return path
TCP SYN to 443Same TTL/ICMP machinery carrying the app's five-tupleTLS handshake, HTTP response
ICMP echoWhether the hop answers echoAnything about the app path

Curated: · Written: · Reviewed:

Q&A-65Forward traceroute succeeds. The application still fails. What have you not mapped?

Forward traceroute proves one thing: probes from your host reached the destination and something answered. It does not map the return path, the stateful midpoint's policy for your application's five-tuple, or the MTU along the path. Those are the three things that break an app while forward traceroute stays clean.

Why traceroute can be clean and the app dead

Traceroute probes are ICMP or UDP/33434+ with small payloads. Your app is TCP/8443 with full-size segments. A return ACL that matches tcp/8443, a NAT with no session for the flow, or a PMTUD blackhole on a 1500-byte segment are all invisible to the probe.

The four candidates, and how to tell them apart

CandidateSignatureDiscriminator
Return-path drop (ACL/firewall/NAT)SYN arrives, SYN-ACK never reaches clientcapture both ends
PMTUD blackhole / MTUhandshake completes, stalls on first full-size segmentping -M do -s 1472
Asymmetric routing with deliveryresponse arrives, app still failsclient capture shows response
Application/TLS layerTCP completes, then error or hangcurl -v shows handshake then failure

Worked trace: return-path drop

Client 10.20.1.50, server 10.60.2.10, TCP/8443. Forward traceroute from client: clean to hop 12.

Client capture:

12:00:01.100 IP 10.20.1.50.51234 > 10.60.2.10.8443: Flags [S], seq 1000, length 0
12:00:02.100 IP 10.20.1.50.51234 > 10.60.2.10.8443: Flags [S], seq 1000, length 0
12:00:04.100 IP 10.20.1.50.51234 > 10.60.2.10.8443: Flags [S], seq 1000, length 0

Server capture:

12:00:01.101 IP 10.60.2.10.8443 > 10.20.1.50.51234: Flags [S.], seq 5000, ack 1001, length 0
12:00:02.101 IP 10.60.2.10.8443 > 10.20.1.50.51234: Flags [S.], seq 5000, ack 1001, length 0

SYN-ACK leaves the server, never arrives. The return filter matches tcp/8443; the traceroute's ICMP return traffic doesn't match, so the probe path looks healthy. That's the gap.

When the return path is not the cause

If the client capture shows the SYN-ACK arriving and the handshake completing, stop looking at the return path. Then it's MTU (handshake fine, first 1460-byte segment retransmitted forever: check with ping -M do -s 1472), or the application itself (TLS handshake completes, then a 500 or a hang: curl -v and server logs).

What to do

Capture at both ends simultaneously, keyed on the five-tuple. If the response leaves one end and doesn't arrive at the other, name the hop where it dies: traceroute from the server back to the client, or the firewall's session table showing no entry or the wrong ingress interface. If it arrives, the return path is exonerated and you move to MTU or the app.

Curated: · Written: · Reviewed:

Q&A-67ICMP port unreachable comes back and the UDP application does not retry. Whose job is that?

The application's job. UDP has no retransmission and no connection state, so nothing below the application will resend. The kernel's only obligation is to surface the ICMP error to a socket that asked for it; whether anything acts on it is the application's decision.

Mechanism. When the destination host has no socket bound to the target port, its IP stack replies with ICMP type 3 code 3 (port unreachable), quoting the original UDP header. The sender's kernel matches that quote back to the socket that sent the datagram, source and destination ports plus the local address, and then behaviour depends on socket state:

  • Connected UDP socket (connect() called): the error is stored on the socket and returned to the application on the next send() or recv() as -1 with errno ECONNREFUSED. One error, consumed once.
  • Unconnected socket: by default the error is discarded. On Linux you must set IP_RECVERR, then read it with recvmsg(..., MSG_ERRQUEUE), which returns the queued error in msg_control as a sock_extended_err. Without that, the application never learns.

So "the stack may surface it" is precise: it surfaces it only to a connected socket, or to an unconnected one that opted in. ICMP errors are also rate-limited (net.ipv4.icmp_ratelimit, default 1000 ms) and masked by icmp_ratemask, so a burst of failures can be coalesced or dropped: absence of an error is not proof of delivery.

What the application does with it. Three credible choices:

  1. Retry in the application. Correct when the payload is idempotent or carries its own request ID: DNS resolvers, NFS/RPC, and most RPC frameworks do exactly this. Cost: you own the timeout, backoff, and duplicate suppression.
  2. Move to a transport that retries. TCP gives ordered, retransmitted delivery but adds a handshake RTT, head-of-line blocking, and per-peer connection state. QUIC gives retransmission with TLS 1.3 built in and no HOL blocking across streams, at the cost of a userspace library and more CPU. Right when you need reliability and can pay for it.
  3. Accept the loss. Right for syslog, statsd-style metrics, and sampled telemetry, where a dropped datagram is cheaper than the retry machinery. Then the ICMP error is a signal to your monitoring, not to the sender.

The wrong move is to try to make the network retransmit. Routers forward or drop; they do not retransmit. "Fixing UDP" in the network means adding a proxy or a TCP tunnel, which is a different design decision, not a fix.

Failure mode to name. A collector restarts and nothing is bound to port 514. The sender's connected socket gets ECONNREFUSED on the next send; if the code ignores the return value, the operator sees silence and hunts packet loss. Detection: tcpdump on the sender showing ICMP type 3 code 3 quoting the datagram, plus the application's own send counter not incrementing retries. On Linux, ss -u -a shows the socket and strace shows the send returning -1 ECONNREFUSED.

How I'd settle it in an interview. State that it is the application's job, name the socket state that decides whether the error is even visible, then pick retry / TCP / QUIC / accept-loss based on idempotency and latency budget.

Curated: · Written: · Reviewed:

Q&A-69A TCP connection starts at one anycast PoP and a later packet arrives at another. What state does the second PoP have?

None. The second PoP has no connection state: no TCB, no socket, no sequence numbers, no receive window, no retransmit timers, no congestion window. Anycast replicates an address across PoPs; it does not replicate the TCP control block, which lives in one kernel's memory. The second PoP is a router with a local TCP stack that has never seen this flow.

What it does with the packet is mechanical. The kernel hashes the 4-tuple (src IP, src port, dst IP, dst port) into its established-connection table, misses, then falls back to the listening-socket lookup. If nothing is bound to that VIP:port, or something is bound but has no matching connection, RFC 793 §3.9 applies: a segment matching no connection and carrying ACK gets a RST back. The reset takes its sequence number from the incoming segment's ACK field, so it lands inside the client's expected receive window and is accepted rather than ignored. The client's stack aborts the connection and the application gets ECONNRESET.

Trace: client 203.0.113.7:40000 to VIP 198.51.100.1:443, established through PoP A, server's next expected byte 5001:

  1. A route change moves the flow's next hop to PoP B.
  2. Client sends seq=1001, len=200, ACK=1, ack=5001.
  3. PoP B: connection lookup misses; no established socket for that 4-tuple.
  4. PoP B replies RST, ACK, seq=5001 (copied from the incoming ACK field), ack=1201 (1001+200).
  5. Client accepts the RST, seq 5001 is exactly what it expected next, and tears down.

The client cannot tell this apart from the server crashing. Same VIP, same port, valid-looking reset.

Options, and what decides between them:

ApproachKeeps flow alive?CostUse when
Pin flows to a PoP (ECMP hash on the 5-tuple)Yes, while the path set is stableAny change to the ECMP set (link flap, PoP drain, capacity change) rehashes and moves a slice of flowsYou control the fabric and long-lived flows are a small share of traffic
Connection-aware front / shared connection tableYes, across PoP movesExtra hop, and the front tier itself must be consistentFlows are long and a reset is expensive (streaming, DB sessions, gRPC streams)
Accept resetsNoOne retry per reset flowFlows are short relative to churn rate: HTTP request/response over a few RTTs
Unicast VIP + DNS steering (GeoDNS, latency-based)Yes, client sticks to one IPFailover waits on DNS TTL and resolver cachingYou need correctness more than instant failover

The deciding question is flow lifetime versus churn rate. If a flow lives 30 seconds and your fabric changes hourly, resets are noise. If it lives 30 minutes, one routing change resets a meaningful fraction of them at once.

Failure modes worth naming:

  • Silent blackhole. If a firewall drops the stray segment instead of the kernel resetting, the client retransmits until RTO exhaustion: Linux tcp_retries2=15 gives roughly 15 minutes before ETIMEDOUT. Detect with retransmit counters and no RST on the wire.
  • Reset storm. A routing change resets many flows at once; every client retries simultaneously. Watch for a synchronized RST spike plus a connection-rate spike.
  • Asymmetric listener. If the app is deployed at every PoP, the second PoP has a listener but no TCB, so it still resets, and it looks like a server-side crash in your logs.

Curated: · Written: · Reviewed:

Q&A-63You connect by IP to a name-based HTTPS virtual host. Which certificate do you get?

You get the listener's default certificate: the one the terminator falls back to when the ClientHello carries no server name. In nginx that is the default_server block on :443, or the first listen 443 ssl block if none is marked. In Apache it is the first <VirtualHost *:443> listed for the address. On an ALB or a CDN it is the certificate configured as the listener's default. It is normally the certificate of some other name, or an operator placeholder, so the handshake reaches the Certificate message and then the client fails name verification. The one case where dialling the IP works: the fallback certificate carries an iPAddress SAN covering that IP.

Selection happens during the handshake, before any HTTP request exists, so a Host: header or a path cannot influence it. SNI is the only in-band way to name a vhost, and RFC 6066 defines its host_name field as a DNS hostname, so an IP literal has nothing to send. TLS 1.3 encrypts the Certificate message but not the SNI that chose it.

What each client sends

InvocationSNICert servedName check
curl https://203.0.113.10/nonelistener defaultfails: no iPAddress SAN for 203.0.113.10
curl --resolve api.example.com:443:203.0.113.10 https://api.example.com/api.example.comapi.example.com certpasses
curl -k https://203.0.113.10/nonelistener defaultskipped, so a green probe against the wrong vhost
openssl s_client -connect 203.0.113.10:443nonelistener defaultonly if you add -verify_hostname
openssl s_client -connect 203.0.113.10:443 -servername api.example.com -verify_hostname api.example.comapi.example.comapi.example.com certchecked

Pass -servername explicitly rather than trusting the client to infer one. On the IP rows there is nothing to infer from.

Handshake trace for curl https://203.0.113.10/

  1. TCP connect to 203.0.113.10:443.
  2. ClientHello with no server_name extension. The reference identity curl will verify against is the IP literal.
  3. The terminator has no SNI key to look up, so it binds the socket to the default SSL context and sends that certificate. Nothing sent later changes the choice.
  4. Chain validation, then identity match: RFC 6125 requires an IP to appear as an iPAddress SAN. A dNSName entry never matches an IP, and browsers ignore the legacy CN once a SAN is present, so CN=api.example.com does not save it.
  5. curl aborts reporting that no alternative subject name matches the target IP address; Chrome shows ERR_CERT_COMMON_NAME_INVALID.

Where the default comes from

server {
    listen 443 ssl default_server;      # no SNI -> this context
    server_name _;
    ssl_certificate     /etc/nginx/fallback.crt;
    ssl_certificate_key /etc/nginx/fallback.key;
}
server {
    listen 443 ssl;
    server_name api.example.com;        # reachable only with SNI = api.example.com
    ssl_certificate     /etc/nginx/api.example.com.crt;
    ssl_certificate_key /etc/nginx/api.example.com.key;
}

Without default_server, the first block on that address:port becomes the default, which is how a vhost ends up handing its certificate to IP traffic by accident. Apache behaves the same way: first-listed *:443 vhost serves the no-SNI case.

Failure modes

  • Health check dialling the IP with verification off. It gets a 200 from the wrong vhost, so the probe is green while real clients fail. Detect it by asserting the presented SAN in the check, or by pointing the probe at the name through --resolve or a hosts entry rather than the IP.
  • Fallback certificate disclosure. The default is often a self-signed placeholder whose SAN lists internal hostnames, and it is handed to anyone who connects by IP. Connect from outside and read it to confirm.
  • No TLS error at all. With -k or a permissive SDK the handshake succeeds and the request lands on the default vhost, answering with another site's page or a 404. Compare body and headers against the named vhost, not status codes.
  • Genuinely IP-only environments. No DNS and a pinned IP means the certificate has to carry the IP in an iPAddress SAN, issued that way by the CA. You cannot fix it at the terminator, and a Host: override does not change certificate selection.

Run the same s_client twice, once with -servername and once without, and diff the SAN lines. If they differ, the selection is SNI-driven, and any probe in your pipeline that dials the IP has been testing the fallback all along.

Curated: · Written: · Reviewed:

Q&A-89The classifier matches DSCP EF internally and the WAN still shows CS0. What was missing?

Classification and marking are separate steps, and this symptom, class-map hits climbing while the WAN capture shows CS0, means the packet was classified and then left with its original DSCP. The usual cause is a policy-map class with no set action, or a policy applied in a direction the packet never traverses.

class-map match-any VOICE
 match dscp ef
policy-map WAN-OUT
 class VOICE
  set dscp ef          ! without this line the class matches and does nothing
interface GigabitEthernet0/0/1
 service-policy output WAN-OUT

Checked against IOS-XE 17.x. show policy-map interface GigabitEthernet0/0/1 output showing matches in class VOICE with no marking is the tell; a capture on the WAN interface is the proof.

Where to mark: trust boundary vs explicit marking

ApproachWhat it doesRight whenCost
Trust DSCP at accessaccess switch/router preserves the endpoint's DSCPyou own the endpoints and they mark correctly (hard phones, managed softphones)any host can spoof EF; one access port with trust off resets the flow to CS0
Classify + mark at WAN edge ingressrouter matches ACL/NBAR and sets DSCPyou don't control endpoints, or they don't markcan't reliably see inside TLS/QUIC; RTP ports are negotiated, so port ACLs drift; each new app needs a rule
Mark at WAN egress onlysets DSCP on the way outLAN is over-provisioned and only the WAN needs the markLAN hops can't queue on it, and any intermediate re-mark wipes it

Ingress marking is the stronger choice when you can do it: the mark exists for every downstream hop, so you can shape and queue per class end to end. Egress marking at the WAN edge is the last chance, and it is exactly where a missing set shows up as CS0 on the wire.

Failure modes and how to catch each

  • Class matched, no set action. Hits increment, DSCP unchanged. Detect: show policy-map interface per direction, then capture on egress.
  • Policy on the wrong interface or direction. Counters stay at zero for the class. Detect: same command, per interface and direction.
  • Trust boundary reset mid-path: an access port with QoS trust off, or a set dscp default in class-default on an intermediate router. Detect: capture at access, distribution and WAN edge and compare.
  • Provider rewrite. Carriers commonly zero or re-map DSCP unless the contract specifies per-class treatment. Detect: capture on both sides of the provider cloud.
  • Tunnels. The mark must be set before encapsulation; the outer header copies the inner DSCP. Detect: decode both headers in the capture.
  • L2 WAN (Metro Ethernet). The wire mark is CoS in the 802.1Q tag, not DSCP. Detect: decode the tag rather than reading the IP header.

Verification

Capture on the WAN for a known EF flow and require EF in the IP header (or CoS 5 in the tag). A class-map hit counter proves classification happened; it says nothing about what left the box.

Curated: · Written: · Reviewed:

Q&A-90NetFlow uses random 1:1024 packet sampling. A 3-second 400-packet burst may not appear. What can you still claim?

With random 1:1024 packet sampling, a 3-second burst of 400 packets has about a 32% chance of producing at least one sampled packet. So if NetFlow shows no record for that interval, the only defensible claim is: no packet from that burst was selected by the sampler. You cannot claim the burst did not occur, and you cannot claim the flow was zero. The absence is the expected outcome roughly two times out of three.

Each packet is included independently with p = 1/1024. For 400 packets, expected sampled packets = 400/1024 = 0.39. P(0 sampled) = (1 - 1/1024)^400 ≈ e^{-0.39} ≈ 0.68. P(at least one) ≈ 0.32. To reach 95% detection probability, you need N such that (1 - 1/1024)^N ≤ 0.05, which gives N ≈ 3066 packets. In a 3-second window that is about 1022 pps. A 400-packet burst is well below that.

Burst size (packets)Expected sampledP(0 records)P(≥1 record)
4000.3968%32%
10000.9838%62%
30662.995%95%

What you can still claim:

  • The sampling design's detection limit: a burst of this size is likely to be missed, and the negative result is uninformative for that 3-second interval. It does not contradict the burst.
  • If a record exists, at least one packet was sampled. A single sampled packet implies an estimated 1024 packets, but the 95% Poisson interval is roughly 26 to 5,700 packets, so it is weak evidence of volume.
  • Over longer windows, aggregate rates are still valid estimates with error bars that shrink as the sampled packet count grows. Top-talker rankings are noisier than the aggregate totals.
  • The sampling rate itself, if configured correctly, tells you the probability of missing a burst of a given size. That is a claim about the measurement system, not about the traffic.

What you cannot claim: no traffic, zero packets, or a closed incident. A zero-record NetFlow interval is not an audit log. If the sampler is actually deterministic (hash-based flow sampling), the same short flow can be missed every time, so absence is even less informative.

To prove presence or absence for the 3-second window, use unsampled interface counters (SNMP IF-MIB ifHCInOctets/ifHCOutOctets), packet capture, or NetFlow with sampling disabled. NetFlow is for aggregate traffic characterization, not per-burst forensics.

Curated: · Written: · Reviewed:

Q&A-97You SPAN 10 Gbps of sources to a 1 Gbps analyser port. How much of the evidence is missing?

Core answer (first 90 seconds)

Assume the sources really do offer 10 Gbps aggregate into the session and the destination is one 1 Gbps port. Then the destination carries 1 Gbps, so at best 10% of the copied frames arrive and at least 90% are gone. In practice it is worse than 90%, and the missing 90% is not a random sample: it is biased toward the bursts, which is usually the evidence you came for.

Why it happens

SPAN copies are made at the source and queued on the destination port's egress. That queue drains at 1 Gbps. Offered at 10 Gbps it fills in under a millisecond: with ~1 MB of shared buffer, 1 MB × 8 / 9 Gbps ≈ 0.9 ms, and from then on the ASIC tail-drops copies. Production forwarding is untouched; the copy is what gets discarded. On some platforms the copy is dropped at the source instead, once the destination is congested, which is the same outcome from the analyser's side. The switch keeps forwarding 10 Gbps while the analyser sees a 1 Gbps tail-dropped stream.

Arithmetic: 10 Gbps of 1500-byte frames is roughly 830k pps; 1 Gbps is roughly 83k pps. Over a 60-second capture that is ~50M frames offered, ~5M captured, ~45M dropped.

QuantityValue
Offered to SPAN10 Gbps ≈ 830k pps
Destination line rate1 Gbps ≈ 83k pps
Captured~10%
Dropped~90% (≈45M frames/min)
Queue fill time at 1 MB buffer≈0.9 ms

The bias matters more than the ratio. Tail drop hits the peaks first. Ten flows at 1 Gbps each lose roughly evenly; one 5 Gbps flow beside nine at 0.55 Gbps loses disproportionately on the big flow. Either way, microbursts, the thing you are usually hunting, are the first casualty.

Choosing a remedy

  • Size the destination: aggregate monitored traffic must sit under the destination rate with burst headroom. Right when you can get a 10G/40G destination and the aggregate is known. Wrong when the aggregate is the unknown you are trying to measure.
  • TAP: lossless copy at line rate, no switch involvement. Right when you need every frame and can afford the hardware and a maintenance window (copper TAPs break the link; optical TAPs do not). It does not aggregate, so you still need a destination that can absorb the sum.
  • Filter the session: capture only the VLAN, ACL or host you care about. Right when you already know the question. Wrong for discovery: you cannot see what you filtered out.

SPAN is still the right tool when monitored traffic sits comfortably below the destination rate, when you need a quick non-intrusive look, or when the switch is the only copy point (stacked or virtual switches, control-plane traffic).

Detecting it

Compare production interface counters against analyser received frames over the same window: 10 Gbps out versus 1 Gbps in is the 90%. On the switch, the destination port's output-discard counter, and a dedicated SPAN/mirror drop counter where the platform exposes one, climbing while production counters stay clean is the signature. Check the analyser too: NIC ring overruns and capture-tool drops undercount further, so the true figure is worse than the switch reports. ERSPAN adds encapsulation, so a 1 Gbps destination carries less than 1 Gbps of original traffic.

Curated: · Written: · Reviewed:

Q&A-91Syslog shows 0 ACL denies and a TAP shows 12,000 hits. Which one forwards?

The TAP is the forwarding record. Syslog is a report about what the device chose to log. If the TAP shows 12,000 packets hitting the deny rule, those 12,000 were dropped, whatever the log buffer says. But that conclusion only holds once you have confirmed the TAP is looking at the same traffic the ACL matches: otherwise neither number answers the question.

State these assumptions before you compare:

  • The TAP/SPAN sits on the ingress side of the ACL, so it sees packets before the deny is applied.
  • The 12,000 "hits" are packets matching the deny rule's 5-tuple, not all traffic on the link.
  • Both cover the same window (the table below assumes 10 minutes).

If the TAP is on the egress side, or is counting permitted traffic, the comparison is meaningless and you should not draw a conclusion from it.

Why syslog reads 0 while the wire shows 12,000

FailureSymptomWhere to look
logging trap set above informationalDenies never leave the boxshow logging prints the trap level
ACL entry has no log / log-input keywordCounters climb, no syslog evershow ip access-lists shows no (log) tag
Rate limit on identical messagesFirst few denies logged, rest suppressedSuppressed-message count in show logging
Buffered log wrapsRecent denies overwritten by other eventsBuffer size vs. event rate
UDP 514 loss to collectorDevice logged it, server has a gapCompare device buffer to collector

ACL deny logging is severity 6 (informational) on IOS/IOS-XE, so a logging trap of 4 (warning) silently discards every deny. That single misconfiguration produces exactly the symptom in the question.

Arithmetic worth doing: 12,000 denies in 10 minutes is 20/s. If the platform's rate limit is set to 10 messages per second, half the denies never reach the collector: the log is not empty because nothing happened, it is empty because the device stopped talking.

When syslog is the better tool

Syslog wins on cost, scale and history. Each event is a few hundred bytes, it is already centralised, and you can query last quarter. A TAP needs a free SPAN port, a capture appliance, and storage; at 10 Gbps and above you are either sampling or dropping. Syslog is also tunable per rule: you can enable log on one ACL entry and leave the rest quiet, or use log-input to capture the ingress interface and source MAC alongside the match. In virtualised or cloud environments where SPAN is awkward or unavailable, syslog may be the only per-rule record you have.

When the TAP wins

A TAP is ground truth for what crossed the wire, including packets the device never logged: hardware-forwarded drops, control-plane policer drops, and ASIC-level discards that never reach the logging path. It also gives you per-packet timing, TCP flags and payload, where syslog gives you a count.

Where the ranking flips

  • "Did this policy drop traffic?" → TAP or ACL counter.
  • "What has this rule been doing for the last 90 days?" → syslog.
  • If the TAP is sampled (sFlow/NetFlow at 1:1000), it is no longer ground truth for a 12,000-packet event. It becomes an estimate, and a correctly configured syslog outranks it.

So: read the ACL counter and the capture first, then syslog, and never close a deny investigation on an empty log buffer. The counter and the wire are the policy; the log is a report about it.

Curated: · Written: · Reviewed:

Q&A-94The IPv4 ACL is tight and the IPv6 ACL is absent. What did dual-stack just open?

Core answer

The IPv6 ACL being absent means nothing filters IPv6 packets to that service. The IPv4 ACL is a separate table and never sees an IPv6 packet, so the moment both host and client are dual-stack, the client's Happy Eyeballs logic (RFC 8305) tries IPv6 first and succeeds. The traffic the IPv4 ACL was written to stop now arrives on a path with no rules at all.

What is open is not just the one port. It is everything the host answers on IPv6: the service's listening socket if it binds the wildcard address (many do once dual-stack is on), ICMPv6, and any other daemon bound the same way. There is no NAT in the path, so the address in the packet is the host's own address: no translation layer to accidentally drop it.

Mechanism

ACLs are per-protocol. A router or host firewall evaluates an IPv4 packet against the IPv4 table and an IPv6 packet against the IPv6 table; there is no inheritance between them. Add SLAAC: hosts on the segment configure their own IPv6 addresses from Router Advertisements (RFC 4861), so a new host appears with a routable address without anyone editing a config. If the prefix is advertised and routed, the service is reachable.

Options, and when each is right

  1. Mirror the IPv4 policy in IPv6. Right when the service genuinely needs to be reachable over IPv6 and you have the tooling to test and monitor it. Do not translate the IPv4 rules literally: a rule permitting 10.0.0.0/8 has no IPv6 equivalent, and a mechanical translation can end up as a broad permit.
  2. Disable IPv6 where it is not needed. Right when nothing on that host or segment is meant to speak IPv6; it is a smaller surface than maintaining a second policy. On Linux that is net.ipv6.conf.all.disable_ipv6=1; on a switch port, do not advertise a prefix and drop RAs (RA Guard, RFC 6105).
  3. Keep IPv6 and filter elsewhere: host firewall or first-hop device. Right when the network ACL is owned by a different team than the host.

Deciding criteria: is the prefix routed to this service at all? Does the service bind IPv6 and log it? Can you probe it from outside over IPv6? If the first answer is no, option 2 is cheaper and safer than option 1.

Failure modes

  • Silent exposure. Detect by probing the service from an external IPv6 address and comparing with the IPv4 result, and by counting IPv6 hits in flow logs. Zero IPv4 hits alongside steady IPv6 hits is the signature.
  • Over-blocking ICMPv6. Dropping all of it breaks Neighbor Discovery and Path MTU Discovery; RFC 4890 lists what must be permitted. Detect by connections that hang on large payloads and ND entries that never resolve.
  • Temporary addresses. Privacy addresses (RFC 8981) rotate, so per-address rules go stale. Filter by prefix, not by host address.
  • Renumbering. A new RA prefix moves hosts out of the filtered range. Detect by alerting on prefix changes.

Worked example (hypothetical)

FamilyACLResult
IPv4deny any to 203.0.113.10 tcp/443 except 198.51.100.0/24only the allowlisted subnet connects
IPv6noneany host with a route to 2001:db8::10 connects

A client inside 198.51.100.0/24 and a client outside it both reach the service over IPv6; only the first reaches it over IPv4. Same service, two different answers.

Curated: · Written: · Reviewed:

Q&A-106Walk me through everything that happens between typing a URL and the page rendering, and at each step, name what breaks there.

Assume a cold cache, IPv4, HTTPS, no proxy, one A record. The path is DNS → route lookup → ARP → TCP → TLS → HTTP → teardown. Each step has a distinct failure signature, and that is the framework.

DNS. The stub resolver checks /etc/hosts, then its cache, then sends a UDP query to the recursive resolver in /etc/resolv.conf. On a miss the resolver iterates: root (.), TLD (.com), authoritative. Every answer carries a TTL; negative answers are cached per the SOA minimum (RFC 2308). Breaks here: UDP 53 blocked (query times out, then falls back to TCP 53), DNSSEC validation failure (SERVFAIL), NXDOMAIN cached for the SOA minimum, split-horizon returning an internal address, or a resolver that ignores TTL. Detection: dig +trace, dig @resolver, check TTLs.

Route lookup. The kernel does longest-prefix match in the FIB. No route → EHOSTUNREACH. A blackhole route → silent drop. Asymmetric routing → return path differs.

ARP. If the next hop is on-link, the host broadcasts an ARP request and expects a unicast reply. Incomplete ARP means no reply: the packet is queued and dropped. Breaks: proxy ARP, ARP cache poisoning, static ARP mismatch, or the gateway is in a different subnet.

TCP. SYN carries MSS, window scale, SACK-permitted. SYN-ACK, ACK. Breaks: SYN dropped by a stateless firewall (no SYN-ACK), SYN-ACK dropped because the return path is asymmetric and the firewall never saw the SYN, RST from a middlebox, or MSS too large for a hop. Detection: tcpdump on both ends, check for SYN retransmits.

TLS. ClientHello carries SNI, cipher suites, ALPN. ServerHello, certificate, key exchange, Finished. TLS 1.3 is 1-RTT. Breaks: expired or name-mismatched cert, SNI-based blocking, a certificate chain larger than the path MTU (ServerHello fragments and may be dropped if ICMP is filtered), or a middlebox that does not speak TLS 1.3. Detection: openssl s_client -connect host:443 -servername host.

HTTP. GET / with Host header. Breaks: 503 from the backend, wrong virtual host, redirect loop, 401. Detection: curl -v, check response codes.

Teardown. FIN/ACK, then TIME_WAIT on the active closer.

Where MTU, NAT, firewalls sit: MTU is per-hop at L3; if DF is set and a hop is smaller, ICMP frag-needed comes back, if filtered, PMTUD black-holes. MSS clamping at the edge fixes it. NAT rewrites source IP/port at the edge, needs state, and breaks inbound unless a static mapping exists; ALGs handle FTP/SIP. Stateful firewalls need to see both directions; asymmetric routing breaks the state table.

StepProtocolWhat breaksHow you see it
DNSUDP/TCP 53resolver unreachable, NXDOMAIN cached, DNSSEC bogusdig +trace, SERVFAIL
Route lookupIPno route, blackhole, asymmetricip route get
ARPARPincomplete entry, proxy ARPip neigh
TCPTCPSYN dropped, RST, MSS too largetcpdump, SYN retransmits
TLSTLScert expired, SNI blocked, chain > MTUopenssl s_client
HTTPHTTP503, wrong vhost, redirect loopcurl -v
TeardownTCPFIN not seen, TIME_WAITss -tan

A cold-cache trace looks like this (times in ms): 0 DNS query, 20 DNS response, 21 SYN, 41 SYN-ACK, 42 ACK, 43 ClientHello, 63 ServerHello, 64 Finished, 65 GET, 85 200 OK. If the 21 ms SYN has no 41 ms SYN-ACK, the fault is before the listener or on the return path, not in TLS or HTTP.

Curated: · Written: · Reviewed: —

Q&A-100Twelve hops look fine from one ping. What does a complete validation still have to prove?

Core answer (first 90 seconds). A ping proves one flow works. A complete validation has to prove the path works for every flow the application will use, in both directions. Concretely, that means: every ECMP member the hash can select, the smallest MTU on the path, the application's real five-tuple (src/dst IP, src/dst port, protocol), the VRF or tenant the user sits in, and that the forwarding plane carries it, not a control-plane reply from a router's CPU. One ping covers one hash, one size, one direction, one protocol.

Why one ping is one point. ECMP hashes on the five-tuple (some platforms add flow label or ingress port), so a single source port pins one member. A default Linux ping sends a 56-byte payload (64-byte ICMP, 84 bytes on the wire) far below any tunnel MTU, so it never exercises fragmentation. And an ICMP echo to a router interface is often punted to the control plane, taking a different path than forwarded traffic.

What to run.

ProbeCoversMisses
ping (56 B payload)one hash, one direction, control-plane replyother ECMP members, large MTU, return path
ping -M do -s 1472 (Linux; -f -l 1472 Windows, -D -s 1472 macOS)1500-byte path, one hashother members, return path
traceroute -F -q 1 repeated with varying source portper-hop MTU, ECMP member enumerationapplication five-tuple, return path
MPLS LSP ping / LSP traceroute (RFC 8029)data-plane forwarding per LSPIP-layer MTU, app ports
BFD (RFC 5880/5881)continuous data-plane livenessMTU, ECMP member coverage

For the application five-tuple, nping or hping3 let you set source port, protocol and payload size; iperf3 with the real MTU confirms bulk behaviour.

Trade-offs and sampling. The full matrix is members × sizes × directions × protocols. Sixteen ECMP members, three sizes, two directions, two protocols is 192 probes: usually too many to run per change. Sample instead: enumerate members once with multipath traceroute, then probe the two MTU extremes (e.g. 1280 and 1500, or 9000 on jumbo fabrics) in both directions with the real five-tuple. A reduced set is defensible when the path is single-member, symmetric, and MTU-uniform: a single-homed 1500-byte path needs one direction and one size. It is not defensible across ECMP, tunnels, or asymmetric routing.

Failure modes and detection.

  • MTU black hole: large packets dropped, ICMP "fragmentation needed" filtered. Detect: -M do -s 1472 fails while -s 1400 succeeds; traceroute -F shows the hop where it stops. VXLAN adds 50 bytes, GRE 24; IPsec varies with mode and cipher.
  • Asymmetric return: forward path clean, return drops. Detect: probe from both ends, or run traceroute from the far side.
  • Bad ECMP member: one hash drops. Detect: vary source port to force different hashes.
  • Control-plane punt: ping answered by CPU while forwarding is broken. Detect: MPLS LSP ping or BFD, which exercise the data plane.
  • VRF/tenant mismatch: probe from the wrong VRF. Detect: run from the actual VRF (ip vrf exec on Linux).

Sign the path only when the recorded matrix shows members, directions and sizes covered, or when you can state why the reduced set is sufficient.

Curated: · Written: · Reviewed:

Q&A-108A team wants to move a lossy telemetry feed from TCP to UDP. Walk me through the transport-layer trade-off, and what a packet capture of each actually shows.

Core answer (first 90 seconds). I'd start by asking what the application needs from the transport: every byte in order, or the newest sample as fast as possible? TCP is a reliable, ordered byte stream with congestion control built into the kernel; UDP is a datagram service that gives you ports and a checksum and nothing else. Almost every TCP-vs-UDP decision falls out of that one question, plus whether you need one-to-many delivery.

What the handshake costs. TCP opens with SYN, SYN-ACK, ACK: one full RTT before the first payload byte, and the SYN consumes a sequence number so it is retransmitted on a doubling timer (Linux net.ipv4.tcp_syn_retries=6). The options exchanged there set the rules for the connection: MSS, window scale, SACK permitted. On a 40 ms RTT path that is 40 ms of dead time before the request starts; TLS 1.3 adds another RTT unless you use 0-RTT.

Reliability and ordering are not free. TCP numbers every byte, ACKs cumulatively, retransmits on loss or timeout, and reassembles in order. Congestion control (CUBIC by default on Linux, BBR where deployed) treats loss as a congestion signal and cuts cwnd, which is exactly wrong on a lossy radio or congested Wi-Fi hop where loss is not congestion. And because delivery is in order, one lost segment stalls every later segment sitting in the receive buffer: head-of-line blocking. That is why HTTP/2 over a single TCP connection can lose to six parallel ones on a lossy path.

Where UDP is the right base. No handshake, no connection state, no ordering, no congestion control, no retransmission. That fits:

  • Loss-tolerant, freshness-first traffic: VoIP/RTP, video, game state. A retransmitted voice sample arrives too late to play.
  • One datagram each way: DNS over UDP/53, with the app retrying and falling back to TCP when the TC bit says the answer was truncated.
  • No IP address yet: DHCP DISCOVER is a broadcast from 0.0.0.0 to 255.255.255.255; TCP cannot do that.
  • Multicast or broadcast: mDNS, IPTV, routing hellos.
  • Building reliability yourself: QUIC runs on UDP/443 with its own loss recovery, per-stream flow control and TLS 1.3, so one lost packet blocks one stream rather than the connection, and it traverses middleboxes that only pass UDP/443.
TCPUDP
Setup3-way handshake, 1 RTTnone
Deliveryordered byte stream, retransmitdatagrams, best effort
Congestion controlCUBIC/BBR in kernelnone: the app's job
Header20–60 B8 B
Fan-outunicast onlyunicast, multicast, broadcast
Capture signatureSYN/SYN-ACK/ACK, then Seq/Ack/Winone datagram each way; ICMP type 3 code 3 if closed

What a capture shows. TCP: SYN, SYN-ACK, ACK, then PSH-ACK data carrying Seq/Ack/Window and options (MSS=1460, SACK permitted, WS=7). A retransmission is the same Seq sent twice; a zero window means the receiver, not the network, is the bottleneck. UDP: one datagram out, one back, no state, and if nothing is listening, ICMP type 3 code 3 (port unreachable) is the only "connection refused" UDP has.

Failure modes worth naming. UDP has no congestion control, so a naive sender can crush a link, and it is the amplifier of choice for reflection attacks (DNS, memcached: small query, huge response). UDP NAT mappings expire in roughly 30–120 s versus hours for established TCP, so keepalives matter. IPv4 UDP checksum is optional, zero means not computed, while IPv6 makes it mandatory.

The judgement call. Default to TCP unless you can name the specific reason UDP is required. If you pick UDP, say who now owns retransmission, ordering, congestion control and security, because you just took all four.

Curated: · Written: · Reviewed: —