Cloud-Native Network Security — Concept Guide

Review status: rewritten after review; coverage and framing reworked.

The mental model: layers, not a wall

Cloud-native network security is a stack of enforcement points, each answering a different question:

• Underlay / VPC design: which networks exist, how they're addressed and routed. This decides what can talk to what.
• Security groups (SGs): stateful, attached to a workload or interface. Allow-list only. Return traffic for an allowed flow is automatically permitted.
• Network ACLs (NACLs): stateless, applied at the subnet boundary, numbered and ordered, allow and deny. Both directions and the ephemeral-port range on the return path must be handled explicitly.
• Managed firewalls (AWS Network Firewall, Azure Firewall, GCP Cloud NGFW): centralized, often with L7 features, inspection, and logging — the place for egress policy and east-west chokepoints.
• Container/overlay networking (CNI): pod-to-pod traffic that may never traverse a subnet boundary, so SGs and NACLs don't see most of it. Kubernetes NetworkPolicy lives here.
• Service mesh and identity-aware access: mTLS and L7 authorization between workloads, and identity-based access to services regardless of network location.

The mistake interviewers watch for is treating SG/NACL as the topic. They're one layer. A senior answer names the layers, says what each can and cannot decide, and then talks about identity — because network placement authenticates nobody. A packet that arrives on a "trusted" subnet is still just a packet.

How the primitives actually behave

Stateful vs. stateless is the classic probe. An SG tracks connection state: allow inbound TCP/443 from 10.0.1.0/24 and the return traffic (destination ephemeral port) is permitted automatically. A NACL doesn't track anything: allow inbound 443 and you must also allow outbound to the client's ephemeral range (1024–65535 on Linux) or the handshake dies at the subnet edge. Weak answers mix these up or claim NACLs are stateful because they "remember" rule numbers.

Provider differences matter and names lie:

• AWS: SGs on ENIs, NACLs on subnets, evaluated lowest-rule-number-first, default NACL allows all.
• Azure: NSGs are stateful five-tuple rules on subnets or NICs — closer to an SG than an AWS NACL despite the name.
• GCP: hierarchical firewall policies (org → folder → project) with priorities and go-to-next-level delegation; VPC firewall rules are distributed and stateful, targeted by tags/service accounts.

Control-plane power beats data-plane rules. Whoever can attach an SG, change a GCP service account or tag, or modify a hierarchical policy can bypass your narrowest allow. Guard those permissions like the rules themselves.

Kubernetes: where subnet thinking breaks

Pod traffic is overlay traffic. Two pods on the same node may never leave it; two pods on the same subnet may be different tenants. SGs and NACLs are nearly blind here. The controls are:

• Default-deny NetworkPolicy. Kubernetes allows all pod traffic by default. Until you apply a policy selecting a pod, it accepts everything. First move in any cluster: deny-by-default per namespace, then add allows.
• Selectors, not IPs. Policy selects pods by label and namespace, and ingress vs. egress are separate policy types — you can lock one without the other, which surprises people.
• L3/L4 only. Standard NetworkPolicy cannot say "allow HTTPS to api.example.com" — only IPs/CIDRs. Since most egress targets are behind rotating IPs, teams either over-permit CIDRs or need a CNI with FQDN policy.
• DNS is load-bearing. A default-deny egress policy that forgets to allow UDP/TCP 53 to the cluster's CoreDNS pods breaks everything, silently, at the next DNS TTL.
• CNI differences are real. Calico and Cilium both enforce NetworkPolicy but Cilium adds L7 and FQDN rules via eBPF; some CNIs enforce nothing at all. "Does your CNI actually enforce policy?" is a fair follow-up.

A minimal correct pattern:

# default-deny in a namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: payments
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

Then a second policy allowing ingress from the frontend namespace on 8443, and egress to CoreDNS on 53 plus the specific service. Interviewers probe the DNS rule and the ingress/egress asymmetry more than the YAML.

Service mesh and workload identity

NetworkPolicy says "this pod may talk to that pod on port X." A mesh adds:

• mTLS between services: every connection is mutually authenticated with short-lived workload certificates, so a compromised pod can't impersonate the payments service.
• L7 authorization policy: allow GET /v1/balance from frontend to payments, deny POST /admin — impossible with L3/L4 rules.
• Certificate lifecycle: issuance and rotation (Istio's istiod signs SPIFFE-style identities; rotation is automatic and frequent) — no more shared passwords between services.

Trade-off: sidecars (Envoy per pod) cost CPU, memory, latency, and upgrade pain; sidecarless models (Cilium ambient mesh, Istio ambient) use per-node or per-node-pair proxies and cut that overhead but change the trust and debugging model. A weak answer says "mesh = mTLS" and stops; a strong one names what NetworkPolicy can't express and when a mesh is overkill — a 20-service system with a flat trust domain probably doesn't need one.

Zero trust: identity as the perimeter

Flat network trust — "it's on the corporate network / in the VPC, so it's fine" — is the thing being replaced. Per-request identity means every call carries who is calling and gets authorized on it.

• Private endpoints (AWS PrivateLink, GCP Private Service Connect, Azure Private Endpoints) map a private address to a specific managed-service interface, keeping traffic on provider backbone. They reduce exposure; they do not authorize anyone. Validate private DNS resolution from every consumer network, then disable public access on the service.
• IAM condition keys tie authorization to network path: e.g. an AWS policy with aws:SourceVpce restricting an S3 bucket to traffic from a specific VPC endpoint. That's identity and network, checked together.
• Admin access without bastions: SSM Session Manager (or equivalent) gives audited, IAM-authorized shell access with no inbound ports open at all — the modern answer to "how do admins reach instances?" A bastion with port 22 open to a corporate CIDR is the answer interviewers expect you to improve on.
• Third parties get scoped private endpoints or identity-federated access, never a peering link that implies blanket trust.

The internet-facing edge

Anything exposed to the internet needs an intentional edge, not a port opened on a workload:

• DDoS protection (provider-managed like AWS Shield, Azure DDoS Protection, or a fronting CDN) absorbs volumetric attacks before they reach your origin.
• TLS terminates at the edge, at a reverse proxy or load balancer, with only the required listener ports open — 80/443, not 22 or 3306 "temporarily."
• A WAF in front of the application filters L7 attacks (SQLi, XSS, bot patterns) that L3/L4 rules can't see.
• Backends accept traffic only from the edge — the ALB's subnets, the proxy's SG, or the forwarding rule's ranges — never from the whole internet. Note the asymmetry: provider firewall rules can protect load-balancer backends without governing the proxy frontend itself; configure the forwarding rule and the edge service separately.
• Client identity through the proxy: forwarded headers like X-Forwarded-For are spoofable by anyone who can reach the backend directly. Only trust them when the backend's network policy guarantees the request came from a known proxy boundary — otherwise an attacker sets the header themselves and your rate limiting or audit trail is fiction.

A weak answer treats the edge as "we have a load balancer." A strong one walks the path: DDoS layer → TLS listener → WAF → backend restriction → header trust, and can say which layer stops which attack class.

Egress: the boundary everyone forgets

Unrestricted egress is the most common real-world finding. A default route through a NAT gateway gives outbound connectivity and address translation — no authorization whatsoever. A compromised pod with open egress exfiltrates to any URL and, worse, to the metadata service.

• Inventory dependencies first: DNS, identity, package registries, telemetry, payment APIs. Then apply destination/port controls, FQDN filtering (managed firewall or CNI), or service-endpoint policies sized to that list.
• DNS is both a control point and a blind spot. FQDN egress rules are only as good as DNS: workloads resolving through their own resolver, or direct-to-IP traffic, bypass them. Log and control DNS egress too.
• Block the metadata service. 169.254.169.254 from a pod is the SSRF-to-credentials path. Defenses: IMDSv2 (session-oriented, requires a token — plus a hop limit of 1 so containerized workloads can't reach it), NetworkPolicy denying the link-local range, or CNI-level metadata guards.
• Plan for fail-open pressure. If egress controls break a certificate check or an update path, someone will widen them to allow-all permanently. Design the exception process (scoped, expiring) before the incident.

Failure modes, diagnosis, and what changes at scale

Diagnose one layer at a time, in order: DNS → route → NAT/translation → load balancer → firewall/SG/NACL → host listener → TLS → application auth. Most debugging failures come from testing two layers at once.

Common failure modes:

• Stateless return-path gaps: NACL allows inbound 443, forgets outbound ephemeral — intermittent-looking failures that are actually deterministic.
• Non-transitive peering: A↔B and B↔C does not give A↔C — until a transit hub or route propagation does, unintentionally.
• Overlapping CIDRs and asymmetric routing in hybrid setups; on-prem firewalls seeing only one direction.
• Eventual propagation of centrally managed rules: a change "not working" may just not have landed yet — which is why you stage org-level policy progressively.
• Flow logs as ground truth: they show observed tuples and accept/reject decisions, but may aggregate, omit some traffic types, and carry no application identity. Combine with firewall logs, DNS, LB logs, and identity telemetry. Configuration review alone misses interactions; run positive and negative probes to prove a forbidden path is actually forbidden.

At scale the shift is from rules to policy as code: owners, purpose, expiry per rule; automated checks for broad CIDRs, IPv4/IPv6 parity, dangerous ports, and shadowed priorities; drift detection for console-made changes. Removal needs evidence — flow-log silence can miss seasonal traffic, so pair telemetry with owner confirmation and a reversible disable window. Measure attack surface as internet-reachable services, unrestricted rules, missing egress controls, stale exceptions, time to contain — not rule count.

Incident response: what to do when the network was the problem

During exposure or compromise, the network evidence is perishable:

• Preserve before you change: snapshot rules, routes, flow logs, and identity events covering the window. Tearing down the exposed resource first destroys the record of who reached it.
• Reconstruct the blast radius: determine exactly which sources could reach which listeners during the window — flow logs, LB access logs, and DNS queries together. "Who could have connected" matters as much as who did.
• Close the path through a trusted control plane — an attacker may hold the credentials you'd use to fix it. Don't block your own responder access or sever the logs you need.
• Rotate credentials and inspect application activity. Network access is one layer; assume tokens issued to anything reachable in the window are stolen.
• Validate closure from every path: external, peer networks, hybrid/on-prem, and IPv6 — the forgotten path that reopens the exposure. Then fix the provisioning guardrail that allowed it, or the same rule reappears next deploy.

Interviewers use incident scenarios to test whether you think in evidence and blast radius, or just start deleting rules.

What interviewers probe, and what a weak answer sounds like

• "Walk me through why this connection fails." They want the layered trace, not a guess. Weak: "check the security group."
• "SG vs. NACL." Weak: getting statefulness backwards. Strong: stateful allow-list on the ENI vs. ordered stateless allow/deny at the subnet edge, plus the ephemeral-port consequence.
• "How do you secure pod-to-pod traffic?" Weak: "we use security groups." Strong: default-deny NetworkPolicy, selectors, the DNS egress rule, CNI enforcement, and where mesh takes over.
• "Is a private endpoint enough?" Weak: yes. Strong: it removes public exposure but authorizes nothing; pair it with endpoint policies, IAM conditions, and disabled public access.
• "How does a compromised pod steal credentials?" Weak: shrug. Strong: open egress + IMDS, and the specific mitigations (IMDSv2, hop limit, egress FQDN policy).
• "What breaks when you centralize firewall policy?" Propagation lag, blast radius, exception process — the operational answer, not just the design.
• "An instance was exposed for six hours — what do you do?" Weak: close the port and move on. Strong: preserve evidence, reconstruct reachable sources, rotate credentials, validate closure including IPv6, fix the guardrail.

Likely follow-ups: IPv6 parity, cost/cardinality of flow logs, how you'd prove a rule is safe to delete, and how zero trust changes admin access patterns.

Worked example: a connection that fails for a non-obvious reason

Setup (AWS, hypothetical but typical): an ALB in public subnets (10.0.0.0/24) targets instances in private subnets (10.0.2.0/24). Clients report intermittent 504s. The instance SG allows inbound 443 from the ALB's subnets. The NACL on 10.0.2.0/24 was recently tightened to:

100  ALLOW  TCP 443  from 10.0.0.0/24
*    DENY   ALL  ALL

Trace the handshake: ALB (source port, say, 51234) → instance:443. Inbound rule 100 matches — fine. The instance replies 443 → 51234. That packet leaves the subnet: outbound evaluation hits the default DENY. SYN-ACK is dropped. Every connection fails, deterministically — the "intermittent" report was clients retrying.

Fix: add ALLOW TCP 1024-65535 to 10.0.0.0/24 outbound (or scope to the ALB's documented ephemeral range). The SG would have handled this automatically — the NACL does not, because it's stateless. This is the single most common NACL mistake, and being able to narrate the dropped SYN-ACK step-by-step is what separates a memorized definition from understanding.

Same lesson generalizes: every stateless boundary — NACLs, some on-prem firewalls in hybrid paths — needs both directions accounted for explicitly.