Containers package processes; orchestration manages desired state under change and failure

A container is a process (or process group) executed from an image with configured filesystem, namespaces, cgroups, identity, mounts, capabilities, network and environment. It shares the host kernel; it is not a virtual machine and not automatically a security boundary. An image is content-addressed configuration plus immutable layers, while a running container adds writable state and runtime configuration. Build reproducibly, pin base images and dependencies by reviewed digest, minimize contents and privileges, scan and sign with stated assurance limits, and rebuild rather than patching a running container invisibly.

Define a container contract: entrypoint and arguments, signal handling, graceful shutdown, non-root UID/GID, read-only filesystem where feasible, required writable paths, ports, health behavior, resource needs, configuration, secrets, logs and data ownership. PID 1 must forward or handle termination and reap children. Avoid embedding credentials or environment-specific data in layers because deletion in a later layer may not remove earlier content. Use multi-stage builds and a small runtime image, preserve SBOM/provenance, and deploy the exact verified digest rather than a mutable tag.

Kubernetes schedules Pods, not isolated containers. Containers in one Pod share a network namespace and can share volumes and lifecycle, creating a tight trust and fate boundary. Put processes together only when they must be co-scheduled and communicate as one operational unit; unrelated services deserve separate Pods. Init containers perform ordered setup, while sidecars provide supporting behavior but add startup, shutdown, resource and security coupling. Ephemeral containers support diagnosis and need exceptional access governance.

Controllers reconcile actual state toward desired state. A Deployment manages ReplicaSets and rolling changes for stateless or externally stateful workloads; StatefulSets and Jobs answer different identity/order/completion needs. Reconciliation is eventually consistent and can amplify a bad desired state. Use reviewed manifests, immutable images, selectors that do not overlap unintentionally, revision history, progress deadlines, bounded surge/unavailable settings and explicit rollback or roll-forward criteria. A controller reporting available replicas does not prove user journeys or data correctness.

Services provide stable discovery and a traffic abstraction over changing endpoints. Understand selector and port mapping, EndpointSlice readiness, DNS, session affinity and internal/external traffic behavior. Network reachability is not authorization; apply authenticated application policy and network controls with default-deny where appropriate. Test endpoint churn, zone skew, connection draining, DNS caching, dual-stack and load-balancer health. Preserve client source/identity only when the architecture and policy explicitly support it.

Health probes have distinct purposes. Startup prevents liveness/readiness from acting before slow initialization completes. Readiness removes an endpoint from normal Service traffic when it should not receive new work. Liveness restarts a stuck container when restart is a safe remedy. A shared deep dependency check can cause every replica to restart during a dependency outage and create a cascade. Make probes cheap, bounded and representative of their decision; protect probe paths from abuse and observe failures. Readiness does not certify complete user behavior, and liveness cannot repair corrupted data.

Resource requests guide scheduling and establish capacity assumptions; limits constrain runtime use according to resource semantics. CPU throttles while memory pressure may trigger OOM termination; limits are not reservations of performance. Size from representative load and failure states, account for sidecars/init/Pod overhead, use quotas and ranges, and monitor throttling, working set, OOM, eviction, disk/ephemeral storage, queueing and objectives. Overcommit and autoscaling need headroom and recovery tests. A Pod that fits on paper may contend on network, storage, kernel or shared dependencies.

Treat scheduling and disruption as reliability controls. Use topology spread, affinity/anti-affinity, taints/tolerations and priorities deliberately; incorrect constraints can leave Pods pending or concentrate them in one failure domain. PodDisruptionBudgets limit some voluntary disruptions, not involuntary node/zone failure and not application correctness. Test node drain, loss, zone outage, rescheduling, image-pull failure and capacity exhaustion while measuring user/data outcomes. Protect control-plane and system workloads separately.

Apply defense in depth: Restricted-style policies where possible, non-root and no privilege escalation, drop capabilities, seccomp, constrained volume types, read-only root, least-privilege service accounts, scoped RBAC and network policies. Do not mount host paths, container runtime sockets or broad credentials without explicit exceptional review. Image scanning cannot detect every runtime or configuration risk; admission and runtime detection complement build controls. Namespace separation is administrative, not sufficient multi-tenant isolation for every threat model.

Kubernetes Secrets are API objects, not a complete secret-management solution. Enable encryption at rest and strict RBAC, avoid broad list/watch and environment/log leakage, prefer short-lived external identity and secret-store integration where appropriate, rotate and audit. A base64 representation is not encryption. Mount only what a workload needs, understand update/reload behavior, and remove old material. Back up and restore control/data state under protected procedures.

Operate through observable user outcomes. Emit structured stdout/stderr or approved telemetry without secrets, propagate correlation, expose useful metrics and preserve audit. Deploy progressively, verify the running digest, critical journeys, data integrity, security and capacity, then observe shutdown/drain and recovery. Test control-plane outage, registry failure, credential expiry, node pressure, network partition and partial rollout. Containers improve packaging consistency; orchestration automates decisions, including bad ones. Neither proves resilience without scoped end-to-end evidence.