Put a deterministic security kernel around probabilistic decisions

An LLM agent observes state, decides whether and how to use tools, receives results, updates its plan, and eventually returns an answer or stops. Tool use can make a model useful beyond text—searching, computing, changing files, operating applications, or coordinating workflows. It also converts model errors and prompt injection into real side effects. The model is an untrusted planner. Deterministic application code must own identity, authorization, validation, budgets, state transitions, execution, confirmation, auditing, and recovery.

Every tool is a capability contract. Give it a narrow name and description, explicit JSON schema, field units/formats/bounds, authentication context, effect classification, idempotency semantics, timeout, result and error schema, data sensitivity, concurrency rules, and rollback story. Avoid generic shell, SQL, HTTP, filesystem, or “execute” tools when a constrained domain operation suffices. Tool descriptions are part of the model interface, while server-side enforcement is the security boundary.

Never trust model-produced arguments. Resolve the authenticated subject and tenant outside the prompt, reject identity or scope overrides, validate structure and semantics, normalize canonical forms, authorize the exact object/action/state, and enforce rate, value, time, destination and data bounds. Recheck immediately before execution to prevent time-of-check/time-of-use races. A valid schema can still contain a forbidden account, path, recipient, query, amount, or transition.

Observations are untrusted too. Web pages, documents, emails, tool errors, API responses and prior memory can contain instructions aimed at the agent. Delimit content as data, retain source/provenance, minimize what enters context, and never let observations change system policy, grant capability, disclose secrets, or self-authorize follow-on actions. Treat model output passed to code, templates, SQL, shells, browsers, or other agents with context-appropriate escaping and validation.

Planning should be bounded and inspectable. The application controls maximum steps, wall time, tokens, cost, tool calls, parallelism, recursion, bytes, retries and side effects. Detect repeated identical actions, oscillation, no-progress loops, expanding search and conflicting plans. Use explicit task state, preconditions, postconditions and completion checks rather than relying on the model to declare success. A tool returning 200 or exit zero does not prove the user's goal was achieved.

Side effects need transaction semantics. Attach stable operation and idempotency keys to retries, record requested/accepted/applied outcome, and distinguish timeout from failure. A request may commit and lose its response; blindly retrying can duplicate payment, message, deletion or deployment. Prefer prepare/preview/commit for consequential actions, optimistic version checks, atomic writes and compensating or rollback operations. Do not promise rollback where the outside world cannot be undone.

Human approval is a control, not a dialog decoration. Require it for irreversible, high-impact, high-cost, external communication, privilege change, sensitive disclosure, or ambiguous actions according to policy. Show the exact target, effect, data, scope, price, reversibility and reason at the moment of decision; bind approval to a canonical action hash and expiry. Changes after approval require reapproval. Avoid approval fatigue by eliminating unnecessary capability and batching only truly atomic related effects.

Least privilege applies per task and per step. Issue short-lived, audience-bound credentials for exact operations and resources; keep secrets outside model-visible context, logs and errors. Separate read from write, draft from send, stage from deploy, and request from authorize. Sandbox code and file access with CPU, memory, network and filesystem limits, immutable inputs where possible, egress allowlists and cleanup. A sandbox reduces blast radius but is not a substitute for authorization or dependency security.

Concurrency complicates agent state. Parallelize only independent read operations or effects with explicit ordering and conflict controls. Use versioned state, locks or compare-and-swap where required, stable correlation IDs, cancellation propagation and bounded result joining. Two individually authorized operations can violate an invariant together. Nested agents do not gain authority by delegation; pass the minimum capability and provenance, and enforce depth and aggregate budgets.

Memory must have provenance, scope and expiry. Separate current conversation, task state, user preferences, organizational facts and retrieved evidence. Do not persist hidden model reasoning, secrets, transient authorization or unverified claims as durable memory. Let users inspect, correct and delete permitted memory, bind it to tenant/identity, and prevent cross-user retrieval. Treat memory as an input that can be stale or poisoned.

Operational failures are normal. Tools time out, return partial pages, rate-limit, change schemas, require reauthentication, or expose uncertain results. Use structured typed errors, bounded retries with backoff/jitter for safe cases, circuit breaking and explicit fallback. Preserve partial progress and resume from verified state instead of rerunning the whole plan. On cancellation, stop launching work, propagate aborts, reconcile in-flight side effects, and report what is known, unknown and recoverable.

Audit every effect without leaking secrets. Record authenticated actor, delegated agent/version, user intent, plan/task ID, tool/schema version, canonical sanitized arguments, policy decision, approval reference, idempotency key, start/end, target, result, applied state, error, rollback and evidence. Tamper-protect high-value records and correlate external provider IDs. Logs should enable reconstruction of why an action occurred while minimizing sensitive prompts, payloads and credentials.

Evaluate agents in executable, resettable environments. Build tasks from real workflows with exact initial state, allowed capabilities, success invariants, forbidden side effects, cost/time budget and cleanup. Score final state and process: task success, argument validity, policy violations, excess calls, latency/cost, recovery, calibration and human intervention. Include prompt injection, malicious tool output, ambiguous intent, stale memory, tool/schema drift, partial failures, uncertain commit, concurrency conflict, cancellation, privilege escalation and sandbox escape attempts. Model-graded traces are useful only when calibrated against deterministic checks and human review.

Monitor tool selection and invalid arguments, authorization/approval denials, side-effect attempts and outcomes, retries/idempotency conflicts, loop/no-progress stops, step/token/cost/latency, tool availability/schema errors, confirmation abandonment, policy incidents, memory writes/reads, sandbox limits, rollback and task success. Roll out model, prompt, tool and policy versions independently with trace replay, shadowing, canaries, kill switches and atomic rollback. Safer autonomy comes from narrow reliable capabilities and observable state, not longer hidden reasoning.

Worked example: refund $50, the 200 never comes back

User asks to refund order 4419. Policy: auto-refund under $50 to the original payment method only. The first refund POST times out after the processor commits.

step	model proposes	kernel	money moved
1	{order:4419, amount:50, dest: original}	schema + session tenant + $50 gate	$50, then timeout
2 retry, no idempotency key	same JSON again	second POST	$100
2 retry, key refund:4419	same JSON again	duplicate of step 1	still $50

A 200 from the model is not the interview. Whether the second call can spend twice is.