LLM Agents & Tool Use

The model reasons in text about the current state and goal, decides on an action (a tool call with arguments), the application executes it, and the result (observation) is appended to the model's context. The model then repeats this reason-act-observe cycle until it decides it has enough information to produce a final answer.

LLMs generate text/tokens; they have no runtime execution capability or sandboxed access to external systems on their own. Letting a model directly execute arbitrary code or API calls, instead of a controlled application-layer executor, would remove any validation, sandboxing, or permission checks — creating major security and correctness risks.

If a tool returns untrusted content (e.g. scraped webpage text) containing text like "ignore previous instructions and do X," the model may follow it as a legitimate instruction, since it's just more tokens in its context. Defenses include treating tool output strictly as data rather than instructions (via prompt structure and delimiters), validating/sanitizing tool outputs, restricting which actions can follow from tool-derived content, and requiring human approval for sensitive downstream actions.

Narrow tools are individually easier for the model to select correctly and reason about, since each has a clear, unambiguous purpose, but too many of them increases prompt size and can make tool selection itself harder. Broad tools reduce the tool-selection decision, but push more reasoning burden and ambiguity onto correctly using a single flexible tool.

Without a hard limit, a model that can't determine task completion, or that gets stuck looping between actions, will keep invoking tools indefinitely — incurring runaway API cost and latency. A step limit forces a bounded outcome, giving the system a defined failure mode instead of an open-ended cost.

Use an external scratchpad or state store, and periodically summarize completed steps/findings into a condensed form that gets re-injected into subsequent prompts, rather than replaying the full raw history. This trades some detail loss for staying within the context limit as the task grows longer.

Planning-first has the model sketch the full multi-step plan upfront, then execute it — giving more coherent multi-step behavior but adding upfront latency and risk from a wrong early plan. Step-by-step decides one action at a time based on the latest observation, which is more adaptive to unexpected results but can lose track of the bigger picture. Planning-first suits well-defined multi-step tasks; reactive suits exploratory or uncertain environments.

For example, the agent misreads a tool's numeric result and carries the wrong number through subsequent calculations and tool calls, producing a confidently wrong final answer. Guards include validation checks after each step, self-reflection/verification prompts, or bounding how much unchecked state can propagate before a checkpoint forces re-verification.

Agent reasoning is imperfect and can be manipulated (e.g. by injected content) or simply wrong. A human checkpoint before irreversible actions — deleting data, sending money, sending external communications — provides a safety backstop that fully autonomous execution lacks.

The schema tells the model what the tool does, what arguments it expects, and its constraints, directly shaping whether the model selects it correctly and calls it with valid arguments. A vague or ambiguous description leads to wrong tool selection or malformed/incorrect argument values.

A single agent handles the full task itself, choosing among all available tools. A multi-agent architecture splits work across specialized sub-agents (e.g. a research agent, a coding agent, a review agent) coordinated by an orchestrator — this helps when subtasks require very different tool sets or expertise, or when isolating context per sub-agent avoids overloading one model's context and attention with everything at once.

Check task completion against ground truth (correct final state or correct tool calls for a labeled set of scenarios), track intermediate step correctness (right tool, right arguments), and measure cost/latency/step count. A plausible-sounding final answer can still reflect wrong intermediate actions that happened to land on a correct-looking result by luck.

For example, calling a flaky external API that occasionally times out. The agent loop should catch the failure, decide whether to retry (with backoff), try an alternative tool, or surface a clear failure/ask-for-help state to the model or user — rather than silently proceeding as if the call had succeeded.

A wrong or manipulated call to a read-only tool is generally recoverable — you just get wrong information. A wrong call to a destructive tool can be irreversible, so write-capable tools usually warrant stricter guardrails: allowlisting, confirmation prompts, dry-run modes, or human approval.

Clearly document each tool's purpose, arguments, and constraints; provide few-shot examples of correct tool-call patterns; explicitly instruct the model to only call tools it's actually been given rather than invent new ones; and instruct it to ask for clarification or stop rather than guess when it's uncertain.

A lightweight initial step classifies the incoming request and dispatches it to the most appropriate specialized agent or toolset. This is preferable because it keeps each downstream agent's context and tool list focused and simpler to reason about correctly, rather than overloading a single agent with every possible tool and use case at once.

After producing an intermediate result, the model is prompted to review or critique its own output against the task requirements before proceeding — for example, asking "does this answer actually address all parts of the question?" This extra pass can catch errors the initial generation missed, at the cost of an additional model call.

Discrete tools are pre-defined, scoped functions with fixed schemas — safer and easier to validate, but limited to what's been built. Code execution lets the model write and run arbitrary code, far more flexible and general-purpose, but with a much larger security surface and harder-to-sandbox behavior.

The agent loop adds latency and cost (multiple model calls) and additional failure surface (tool selection errors, looping) compared to a single-shot generation. If the task doesn't actually need external actions or multi-step reasoning, a single prompt is faster, cheaper, and has fewer ways to fail.

Inspect the tool descriptions for ambiguity or overlap with other tools, check whether the system prompt and few-shot examples clearly disambiguate when each tool applies, and consider whether the tool set itself should be narrowed, renamed, or a routing step added to reduce the ambiguity the model faces at decision time.

Without safeguards, the model can alternate between two similar actions (e.g. searching, then searching again with a near-identical query) without recognizing it's stuck, burning cost and time with no progress. Prevent it with a step limit, tracking repeated or near-identical actions to force a different strategy, or a max-no-progress counter that triggers termination.

For example, a call to a flaky API could return an error page's HTML instead of expected JSON, or truncated/malformed data. Feeding that unvalidated into the model's context could cause it to misinterpret the error as valid data and reason incorrectly on top of it — validating the format and expected fields catches this before it corrupts the agent's state.

Within-run memory (a scratchpad or running state) only needs to survive one task's execution and can be ephemeral. Cross-session memory needs durable storage and raises additional questions about what's worth persisting, privacy, and staleness — conflating the two can lead to either losing useful within-task context or over-persisting irrelevant details long-term.

Cost scales roughly linearly (or worse) with the number of model calls in the loop, since each step is a full LLM invocation. Reducing it without hurting success usually means better upfront tool selection and prompting to need fewer steps, using a cheaper/faster model for simpler intermediate steps while reserving a stronger model for final synthesis, or caching repeated sub-results.

An agent can run to completion — hit a stopping condition, use all its allotted steps, return some final text — while still having failed the actual goal, e.g. via wrong tool calls, a wrong final answer, or giving up prematurely. Evaluation must check the actual task outcome against ground truth, not just that the loop terminated cleanly.