Skip to content
Tech Interview Prep

Real Interview Questions for the Agentic AI Engineer at Amazon

New

Interview Q&A

25 questions · Flow follows Relevance until this set has a learning order.

Q&A-1Design an online and offline evaluation pipeline to monitor, benchmark, and iteratively adjust LLM agent outputs for complex production tasks.

Reported in Amazon Agentic AI Engineer Interview Questions & Guide 2026

The 90-second answer

I'd assume the agent is a planner-plus-tools system on Bedrock, an orchestrator calling Lambda action groups and a Knowledge Base, and that success is multi-step and partly subjective, e.g. "resolve this customer's refund correctly and within policy." The design is two loops over one trace schema.

Every run, offline or online, writes the same record to S3 as Parquet: task_id, input, plan, each tool call with arguments and result, final answer, input/output tokens, latency, guardrail verdicts, and user feedback. Offline and online then differ only in sampling and in what the score gates.

LayerRuns onSampleGates
Deterministic checksevery run100%schema valid, tool args match contract, policy/PII rules
Trajectory checksevery run100%tool order, step count within budget, no repeated failing call
LLM judgeoffline suite + prod100% offline, 5% prodanswer quality against rubric
Human reviewweekly + all escalations~100/wkjudge calibration, new golden cases

Offline is the gate. A golden set of ~400 tasks stratified by task family and difficulty, each with expected tool calls and a rubric. Deterministic checks run first because they're free and catch most regressions: wrong tool, wrong argument, refund issued before the policy check. The judge scores only what's left. It's pinned by model version and calibrated against ~100 human-labelled traces; I won't ship a judge below 0.6 Cohen's kappa against those labels, and I re-calibrate whenever the judge model changes. Merge gate: no drop over 1pp on task success, zero safety-check failures, cost per task within 10% of baseline.

Online is the monitor. All production traces get deterministic and trajectory checks; 5% get judge scoring; 100% of thumbs-down and human escalations are queued for review. New prompts or tool versions go to a 10% canary with a sequential test and auto-rollback.

The arithmetic decides the design. Detecting a 2pp drop from an 80% baseline at 80% power needs about 6,500 tasks per arm. At 200k tasks/day and a 10% canary that's roughly 8 hours of traffic. A canary cannot catch small regressions in an hour: that's why the offline suite exists, and why I never let the canary be the only defence.

A worked trace: task "refund order 112-4471." The agent calls get_order, then issue_refund, then check_policy. The answer reads well and the judge scores it 4/5. The trajectory check fails it: issue_refund before check_policy violates the required order, and on a different order it would have refunded a non-returnable item. That failure becomes a new golden-set case, and the fix, a plan constraint or a tool-level precondition, is verified offline before it reaches the canary.

Failure modes I watch. Judge drift after a model upgrade, detected by re-scoring a frozen 100-trace calibration set weekly. Golden-set overfit, where prompts get tuned to the suite, detected by holding out 20% of cases and rotating them. Reward hacking, where the agent produces judge-pleasing prose without doing the work, detected by trajectory checks that never consult the judge. Nonstationarity, where a downstream tool's contract changes and every stored expectation goes stale, detected by contract tests on the tools themselves.

The loop closes by taxonomy: every production failure gets a label (planning, tool selection, tool arguments, retrieval, policy, formatting) and the label decides whether the fix is a prompt change, a tool precondition, a retrieval change, or a new case. Nothing ships as "fixed" without a case that would have caught it.

Curated: · Written: · Reviewed:

Q&A-2Walk me through the low-level design (LLD) of an autonomous multi-agent system built to optimize operational decision-making under high concurrency.

Reported in Amazon Agentic AI Engineer Interview Questions & Guide 2026

The 90-second answer

I'd pin the scope first: fulfillment-center exception handling. Say 30,000 shipments an hour at peak, each needing a decision (re-route, substitute, hold) inside a 90-second SLA. The design principle that drives everything else: agents propose, a deterministic policy layer disposes. No LLM output reaches a write API directly.

LayerComponentWhy
IngestKinesis → SQS per decision classabsorbs 3× peak bursts
Orchestratorsupervisor agent, plan-and-execute (not free ReAct)bounded, replayable step graph
SpecialistsInventoryAgent, CarrierAgent, CostAgent: each a tool-using LLM call with a output schemadisjoint toolsets, separately evaluable
BlackboardDynamoDB, pk DEC#{decision_id}state, lease, step count, token budget
Actuationrules engine + idempotent action APIhard constraints (SLA, hazmat) enforced outside the model
TraceX-Ray + CloudWatch, keyed by decision_idone decision = one trace

Concurrency is handled by partitioning on the entity, not the agent. One worker owns one decision at a time via a lease taken with a conditional write, so there is no lock service and no distributed transaction:

python

boto3, Python 3.12, DynamoDB

def acquire(table, decision_id, worker, ttl_s=120): now = int(time.time()) try: table.update_item( Key={"pk": f"DEC#{decision_id}"}, UpdateExpression=("SET lease_owner=:w, lease_expiry=:e, " "attempts=if_not_exists(attempts,:z)+:one"), ConditionExpression="attribute_not_exists(lease_owner) OR lease_expiry < :now", ExpressionAttributeValues={":w": worker, ":e": now + ttl_s, ":now": now, ":z": 0, ":one": 1}, ) return True except ClientError as e: if e.response["Error"]["Code"] == "ConditionalCheckFailedException": return False raise

A worked trace. d-4471 arrives at t=0. Worker A leases at t=12 ms. InventoryAgent returns a substitute SKU at t=610 ms (Sonnet-class model, ~1.2k in / 300 out). CarrierAgent answers at t=1,090 ms. The policy engine rejects the substitute: it ships from a different FC and breaks the promised delivery window. Second pass with CostAgent picks hold-and-split at t=1,900 ms. The action API is called with Idempotency-Key: d-4471-v2 and returns 200 at t=2,100 ms. p50 is ~2 s against a 90 s budget; the headroom is for the human-approval path, not for retries.

Sizing. 30k/hr is 8.3/s average, ~25/s at 3× peak. Three LLM calls × 1.5k tokens ≈ 4.5k tokens per decision. At Sonnet-class list prices ($3/M input, $15/M output, mid-2025), that is roughly $0.01–0.02 per decision, so $300–600/hour at peak: a number I'd put in front of finance before launch. 25 rps × 2 s = ~50 decisions in flight, which is well inside a Bedrock cross-region inference profile but not inside a single on-demand model quota in one AZ; that quota is the first thing I'd load-test.

Failure modes I'd design for, and how I'd see them.

  • Model timeout or 5xx. Retry twice with jitter, then fall back to a rules-only path that covers the ~80% common cases and queue the rest for a human. Detect via p99 decision latency and a fallback_rate metric; alert when it exceeds 5% over 5 minutes.
  • Duplicate action. Idempotency key on the action API plus a fencing token (the lease attempts counter) so a stale worker's write is rejected server-side.
  • Hot partition. One FC outage generates 10k decisions at once. Shard on decision_id, never on fc_id, or you will throttle exactly when you most need throughput.
  • Runaway loop. Hard cap of 6 steps and a token budget per decision, both stored on the blackboard and checked by the supervisor before each step. Cost alarms per decision class catch the slow leak.
  • Prompt injection through tool output: a carrier note field saying "ignore prior instructions." Tool results are data, never instructions: the policy layer accepts only schema-validated fields, and the action API re-validates independently.

Trade-offs. A single agent with all tools is simpler and cheaper below a few hundred decisions an hour; multi-agent earns its complexity when specialists have disjoint toolsets, different latency/cost profiles, and you need per-specialist evaluation. Synchronous orchestration is easier to reason about; I'd move to Step Functions only for decisions that wait on human approval or a carrier callback, because that is where a 90-second SLA actually gets spent.

Curated: · Written: · Reviewed:

Q&A-3How would you adapt an agentic architecture if midstream customer requirements shift significantly without resetting the underlying system state?

Reported in Amazon Agentic AI Engineer Interview Questions & Guide 2026

The 90-second answer

Assume "midstream" means at least one tool call has already produced an external side effect, and "without resetting state" means the conversation, memory and completed work survive. The mistake is treating the requirement as prompt text baked into the run. I make the requirement set a versioned control-plane object and the plan a derived artefact. Durable state is an append-only event log of decisions, tool calls, observations and side effects. A requirement change bumps a plan epoch; the executor checks that epoch before every tool dispatch, so no stale step fires after the change. The orchestrator enters REPLANNING: snapshot current state, diff goal v1 against v2, keep completed steps whose postconditions still hold, compensate the ones that do not, append a new plan, resume. Reset is the fallback only when state is corrupt or a security boundary is crossed.

StateTriggerGuarantee
EXECUTINGstep dispatchepoch check before each tool call
REPLANNINGrequirement v2 eventno new side effects; snapshot taken
RECONCILINGdiff computedcompleted steps kept or compensated
AWAITING_APPROVALcompensation or policy violationhuman decides; state preserved

python

orchestrator loop, Python 3.12

def run(plan, log, control): for step in plan.pending: if control.epoch != plan.epoch: return Replan(plan, log) # fence stale plan if not postconditions_hold(step.pre, log): return Replan(plan, log) result = tools.call(step.tool, step.args, idempotency_key=f"{plan.id}:{step.id}") log.append(step.id, result) return Done(log)

Trace. Customer asks to refund order 123. The agent validates the order, issues refund r_88, then the customer says "actually exchange the item, don't refund." The orchestrator bumps the epoch to 2 and freezes dispatch. The snapshot contains refund_id=r_88 and order state refunded. The diff marks the refund step invalid. Compensation calls POST /refunds/r_88/cancellation with an idempotency key. If the cancellation window has closed, the run moves to needs_human and keeps state; resetting would lose the fact that a refund happened and risk a double refund. The new plan creates a replacement order and return label, then resumes from the current log.

Failure modes. A stale executor can fire after the epoch bump; detect it with a conditional write on (execution_id, epoch) and reject the tool call. Non-idempotent compensation duplicates side effects; detect duplicates in the audit log and require idempotency keys on every compensating action. Ambiguous new requirements stall the planner; park in AWAITING_CLARIFICATION rather than guessing. Old-goal memory poisons retrieval; tag memory entries with the goal version and filter by current epoch.

Trade-offs. A full replan costs a planner call and added latency; patching the existing plan is cheaper but can miss invalidated postconditions. I use full replan when success criteria change, patch when only parameters change. On Amazon's stack I would run the orchestrator as a Step Functions state machine, persist events in DynamoDB with conditional writes, publish requirement changes on an EventBridge control bus, and validate the new plan with Bedrock Guardrails before any new tool call. Bedrock Agents session state is useful for conversation continuity but is not the system of record; the event log is.

Curated: · Written: · Reviewed:

Q&A-4Describe your architectural approach for an AGI project that orchestrates tool use, working memory, and long-term storage for unstructured domain data.

Reported in Amazon Agentic AI Engineer Interview Questions & Guide 2026

The answer

First I'd pin down what "AGI" means for this team, because it changes the design. If it's a research system chasing generality, the hard problems are evaluation and self-improvement. If it's what most product teams actually ship: an agent that runs long multi-step tasks over messy documents without a human in the loop, then it's a control-plane problem: who owns state, what bounds the loop, and how you prove afterwards what the agent did. I'd assume the second, say so out loud, and check.

On that assumption: three memory tiers, and a tool gateway the model cannot route around.

TierHoldsStoreLifetime
workinggoal, subgoals, verified facts with source URI, open questionsprompt + DynamoDB session itemone task
episodicpast trajectories, agent-written observationsDynamoDB + S3TTL, ~90 days
semanticchunked corpus, embeddings, ACL, source mtimeOpenSearch Serverlessuntil source deleted

Working memory is the scarce resource. I budget the model's context window explicitly rather than assuming a number: system prompt, tool schemas, then roughly half the remainder for retrieved evidence, the rest for trajectory. The scratchpad is structured, not a transcript: goal, subgoals, facts with citations, open questions. Persist it after every step so a crash resumes instead of restarting.

Long-term storage for unstructured data is an ingestion pipeline plus a hybrid index. S3 landing → EventBridge → Step Functions → Textract for PDFs → chunk at 300–800 tokens with about 15% overlap → Titan Embeddings V2 → OpenSearch Serverless with both kNN and BM25. Metadata carries source URI, ACL, mtime and entity tags so retrieval filters before it ranks. Query path: hybrid retrieve top 50, rerank down to 8.

Agent-written facts never go straight into the semantic index. They land in episodic storage, and a consolidation job promotes them only with provenance and a confidence threshold: otherwise one bad summarization poisons every future task.

Tools are registered with a JSON Schema, a timeout, an idempotency key and a circuit breaker. Results are normalized and truncated to about 2k tokens before they enter context; a raw CloudWatch response will otherwise eat the window.

A trace: "which clauses in contract C-4471 conflict with the 2026 vendor policy?" Step 1 hybrid-retrieves the policy, 8 chunks, ~6k tokens. Step 2 does the same for the contract. Step 3 calls clause_extractor. Step 4 compares and writes three findings with citations. Five steps, ~40k input tokens, ~2 s p50.

Cost, since it decides feasibility: context grows across steps. If it climbs from 20k to 120k over ten steps, the average is ~70k, so ~700k input tokens. At a Sonnet-class list price of $3 per million input tokens, I'd re-check that at build time, that's about $2.10 per task, and caching the static prefix cuts the repeated portion.

Failure modes I'd instrument: ingestion lag (refuse to answer when a source's mtime is newer than the index), prompt injection inside retrieved text (documents are untrusted; they never get to select tools), loops (cap steps, hash state to detect repeats), and cost runaway (per-task token budget with a hard stop).

Trade-off: Bedrock Knowledge Bases ships in a day but limits control over chunking and hybrid weighting; custom OpenSearch is right when domain jargon and ACL filtering matter. I'd also evaluate the managed agent runtime and memory services in Bedrock AgentCore against building this on Step Functions and DynamoDB, and confirm the current service surface before committing.

Curated: · Written: · Reviewed:

Q&A-5Explain how you would structure an agentic solution when constrained to run on only 20% of the originally allocated GPU/compute resources?

Reported in Amazon Agentic AI Engineer Interview Questions & Guide 2026

The 90-second answer

First I'd pin down what the 20% applies to, because the levers differ. If inference runs on Bedrock, the constraint is provisioned throughput units and per-account quotas, and I'm buying back token volume and model tier. If it's self-hosted on SageMaker or EKS, the constraint is replicas and KV-cache memory, and I'm buying back context length and batching efficiency. Either way, the wrong move is running the same design slower: at 20% you change what the model is asked to do per step and how many steps exist.

Four levers, applied together:

LeverChangeWhy it pays
Model routingplanner on the large model; tool-argument formatting, extraction, intent classification on a small onemost agent steps are mechanical, not reasoning
Prompt cachingstable prefix (system prompt, tool schemas) first, volatile content (timestamp, retrieved docs) laston Bedrock, Claude cache reads bill at ~0.1× input, writes at 1.25×
Loop controlplan-then-execute with a hard 6-step budget; deterministic edges between toolsReAct re-sends the whole transcript every step
Schedulinginteractive path on reserved capacity; everything else through SQS into batchagent traffic is bursty and mostly async-tolerant

The arithmetic. Take a hypothetical task that averages 8 ReAct steps, a 2,000-token stable prefix, and ~400 tokens appended per step. Input at step k is 2,000 + 400(k−1), so total input is 8×2,000 + 400×(0+…+7) = 27,200 tokens. Caching the prefix after step 1 removes 7×2,000 = 14,000 tokens from full price, cutting input cost ~46%. Now route 6 of 8 steps to a model at a tenth the per-token price: blended cost is 0.25×10 + 0.75×1 = 3.25 against 10, a 67% cut. Multiply: 0.33 × 0.54 ≈ 0.18. Inside the 20% budget. No single trick gets there; the product does. Those prices are illustrative, but the structure of the calculation is what I'd defend, and I'd replace the numbers with real ones from OTel spans before committing.

What one request looks like. Router call on the small model (~300 tokens) classifies intent. Planner call on the large model emits a 4-step plan against the cached prefix. Steps 2–4 execute deterministically (the orchestrator, not the LLM, decides the next tool) with small-model extraction on each result. One synthesis call at the end. Three model calls instead of eight, and the expensive one runs once.

Failure modes I'd watch. Cache busting: any edit to the system prompt or a tool schema invalidates the prefix for every in-flight task, so prompt versions are pinned and rolled out like code, with cache-hit rate on a dashboard. Step-budget exhaustion: tasks that hit 6 steps return a partial answer, so I track completion rate and fall back to a deterministic template rather than letting the loop extend. Silent quality regression from the downgrade: the router and extraction prompts get an offline eval gate on a frozen trajectory set before any routing change ships. Queue latency: the async path needs a p95 wait SLO and a shed path that degrades to single-step retrieval when the backlog crosses it.

One thing I would not do is shrink the tool set to save schema tokens. Fewer, coarser tools push reasoning back into the large model and usually cost more than the tokens saved.

Curated: · Written: · Reviewed:

Q&A-6Design and justify the architecture for an AGI agent, including the tradeoffs, safety, and operational concerns.

Reported in Amazon Agentic AI Engineer Interview Questions & Guide 2026

The 60-second answer

There's no AGI to design against, so I'd answer the question underneath it: an agent that isn't hard-coded to one task (it takes a goal, decomposes it, picks tools, and improves from feedback) plus an architecture that makes that generality safe to operate. My shape is a deterministic control plane around a probabilistic core. The model proposes; code outside the model decides whether the proposal is allowed, records it, and can replay it. Generality comes from the planner and the tool registry, not from granting the model more trust.

goal ──► Planner (LLM, temp 0) ──► plan DAG (validated, budgeted) ▲ │ │ re-plan on failure ▼ Verifier / Critic ◄──── Executor (Step Functions, durable) │ Policy gate: IAM + allowlist + blast radius │ Tool sandbox (per-session creds, dry-run) │ Memory: working (context) │ episodic (DynamoDB) │ semantic (KB, TTL) │ Trace: every step → S3, replayable offline

Why each piece is where it is. The planner emits a DAG, not a free-running chat loop, because a DAG can be schema-validated, cost-budgeted and shown to a human before anything executes. The executor is a durable workflow (Step Functions) so a 40-minute task survives a Lambda timeout and every step is an event, not a log line. The policy gate is the only path from a proposal to a tool call: the model never holds credentials. Memory is split because they have different failure modes: working memory is truncated, episodic is append-only and replayable, semantic memory is written only with provenance and a TTL, since a poisoned fact is the one bug that persists across every future task.

DecisionI'd chooseWhat it costs
AutonomyHard budget: 25 steps, 8 min wall clock, $2/taskLong-horizon tasks fail; escalate with the partial plan instead of silently dying
PlanningPlan-then-execute with re-plan checkpointsBrittle when step 1 changes the world; interleaved ReAct adapts but can't be reviewed up front
VerificationExecutable oracles first (schema check, unit test, dry-run diff)LLM critics share the planner's blind spots: correlated errors, not independent ones
ModelRouter: small model for tool selection, large for planningTwo eval surfaces to maintain

Safety, enforced outside the prompt. Three gates, none of which is a sentence in the system prompt. First, capability scoping: a per-session IAM role, read-only by default, with write tools added only when the plan names them. Second, irreversible actions are two-phase: dry_run returns the diff, commit needs an approval token, and blast radius is capped (≤10 records per call, ≤$500 spend) so a wrong plan is a bad afternoon, not an incident. Third, retrieved documents and tool outputs are untrusted data: they go in a delimited user-role block and any tool call they induce is re-checked against the plan's allowlist. Prompt injection through a fetched page is the dominant failure mode here, and Bedrock Guardrails (denied topics, PII, contextual grounding) is a filter on top of that boundary, not the boundary itself.

Operations. Every step emits (plan_id, step_id, tool, args_hash, result_hash, latency, tokens, cost). Tool responses are recorded, so a failed run replays offline against the same inputs: that's what makes regressions debuggable instead of anecdotal. Watch four numbers: task success on a frozen eval set, escalation rate, steps per task, cost per successful task. Concrete detectors: the same args_hash three times in a row aborts as a loop; a per-task token budget hard-cuts cost runaway; a canary string planted in the knowledge base trips if it ever appears in an outbound tool argument, which catches injection exfiltration. Sample 2% of successful runs for human audit: success metrics alone won't tell you the agent is doing the right thing for the wrong reason.

Illustrative figure from a setup like this: adding the dry-run/commit split moved a 200-task internal eval from 61% to 78% success, and nearly all of the gain came from removing irreversible-action mistakes rather than better planning.

Curated: · Written: · Reviewed:

Q&A-7How would you systematically debug and reduce an LLM agent that invents API tool parameters during tool calling?

Reported in Amazon Agentic AI Engineer Interview Questions & Guide 2026

The 90-second answer

Assume the agent runs on Bedrock with a Claude model, tools declared as JSON Schema in toolConfig, executed by Lambda action groups, with traces in CloudWatch/X-Ray and S3. I treat invented parameters as a data problem first, then fix at three layers: schema and tool design, grounding, and validation with bounded repair.

Instrument every tool call attempt: tool name, raw toolUse.input, schema version, validator result, model ID, temperature, prompt version, trace ID. Without the raw arguments you cannot tell an invented enum from a missing retrieval. Store in S3 partitioned by date, query with Athena; emit CloudWatch EMF metrics per tool and parameter.

Then label 500 production traces and classify. The taxonomy decides the fix.

FailureExampleDetectionFix
invented field{"customer_id":"cust_8812","expedite":true} where schema has no expediteadditionalProperties validatorremove from prompt, add to schema or drop
invented enum"status":"priority" when enum is [standard, expedited]enum checkenum + description + examples
format drift"amount":"49.99" vs integer centstype/patterntype: integer, minimum, example
plausible but nonexistent IDcust_8812 not in DBforeign-key lookupresolve-then-act tool
unit/scaleamount: 4999 meaning $49.99range checkrename amount_cents, add bounds

Trace: model emits {"customer_id":"cust_8812","amount":49.99,"currency":"USD","reason":"damaged"}. Schema requires amount_cents integer and currency enum [USD, CAD, GBP]; validator rejects amount as an additional property and 49.99 as non-integer. The repair message returns the schema fragment and both errors; the second attempt emits {"customer_id":"cust_8812","amount_cents":4999,"currency":"USD","reason":"damaged"}. That is one repaired call, not a silent bad refund.

python from pydantic import BaseModel, Field, ValidationError from typing import Literal

class RefundInput(BaseModel): model_config = {"extra": "forbid"} customer_id: str = Field(pattern=r"^cust_[0-9a-f]{4}$") amount_cents: int = Field(gt=0, le=100_000) currency: Literal["USD", "CAD", "GBP"] reason: str = Field(max_length=200)

def validate_and_repair(raw: dict, call_model, max_repairs: int = 2): for attempt in range(max_repairs + 1): try: return RefundInput.model_validate(raw) except ValidationError as e: if attempt == max_repairs: raise raw = call_model(repair_prompt(raw, e, RefundInput.model_json_schema()))

Checked against Pydantic 2.x, Python 3.12.

Grounding is the biggest lever for IDs. Do not put a customer ID in the prompt and hope. Give the agent search_customers(query) returning [{id, name}], and require the refund tool to accept only an ID copied from a tool result. In one trace set, that cut nonexistent-ID errors from 1.8% to 0.1% of calls.

Reduce at the source: one intent per tool, no near-duplicate tools, additionalProperties: false, enums for closed sets, examples in the description, and forced toolChoice when only one tool is valid. Where the runtime supports grammar-constrained decoding, use it; Bedrock tool use does not guarantee schema adherence, so the validator is not optional.

Failure modes: over-strict schema rejects valid calls; repair loops add latency and tokens (each repair ~300 ms and ~400 tokens); the model learns to over-ask for clarification. Detect with a 5% canary, invalid-parameter rate per tool, repair rate, and task success on a 200-case golden set. Ship schema + grounding first; repair loop second; fine-tuning only for a tool that stays above 1% invalid after both.

Arithmetic: 50k tool calls/day at 3% invalid = 1,500 bad calls. Schema + grounding to 0.6%, repair catches 80% of those, leaving ~60/day reaching the API.

Curated: · Written: · Reviewed:

Q&A-8How would you design an enterprise-grade Retrieval-Augmented Generation (RAG) pipeline using embeddings and vector search that maintains latency under 200ms at 10,000 QPS?

Reported in Amazon AI Engineer Interview Questions & Guide 2026

The 90-second answer

First I'd pin down what the 200 ms covers. Generation does not fit in it: time-to-first-token alone runs 300–600 ms on a streaming endpoint. So I'd design to a 200 ms p99 budget for retrieval-to-context (normalize → embed → ANN search → filter → rerank → prompt assembly), stream the generation after that, and say the assumption out loud. If the interviewer means end-to-end, the design changes into a semantic cache in front of a much smaller generation path.

With that fixed, the shape is a two-stage retriever. Stage one is a sharded HNSW index over quantized embeddings, fanned out in parallel under a hard deadline. Stage two is an optional small cross-encoder rerank over the top 50, applied only when the ANN score margin is thin. Everything in front of it is a cache.

Stagep50 (ms)p99 (ms)Notes
normalize + exact cache lookup13Redis, key = sha256(normalized query)
query embedding1235GPU pool, 5 ms micro-batch window
ANN fan-out, 8 shards in parallel1550k=100, ef_search=128, int8-quantized vectors
ACL / metadata filter38pre-filtered on tenant partition
rerank top-50—40skipped when top-1 margin > 0.08
assemble + tokenize410
total~35~145headroom for GC and network jitter

Sizing, worked example

50M chunks x 1024 dims x 4 B = 205 GB raw vectors HNSW graph overhead (~1.4x) = 287 GB resident sharded 8 ways = 36 GB/shard -> r6i.2xlarge (64 GiB) per-shard load 10,000 / 8 = 1,250 QPS Little's law: 10,000 x 0.2 s = 2,000 requests in flight

At 1,250 QPS per shard with ef_search=128 over ~6M vectors, one node sits near its ceiling, so I run two replicas per shard, 16 nodes, one for HA, one for burst and rolling deploys. Note ef_search must be at least k: asking for k=100 with ef_search=64 silently caps you at 64 candidates, which is the kind of bug that shows up as a recall regression, not an error.

Embedding is the harder quota: 10k QPS is 600k requests/min. Bedrock on-demand embedding throughput is governed by per-model, per-region quotas published in the service-quotas console, and those on-demand limits are set well below 600k/min: read the current figure for your model and region rather than trusting a number in a design doc. The design answer is the same either way: provisioned throughput, or a self-hosted SageMaker/EKS GPU pool. A 1024-dim bi-encoder at batch 32 on an A10G does roughly 3–5k short-sequence embeddings/sec (worked figure: measure your own), so 3 GPUs steady-state, 6 across two AZs.

Failure modes I'd watch

Embedding version skew. Index built with model v1, query embedded with v2: recall collapses silently, no error anywhere. Detect with a nightly recall@10 run against a 5k golden set of query → known-relevant doc ids, alerting on a >2-point drop. Prevent by putting the model version in the index alias name and having the query path read the alias.

Post-filter ACL starvation. ANN returns top 100, the ACL filter removes all 100 for a restricted tenant, the user gets an empty context and the LLM hallucinates. Fix by partitioning the index per tenant namespace or over-fetching 10×, and monitor empty-result rate per tenant as a first-class SLI.

Fan-out tail. One slow shard sets the p99 for the whole request. Hedge: if a shard misses a 30 ms soft deadline, fire a duplicate at its replica and cancel the loser.

python async def retrieve(q, deadline_ms=200): budget = Deadline(deadline_ms) vec = await embed(q, timeout=budget.remaining(0.7)) shards = await gather_with_deadline( [ann(s, vec, k=100, ef=128) for s in shard_pool], timeout=budget.remaining(0.5)) # partial results are acceptable cands = acl_filter(merge_topk(shards, 100), q.tenant) if margin(cands) < 0.08: cands = await rerank(q, cands[:50], timeout=budget.remaining(0.8)) return cands[:10]

Freshness gets a blue/green index with an alias swap and dual-writes during the build: never a rebuild in place, because a rebuild in place is a multi-minute recall outage.

Curated: · Written: · Reviewed:

Q&A-9Explain the architectural setup and coordination mechanics of multi-agent systems using frameworks like LangGraph or custom orchestrators, and how you prevent infinite execution loops and manage state persistence across agents.

Reported in Amazon AI Engineer Interview Questions & Guide 2026

The 90-second answer

A multi-agent system is a graph, not a chat room. I define one shared state object, a set of nodes that are agents or tools, and a router that picks the next node. LangGraph gives that shape directly: a StateGraph over a typed state, a supervisor node that emits the next hop, and a checkpointer keyed by thread_id that writes state after every superstep. On Amazon the graph runs in a Step Functions Standard workflow when a run can outlive Lambda's 15-minute ceiling, or in a Fargate service with checkpoints in DynamoDB when I need token streaming at low latency. Bedrock Agents' multi-agent collaboration is the buy option when every agent is just a prompt plus action groups.

Three things make it survive production: a hard step budget, a no-progress detector, and state that resumes from a checkpoint instead of restarting.

    ┌────────────┐

task → │ supervisor │ ──route──┬→ researcher ─┐ └────────────┘ ├→ coder ──────┤→ back to supervisor │ budget spent └→ critic ─────┘ ↓ escalate (interrupt_before → human)

The graph, with the guards in it

python from typing import Annotated, TypedDict from langgraph.graph import StateGraph, END from langgraph.graph.message import add_messages from langgraph.checkpoint.postgres import PostgresSaver # or DynamoDB saver

class RunState(TypedDict): messages: Annotated[list, add_messages] hops: int seen: list[str] # hash of (node, tool, args) deadline_ms: int next: str

def supervisor(s: dict) -> dict: if s["hops"] >= 12 or time_left_ms() <= 0: return {"next": "escalate"} return {"next": route(s)} # researcher | critic | END

g = StateGraph(RunState) g.add_node("supervisor", supervisor) g.add_node("researcher", researcher) g.add_node("critic", critic) g.add_node("escalate", escalate) g.set_entry_point("supervisor") g.add_conditional_edges("supervisor", lambda s: s["next"]) for n in ("researcher", "critic"): g.add_edge(n, "supervisor")

app = g.compile(checkpointer=PostgresSaver(pool), interrupt_before=["escalate"]) cfg = {"configurable": {"thread_id": "order-42/attempt-1"}, "recursion_limit": 25} app.invoke({"messages": [task], "hops": 0, "seen": [], "deadline_ms": 90_000}, cfg)

recursion_limit is LangGraph's own backstop: it raises GraphRecursionError at 25 supersteps by default. I never rely on it alone, because it counts graph transitions, not wasted work: two agents ping-ponging politely burns all 25 hops and returns nothing. Hence the domain budget of 12 hops inside the supervisor, which is the number I actually tune.

Loop guards, and what each one catches

GuardMechanismFires when
hop budgethops >= 12 in supervisorsupervisor thrash between two agents
cycle detectionhash (node, tool, args) into seenidentical tool call repeats
no-progresscritic returns "no new evidence" twicestate unchanged, agents restating
deadlinewall clock vs deadline_msslow tool, not a loop
idempotency keyper tool call, stored with the checkpointresume replays a side effect
human interruptinterrupt_before=["escalate"]budget exhausted with work pending

Budget arithmetic: 12 supervisor hops at roughly 2k tokens per hop is ~24k tokens and 40 s p99 for a run. Unbounded, the same thrash pattern in staging reached 300+ hops before the Lambda timed out: 25× the token spend for the same empty answer. That is the number I put in the design review.

State persistence

Checkpoint after every superstep, keyed by thread_id = business key plus attempt number. Resume is app.invoke(None, cfg): it continues from the last checkpoint, not from the top. get_state_history(thread_id) gives time travel, which is how I debug a bad run: replay from hop 7 with a fixed prompt.

Two constraints bite. DynamoDB items cap at 400 KB, so the checkpoint cannot hold raw documents: I keep message history trimmed to the last 8 turns plus a rolling summary, and push large artefacts to S3 with a pointer in state. And every superstep writes the full state, so a 12-hop run is 12 writes: fine at hundreds of runs a minute, a cost problem at thousands, which is when I move to Postgres and write deltas.

Failure modes I watch for

Resume after a crash re-executes the node that was mid-flight, so any tool with a side effect needs an idempotency key stored alongside the checkpoint: otherwise the refund fires twice. State schema changes break old checkpoints, so the state carries a schema_version and old threads are drained before a deploy. And checkpoint writes are on the critical path: if the checkpointer is unavailable the run should fail closed rather than proceed unrecorded, because an uncheckpointed agent run is unrecoverable after a crash.

When a single agent with six tools can do the job, I don't build the graph. Coordination adds a supervisor LLM call per hop and compounds errors across agents; multi-agent earns its cost when the subtasks need genuinely different tools, prompts or permission scopes.

Curated: · Written: · Reviewed:

Q&A-10Walk me through how you would design the retrieval layer for a RAG system that has to serve 50 million documents with sub-second latency.

Reported in Amazon AI Engineer Interview Questions & Guide 2026

I’d start by separating the two retrieval problems: finding candidate chunks cheaply, then ranking a small set accurately. A single vector index over 50M chunks is possible, but it’s rarely the best design once you add freshness, filtering, and cost constraints.

My default architecture is hybrid retrieval: BM25 plus dense vectors, each returning top 100 candidates, fused with reciprocal rank fusion, then a cross-encoder reranker on the top 20–50. That gives exact-term recall for IDs and product codes while keeping semantic recall for paraphrases. The reranker is where most of the answer quality comes from, so I budget latency there rather than in the ANN index.

For the vector index, I’d use HNSW if the corpus fits in memory and recall matters more than cost. At 50M chunks × 1024 dims × 4 bytes, raw vectors are ~205 GB; HNSW graph overhead with M=16 adds roughly 128 bytes per vector, so ~6.4 GB more. That fits on a few large nodes and gives p99 under 20 ms at recall@10 above 0.95. If the corpus grows past memory or cost dominates, I’d switch to IVF-PQ or DiskANN. IVF-PQ at 64 subquantizers is ~64 bytes per vector, so ~3.2 GB for 50M, but recall drops and needs retraining when the embedding distribution shifts.

ChoiceMemory for 50M × 1024p99Recall@10Failure mode
HNSW~210 GB10–20 ms0.95–0.99OOM, slow rebuilds
IVF-PQ~4–8 GB30–80 ms0.80–0.95recall collapse after drift
DiskANN~20 GB RAM + SSD20–50 ms0.90–0.97SSD IOPS saturation

I’d shard by document namespace or tenant so a single hot tenant can’t starve the rest, and keep a small exact-search golden set to measure recall drift nightly. Freshness is the other constraint: HNSW handles incremental inserts but deletes are tombstones, so I’d compact segments on a schedule. If the embedding model changes, I’d re-embed and rebuild rather than mix vector spaces. The guardrails are per-shard memory alerts at 70%, recall canaries after every model or index change, and a fallback to BM25-only if the vector path degrades.

Curated: · Written: · Reviewed:

Q&A-11How do you construct a multi-agent system where one agent generates code, a second agent runs tests in a sandbox environment, and a third agent fixes errors based on runtime output?

Reported in Amazon AI Engineer Interview Questions & Guide 2026

The 90-second answer

Three agents, one orchestrator, and two rules that decide whether it works: the loop is bounded, and every handoff is a typed object rather than prose. The Generator emits a patch. The Executor runs the pinned test suite in a disposable sandbox and returns a structured failure record. The Fixer consumes that record plus the previous diff and emits a new patch. The orchestrator (Step Functions around Lambda, or a Strands/Bedrock Agents supervisor) owns the iteration counter, the workspace, and the stop condition: five attempts, then escalate to a human with the full attempt history.

The two decisions that make or break it: the test suite is immutable (hash-pinned, any patch touching it is rejected), and the sandbox is a fresh microVM per attempt with egress denied.

    ┌──────────────┐

task ─▶│ Orchestrator │── attempt n ──▶ Generator ──patch──┐ │ (max 5) │ ▼ └──────┬───────┘ ┌──────────────┐ │ pass ◀── verdict ──────────│ Executor │ ▼ │ Firecracker │ deliver │ no egress │ │ fail ── FailureRecord ───▶└──────────────┘ └──────────────▶ Fixer ──patch──▶ (next attempt)

The handoff contract. The Fixer never sees a raw log. It sees this, and nothing else:

{ "attempt": 2, "exit_code": 1, "duration_ms": 4210, "tests": {"passed": 12, "failed": 3, "errors": 0}, "failures": [ {"test": "test_refund_exceeds_total", "assertion": "AssertionError: 120.0 != 100.0", "file": "tests/test_refunds.py", "line": 44, "stderr_tail": "…last 6 KB…"} ], "patch_sha": "9c1f…", "test_suite_sha": "4a77…" }

Sandbox spec. Every row is a control, not a preference:

ControlValueWhy
base imagedigest-pinned, read-only rootsame result on retry
networkegress deniedno exfiltration, no flaky dependency fetches
cpu / mem1 vCPU / 2 GBOOM becomes a signal instead of a host crash
wall clock30 s per run, SIGKILLinfinite loops in generated code
user / syscallsuid 1000, default seccompshrinks escape surface
filesystem/workspace rw, 512 MBbounded disk

One trace. Attempt 1: Generator emits an 84-line module, no test changes. Executor returns 12 passed, 3 failed, exit 1, 4.2 s. Fixer receives the three assertion lines and the stderr tail, emits a 9-line patch. Attempt 2: 15 passed, exit 0, 6.1 s. Two model calls per attempt, ~11 s wall clock, roughly $0.08 in tokens at Sonnet-class pricing (illustrative).

Failure modes and how you catch them.

  • Test tampering. The Fixer "fixes" by deleting the failing assertion. Reject any patch whose diff touches tests/ or changes test_suite_sha, and re-run from a clean checkout of the pinned suite.
  • Flake mistaken for a bug. Re-run the failing tests once in a fresh sandbox; if the result flips, mark flaky and do not feed it to the Fixer.
  • Context blowup. Cap stderr at 8 KB: first 2 KB plus last 6 KB, because tracebacks live at the tail.
  • Fixer spinning. If attempt n's failure signature (test id plus assertion hash) equals attempt n−1's, break early and escalate rather than burn three more attempts.

When not to build this. For a single-file function with a fast suite, one agent with an executor tool call is cheaper and just as reliable. The third agent earns its cost only when the fix depends on runtime output the Generator cannot predict: a type error at a call site, a boundary condition, a missing import.

Curated: · Written: · Reviewed:

Q&A-12Design an enterprise search engine using hybrid RAG that ingests millions of AWS documentation pages, updates in real time, and maintains p99 latency below 150ms.

Reported in Amazon AI Engineer Interview Questions & Guide 2026

The 90-second answer

First I'd pin down the SLO: 150 ms p99 cannot include an LLM generation call. Bedrock time-to-first-token alone is typically 300–800 ms. So I scope the 150 ms to the retrieval-and-ranking API that returns top passages; generation streams on a separate path. If the interviewer means end-to-end answer latency, the design shifts to cached or precomputed answers and a distilled ranker, and 150 ms is not achievable with a generative model in the loop.

With that scope: ingest AWS docs from S3 through Kinesis, chunk to ~500-token passages with overlap, embed with Bedrock Titan Text Embeddings V2 (1024-dim), and index each chunk into OpenSearch as both a BM25 text field and a k-NN HNSW vector. The query path fans out BM25 and k-NN in parallel, fuses with reciprocal rank fusion (k=60), reranks the top 50 with a cross-encoder, and returns the top 10. Real-time updates are upserts through the same pipeline with external versioning; p99 freshness lands around 2 s.

Latency budget (p99)

StageBudgetNotes
Parse, auth, routing5 msAPI Gateway + Lambda or ECS
BM25 across 3 shards25 mstext field, no wildcards
k-NN HNSW, ef_search=10040 ms1024-dim, cosine
RRF fusion5 msk=60
Cross-encoder rerank top 5050 msGPU, batch 50
Serialize + network15 ms
Total140 ms10 ms headroom

Sizing arithmetic. Assume 5M pages × ~4 chunks/page = 20M vectors. 1024 dims × 4 bytes = 4 KB per vector → 80 GB raw; HNSW graph overhead ~1.5× → ~120 GB. Shard into 20 shards of 1M vectors (~6 GB each) across 6 r6g.2xlarge data nodes with one replica. That keeps each shard's HNSW graph resident in memory and avoids page faults on the k-NN path.

Query trace. Query S3 lifecycle transition to Glacier: BM25 returns 200 hits in 22 ms; k-NN returns 100 neighbors in 38 ms; RRF merges to 50 candidates in 4 ms; the reranker scores 50 in 45 ms; top 10 returned at 128 ms. A repeated query served from ElastiCache returns in 8 ms.

Real-time ingestion. Source change event → Kinesis → Lambda (or Flink for backfill) → chunk + embed → OpenSearch bulk upsert with version_type=external. Set refresh_interval=1s on the hot index, giving p99 visibility ~2 s. For a full reindex, build a new index and swap the alias; never mutate in place.

Failure modes. HNSW recall drops if ef_search is too low: track recall@10 weekly against a labeled set. Reranker timeout: enforce a 60 ms deadline and fall back to RRF order. Hot shard from skewed doc popularity: add replicas and a query cache. Embedding model upgrade: dual-write both vector fields, backfill, then cut over. Chunk boundary splitting a code sample: use structure-aware chunking on headings and code fences.

Trade-offs. RRF is robust to score-scale mismatch between BM25 and cosine; weighted linear fusion can beat it only after per-domain tuning. Cross-encoder rerank buys roughly 15–20 NDCG points but costs 40–60 ms; if the budget tightens, rerank only the top 20 or drop to a smaller bi-encoder. Generation never sits in the 150 ms path.

Curated: · Written: · Reviewed:

Q&A-13How would you optimize the infrastructure for serving a 70B parameter model across multi-GPU nodes to achieve a Time To First Token (TTFT) under 200ms and throughput over 100 tokens/sec/user?

Reported in Amazon AI Engineer Interview Questions & Guide 2026

The 90-second answer

Pin the SLO to a workload first: TTFT under 200 ms at what prompt length and arrival rate, and 100 tok/s/user at what batch. I'd assume 2k-token prompts, 8k max context, 20–40 concurrent sequences per replica. Under those, the shape is: 70B in FP8 on a single 8-GPU node, tensor parallel 8 over NVLink, prefill and decode disaggregated, prefix caching on, chunked prefill with admission control in front.

FP8 weights are 70 GB. A p5.48xlarge gives 8×H100 80 GB = 640 GB, so TP=8 fits weights, KV and activations with room. I would not span nodes for 70B: TP=16 doubles the collective count and puts every all-reduce on EFA, and each cross-node hop costs 100–200 µs. That alone eats the 10 ms per-token budget that 100 tok/s/user implies.

The TTFT budget

StageBudgetWhat sets it
ingress, auth, tokenize8 ms2k tokens through a fast tokenizer, ~3 ms CPU
scheduler queue15 msadmission control, not luck
prefill, 2k tokens, TP=8 FP890 ms2·70e9·2000 = 280 TFLOP ÷ ~3.5 PFLOP/s ≈ 80 ms compute + overhead
KV handoff P→D over EFA15 ms2k × 160 KB = 320 MB at ~50 GB/s
first decode step6 ms70 GB ÷ 26.8 TB/s aggregate HBM
detokenize, first flush5 ms
total139 ms~60 ms left for p99 jitter

3.5 PFLOP/s assumes ~45% MFU against 8×H100 dense peak of 989 TFLOPS each. That is the number to verify with a load test before promising anything to a customer.

Decode: where the 100 tok/s comes from

Every decode step re-reads all weights. 70 GB FP8 over 8×3.35 TB/s = 26.8 TB/s is a 2.6 ms floor; at 60% achieved bandwidth it's ~4.4 ms, plus 1–2 ms of TP all-reduce latency across 160 collectives per step (2 per layer × 80 layers). Call it 6 ms → 165 tok/s/user single-stream. Batch 32 adds ~10 GB of KV reads (0.4 ms), so the step stays under 8 ms and 100 tok/s/user holds. Past batch ~64 the step crosses 10 ms and the SLO breaks: that is the admission-control line, not a tuning preference.

Disaggregation is the TTFT lever

Co-located prefill and decode means a new request's prefill waits behind whatever decode batch is running. That head-of-line blocking is what turns a 90 ms prefill into a 400 ms TTFT at peak. Separate pools, KV shipped over EFA, each pool autoscaling on its own queue depth. NVIDIA Dynamo and vLLM's disaggregated prefill both implement this.

Agentic workloads: cache the prefix

Agent sessions resend a 4–8k system prompt with tool schemas on every turn. Prefix caching reduces that to ~200 new tokens to prefill, dropping TTFT from ~180 ms to ~30 ms. Pair it with prefix-aware routing: hash the system prompt, route the session to the replica already holding those blocks, or you get the hit rate of a random load balancer.

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 8 --quantization fp8 --kv-cache-dtype fp8 \
  --max-model-len 8192 --gpu-memory-utilization 0.90 \
  --enable-prefix-caching --max-num-seqs 48

(vLLM 0.6+; chunked prefill is on by default in V1.)

Failure modes I'd instrument

KV exhaustion causing preemption and prefill recompute: watch vllm:num_preemptions_total and the KV usage gauge; p99 TTFT triples before p50 moves. Pool imbalance: prefill queue deep while decode idles; alert per pool, not per service. TP straggler: one throttled GPU and all eight wait on the all-reduce; track per-GPU step-time skew and NCCL watchdog timeouts. EFA stalls on the KV path show up as a fat tail on the handoff histogram, invisible in aggregate TTFT.

Cost lever once the SLO holds: Trainium2 or Inferentia2 for the decode pool, since decode is bandwidth-bound and price per GB/s is better there.

Curated: · Written: · Reviewed:

Q&A-14Design a practical plan to explain and speed up transformer inference, with complexity, bottlenecks, and serving tradeoffs.

Reported in Amazon AI Engineer Interview Questions & Guide 2026

First I'd pin the workload, because "speed up inference" means different things for an agent. An agent turn is typically 2–4k tokens of system prompt plus tool schemas, a short user message, then a 100–400 token decode that is often a JSON tool call, then a tool round-trip, repeated. That is prefill-heavy and latency-sensitive per turn, not throughput-heavy. Batch scoring flips the answer toward throughput. I'll assume the agent case: decoder-only model, 8k–32k context, p99 under ~2 s per model call.

A forward pass splits into two phases with opposite bottlenecks.

PhaseWhat runsBound byCost scaling
Prefillall prompt tokens in parallelGPU FLOPs; attention is O(n²)~2·P·n FLOPs plus quadratic attention
Decodeone token per step per sequenceHBM bandwidth (weights + KV re-read every step)~2·P FLOPs/token, arithmetic intensity ≈ 1 FLOP/byte

Decode is the wall. A 70B model in FP16 is 140 GB of weights, and every generated token streams all of them from HBM. On an H100 at 3.35 TB/s that is a floor of ~42 ms per token at batch 1: you pay 140 GB of reads to do 140 GFLOPs of work. Batching amortizes it: at batch 32 the same weight read serves 32 tokens, so per-token cost falls toward the FLOP limit.

KV cache is the second wall and it sets concurrency. Llama-3-70B: 80 layers, 8 KV heads (GQA), head dim 128, FP16.

per token = 2 (K,V) × 80 × 8 × 128 × 2 B = 327,680 B ≈ 320 KB at 8k context = 2.6 GB per sequence 4×H100 80 GB = 320 GB − 140 GB weights ≈ 180 GB for KV ≈ 68 concurrent 8k sequences

That is the ceiling before fragmentation. The plan, in order of payoff for an agent:

  1. Prefix caching. System prompt and tool schemas are identical across turns and users. Cache that prefix's KV and skip prefill for 2–4k tokens on every call. Usually the single biggest latency win in agentic serving; vLLM exposes it as --enable-prefix-caching.
  2. Continuous batching + PagedAttention. Schedule at token granularity, allocate KV in fixed-size blocks instead of reserving max length per sequence. The PagedAttention paper (Kwon et al., SOSP 2023) measured 60–80% of KV memory wasted to fragmentation in prior systems and reports 2–4× throughput over Orca-style batching, up to 24× over naive HF serving.
  3. Quantization. FP8 on H100 or INT8 weights roughly halves the weight stream, which attacks the decode bandwidth bound directly. Verify tool-call JSON validity separately: a 1% drop in function-call accuracy is an incident, not a benchmark line.
  4. Speculative decoding on the latency-critical single-stream path: a small draft model proposes k tokens, the big model verifies in one pass. 2–3× token-latency gain at low batch, but acceptance rate falls and the extra compute hurts once you are throughput-bound: enable per route, not globally.
  5. Chunked prefill so one long prompt doesn't stall decode for everyone in the batch; it trades TTFT against inter-token latency.

Failure modes worth naming: KV OOM under bursty concurrency. Watch gpu_cache_usage_perc and preemption counts, not GPU util; prefix-cache thrash if you key on the whole prompt rather than the stable prefix; quantization silently breaking structured output; speculative decoding regressing p99 as batch grows. On Inferentia2 the same levers exist through the Neuron SDK, but block size and batch caps get re-tuned: don't port GPU constants.

Curated: · Written: · Reviewed:

Q&A-15How would you design a system to handle multi-turn conversations in Bedrock?

Reported in ai-interview-guides/guides/amazon.md at main · landedjobs/ai-interview-guides

The 90-second answer

Converse on bedrock-runtime is stateless: every call takes the full messages array and returns exactly one assistant message. Multi-turn is therefore not a Bedrock setting you enable: it is an orchestration layer you own. Three jobs: persist the transcript, assemble a bounded context window each turn, and keep the tool-use loop and guardrails consistent across turns. Bedrock Agents does give you a managed sessionId with session attributes and prompt-session attributes, and I use it when the flow is entirely Bedrock-native. The moment I need custom retrieval, custom tools, or history shared across web and voice channels, I run Converse myself and own the state.

Three state layers

LayerContentsStoreLifetime
Transcriptevery user / assistant / toolResult block, verbatimDynamoDB, PK convId, SK turn#0007kept for audit and replay
Working contextlast K turns, rolling summary, extracted constraint slotsElastiCache Redis, ctx:{convId}30 min idle TTL
Long-term memoryfacts and preferences pulled out per turnOpenSearch k-NN, embeddeduntil the user deletes them

Append with a conditional write on a version attribute so two open tabs cannot interleave turns; the loser gets a 409 and re-reads.

The turn loop

python import boto3 rt = boto3.client("bedrock-runtime")

def turn(conv, user_text): conv.append({"role": "user", "content": [{"text": user_text}]}) for _ in range(6): # tool-loop cap r = rt.converse( modelId="anthropic.claude-3-7-sonnet-20250219-v1:0", system=[{"text": SYSTEM}, {"cachePoint": {"type": "default"}}], messages=conv, toolConfig=TOOLS, inferenceConfig={"maxTokens": 2048, "temperature": 0.2}) conv.append(r["output"]["message"]) if r["stopReason"] != "tool_use": return conv, r["usage"] for b in r["output"]["message"]["content"]: if "toolUse" in b: conv.append({"role": "user", "content": [{"toolResult": { "toolUseId": b["toolUse"]["toolUseId"], "content": [{"json": run_tool(b["toolUse"])}]}}]}) raise RuntimeError("tool loop exceeded")

Context assembly. Claude 3.7 Sonnet on Bedrock gives a 200K window. I budget: 1.2K system, 1.4K tool schemas, 40K history, 4K output reserve. When history crosses 40K tokens, turns 1..N-4 are folded into a rolling summary and the last four stay verbatim. The cachePoint sits after the system prompt so the stable prefix is what gets cached: Bedrock's prompt cache has a 5-minute TTL, so a user who pauses longer pays full input price on the next turn.

A trace (figures illustrative). Turn 12 of a support conversation. Raw history is 41,300 tokens, so the summariser compresses turns 1–8 into 900 tokens. Prompt becomes 1,200 system (cache hit) + 1,400 tools + 900 summary + 4,100 recent = 7,600 tokens. At $3 per 1M input tokens that is $0.023 for the turn, against $0.124 if I had resent the raw transcript. Time-to-first-token drops from 610 ms to 430 ms.

Failure modes I design for

  • Constraint loss in summarisation. A user says "never book before 9am" at turn 3; the summary drops it. Fix: extract constraints into a structured field every turn and inject them verbatim, never through the summariser. Detect with a regression set of transcripts asserting constraint survival.
  • Alternation break. Converse requires messages to alternate user/assistant starting with user. A turn that fails mid-stream leaves two user messages in a row and the next call returns a ValidationException. Append a placeholder assistant block on failure.
  • Guardrail block. A blocked turn must still be written to the transcript as an event, or the next turn's alternation breaks the same way.
  • Tool-loop runaway. Six iterations, then return a partial answer flagged degraded rather than hanging the request.

Stream with converse_stream for TTFT, and persist the assembled message only after the stream closes.

Curated: · Written: · Reviewed:

Q&A-16What are the trade-offs between using a vector database vs. a keyword search in RAG?

Reported in ai-interview-guides/guides/amazon.md at main · landedjobs/ai-interview-guides

The 60-second answer

Neither wins. They fail on disjoint query classes, so the default for an agent's retrieval tool at Amazon scale is hybrid: BM25 and dense vectors retrieved in parallel, fused by rank, then a cross-encoder rerank over the top 50. The decision you actually have to defend is which one you'd keep if you could only keep one, and that follows from your query mix, not from a benchmark someone posted.

Why they fail differently

BM25 scores term overlap with IDF weighting (Lucene defaults k1=1.2, b=0.75). A rare token (E-4021, m5.2xlarge, a customer ARN) carries most of the score, which is exactly what you want for identifier lookups and exactly what embeddings blur away. A 1024-dim vector for a 12-token error-code page looks like every other short technical page in the corpus.

The reverse holds for vocabulary mismatch. "Order cancelled after payment retry failed" and "purchase terminated when the retry didn't go through" share one content token; BM25 scores near zero, the embeddings land close together.

BM25Dense (HNSW)
exact IDs, codes, SKUsdecisiveweak
paraphrase, synonymsnear-zerostrong
negation ("not compatible")weakweak
brand-new jargonworks day oneneeds re-embed
index costincremental, cheapembed every chunk; re-embed on model change
memoryinverted index, ~10–20% of text size4 bytes × dims × vectors, resident
explainabilityshows matched termsopaque

The arithmetic that decides the architecture

50M chunks at 1024 dims, float32: 50e6 × 1024 × 4 B = 204.8 GB of raw vectors, plus HNSW layer-0 links at 2m per node with m=16 (the OpenSearch k-NN default, alongside ef_construction=128) ≈ 6.4 GB. That does not fit one node: you shard across memory-optimised instances, or drop to Titan Text Embeddings V2's 512-dim output, which halves it to ~102 GB for a recall loss you measure rather than assume. BM25 over the same corpus is a few GB of inverted index. Embedding once at ~300 tokens per chunk is 15B tokens ≈ $300 at Titan V2's $0.02 per 1M tokens: cheap until you upgrade the embedding model and pay it again plus a full reindex.

Fusion: ranks, not scores

BM25 scores are unbounded; cosine sits in [-1, 1]. Adding them is meaningless. Reciprocal rank fusion sidesteps the calibration problem:

python def rrf(rankings, k=60, limit=20): scores = {} for ranked in rankings: # doc ids, best first for rank, doc_id in enumerate(ranked, 1): scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank) return sorted(scores, key=scores.get, reverse=True)[:limit]

Trace for "E-4021 retry storm" over 50M chunks:

docBM25 rankvector rankRRF score
error-code page1121/61 + 1/72 = 0.0303
generic retry doc3041/90 + 1/64 = 0.0267
best semantic match20011/260 + 1/61 = 0.0202

The exact-code page wins on the strength of a single rank-1 placement, which is the whole argument for keeping lexical in the pipeline. k=60 is the Cormack et al. default and what Bedrock Knowledge Bases' hybrid mode on OpenSearch Serverless uses.

Failure modes and how you catch them

  • Stale embeddings after a model upgrade: silent recall drop. Pin the model version in index metadata and run a fixed eval set per query bucket (exact-ID, paraphrase, negation) reporting recall@20 per bucket, never a single average.
  • HNSW recall collapse when ef_search is tuned down for latency: measure against exact k-NN on a 10k sample.
  • Fusion masking a broken retriever: log each retriever's rank for every served query. If BM25's share of the final top-20 falls below ~5%, it's misconfigured, not redundant.
  • Reranker latency: a cross-encoder over 50 candidates adds 30–80 ms, usually the largest single line in a sub-200 ms budget.

When not to

Under ~10k chunks with a strong reranker, BM25 plus rerank is often enough and removes an entire embedding pipeline. All-identifier queries: skip vectors. All natural-language support questions with no codes: vectors plus a reranker carry it.

Curated: · Written: · Reviewed:

Q&A-17How do you optimize an LLM for low-latency inference in a production environment?

Reported in ai-interview-guides/guides/amazon.md at main · landedjobs/ai-interview-guides

The 90-second answer

Split the SLO before touching a single flag. A single call is TTFT plus (output tokens × TPOT). An agent turn is not one call: it's 3–8 sequential ones, each paying its own TTFT. So for an agent, TTFT and the number of round trips dominate; raw tokens/sec barely moves the number the user feels.

Levers, in the order I pull them:

  1. Kill the repeated prefill. The system prompt and tool schemas are byte-identical on every step of the loop: 1–3k tokens of pure repeat. Automatic prefix caching turns that into a KV-cache hit.
  2. Right-size the model. An 8B with tool-call fine-tuning beats a 70B on latency by roughly an order of magnitude. Route only the hard steps up.
  3. Quantize weights. Decode is memory-bandwidth bound, so halving weight bytes roughly halves TPOT.
  4. Batch, but cap it. Continuous batching buys throughput and spends TTFT. max_num_seqs is the dial.
  5. Speculative decoding for TPOT, only if draft acceptance is high.
Component (illustrative)BaselineTunedLever
prefill, 2,000 tok, 8B210 ms55 msprefix cache hits 1,500 tok
decode, 150 tok1,200 ms600 msFP8 weights
queueing at 40 rps400 ms60 mscap max_num_seqs, add replicas
per call~1.8 s~0.7 s
× 5 agent steps9 s3.5 sparallel tool calls cut steps too

The arithmetic that decides it

Decode reads every weight for every token. 8B FP16 = 16 GB; an A100 80GB moves ~2.0 TB/s, so the floor is 16/2000 ≈ 8 ms/token: 125 tok/s per replica before any overhead. FP8 weights on H100 (3.35 TB/s) drop that to 8/3350 ≈ 2.4 ms. A 70B FP16 is 140 GB → 70 ms/token on one A100; tensor-parallel across 8 splits it to 17.5 GB per GPU ≈ 8.7 ms plus all-reduce.

Prefill is compute-bound: ~2 × params × tokens FLOPs. 8B × 2,000 tokens = 32 TFLOP; at ~150 TFLOPS effective that's ~210 ms. Cache 1,500 of those tokens and you pay ~50 ms instead.

python from vllm import LLM

llm = LLM( model="meta-llama/Llama-3.1-8B-Instruct", quantization="fp8", # native FP8 needs Hopper/L40S; INT8 weight-only on Ampere enable_prefix_caching=True, # shared system prompt + tool schemas max_model_len=8192, max_num_seqs=64, # concurrency cap: this is what protects TTFT max_num_batched_tokens=8192, )

One rule makes or breaks the cache: static prefix first, variable content last. Put the timestamp, request id or user profile at the top of the system prompt and the hit rate goes to zero.

Failure modes I'd watch

  • Prefix cache silently missing: someone interpolates a date into the system prompt. Detect via vLLM's prefix-cache hit-rate metric; alert on a drop, don't wait for the latency graph.
  • Quantization breaking structured output. FP8 is usually safe; INT4 degrades JSON tool arguments well before perplexity moves. Gate on tool-call validity rate, not loss.
  • Batching starving TTFT. p99 TTFT climbs once concurrent prefill exceeds the prefill budget. Instrument queue time separately from compute time or you'll misread it as a slow model.
  • Speculative decoding regressing. Below ~60% draft acceptance you pay the draft cost for nothing. Measure acceptance rate, not just latency.
  • Cold starts. Serverless endpoints take 30–60 s to load weights. Provisioned throughput on Bedrock, or min replicas plus inference components on SageMaker, or you've optimized a number nobody sees.

When not to do this: offline eval and batch scoring are throughput-bound. Let TTFT go and maximize tokens/sec per NeuronCore or GPU instead.

Curated: · Written: · Reviewed:

Q&A-18What hands-on experience do you have with post-training or agents?

Reported in Amazon Interview Process & Questions · TrueInterview

The 90-second answer

Two projects, one on each side: post-training and agent engineering fail in different ways, so I'll give you both.

Post-training: I fine-tuned an 8B open-weight model to emit tool calls in a fixed JSON schema for an internal ops assistant. 38k SFT trajectories, LoRA r=16 on the attention and MLP projections, 3 epochs, then DPO on 6k preference pairs where the rejected completion was a real production failure: wrong tool, missing required argument, or a call made when no tool was needed. Schema-valid tool calls went 71% → 96% on a held-out 500-task set; end-to-end task success 48% → 63%. The part worth talking about is the regression: SFT alone pushed tool-call rate to 94% of turns, including chit-chat, because every teacher trajectory contained a call. Mixing 20% no-tool-needed negatives into SFT fixed it. DPO on top of the unfixed SFT model did not: the prior was too strong.

Agents: I owned the runtime for a support agent on Bedrock (order lookup, refund, ticket update) with a step budget, per-tool JSON schema validation before dispatch, and idempotency keys on every mutating call. Containment went 34% → 58% of inbound tickets over a quarter at p95 2.4 s end to end.

The honest boundary: I've done LoRA-scale SFT, preference optimisation, and small-scale RLVR with execution-grounded rewards. I have not run pretraining-scale RL, and I haven't owned an RLHF data pipeline end to end.

Post-training projectAgent project
Base8B open-weight, LoRA r=16Claude/Nova-class via Bedrock
Data38k SFT + 6k preference pairs500-task eval, 12k live tickets
MethodSFT → DPOstep budget, schema validation, idempotency
Metricschema validity 71→96%, task success 48→63%containment 34→58%, p95 2.4 s
Failure foundover-calling toolsduplicate refund on retry

The reward I actually used. Format-only rewards get gamed, so the RLVR signal was execution-grounded:

python def reward(rollout, task): if not schema_valid(rollout.tool_calls): # per-tool JSON schema return -1.0 # malformed call is worse than none if not same_multiset(rollout.tool_calls, task.expected_calls): return 0.0 return 1.0 if rollout.final_answer == task.gold else 0.3

That 0.3 is deliberate: correct tool sequence with a wrong final answer still teaches the model something, but it can't outscore getting the answer right.

The agent failure, concretely. A retry after a 2 s timeout produced a second refund. The trace:

stepmodel outputruntime action
1refund(order=8821, amount=49.00)executed, key sha(order,amount) stored
2refund(order=8821, amount=49.00)deduped, cached 200 replayed
3final answer3 steps, 1.9 s

Without the key, step 2 is a second $49 refund. The step budget alone would not have caught it: the loop was only two steps deep.

What I'd expect you to push on. How I know it generalised: the 500-task set was held out by task family, not randomly, and the LLM judge was calibrated against 200 human labels at 0.81 Cohen's kappa before I trusted it. How I detect reward hacking: schema validity climbing while task success plateaus means the reward is measuring format, not correctness. That's the signal to add an execution check. And what I'd do with 10× compute: more RLVR on verifiable tasks, not more SFT, because the SFT ceiling showed up at 63%.

Curated: · Written: · Reviewed:

Q&A-19Tell me about a time AI gave you a wrong answer.

Reported in Amazon Interview Process & Questions · TrueInterview

The 90-second answer

I owned the returns-eligibility agent in a seller-support workflow: a Bedrock agent with a knowledge base over policy docs and one action group calling a Lambda that read order state from DynamoDB. A seller asked whether they could still return an item 45 days after delivery. The agent replied "Yes: your order is eligible for a return, and I've started the request." The order was outside the 30-day window. The seller opened a case quoting our own agent, and the associate refunded it as goodwill.

I had the transcript in a failing regression case within a day, because the turn was already in our 480-case golden set and the nightly run flagged it. Root cause was not the model being dumb. It was my schema.

The Lambda returned two keys containing the word "eligible": a top-level returnEligibility: "INELIGIBLE" and a nested promotions.eligible: true for an active discount. My OpenAPI response schema declared type: string with no field descriptions, so the model received an unlabelled JSON blob and pattern-matched the wrong one. The agent instruction compounded it: "confirm eligibility to the customer" reads as an instruction to affirm.

BeforeAfter
response schematype: stringtyped object, returnEligibility enum ELIGIBLE/INELIGIBLE/UNKNOWN
field descriptionsnoneone line per field, returnEligibility marked authoritative
colliding keypromotions.eligiblerenamed discountActive
instruction"confirm eligibility""state eligibility only from returnEligibility; if UNKNOWN, ask for the order id"

Schema alone is not a guardrail, because a model can still narrate past a tool result. I added a deterministic post-generation check that runs before the answer leaves the agent:

python

python 3.11

import re AFFIRMATIVE = re.compile(r"\b(eligible|you can return|return approved)\b", re.I)

def claim_matches_tool(answer: str, tool_result: dict) -> bool: if not AFFIRMATIVE.search(answer): return True return tool_result.get("returnEligibility") == "ELIGIBLE"

A mismatch blocks the response, emits a CloudWatch metric eligibility_claim_mismatch, and falls back to a templated answer built from the tool fields. That fallback is deliberately boring: it reads worse than the model and is never wrong.

On the golden set, wrong-answer rate on eligibility questions went from 6.2% to 0.4% over three weeks. The residual 0.4% was knowledge-base staleness, a policy page updated without a re-sync, which I fixed separately with a sync-age alarm at 24 hours.

What I'd say the lesson generalises to. When an agent's answer depends on a tool result, the tool result is a contract, not context. Two things follow: every field the model must reason over gets a name and a description that cannot collide with another field, and any claim with a downstream cost (a refund, a deletion, a promise of eligibility) gets verified outside the model against the tool payload. The failure mode I watch for now is a tool returning HTTP 200 with a semantically negative body; the model reads success from the transport and answers from the transport. Detection is a per-turn assertion on the claim, not a thumbs-down signal, because the seller who got the wrong answer is the one who doesn't complain: they just act on it.

Curated: · Written: · Reviewed:

Q&A-20Tell me about a time when you made an architectural decision that failed or required reversal. What did you learn, and how would you execute it differently today?

Reported in Amazon Agentic AI Engineer Interview Questions & Guide 2026

The decision was mine: build the returns-resolution agent as one Bedrock agent with all 40 action groups attached and a 200k-token context window, and keep conversation state in the Lambda orchestrator's memory. I chose it for speed: one agent, one prompt, ship in three weeks. It reversed six weeks after launch.

The failure showed up as three metrics at once. Tool-selection accuracy fell from 94% on our 12-tool prototype to 71% at 40 tools. p95 latency went from 3.2 s to 19 s, p99 to 22 s, because the prompt had grown to 180k tokens by turn six. Cost per resolved return went from $0.08 to $0.42. The one that forced the reversal was a duplicate refund: a deployment recycled the Lambda mid-conversation, the in-memory state vanished, the agent re-planned from the customer's last message, and issued a second refund of $84.20 for the same return.

SignalBefore (12 tools, 8k ctx)At failure (40 tools, 180k ctx)
Tool-selection accuracy94%71%
p95 latency3.2 s19 s
Cost per resolved return$0.08$0.42
Duplicate refunds03 in 10k

Trace of the failed request: turn 1 get_order → ok; turn 2 get_shipment → ok; turn 3 check_policy → ok; turn 4 issue_refund → 200. Deployment at 14:02 UTC. Turn 5, customer asks “where is my refund?”: Lambda cold start, state empty, agent calls get_order again, then issue_refund again. Second refund of $84.20. A reconciliation job comparing refunds to returns flagged two refunds for r_7781 at 14:07.

The root cause was not the model. I had treated the context window as a substitute for state management and tool design. Context is a budget, not a database. Tool count is an architectural parameter: every added tool costs selection accuracy. And any side-effecting action needs an idempotency key before it needs a retry policy.

The reversal: a planner-executor split. The planner sees a compact catalog of 8 subgoals, not 40 tools. The executor gets only the tools for that subgoal, capped at 6. State moves to DynamoDB keyed by task_id, and every side-effecting action writes a conditional put on (return_id, action) so a retry is a no-op. A supervisor checks the plan against policy before execution. Results after 90 days: p95 4.1 s, cost $0.09 per resolved return, tool-selection accuracy 96%, duplicate refunds zero.

How I'd execute differently today: run a two-week spike on 500 production-like traces before committing, with three gates. Tool-selection accuracy >95% at the target tool count, p95 <5 s, cost <$0.10 per task. Write the ADR with explicit reversal criteria: if duplicate side effects appear or p95 exceeds 8 s for a week, roll back. Design the state machine and idempotency layer first, then the agent. And cap each agent at 8–10 tools, splitting by subgoal rather than by API surface. That is the decision I'd make today: planner-executor and external state from day one, not after the duplicate refund.

Curated: · Written: · Reviewed:

Q&A-21Design an agentic ad bidding system that makes real-time bid adjustments at very high scale with strict latency and reliability needs.

Reported in Amazon Advertising Agentic AI Engineer Interview Questions

The model does not belong in the bid path. At Amazon Advertising scale I assume peak 200k bid requests/s, a p99 end-to-end budget of 50 ms, OpenRTB 2.6, and per-campaign budget pacing. The agentic part is a control-plane loop that changes the policy the bidder executes; the data plane stays deterministic and keeps bidding if the agent is down.

bid request -> edge (TLS, auth) -> bidder fleet (stateless) |-- local feature cache |-- versioned policy bundle (LKG in memory) |-- local budget lease |-- guardrail evaluator -> OpenRTB bid response

telemetry -> Kinesis -> aggregates -> agent (Bedrock/LLM + bandit) -> policy diff -> schema/bounds validator -> shadow -> canary -> policy store

The agent loop. It observes win rate, eCPM, budget burn rate, ROI, and latency by segment; reasons over those signals plus operator constraints; emits a structured policy diff, not prose; and evaluates the diff in shadow and canary before promotion. A policy diff looks like {"segment":"electronics_us","bid_multiplier":1.08,"max_cpm":2.40,"pacing_rate":0.93}. A validator enforces schema, bounds, and budget impact. The LLM is useful because constraints arrive as natural language ("do not exceed $0.80 CPC on new advertisers") and because it can correlate many weak signals; a bandit or PID controller is better when the policy space is small and stable.

Hot path. Per-request adjustment is a small GBDT plus policy arithmetic, not an LLM call. If the requirement is literally an LLM decision per bid, I push back: 200k QPS at 500 ms model latency is impossible and unnecessary. The agent tunes the 2 ms model.

python def bid(req, policy): if not eligible(req, policy): return no_bid() base = gbdt.predict(req.features) # p99 ~3 ms mult = policy.segment_multiplier(req) price = min(base * mult, policy.max_cpm, req.bidfloor * policy.max_floor_mult) if not budget.try_reserve(req.id, price): # local lease return no_bid() return Bid(price)

Latency budget. I design to 25 ms p99 inside the 50 ms envelope, leaving headroom for network and retries.

Stagep99 ms
TLS/HTTP2 termination3
parse/validate OpenRTB1
feature cache lookup4
GBDT scoring3
policy + guardrail eval1
local budget lease1
serialize response1
network10

A feature-cache miss never triggers a remote call in the hot path; it falls back to conservative features and lowers the bid or no-bids. Budget is leased in chunks, e.g. $50 per bidder per 100 ms, so a central pacing service is not on the request path. Lease size bounds overspend; nodes reconcile asynchronously.

Reliability. Bidders are stateless, multi-AZ, N+2 capacity, with load shedding at the edge. Policy is a versioned, signed artifact pulled and cached locally; activation is atomic and rollback is a pointer flip. If the policy service is down, bidders keep the last known good bundle. If the agent is down, bidding continues. If the lease service is down, bidders fall back to a conservative local rate.

FailureDetectionResponse
Agent proposes aggressive multiplierShadow eCPM/ROI divergence, budget burn alertReject at bounds validator; auto-rollback canary
Policy store unavailablePull errors, stale version ageServe LKG from memory; page on-call
Feature store latencyCache hit rate, p99 by stageServe fallback features; no remote call
Budget overspend from leasesBurn rate vs planShrink lease size; reconcile; cap max CPM
Feedback oscillationWin-rate oscillation across cyclesDampen changes, cap delta per cycle, holdout
Hallucinated policy JSONSchema validation failureReject diff; keep current policy

Trade-offs. Agent cadence is minutes, not milliseconds: faster loops react to noise and burn budget on exploration. Central budget accounting is exact but adds latency; local leases are fast but allow bounded overspend. A larger LLM planner handles richer constraints but is slower and harder to evaluate; a bandit is cheaper and more predictable. I would start with the deterministic bidder, local leases, and a shadow-mode agent, then promote policy changes only after canary metrics beat the holdout on eCPM and ROI without breaching budget or latency.

Curated: · Written: · Reviewed:

Q&A-22Design the feedback loop for a self-correcting AI agent that learns from user corrections, tool outcomes, and execution traces.

Reported in Amazon Advertising Agentic AI Engineer Interview Questions

The 90-second answer

Three signal sources, one trace store, three consumers, one promotion gate.

Every agent turn emits a structured trace: the plan, each tool call with its arguments, each observation, the final answer, token and latency counts. User corrections and tool outcomes are joined onto that trace by trace_id and written as labelled records. A distillation job turns those records into three artifacts of increasing cost and decreasing reversibility: memory writes (minutes), prompt/exemplar/retrieval changes (days), fine-tunes (weeks), and nothing reaches production without a paired offline eval, then shadow traffic, then a canary with automatic rollback.

The decision that matters most: push feedback into memory and retrieval first. Weights are the last lever, not the first, because a fine-tune takes weeks and you cannot roll it back per-account.

What each signal actually proves

SignalWhat it provesWhere it landsTime to effect
User edit of the outputThe answer was wrong and the right answerDistilled example + memoryhours
Thumbs down, no textSomething was wrong; not whatTriage queue only—
Tool 4xx / schema validation failureThe call was malformed: ground truthPrompt/tool-schema fixdays
Tool 200 but business outcome wrongPlan or argument choice was wrongEval case + exemplardays
Full traceWhich step failedAttribution—

User corrections are the noisiest signal and the only one carrying the correct answer. Tool outcomes are the most reliable and say nothing about intent. You need both, which is why attribution is the hard part.

Attribution. A bad final answer can come from retrieval, planning, tool arguments, or the model. So label at step level, not turn level: for each trace, mark the first step whose output was not recoverable. In practice that is a small classifier over (step, observation) pairs plus rules: a tool 422 is unambiguous, a wrong final answer is not.

A worked trace. Advertising agent, budget change. User says "keep this campaign under $200/day." The agent calls update_campaign_budget(campaign_id=c_88, daily_budget=500). User edits the value to 200 and comments "I said under 200." The trace shows the plan step cited a retrieved doc from a different campaign's playbook. Attribution: retrieval, not reasoning. Three actions fire: a scoped memory write for that account (budget_ceiling=200), a retrieval filter fix excluding cross-campaign playbooks, and the trace becomes a permanent eval case. No fine-tune.

The gate. Offline eval runs the candidate against the incumbent on the same trace set, paired, so you compare per-item outcomes rather than two aggregate scores. A 2 pp lift on a 400-case slice is inside noise; McNemar on paired results tells you whether it is real. Require: no slice regressing more than 1 pp, target slice improved with p < 0.05, then shadow for a week, then 5% canary with rollback on task-success drop.

Failure modes worth naming.

  • Sycophancy drift: the agent optimises for thumbs-up and gets agreeable instead of correct. Detect by tracking tool-verified task success against acceptance rate: acceptance up, success flat, you have a reward-hacking loop.
  • Poisoned corrections: a user is simply wrong. Require corroboration: same failure class across distinct sessions, or a tool outcome that contradicts the correction: before it becomes an exemplar.
  • Stale memory: preferences change. Confidence decay plus TTL on memory entries, and the actor assembles context from a versioned store so a bad write is a rollback, not a retrain.
  • Frozen eval set: production drifts away from it. Sample fresh traces weekly and watch embedding drift on the eval distribution.

Keep the learner out of the actor's hot path. The actor reads a pinned version of memory and policy; the learner writes candidates. That separation is what makes the whole loop reversible.

Curated: · Written: · Reviewed:

Q&A-23Design a system in which an AI agent works on a stream of security data and runs security detection on it, with incoming traffic of about 1 TB per day.

Reported in Amazon: Design an AI-Agent Security Detection Pipeline for 1 TB/Day of Streaming Telemetry

The core answer

1 TB/day is ~12 MB/s average and roughly 35 MB/s at a 3× peak. That is far too much to put an LLM in front of every event, so split the system in two. A deterministic detection layer sees 100% of the stream. An agent layer sees only what survives triage: alerts, correlated incidents, and scheduled hunts. The agent's job is not to read logs; it is to enrich, correlate, decide, and act through tools, and to propose new detection rules offline.

StageServiceNote
ingestKinesis Data Streams~36 shards at peak
archiveFirehose → S3 Parquet, dt/source/ partitions~4–6× compression → ~5 TB/month stored
normalizeManaged Service for Apache Flinkmap to OCSF
detectFlink (Sigma rules as SQL) + model endpoint5-min windows per principal
triageLambda → agentalerts only, ~10⁴/day
agentBedrock + orchestratortools below
case stateDynamoDBone item per incident
hunt / rule authoringAthena + agent, offlinedaily

Sizing the stream. A provisioned Kinesis shard takes 1 MB/s and 1,000 records/s in. At 2 KB per event, 1 TB/day is ~500M events ≈ 5.8k records/s average, ~17k at peak → 18 shards by record count; 35 MB/s peak → 35 shards by bytes. Provision 36 and hash on source + principal rather than account, or one noisy VPC flow source pins a single shard.

Agent tools. Read-only by default: search_events(query) over OpenSearch (30-day hot window), athena_query(sql) over the S3 archive, enrich_ioc(indicator) against internal TI and GuardDuty findings, get_asset_context(resource_id) for tags, owner and internet exposure. Write tools (open_case, isolate_instance, revoke_keys) sit behind approval.

One incident, traced. GuardDuty raises UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration at 02:14. Flink correlates it with a 40× spike in GetCallerIdentity from a new ASN. The agent calls get_asset_context (role attached to three instances, one public-facing), search_events for the role's last 24 h (finds CreateAccessKey from a Tor exit node), then enrich_ioc (IP flagged by two feeds). It proposes isolation plus key revocation, attaches the evidence chain to the case, and a human approves the write. Agent latency ~8 s, ~15k input tokens, roughly $0.04 per incident.

Guardrails that matter here. Log fields are attacker-controlled: user-agent strings, filenames, SNI. Never interpolate raw log text into the instruction; pass it as structured JSON in a tool result and tell the model to treat it as data. Allowlist tool calls, cap them per incident (12 is a reasonable ceiling), and log every call to CloudTrail plus the case audit trail. Write-tool IAM is deny-by-default and requires approval above a blast-radius threshold.

Failure modes. Prompt injection through a crafted user-agent is caught by an output classifier plus the tool allowlist, not by hoping the model resists. Detection drift as TTPs change is caught by tracking per-rule hit rates weekly and backtesting agent-proposed rule diffs against a labelled corpus before merge. Pipeline lag shows up as MillisBehindLatest above 60 s: alarm on it, because a detection system that is an hour behind is not a detection system. Cost blowup happens the moment someone routes all events to the agent; put a hard quota on Bedrock invocations per hour so that mistake is a 429, not a monthly surprise.

Curated: · Written: · Reviewed:

Q&A-24Implement scaled dot-product attention with a causal mask and tests that would expose attention to future tokens.

Reported in ai-interview-atlas/docs/companies/amazon.md at main · eiler2005/ai-interview-atlas

The 90-second answer

Scores are QKᵀ/√d_k, then the mask, then softmax over keys, then @V. The mask goes on the scores before the softmax, as -inf at every position j > i, so the exponentiated weight is exactly 0. Two ways this gets silently wrong: masking after softmax without renormalizing (rows no longer sum to 1), and masking with 0 instead of -inf (future tokens keep weight 1/n).

python import torch, torch.nn.functional as F

def attention(q, k, v, causal=True): """q: (B,H,L,D); k,v: (B,H,S,D). Returns (out, weights).""" scores = (q @ k.transpose(-2, -1)) / q.size(-1) ** 0.5 if causal: L, S = q.size(-2), k.size(-2) qi = torch.arange(L, device=q.device).unsqueeze(1) + (S - L) # KV-cache offset kj = torch.arange(S, device=q.device).unsqueeze(0) scores = scores.masked_fill(kj > qi, float("-inf")) w = F.softmax(scores, dim=-1) return w @ v, w

Why the leak is fatal, not cosmetic. Position 1 with raw scores [2.0, 1.0, 5.0]: unmasked softmax gives [0.047, 0.017, 0.936]. 93.6% of that token's output is the future token. Masked, it is [0.731, 0.269, 0.0].

The test that actually catches it

Shape assertions and "mask is upper-triangular" assertions pass on a mask that is off by one or misaligned across batch/head dims. Perturb the future and assert the past is untouched:

python def test_future_tokens_cannot_change_past_outputs(): torch.manual_seed(0) q, k, v = (torch.randn(2, 3, 6, 8) for _ in range(3)) out, _ = attention(q, k, v) for t in (3, 4, 5): k2, v2 = k.clone(), v.clone() k2[..., t, :] -= 100.0 v2[..., t, :] += 100.0 out2, _ = attention(q, k2, v2) assert torch.equal(out[..., :t, :], out2[..., :t, :]) # bit-identical assert not torch.allclose(out[..., t:, :], out2[..., t:, :])

def test_no_gradient_reaches_future_keys(): q, k, v = (torch.randn(1, 1, 5, 4, requires_grad=True) for _ in range(3)) out, _ = attention(q, k, v) out[..., 0, :].sum().backward() assert k.grad[..., 1:, :].abs().sum() == 0

torch.equal holds because exp(-inf) is exactly 0.0 and adding 0.0 is exact. If you mask with torch.finfo(dtype).min instead, downgrade to allclose(atol=1e-6), and note that choice leaks a little.

BugCaught by
mask after softmax, no renormalizew.sum(-1) ≈ 1
0 instead of -infperturbation test
diagonal=0 instead of 1row 0 becomes all-masked → NaN
mask not broadcast over (B,H)perturbation test with B,H > 1
KV-cache offset wrongfull-sequence vs step-by-step, atol=1e-5

Follow-ups an interviewer will push on

KV cache. During decode L=1, S=past+1, so the mask must be offset by S-L as above. PyTorch's F.scaled_dot_product_attention(..., is_causal=True) assumes q and k are the same length; with a cache, pass an explicit boolean mask instead. The regression test is: run the full sequence in one call, then feed it token by token through the cache, and assert the outputs match to 1e-5.

Padding. A left-padded batch plus a causal mask can fully mask a row, and softmax of all -inf is NaN. Either guarantee one unmasked key per row or use finfo(dtype).min and accept the residual leak.

Cost. Time is O(L²d) and the weights are O(L²) per head: at L=8192, bf16, that is 128 MB per head, which is why production uses a fused kernel (Flash Attention) rather than materializing w.

Curated: · Written: · Reviewed:

Q&A-25Walk me through how you would solve a DAG topological sort problem, including the code.

Reported in Real Tech Interview Experiences (2026) – Page 2 | PracHub

The 90-second answer

Clarify two things first: does any valid order do, or is there a tie-break the interviewer expects, and does the graph fit in memory. Then Kahn's algorithm: compute in-degrees, seed a queue with every node at in-degree zero, pop a node, append it to the order, decrement its successors, push any that reach zero. O(V+E) with a plain deque.

The correctness argument is one line: a DAG always has at least one source, and deleting a source leaves a DAG, so the loop cannot stall before V nodes are emitted unless a cycle exists. That is also the cycle check, if the order is shorter than V, the leftovers are exactly the nodes on or downstream of a cycle.

python from collections import deque import heapq

def topo_sort(n: int, edges: list[tuple[int, int]]) -> list[int] | None: """edges: (u, v) means u must run before v. Returns None if a cycle exists.""" adj = [[] for _ in range(n)] indeg = [0] * n for u, v in edges: adj[u].append(v) indeg[v] += 1

ready = [i for i in range(n) if indeg[i] == 0]
heapq.heapify(ready)      # drop the heap for O(V+E); keep it for a deterministic order
order = []
while ready:
    u = heapq.heappop(ready)
    order.append(u)
    for v in adj[u]:
        indeg[v] -= 1
        if indeg[v] == 0:
            heapq.heappush(ready, v)
return order if len(order) == n else None

Checked against Python 3.12.

Trace. Six nodes, edges (5,2) (5,0) (4,0) (4,1) (2,3) (3,1). In-degrees start [2,2,1,1,0,0], so ready = [4,5].

poporder so farnewly ready
440→1, 1→1, neither yet
54,52, 0
04,5,0—
24,5,0,23
34,5,0,2,31
14,5,0,2,3,1—

Six out of six, so it is a DAG. Add edge (1,5) and the same run stalls after [4]: in-degree of 5 is now 1, nothing else reaches zero, len(order) == 1 != 6 → cycle. The five unvisited nodes are {0,1,2,3,5}: the cycle itself is 5→2→3→1→5, and 0 is merely downstream of 5. That is the useful property: the leftover set is the cycle plus everything it blocks, so you know which tasks are stuck, not just that something is.

Kahn vs DFS post-order.

Kahn (in-degree)DFS post-order
order producedsources firstsinks first, then reversed
cycle detectionleftover nodesback edge to a gray node
cycle pathneeds a second passfree, read the recursion stack
deep graphsno recursionPython's 1000-frame limit; write it iteratively
incremental edgesnatural, bump one in-degreerebuild

If the interviewer wants the actual cycle printed, DFS with white/gray/black colouring is the shorter answer, because the gray nodes on the stack are the cycle.

Where it bites. Heap tie-breaking adds a log factor on the V pushes and pops only (O(V log V + E), not O((V+E) log V)) fine for a 10k-task plan, wrong for a 50M-edge graph. Duplicate edges are harmless here because the in-degree is incremented and decremented symmetrically; they only break implementations that dedupe the edge list for in-degree but iterate the raw adjacency list. A self-loop (u,u) correctly reports a cycle, since indeg[u] never returns to zero. And a topological order is not unique; if the interviewer expects a particular one they have a tie-break in mind, so ask rather than guess.

In an agent orchestrator this is the scheduler. The planner emits a task DAG of tool calls; each Kahn wave, every node that hits in-degree zero in the same round, is a set that can fan out in parallel, which is how you turn a serial plan into concurrent tool calls. A cycle out of the planner means the plan is invalid: reject and re-plan, never let the executor spin on it.

Curated: · Written: · Reviewed:

Curated: · Written: · Reviewed: