Skip to content
Tech Interview Prep
Technical interview guide

On-Call & Alerting Design

Designing alerts that page for what actually needs a human, and structuring on-call sustainably.

Read
28 min
Practice MCQs
25
Interview Q&A
25
Edition
v5
Editorial status
Reviewed

Scope: Google SRE, Prometheus, Alertmanager, OpenTelemetry, and NIST guidance current 2026-08-31.

Curated: · Written: · Reviewed:

25 questions

MCQ-19An SRE writes `alerts_test.yml` with `input_series` and `exp_alerts`, runs `promtool test rules alerts_test.yml`, and it passes. What has that run actually verified?
MCQ-24An SRE team audits one quarter of paging history before deciding which alert rules to keep. Which measurement should the audit report as paging precision?
MCQ-25Three monitoring sources (a host agent, a blackbox probe, and a cloud provider feed) all fire for the same failed checkout service. How should the pipeline collapse them before a page is sent?
MCQ-7An Alertmanager instance routes to a single on-call pager with `group_by: [alertname, cluster, service]`, `group_wait: 30s`, `group_interval: 5m`, `repeat_interval: 4h`. A shared storage node fills up: within 20 seconds, 60 `DiskFull` alerts fire. 10 from each of six services, all on cluster `prod-eu`. Three minutes later, a seventh service on the same cluster fires its first `DiskFull` alert. What reaches the pager in the next four minutes?
MCQ-8Alertmanager is configured with: ```yaml inhibit_rules: - source_matchers: [alertname="NodeDown"] target_matchers: [severity="warning"] equal: ["instance"] ``` At 02:14 four alerts are firing: `NodeDown{instance="node-7:9100", severity="critical"}`, `HighCPU{instance="node-7:9100", severity="warning"}`, `DiskFull{severity="warning"}` (this exporter sets no `instance` label), and `HighCPU{instance="node-8:9100", severity="warning"}`. Which notifications does Alertmanager suppress?
MCQ-9During a two-hour migration you create an Alertmanager silence with the single matcher `service=~"checkout"` to mute noisy checkout alerts. The alerts you meant to mute carry `service="checkout-api"`, and they keep paging for the entire window. Why did the silence not apply?
MCQ-6A payments service exports `http_requests_total` to Prometheus. The request-rate panel for 02:09–02:14 reads 'No data' rather than 0, and the alert `absent(http_requests_total{job="payments"})` fires at 02:14. The Prometheus target page shows the pod's scrape endpoint down since 02:00 with 'context deadline exceeded'. The service's access logs, shipped by a separate agent, record 2,400 requests in the same 02:09–02:14 window, with 0 5xx. What is the correct reading of the panel?
MCQ-20A worker exposes `jobs_total`, a counter that is only ever incremented, scraped every 60 s. The alert `throughput < 1 job/s` pages when the 5-minute rate drops below 1. The dashboard computes throughput as `(jobs_total - jobs_total offset 5m) / 300`. Samples: 12:00 → 4,800; 12:01 → 4,860; 12:02 → 4,920; 12:03 → 0; 12:04 → 150; 12:05 → 300. The alert pages at 12:05. What should the on-call engineer conclude?
MCQ-4A service has a 99.9% availability SLO over a rolling 30-day window, so its error budget is 43.2 minutes. At 14:00 a deploy starts returning HTTP 500 on 2% of requests; before that the error rate was effectively zero. At 14:05 the on-call checks the 1h/5m multiwindow fast-burn alert, configured with a 14.4x threshold. What does that alert report?
MCQ-1checkout-api has a 99.9% availability SLO: 43.2 minutes of error budget per 30-day month. At 14:00 its 5xx rate jumps from 0.05% to 2.0% and stays there. Your team's page threshold is a 1-hour burn rate above 14.4x. The only change in the last 24 hours is the 13:50 deploy; the runbook's rollback takes 4 minutes and has run safely twice this month. What should the on-call do?
MCQ-5A service has a 99.9% availability SLO over 30 days, so its error budget is 43.2 minutes. Its only alert is a multi-window burn-rate alert that pages when the burn rate is at least 14.4 on both a 1-hour window and a 5-minute window. At 10:00:00 a dependency fails hard and returns 100% errors for 20 seconds, then recovers. At 10:02:00 it fails hard again for 40 seconds, then recovers for good. What does the on-call engineer see?
MCQ-16A service pages on `error_rate > 1%`, sampled every 15 seconds. Over 30 days the rule fired 1,240 times: 1,180 of those cleared within 3 minutes with no human action (mostly deploy blips), and 60 were real incidents. Median time-to-acknowledge on the 60 real incidents rose from 4 to 22 minutes as engineers began acknowledging pages without reading them. Which change most directly reduces that burden?
MCQ-2A checkout team pages on 38 cause-based alerts (CPU > 85% for 5 min, GC pause p99 > 200 ms, DB connection pool > 90%, disk queue > 8, and so on). Last quarter there were 5 user-visible incidents. Cause alerts fired first for 2 of them. The other 3: a feature-flag rollout that made checkout return HTTP 500, an expired TLS certificate on the CDN edge, and a payment provider returning 503s: produced no cause alert at all; the support queue reported each one, on average 11 minutes after impact began. In the same quarter, 71% of the 240 pages came from cause alerts that never produced a user-visible error. Checkout success rate normally sits at 99.7%. Which change addresses the gap this data shows?
MCQ-3You're on call for a payments API. Two candidate paging rules are on the table. Rule A: page when `db_pool_wait_seconds > 2`. It fired 60 times last quarter, 55 of those firings were followed by customer-visible checkout failures within 10 minutes, and the runbook fix (raise the pool ceiling) is a 30-second change that is safe at any traffic level. Rule B: page when `disk_used_percent > 80`. It fires weekly, but the only remediation is a multi-hour data migration that needs a maintenance window. Which rule should page, and on what basis?
MCQ-18A payments service's alerting pipeline evaluates rules every 30 seconds and pages the on-call engineer through PagerDuty; the rotation currently absorbs about 5 pages a week. The team wants to add a rule that pages when a service's 5xx ratio exceeds 2% for 10 minutes, because two degradations last quarter were missed. Replaying 30 days of recorded metrics, the rule as written fires 620 times. A 7-day shadow run (rule evaluated, results logged, nothing paged) produces 145 firings; an engineer reviewing them finds 6 that match real degradations lasting 35-90 minutes, while the other 139 are spikes that clear within 10-12 minutes. All 6 real degradations peaked between 3% and 8% error ratio. What should the team do next?
MCQ-10An alerting path has five stages: rule evaluation, routing, provider delivery, device receipt, and human acknowledgement. The only monitoring on it is a probe that checks the evaluator process is up and its rule files parse; that probe has been green for 30 days. During that window a primary database went down, no page was sent, and a customer reported the outage 40 minutes later. What does the green probe prove?
MCQ-21A synthetic alert canary injects a test alert into the evaluator every 5 minutes and marks the run PASS when the provider API returns 202 within 5 seconds of injection. It has been green for 90 days. During a real incident the provider accepted the alert (202 in 0.4s), but the on-call's phone never rang and no escalation fired 10 minutes later. Which change to the canary would have caught this?
MCQ-12A team keeps its incident runbook as a ConfigMap in the same Kubernetes cluster it documents. A regional network partition causes etcd to lose quorum, and for 35 minutes the API server returns 503 to every request, including reads of the ConfigMap. On-call engineers cannot open the runbook and the recovery takes 3x longer than the last drill. Which change would have kept the runbook readable during the partition?
MCQ-11At 03:12 a page fires for `checkout-api`: 5xx rate 31% (baseline 0.4%), p99 2.4 s (baseline 180 ms). The annotation reads `impact: checkout failing`, `safe action: roll back deploy 4821`, `runbook: /rb/checkout`. Deploy 4821 shipped 6 minutes ago; its only change is flipping the `new_tax_calc` flag from off to on. The flag is read per request from the config service, and the deploy changed no code. The responder has never worked on this service. Which `safe action` line belongs in this annotation?
MCQ-13You are the on-call engineer for a checkout service. At 02:14 a p99 latency alert fires: p99 has gone from 40 ms to 180 ms. You have run the runbook: restarted both app instances, confirmed the connection pool is at its configured maximum of 20, and checked the database, which shows no slow queries. Twenty minutes later latency is unchanged. Which of these is functional escalation?
MCQ-14A 7-person platform team has had one staff engineer carry every page for two years: roughly 40 pages a week, most of which auto-resolve, with no second engineer ever on the hook and no written path for who to wake when the primary is stuck. The team has flagged attrition risk if this continues. Which single change most reduces the risk that the on-call structure fails?
MCQ-15At 18:00 you take over on-call. The outgoing engineer's handoff reads: 'INC-4821. Checkout p99 900 ms → 60 ms and errors 7% → 0.4% after raising the DB pool from 20 to 60. Root cause is a retry storm in client lib 4.2; fix PR #3312 is open. The pool change is a temporary override in the config service and auto-reverts to 20 at 20:00. I'm off.' PR #3312 will not be reviewed until tomorrow morning. The change freeze starts at 20:00; after that, config changes need a CAB ticket with 24 h notice. What must you do before 20:00?
MCQ-23`KafkaConsumerLagHigh` pages the platform on-call when consumer lag on group `billing-sync` exceeds 50k for 10 minutes. The migration to `billing-sync-v2` finished 45 days ago. The old group's one remaining member stopped committing at that point, while producers still write about 2k messages/min to the topic. The rule has been in firing state continuously since, and the on-call silences it at the start of every shift. `BillingSyncV2LagHigh` was added during the migration and has fired twice, both real. What is the right disposition for `KafkaConsumerLagHigh`?
MCQ-17A platform org of 12 service teams publishes a monthly leaderboard ranking teams by declared Sev-2 incident count, lowest first. Over two quarters, declared Sev-2s fall 45%, Sev-1 counts are flat, incidents customers report via support tickets are flat, and median time from first customer impact to declaration rises from 6 to 41 minutes. What does this pattern most likely indicate?
MCQ-2202:10. The gateway accepts a forged `X-Forwarded-User` header, so any caller can act as any tenant. No patch exists; the gateway fix ships at 08:00, and the edge team works from `#inc-4471` and needs the header name before then to write a blocking rule. `#inc-4471` has 300 members, 40 of them contractors from a third-party MSSP whose contract does not cover customer identifiers. The incident doc is access-controlled to the responder group. The status page has 12,000 subscribers. The 02:30 update is due. What should it contain?