Skip to content
Tech Interview Prep
Technical interview guide

Metrics, KPIs & North Star Frameworks

Choosing the metric that actually reflects whether the product is succeeding, and avoiding vanity metrics.

Read
31 min
Practice MCQs
25
Interview Q&A
25
Edition
v3
Editorial status
Reviewed

Scope: Google HEART, Scrum.org Evidence-Based Management 2024, DORA metrics guidance, GOV.UK Service Manual, NIST statistical and privacy guidance, and FTC dark-pattern guidance current 2026-08-31.

Curated: · Written: · Reviewed:

25 questions

MCQ-3In a quarterly review, the growth lead reports 'we added 60,000 users this quarter': registered accounts went from 480,000 to 540,000. Monthly active accounts were flat at 90,000, and 12,000 previously active accounts stop logging in each month and never return. Which characterization of the quarter is correct?
MCQ-2A B2B SaaS company sells annual per-seat contracts. Over the last two quarters: | metric | Q2 | Q3 | | --- | --- | --- | | registered accounts | 9,400 | 15,800 | | paying accounts | 2,900 | 3,075 | | accounts with ≥1 weekly active user | 2,450 | 2,600 | | paid seats | 38,000 | 42,200 | | seats with ≥1 edit in the last 7 days | 21,000 | 21,600 | | gross renewal rate | 91% | 86% | Which metric should the team adopt as its North Star?
MCQ-4A conversion-rate metric is defined as: numerator = distinct `user_id` with a `checkout_complete` event; denominator = distinct `user_id` with a `checkout_start` event; window = 7 days from each user's first `checkout_start`; eligibility = exclude `is_internal = true`. Team A applies the definition and reports 12.4% for the week: 1,240 distinct completers out of 10,000 distinct starters. Team B's query returns 18.1% for the same week: 1,810 `checkout_complete` events out of the same 10,000 distinct starters. The extra 570 completions are repeat `checkout_complete` events from users already counted in Team B's numerator. Which change to Team B's query makes its number match Team A's definition?
MCQ-10A checkout dashboard reports 10% conversion last month (400 orders / 4,000 sessions) and 16% this month (400 orders / 2,500 sessions). The only change is that this month the dashboard excludes 1,500 sessions that hit an error page before reaching the cart; last month those sessions were counted. Orders are flat at 400. What should the team conclude?
MCQ-7An engineer is assembling a HEART dashboard for a new in-app wallet. Four metrics are already in place: 18% of new signups used the wallet at least once in their first week; 44% of wallet users used it again within 7 days; 73% of wallet sessions completed the intended action; mean post-use satisfaction is 4.2 out of 5. One HEART dimension is still uncovered. Which metric should be added to cover it?
MCQ-22A B2B collaboration product's North Star is the number of teams that co-edit a shared document in a given week. The team owns onboarding and the editor, cannot change pricing or the sales motion, and has one quarter to improve the metric. Which decomposition should it adopt?
MCQ-1A B2B analytics team ships a 'saved search' feature. In six weeks they must decide whether to fund a follow-up that emails a user when any of their saved searches has new results; it costs 2 engineer-months and only pays off if users come back to their saved searches. Telemetry: 12,000 accounts, weekly active accounts rose from 3,900 to 4,550, and saved searches created rose from 900 to 1,050. Which KPI should they adopt for that decision?
MCQ-6Your team's North Star is weekly orders per active customer. A new onboarding flow runs as a 50/50 test, each arm getting 1,000,000 new customers, both observed for the same full 28 days after rollout. Control: 40.0% of new customers become weekly repeaters, placing 2.00 orders a week. Variant: 38.0% become weekly repeaters, placing 2.06 orders a week; both shifts p < 0.01. Support contacts move from 3.1 to 4.4 per 1,000 orders. Which read of the test is correct?
MCQ-25A weekly active-usage metric rises 6% the week after a release. The release was gated by a flag, so a set of legacy-plan accounts never received it; those accounts match the released accounts on plan tier and prior usage. Which comparison best establishes that the release, not seasonality, instrumentation, or noise, caused the rise?
MCQ-9A redesign shipped to all users. The dashboard shows overall activation rising from 35% to 43%, so the team plans to keep it. The platform split shows iOS activation fell from 50% to 48% and Android fell from 30% to 28%. Before the launch, 1,000 of 4,000 users were on iOS; after, 3,000 of 4,000 were on iOS. What should the team conclude?
MCQ-12A subscription product's Q3 trial-to-paid conversion was 12%. Q4 traffic historically converts 1.2 points below Q3. The growth team targets 18% in Q4 and plans to ship three onboarding changes, each expected to match the best of the last three changes, which moved conversion by +0.5, +0.9 and +1.2 points. What Q4 conversion should they expect, and how does it compare with the target?
MCQ-14A checkout A/B test was configured for a 50/50 split and powered to detect a 0.2pp change in conversion. After 14 days the logs show 480,000 users in control and 520,000 in treatment. Conversion is 10.0% in control and 10.4% in treatment, p = 0.002. What should the team do next?
MCQ-15An A/B test was configured for a 50/50 split. After a week, the assignment logs show 550,000 users in control and 450,000 in treatment (n = 1,000,000). The primary metric shows a 3% lift for treatment (p = 0.01). What should the team do?
MCQ-13Before launching a new ranking model, a team pre-registers these guardrails for a two-week A/B test: p99 latency increase ≤50 ms, error rate ≤0.5%, and revenue per session drop ≤1%. The primary metric is click-through rate. Results: CTR +4.2% (95% CI +1.1% to +7.3%, p=0.01), p99 latency +120 ms, error rate 0.48%, revenue per session -0.3%. What should the team do?
MCQ-16A three-sided ad platform ran an ad-load test: moving from 2 to 3 ads per break lifted RPM 18% over two weeks, while 30-day viewer retention fell 4 points and creator uploads fell 7%. Advertiser ROAS was flat. Leadership wants one metric architecture to steer the next year. Which design should the team adopt?
MCQ-5A payments team has 90 days of checkout incident data. The runbook for database pool exhaustion (raise the pool limit, roll the checkout service) takes about 10 minutes from page to recovery. They want one metric to page on that fires early enough to run that runbook before the checkout SLO breaches. Across past breaches: `db_pool_wait_ms` p99 rises from 20 ms to 180 ms at T-14 min; `checkout_5xx_rate` rises from 0.2% to 1.4% at T-6 min; `refund_rate` rises from 0.4% to 1.1% at T+35 min; `support_tickets` rises at T+50 min; `node_disk_io_wait_ms` rises at T-22 min but only during the 02:00 nightly backup window and is uncorrelated with checkout breaches (0.04). Which metric should they page on?
MCQ-8A team's goal for a new 'bulk approve' feature in an expense tool is to cut the time finance staff spend approving expenses. They instrument `bulk_approve_clicked`, `bulk_approve_confirmed`, `expense_approved_individually`, and `bulk_approve_undone`. In week 1, 4,000 expenses are approved: 3,000 through bulk approve and 1,000 individually. Of the 3,000 bulk approvals, 600 are undone within the hour. The team reports 'bulk approve adoption = 3,000 / 4,000 = 75%' and calls the feature a success. Which statement best explains why 75% cannot show the goal was met?
MCQ-11A growth dashboard's checkout-conversion tile divides purchases by checkout_started. Purchases are settled orders counted by the payments service; checkout_started is a client-side event emitted by the mobile SDK. Week of Feb 17: 12,000 / 20,000 = 60.0%. Week of Mar 3: 12,000 / 15,000 = 80.0%. No checkout, pricing or payment code shipped between those weeks and order volume was flat; the only release was iOS v7.4 on Mar 3, whose notes list an analytics SDK bump from 3.10 to 3.12 and a tracking-plan cleanup. What explains the 80%, and what should the team do with the series?
MCQ-17A team's DORA dashboard for the last quarter shows deployment frequency up from 3/day to 12/day, lead time for changes down from 3 days to 6 hours, change failure rate up from 6% to 24%, and failed-deployment recovery time up from 20 minutes to 90 minutes. The deploy log shows that 9 of the 12 daily deploys are rollbacks of a previous release. What does the dashboard actually show?
MCQ-18A password manager's team is told to raise average session duration from 45 s to 90 s. Telemetry shows users who retrieve a credential in under 20 s renew at 92%, while users who take over 60 s renew at 71%, and every session in the slow group ends in a successful login. What should the team do?
MCQ-19A support org pays agents a monthly bonus on tickets resolved per agent. Baseline: 42 tickets/agent/day, 6% reopen rate, CSAT 4.6/5. Two quarters after the bonus starts: 61 tickets/agent/day, 23% reopen rate, CSAT 3.9/5, and median first-reply time unchanged. Which reading of these numbers is right?
MCQ-20An agent upgrade leaves 24 of 200 hosts no longer shipping metrics. Those 24 are the hosts in the memory-pressured tier, and memory pressure is the suspected cause of the user-reported timeouts. The fleet error-rate dashboard, computed as sum(errors)/sum(requests) over the 176 reporting hosts, reads 0.4%; those hosts carry 70% of request volume. What does the 0.4% tell the on-call engineer about the fleet error rate?
MCQ-23A subscription team cannot see whether customers will renew until contracts expire, so it plans to use weekly admin logins as a stand-in for retention. What has to be true for that proxy to be worth tracking?
MCQ-24A payments team has used checkout API p99 latency as the leading indicator for its 99.95% checkout-success SLO for three years, paging when it exceeds 300 ms. Six months ago the API moved behind an edge tier that retries idempotent requests; since then the service-side p99 has read 480 ms, the edge's end-to-end p99 is 260 ms, and the success SLO has held at 99.95%. What should the team do with the service-side p99 metric?
MCQ-21Your team ships a redesigned 3-step checkout. Over two weeks and 240,000 sessions, overall conversion falls from 68% to 64%. Broken out, desktop holds at 71%, Android holds at 66%, and iOS falls from 66% to 52%; iOS is 30% of sessions. Funnel events put the iOS loss at step 2 (address entry): median time on the step rises from 9 s to 34 s, and 22% of iOS users abandon after a failed form validation. No client-side error event fires on that failure, and support tickets say only that "the Continue button does nothing." The release freeze is one week away. What should the team do next?