Top 100 Data Analyst Interview Questions and Answers

The first question I would ask about join fan-out inflating totals is what decision the answer will change.

A join returns one row per matching pair, so joining a one-row-per-order table to a many-rows-per-order table repeats every order-level value once per item. Any sum of an order-level column after that join is multiplied by the item count.

Concretely, check the grain of both tables before joining, aggregate the many side to the grain of the one side first, and compare row counts before and after the join so fan-out is caught rather than discovered.

The reason for that specificity is a failure I have seen: A revenue query joined orders to order lines and summed order_total, reporting 4.1 million against 1.7 million actually billed because the average order had 2.4 lines.

Row counts expose the fan-out.

Step	Rows	SUM(order_total)
orders alone	10,000	1.7m
joined to lines	24,000	4.1m
lines pre-aggregated	10,000	1.7m

I would not consider it settled without evidence: Compare COUNT(*) and COUNT(DISTINCT order_id) after the join and reconcile the revenue total to the billing system.

Know the grain of every table before you join it.

I would open a WHERE filter on the right table of a LEFT JOIN by checking the denominator before the numerator.

A condition in the WHERE clause on a column from the right-hand table removes every row where that column is NULL, which includes all the unmatched rows the LEFT JOIN was meant to keep. The query silently becomes an inner join.

Concretely, move conditions on the optional table into the ON clause, or test explicitly for NULL, and confirm the output row count equals the left table's count.

The reason for that specificity is a failure I have seen: A report of customers and their 2026 orders filtered order_date in the WHERE clause and dropped 38,000 customers with no orders, so the share of inactive customers read as 0 percent instead of 31 percent.

Where the filter sits decides the answer.

Filter placement	Customers returned	Inactive share
WHERE order_date >= '2026-01-01'	84,000	0%
ON ... AND order_date >= '2026-01-01'	122,000	31%

I would not consider it settled without evidence: Check that the result has exactly as many rows as the customer table and that customers without orders appear with NULLs.

A LEFT JOIN only stays left if the filter lives in the ON clause.

My starting point for COUNT of a nullable column is the row count before and after every step.

COUNT(*) counts rows, COUNT(column) counts rows where that column is not NULL, and COUNT(DISTINCT column) counts unique non-NULL values. Choosing the wrong one changes the denominator of every rate built on it.

Concretely, decide what entity is being counted, pick the form that counts that entity, and check the NULL rate of the column so the gap between the forms is explained rather than surprising.

The reason for that specificity is a failure I have seen: A signup conversion rate used COUNT(email) as its denominator, and because 12 percent of guest sessions had no email the rate was overstated from 3.1 to 3.5 percent.

Three counts over one sessions table.

Expression	Result	Counts
COUNT(*)	50,000	sessions
COUNT(email)	44,000	sessions with an email
COUNT(DISTINCT email)	31,500	people

I would not consider it settled without evidence: Run all three counts side by side on the column and explain every difference between them.

Every count answers a different question, so name the question first.

With WHERE versus HAVING, the headline figure is where I stop trusting and start checking.

WHERE filters rows before grouping and HAVING filters groups after aggregation. A condition on a raw column belongs in WHERE, and a condition on an aggregate such as SUM or COUNT can only go in HAVING.

Concretely, put row-level filters in WHERE so fewer rows are grouped, reserve HAVING for conditions on aggregates, and read the query in the order FROM, WHERE, GROUP BY, HAVING, SELECT.

The reason for that specificity is a failure I have seen: An analyst grouped by status and filtered status = 'refunded' in HAVING instead of WHERE, and the database grouped all 9 million rows before discarding most of them, turning a 4 second query into a 3 minute one.

Same result, different work.

Placement	Rows grouped	Runtime
HAVING status = 'refunded'	9,000,000	3 min
WHERE status = 'refunded'	210,000	4 s

I would not consider it settled without evidence: Read the query plan and confirm row-level filters are applied before the aggregation step.

Filter rows early and groups late.

I would scope ROW_NUMBER, RANK, and DENSE_RANK with the stakeholder before writing a single query.

ROW_NUMBER gives every row a unique number even when values tie, RANK gives tied rows the same number and then skips, and DENSE_RANK gives tied rows the same number without skipping. The choice decides whether a top-3 query returns three rows or more.

Concretely, decide how ties should be treated before writing the query, add a deterministic tiebreaker to ORDER BY when using ROW_NUMBER, and test on data that contains a tie.

The reason for that specificity is a failure I have seen: A top-3 sellers bonus query used ROW_NUMBER without a tiebreaker, and two reps tied on 142 deals were ordered arbitrarily, so the rep paid the bonus changed between two runs of the same query.

One tie, three numberings.

Rep	Deals	ROW_NUMBER	RANK	DENSE_RANK
A	150	1	1	1
B	142	2	2	2
C	142	3	2	2
D	130	4	4	3

I would not consider it settled without evidence: Run the query twice on a fixture with a deliberate tie and confirm the output is identical and matches the stated tie rule.

Ties are a business rule, so write them down before ranking.

The honest answer on running totals with a window frame begins with what the data cannot tell us.

SUM() OVER (ORDER BY day) produces a running total, but the default frame with ORDER BY is RANGE, which treats rows with the same order value as peers and adds them all at once. Duplicate dates therefore jump instead of accumulating row by row.

Concretely, aggregate to one row per day before applying the window, or state ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW explicitly, and fill missing days so the running total has no gaps.

The reason for that specificity is a failure I have seen: A cumulative revenue chart was computed on raw transactions, and because 3 transactions shared a timestamp the curve showed the same 18,400 total on three rows, which a reviewer read as a data freeze.

One row per day, then accumulate.

Day	Revenue	Running total
Sep 1	5,000	5,000
Sep 2	7,200	12,200
Sep 3	6,200	18,400
total check	18,400	18,400

I would not consider it settled without evidence: Check that the last running total equals the plain SUM over the whole period and that there is exactly one row per day.

Pin the grain and the frame before trusting a cumulative line.

I would test month-over-month growth with LAG on a ten-row example I can verify by hand.

LAG(revenue) OVER (ORDER BY month) returns the prior row's value, so growth is revenue divided by the lagged value minus one. The edge cases are the first month, where the lag is NULL, a prior month of zero, and a missing month that makes LAG reach back two months.

Concretely, build a complete calendar of months first, left join revenue to it with zero for empty months, guard the division against zero, and label the first month as having no comparison.

The reason for that specificity is a failure I have seen: A growth report ran LAG over a table with no row for February, so March was compared with January and showed 46 percent growth that was really two months of change.

A missing month doubles the gap.

Month	Revenue	Prior	Growth
Jan	100,000	none	none
Feb	118,000	100,000	18.0%
Mar	146,000	118,000	23.7%
Mar without Feb row	146,000	100,000	46.0%, wrong

I would not consider it settled without evidence: Join to a generated month series and confirm every month appears exactly once before computing growth.

LAG looks at the previous row, not the previous month.

The part of deduplicating to the latest record per key that interviewers probe is the edge case, not the syntax.

Number the rows within each customer ordered by update time descending and keep row number one. GROUP BY with MAX(updated_at) finds the latest time but not the other columns of that row, so it has to be joined back or it mixes values from different rows.

Concretely, use ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY updated_at DESC, id DESC) and filter to 1, with a tiebreaker so identical timestamps resolve the same way every run.

The reason for that specificity is a failure I have seen: A deduplication used MAX on each column separately and produced customer rows whose email came from 2025 and whose plan came from 2026, a combination that never existed for 2,300 customers.

Keep row number one per customer.

customer_id	updated_at	plan	keep
7	2026-03-02	pro	yes, rn = 1
7	2025-11-19	basic	no, rn = 2
9	2026-01-05	basic	yes, rn = 1

I would not consider it settled without evidence: Assert one row per customer_id afterwards and spot-check that every output row exists as a whole row in the source.

Pick a whole row, never a column at a time.

For the Nth highest value per group, I would state the assumption out loud and then try to break it.

DENSE_RANK over salary descending within each department identifies the second distinct salary, while ROW_NUMBER would return the second person even if they earn the same as the first. Departments with fewer than two distinct salaries must still appear if the question asks for every department.

Concretely, rank with DENSE_RANK partitioned by department, filter to rank 2, and left join the result back to the department list so departments without a second salary show NULL rather than disappearing.

The reason for that specificity is a failure I have seen: An interview answer used ORDER BY salary DESC LIMIT 1 OFFSET 1 across the company, which returned one row instead of 14 departments and repeated the top salary whenever the two highest earners tied.

Second distinct value versus second person.

Dept	Salaries	Second distinct	Second person
Sales	90k, 90k, 70k	70k	90k
Ops	60k	NULL	NULL
Tech	150k, 120k	120k	120k

I would not consider it settled without evidence: Test on a department with a tie at the top and on a department with one employee and confirm both give the stated answer.

Say whether second means second person or second value.

What separates a good answer on a cohort retention query is a number someone else can check.

Assign each user a cohort month from their first activity, compute the months between cohort and each later activity, and count distinct active users per cohort and month offset divided by the cohort's size. Retention is always relative to the cohort's starting count, not to the previous month.

Concretely, build the cohort table first, join activity to it, group by cohort month and month number, and divide by the month-zero count, leaving recent cohorts' future months blank rather than zero.

The reason for that specificity is a failure I have seen: A retention chart treated months that had not happened yet as zero activity, so the three most recent cohorts appeared to churn completely and average month-6 retention was reported as 18 percent instead of 27 percent.

Future months stay empty.

Cohort	Size	M1	M2	M3
Jan	1,000	42%	33%	29%
Feb	1,200	40%	31%	not yet
Mar	900	44%	not yet	not yet

I would not consider it settled without evidence: Check that month zero is 100 percent for every cohort and that cells beyond today's date are empty rather than zero.

A cohort cannot be retained in a month it has not reached.

I would treat a funnel conversion query as a claim that has to survive a second slice of the data.

A funnel counts the users who reach each step in order, and each step's conversion is its users divided by the previous step's users. Users must be counted once per step, and a later step should only count users who also completed the earlier ones.

Concretely, take each user's first timestamp per step, require each step's time to be after the previous step's, count distinct users per step, and state the conversion window, such as purchase within 7 days of signup.

The reason for that specificity is a failure I have seen: A funnel counted purchase events without requiring a prior signup, so returning customers who bought without a new signup pushed signup-to-purchase conversion to 64 percent when the ordered figure was 21 percent.

An ordered funnel with a 7-day window.

Step	Users	Step conversion
visited	50,000	none
signed up	6,000	12.0%
purchased within 7 days	1,260	21.0%

I would not consider it settled without evidence: Confirm each step's user count is less than or equal to the previous step's and that the conversion window is stated in the query.

A funnel is a sequence, so enforce the order.

Before I share anything on date truncation and time zones, I would write the one-sentence recommendation.

Timestamps are usually stored in UTC while the business closes its day in local time, so truncating a UTC timestamp to a date moves late-evening orders into the next day. The total over a month barely changes but every daily figure shifts.

Concretely, convert timestamps to the business time zone before truncating to a date, write the time zone into the metric definition, and reconcile one day against finance's figure to the order.

The reason for that specificity is a failure I have seen: A dashboard truncated UTC timestamps for a New York business, so orders between 8 pm and midnight Eastern landed on the next day and daily revenue differed from finance by 6 to 9 percent.

The same order, two dates.

Order time UTC	Eastern time	UTC date	Business date
2026-09-02 01:30	Sep 1 21:30	Sep 2	Sep 1
2026-09-02 14:00	Sep 2 10:00	Sep 2	Sep 2

I would not consider it settled without evidence: Reconcile one full day to the order level against the finance ledger after converting to the business time zone.

Decide whose midnight the day ends at.

The first question I would ask about NOT IN with a NULL in the subquery is what decision the answer will change.

NOT IN compares a value with every value in the list, and a comparison with NULL is unknown rather than false. If the subquery returns a single NULL, no row can pass the condition and the result is empty.

Concretely, use NOT EXISTS, which treats NULLs sensibly, or filter NULLs out of the subquery, and check the subquery column for NULLs before relying on NOT IN.

The reason for that specificity is a failure I have seen: A query for customers who never received a refund returned 0 rows because one refund record had a NULL customer_id, and the retention team briefly concluded every customer had been refunded.

A single NULL changes everything.

Subquery values	Query	Rows returned
1, 2, 3	NOT IN	4,997
1, 2, 3, NULL	NOT IN	0
1, 2, 3, NULL	NOT EXISTS	4,997

I would not consider it settled without evidence: Count NULLs in the subquery column and compare the NOT IN result with a NOT EXISTS version of the same query.

One NULL can empty a NOT IN.

I would open structuring a long query with CTEs by checking the denominator before the numerator.

Common table expressions let you name each step, such as filtered orders, one row per customer, and the final aggregate, so every step can be read and checked on its own. A query that nests five subqueries hides where rows were lost or duplicated.

Concretely, write one CTE per logical step, give each a name that states its grain, and select from each step on its own while developing to check its row count before building the next.

The reason for that specificity is a failure I have seen: A nested query lost 11 percent of customers in a join three levels deep, and nobody found it for a month because no intermediate result could be inspected without rewriting the query.

Each CTE states its grain.

CTE	Grain	Rows
paid_orders	order	81,000
customer_totals	customer	22,400
final	segment	5

I would not consider it settled without evidence: Record the row count and grain of each CTE while developing and keep those checks as comments or tests.

Name each step and check each step's row count.

My starting point for consecutive-day streaks is the row count before and after every step.

The gaps-and-islands technique subtracts a row number from each date. Within a run of consecutive dates the difference stays constant, so grouping by user and that difference gives one group per streak.

Concretely, deduplicate to one row per user per day, compute date minus ROW_NUMBER() days within each user ordered by date, group by user and that anchor, and keep groups with a count of 5 or more.

The reason for that specificity is a failure I have seen: A streak query forgot to deduplicate multiple logins per day, so the row numbers ran ahead of the dates and 3,100 users were credited with streaks they never had.

A constant anchor marks one streak.

Login date	Row number	Date minus row number	Streak
Sep 1	1	Aug 31	A
Sep 2	2	Aug 31	A
Sep 3	3	Aug 31	A
Sep 5	4	Sep 1	B

I would not consider it settled without evidence: Test on a fixture with two logins on one day and a one-day gap and confirm the streak lengths are exactly right.

Deduplicate to the day before counting days.

With median and percentiles in SQL, the headline figure is where I stop trusting and start checking.

Order values are right-skewed, so a few large orders pull the mean well above what a typical customer spends. The median, computed with PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY value), describes the middle customer.

Concretely, report the median with the 25th and 75th percentiles next to the mean, and state whether the percentile function interpolates, because PERCENTILE_CONT and PERCENTILE_DISC differ on small groups.

The reason for that specificity is a failure I have seen: A pricing proposal used a mean order value of 142 dollars as typical, but the median was 64 dollars, so the new free-shipping threshold was set above what 80 percent of customers ever spent.

A skewed order-value distribution.

Statistic	Order value
mean	$142
median	$64
25th percentile	$31
75th percentile	$118

I would not consider it settled without evidence: Plot the distribution and report mean, median, and quartiles together before choosing one as typical.

Typical means the middle, not the average.

I would scope pivoting rows into columns with CASE with the stakeholder before writing a single query.

Conditional aggregation pivots data, since SUM(CASE WHEN month = 'Jan' THEN revenue END) produces one column and grouping by product produces one row per product. Using ELSE 0 rather than leaving the NULL decides whether a missing month reads as zero or as unknown.

Concretely, write one CASE expression per output column, group by the row key, and choose deliberately between zero and NULL for missing combinations, because averages treat the two differently.

The reason for that specificity is a failure I have seen: A pivot used ELSE 0 for products not yet launched, so their average monthly revenue was divided across 12 months instead of 4 and new products looked 67 percent weaker than they were.

NULL marks months before launch.

Product	Jan	Feb	Mar
Basic	12,000	13,500	14,100
Pro	NULL	NULL	9,800

I would not consider it settled without evidence: Check the pivoted totals against the unpivoted SUM and confirm how missing combinations are represented.

A missing month and a zero month are different facts.

The honest answer on DAU, MAU, and stickiness begins with what the data cannot tell us.

DAU/MAU divides the average daily active users by the monthly active users and estimates how many days a typical monthly user is active. A ratio of 0.2 means the average monthly user appears about 6 days a month.

Concretely, define active precisely, such as a meaningful action rather than an app open, compute DAU as the average over the days of the month, count MAU as distinct users over those same days, and compare ratios only across products that use the same definition.

The reason for that specificity is a failure I have seen: A team counted a push-notification open as active, and DAU/MAU rose from 0.18 to 0.31 after a notification campaign while time spent in the product did not change at all.

The definition moves the ratio.

Definition of active	Avg DAU	MAU	DAU/MAU
any open	62,000	200,000	0.31
core action	36,000	200,000	0.18

I would not consider it settled without evidence: Recompute the ratio with a stricter definition of active and check that the trend holds.

Stickiness is only as honest as the definition of active.

I would test writing queries that run efficiently on a ten-row example I can verify by hand.

Most slow analytical queries read far more data than they need. Filtering on the partition or index column, selecting only needed columns, and aggregating before joining usually cut runtime by orders of magnitude.

Concretely, read the query plan, filter on the date partition first, replace SELECT * with named columns, pre-aggregate large tables before joining, and avoid wrapping filtered columns in functions that prevent partition pruning.

The reason for that specificity is a failure I have seen: A query filtered on DATE(created_at) instead of a range on created_at, which disabled partition pruning, so it scanned 2.4 terabytes across three years instead of 18 gigabytes for one week.

Each change cuts the scan.

Change	Data scanned	Runtime
original	2.4 TB	20 min
range filter on partition	18 GB	40 s
named columns only	6 GB	14 s

I would not consider it settled without evidence: Compare the bytes scanned in the query plan before and after each change.

Read less data before asking for more compute.

The part of UNION versus UNION ALL that interviewers probe is the edge case, not the syntax.

UNION removes duplicate rows across the combined result and UNION ALL keeps them. When two sources legitimately contain identical rows, such as two separate 10-dollar payments from the same customer on the same day, UNION silently deletes real data.

Concretely, default to UNION ALL when stacking sources and deduplicate explicitly on a key when duplicates are genuinely errors, so the rule for what counts as a duplicate is visible.

The reason for that specificity is a failure I have seen: A payments report combined card and wallet tables with UNION, which merged identical same-day payments and understated revenue by 1.8 percent across 40,000 customers.

What UNION quietly removed.

Method	Rows	Revenue
UNION ALL	612,000	$9.83m
UNION	601,000	$9.65m
difference	11,000	1.8%

I would not consider it settled without evidence: Compare row counts of UNION and UNION ALL and inspect a sample of the rows that UNION removed.

Deduplicate on purpose, not as a side effect.

For joining on mismatched keys, I would state the assumption out loud and then try to break it.

Join keys often differ in type, case, padding, or format, such as a numeric ID in one table and a zero-padded string in another. The join then drops the unmatched rows without any error.

Concretely, profile both key columns for type, length, and example values, normalize them explicitly by trimming, casting, and lowercasing, and measure the match rate before and after.

The reason for that specificity is a failure I have seen: A CRM export stored customer IDs with a trailing space, and a join to billing matched 58 percent of customers instead of 99 percent, which an analyst first reported as a churn spike.

One TRIM restores the match.

Key handling	Matched customers	Match rate
raw	29,000	58%
TRIM applied	49,500	99%

I would not consider it settled without evidence: Report the match rate of the join and list a sample of unmatched keys from each side.

Measure the match rate of every important join.

What separates a good answer on NULLs in averages and sums is a number someone else can check.

Aggregate functions skip NULLs, so AVG divides by the number of non-NULL values rather than by all rows. If NULL means zero, such as no purchase, the average is overstated, and if NULL means unknown, replacing it with zero understates it.

Concretely, decide what NULL means for each column, use COALESCE to zero only when missing truly means none, and report how many rows were NULL alongside the average.

The reason for that specificity is a failure I have seen: Average spend per user skipped the 70 percent of users with no purchases, reporting 85 dollars when spend per user across everyone was 25.50 dollars.

Two averages, two questions.

Treatment	Users counted	Average spend
NULLs skipped	3,000	$85.00
NULL as zero	10,000	$25.50

I would not consider it settled without evidence: Report the NULL count next to the aggregate and compute the result both ways to show which question each answers.

Decide what a NULL means before you average over it.

I would treat sampling a large table for exploration as a claim that has to survive a second slice of the data.

A LIMIT returns whatever rows the engine reads first, which are often the oldest partition or one customer, so it is not a random sample. A random or hashed sample on the entity key gives a representative subset.

Concretely, sample by hashing the entity ID, such as keeping users where a hash modulo 100 is 0, so each sampled user keeps all their events, and compare a few totals between the sample and the full table.

The reason for that specificity is a failure I have seen: An analyst explored the first 100,000 rows returned by LIMIT, which all came from 2019, and concluded mobile was 20 percent of traffic when it was 64 percent across the full table.

Which rows you get depends on the method.

Method	Date range	Mobile share
LIMIT 100000	2019 only	20%
1% hashed user sample	2019 to 2026	63%
full table	2019 to 2026	64%

I would not consider it settled without evidence: Compare the sample's date range and segment mix with the full table before drawing conclusions.

LIMIT is not a sample.

Before I share anything on late-arriving data in recent periods, I would write the one-sentence recommendation.

Events often arrive hours or days after they happen because of offline devices, batch uploads, or delayed payments. The most recent period is incomplete, so comparing it to complete periods shows a false drop.

Concretely, measure how complete each day is after 1, 2, and 7 days, mark the incomplete window in reports, and compare recent periods only with earlier periods at the same age.

The reason for that specificity is a failure I have seen: A morning report showed a 22 percent drop in yesterday's orders that disappeared by the afternoon, and a marketing campaign was paused because of it.

How complete a day is, by age.

Day age at report time	Share of final orders
8 hours	78%
1 day	94%
3 days	99.5%

I would not consider it settled without evidence: Plot how each day's count grows over the following week and report the typical completeness at the time the report runs.

Recent data is incomplete until proven otherwise.

The first question I would ask about window functions versus GROUP BY is what decision the answer will change.

GROUP BY collapses rows into one row per group, while a window function computes a value across a group and keeps every original row. Showing each order next to its customer's total, or each day next to a 7-day average, needs the row detail and the group value together.

Concretely, use GROUP BY when the output grain is the group, and SUM or AVG OVER (PARTITION BY ...) when the output grain is the row, then check that the row count is unchanged.

The reason for that specificity is a failure I have seen: An analyst computed each order's share of customer spend by grouping and joining back without the customer key, which cross-joined every order to every customer total and produced 4.6 billion rows before the query was killed.

Row detail with the group total beside it.

Order	Customer	Amount	Customer total	Share
101	A	40	100	40%
102	A	60	100	60%
103	B	25	25	100%

I would not consider it settled without evidence: Check that the output has the same number of rows as the input and that shares within each customer sum to 100 percent.

Choose the output grain first, then the tool.

I would open clarifying an ambiguous data request by checking the denominator before the numerator.

Active customers can mean anyone who logged in, anyone who purchased, or anyone with a paid plan, over a day, a month, or a year, and each definition gives a different number. The right one depends on the decision the manager is making.

Concretely, ask what decision the number feeds, propose a definition with its time window in writing, show how much the candidates differ, and get agreement before building anything recurring.

The reason for that specificity is a failure I have seen: Two teams reported active customers of 48,000 and 131,000 to the same board meeting because one used paid plans and the other used logins in 90 days.

One question, three honest answers.

Definition	Window	Active customers
logged in	90 days	131,000
purchased	90 days	74,000
paid plan	today	48,000

I would not consider it settled without evidence: Write the agreed definition and time window into the request and confirm it with the requester before the analysis is delivered.

Agree on the definition before anyone sees a number.

My starting point for defining a metric precisely is the row count before and after every step.

A complete definition states the numerator, the denominator, the unit being counted, the time window, the time zone, the filters and exclusions, and the source table. Two analysts given it should produce the same number to the unit.

Concretely, write the definition as a short spec plus a reference query, include one worked example with real counts, and version it so a changed definition is visible in the history.

The reason for that specificity is a failure I have seen: Conversion rate was defined only as purchases over visitors, and one analyst excluded internal traffic while another did not, so the same week was reported as 2.9 and 3.4 percent.

A definition someone else can rerun.

Element	Example
numerator	distinct users with a paid order
denominator	distinct non-internal visitors
window	calendar week, US Eastern
source	analytics.events v3
last week	3,120 / 104,000 = 3.0%

I would not consider it settled without evidence: Have a second analyst reproduce the number from the written definition alone and resolve any difference.

If two people cannot reproduce it, it is not defined yet.

With diagnosing a sudden metric drop, the headline figure is where I stop trusting and start checking.

Start by ruling out data problems such as late loads, tracking bugs, and definition changes, then check external causes such as holidays and outages, and only then break the drop down by segment to find where it is concentrated. A real drop is usually concentrated in one platform, region, or channel.

Concretely, confirm the data is complete, compare with the same weekday last week, split the change by platform, country, channel, and new versus returning users, and look for the segment that explains most of the gap.

The reason for that specificity is a failure I have seen: An analyst spent a day building a customer-behaviour story for a 15 percent drop that was entirely caused by an Android app release that stopped sending order events.

The drop lives in one platform.

Segment	Last Tuesday	Yesterday	Change
iOS	4,100	4,050	-1%
Android	2,700	1,560	-42%
Web	3,200	2,890	-10%
total	10,000	8,500	-15%

I would not consider it settled without evidence: Decompose the drop by segment and show which segments account for the majority of the missing orders.

Check the data before you explain the behaviour.

I would scope percentage change versus percentage points with the stakeholder before writing a single query.

The rate rose by 1 percentage point, which is a 25 percent relative increase. Reporting 1 percent or 25 percent without saying which kind of change is meant can mislead a reader by a factor of 25.

Concretely, use percentage points for differences between rates and percent for relative change, and state both with the base value when the audience will make a decision on it.

The reason for that specificity is a failure I have seen: A launch email said conversion improved by 25 percent, leadership assumed conversion was now around 29 percent, and the forecast built on that reading overstated revenue by 3.8 million dollars.

One change, two ways to state it.

Measure	Value
before	4.0%
after	5.0%
absolute change	+1.0 percentage point
relative change	+25%

I would not consider it settled without evidence: Write both the absolute and relative change with the starting value in any summary that goes to decision makers.

Name the kind of percentage every time.

The honest answer on Simpson's paradox and mix shift begins with what the data cannot tell us.

An overall rate is a weighted average of segment rates, and if traffic shifts toward a segment with a lower rate, the total can fall while every segment improves. This is Simpson's paradox, and it is caused by mix, not performance.

Concretely, decompose the overall change into a rate effect within segments and a mix effect from shifting volume, and report both so the audience sees which one moved.

The reason for that specificity is a failure I have seen: A marketing team was blamed for falling conversion when both regions had improved, because a new market with a 1 percent rate grew from 10 to 50 percent of traffic.

Both segments up, total down.

Segment	Share before	Rate before	Share after	Rate after
Core	90%	5.0%	50%	5.5%
New market	10%	1.0%	50%	1.2%
total	100%	4.6%	100%	3.35%

I would not consider it settled without evidence: Compute the overall rate with last period's segment mix and this period's segment rates to isolate the rate effect.

When segments and totals disagree, look at the mix.

I would test weighted versus unweighted averages of rates on a ten-row example I can verify by hand.

A simple average of store rates gives every store equal weight, so a store with 20 visitors counts as much as one with 20,000. The company rate is total conversions divided by total visitors, which weights each store by its traffic.

Concretely, sum numerators and denominators across stores and divide once, and use the unweighted average only when the question is explicitly about the typical store.

The reason for that specificity is a failure I have seen: An operations review averaged 50 store rates to 9.8 percent, but three tiny stores with rates above 40 percent inflated it, and the traffic-weighted company rate was 6.1 percent.

Two stores, two very different weights.

Store	Visitors	Conversions	Rate
large	20,000	1,200	6.0%
small	20	9	45.0%
weighted total	20,020	1,209	6.04%
simple average	none	none	25.5%

I would not consider it settled without evidence: Compute both the weighted and unweighted averages and state which question each one answers.

The company rate is a ratio of totals.

The part of monthly churn for a subscription product that interviewers probe is the edge case, not the syntax.

Monthly churn is the number of customers who cancel during the month divided by the customers active at the start of the month. Customers who join during the month belong outside that base, and counting their quick cancellations against it distorts the rate.

Concretely, fix the starting base at the first day, count cancellations only from that base, report new-customer early churn separately, and state whether churn is counted at cancellation or at the end of the paid period.

The reason for that specificity is a failure I have seen: A churn metric divided all cancellations by end-of-month customers, and because a promotion added 30 percent new customers, reported churn fell from 6 to 4.2 percent while retention of existing customers was unchanged.

A fixed base for September.

Item	Count
customers on Sep 1	10,000
of those, cancelled in Sep	600
monthly churn	6.0%
new in Sep, cancelled in Sep	450, reported separately

I would not consider it settled without evidence: Recompute churn from the start-of-month cohort alone and compare it with the reported figure.

Churn needs a fixed starting base.

For customer lifetime value estimation, I would state the assumption out loud and then try to break it.

A simple estimate multiplies average monthly margin per customer by the expected lifetime, which is roughly one divided by monthly churn. It uses margin rather than revenue, and it assumes churn stays constant, which usually overstates value for recent cohorts.

Concretely, compute margin per customer per month, estimate churn from recent cohorts, cap the horizon, for example at 36 months, and compare the formula result with the observed cumulative margin of older cohorts.

The reason for that specificity is a failure I have seen: A marketing plan used revenue rather than margin in an LTV of 1,200 dollars, and at a 40 percent margin the real figure was 480 dollars, so the team was paying 600 dollars to acquire customers worth less than that.

LTV built on margin.

Input	Value
revenue per month	$50
margin at 40%	$20
monthly churn	4%
expected lifetime	25 months
LTV on margin	$500

I would not consider it settled without evidence: Check the formula result against the actual cumulative margin earned by a cohort that is at least two years old.

Lifetime value is margin, not revenue.

What separates a good answer on customer acquisition cost and payback is a number someone else can check.

CAC is acquisition spend divided by new customers acquired in the same period, and payback is CAC divided by monthly margin per customer. Both mislead when spend and customers come from different periods or when organic signups are counted as paid acquisitions.

Concretely, match spend to the customers it produced using a lag or attribution window, separate paid from organic customers, and compute payback on margin rather than revenue.

The reason for that specificity is a failure I have seen: A blended CAC divided paid spend by all new customers, including 60 percent who came organically, and reported 80 dollars when the paid CAC was 200 dollars.

Paid CAC and payback on margin.

Measure	Value
paid spend	$200,000
paid customers	1,000
paid CAC	$200
monthly margin per customer	$25
payback	8 months

I would not consider it settled without evidence: Compute CAC separately for paid and organic customers and check that the time windows for spend and customers match.

Divide paid spend by the customers it paid for.

I would treat north star and guardrail metrics as a claim that has to survive a second slice of the data.

A north star metric captures the value customers get from the product, such as weekly orders delivered, and aligns teams on one outcome. Guardrail metrics, such as refunds, latency, and unsubscribe rate, catch the harm that optimizing the north star alone can cause.

Concretely, pair the north star with two to four guardrails, set a threshold for each, and treat a guardrail breach as a reason to stop a change even when the north star improves.

The reason for that specificity is a failure I have seen: A team raised weekly orders by 6 percent with aggressive discounts while the refund rate rose from 2 to 7 percent, and the net margin per order became negative before anyone looked.

The win that breached two guardrails.

Metric	Before	After	Threshold
weekly orders	100,000	106,000	north star
refund rate	2%	7%	max 3%
margin per order	$4.10	-$0.60	min $3.00

I would not consider it settled without evidence: Report every experiment's result on the north star together with each guardrail and its threshold.

A north star without guardrails invites the wrong wins.

Before I share anything on leading and lagging indicators, I would write the one-sentence recommendation.

Revenue is a lagging indicator that reflects decisions made weeks earlier. Leading indicators, such as trial starts, activation rate, and pipeline created, move first and give time to act, but only if they are shown to predict the outcome.

Concretely, pick candidate leading indicators, check historically how strongly and with what lag they track revenue, and keep the ones that move reliably ahead of it.

The reason for that specificity is a failure I have seen: A team tracked website visits as a leading indicator for two quarters, but visits had no relationship with revenue after a bot surge, and a 12 percent decline in trial activation went unnoticed for eight weeks.

Candidates tested against revenue.

Indicator	Lead time	Correlation with revenue
trial activations	6 weeks	0.81
pipeline created	8 weeks	0.74
site visits	none	0.12

I would not consider it settled without evidence: Measure the historical correlation and lag between each leading indicator and revenue before relying on it.

A leading indicator has to earn the title with data.

The first question I would ask about vanity metrics is what decision the answer will change.

A vanity metric goes up easily and does not change a decision, such as cumulative signups, which can never fall. A useful metric can move in both directions and is tied to value, such as weekly active customers or revenue retained.

Concretely, ask what action would follow if the metric moved either way, and replace cumulative or easily inflated counts with rates or active counts over a fixed window.

The reason for that specificity is a failure I have seen: A startup reported 500,000 cumulative signups to investors while monthly active users had fallen from 60,000 to 41,000 over the same two quarters.

The cumulative line hides the decline.

Quarter	Cumulative signups	Monthly active users
Q1	380,000	60,000
Q2	440,000	52,000
Q3	500,000	41,000

I would not consider it settled without evidence: Show the metric alongside an active or rate-based version and check whether they tell the same story.

If it can only go up, it cannot warn you.

I would open mean versus median for skewed data by checking the denominator before the numerator.

The mean is pulled toward extreme values, so for skewed quantities such as income, session length, or deal size it can describe almost nobody. The median describes the middle case and changes little when a few extreme values appear.

Concretely, look at the distribution first, report the median and a spread measure for skewed data, and keep the mean when totals matter, since total revenue equals mean times count.

The reason for that specificity is a failure I have seen: An average session length of 14 minutes was driven by 2 percent of sessions left open overnight, and the median session was 3 minutes, so a content plan built for 14-minute visits failed.

Session length, heavily skewed.

Statistic	Session length
mean	14 min
median	3 min
90th percentile	11 min
sessions over 2 hours	2%

I would not consider it settled without evidence: Plot a histogram and report the mean, median, and 90th percentile before choosing a summary.

Summarize the shape before you summarize the number.

My starting point for handling outliers is the row count before and after every step.

An outlier is either an error, such as a test transaction or a unit mistake, or a real extreme, such as one very large customer. Errors should be fixed or removed with a documented rule, and real extremes should be kept and analyzed separately rather than deleted.

Concretely, investigate the top values individually, define a written rule for exclusions, report results with and without the extremes, and never remove points just because they spoil the result.

The reason for that specificity is a failure I have seen: An analyst capped revenue at the 99th percentile to tidy a chart and removed the company's 12 largest accounts, which together made up 31 percent of revenue.

Three outliers, three decisions.

Record	Value	Cause	Action
order 88	$1,000,000	test order	remove
order 412	$98,000	enterprise deal	keep, report separately
order 97	$5,400	priced in cents	fix to $54

I would not consider it settled without evidence: List the removed records, their share of the total, and the reason each was removed.

Explain an outlier before you remove it.

With year-over-year versus month-over-month comparison, the headline figure is where I stop trusting and start checking.

Month-over-month comparisons mix real change with seasonality, and for many businesses December is always far above November. Comparing with December last year removes the seasonal pattern and shows whether the business actually grew.

Concretely, use year-over-year for seasonal businesses, align weekdays and holidays, for example by comparing the same retail weeks, and adjust for calendar differences such as an extra weekend.

The reason for that specificity is a failure I have seen: A 40 percent month-over-month rise was celebrated, but December was down 6 percent year over year, and the decline was only noticed when January came in weak.

Up on last month, down on last year.

Comparison	Sales	Change
November 2026	$1.00m	none
December 2026	$1.40m	+40% MoM
December 2025	$1.49m	-6% YoY

I would not consider it settled without evidence: Compare the period with the same period a year earlier, aligned by weekday and holiday.

Compare with the same season before claiming growth.

I would scope segmenting customers for analysis with the stakeholder before writing a single query.

A useful segmentation groups customers by behaviour that matters for a decision, such as recency, frequency, and monetary value, and yields segments large enough to act on. Segments should be few enough to explain and stable enough to track.

Concretely, score customers on recency, frequency, and spend in quintiles, combine the scores into a handful of named segments, and check each segment's size and share of revenue.

The reason for that specificity is a failure I have seen: A segmentation created 125 RFM cells, most with fewer than 50 customers, and the marketing team could not act on any of them, so the work was never used.

Four actionable segments.

Segment	Customers	Revenue share
champions	8%	41%
loyal	17%	28%
at risk	22%	14%
lapsed	53%	17%

I would not consider it settled without evidence: Report each segment's customer count, revenue share, and a planned action before presenting the scheme.

A segment is only useful if someone can act on it.

The honest answer on marketing attribution models begins with what the data cannot tell us.

Last-click attribution gives all credit to the final touch and favours channels such as branded search that capture existing demand. Multi-touch models spread credit across the journey, but every rule-based model is an assumption, and only experiments measure incremental impact.

Concretely, show results under last-click and one multi-touch model, flag channels whose credit changes a lot between them, and test the largest spend decisions with holdout or geo experiments.

The reason for that specificity is a failure I have seen: Last-click credited branded search with 45 percent of sales, and when its budget was doubled, sales did not change, because those buyers were already searching for the brand.

Credit versus measured lift.

Channel	Last click	Linear	Incremental in test
branded search	45%	22%	6%
paid social	12%	24%	19%
email	18%	20%	15%

I would not consider it settled without evidence: Run a holdout or geo experiment for the channel with the largest budget and compare incremental sales with its attributed sales.

Attribution models assign credit, experiments measure it.

I would test revenue decomposition on a ten-row example I can verify by hand.

Revenue equals customers times orders per customer times average order value, so a change can be split into those factors. Decomposing shows whether growth came from more buyers, more frequent buying, or higher prices or mix.

Concretely, compute each factor for both periods, attribute the change to each factor in sequence or with a log decomposition so the parts add up, and investigate the largest part further.

The reason for that specificity is a failure I have seen: A 10 percent revenue rise was credited to a loyalty program, but decomposition showed order frequency was flat and the growth came from a 9 percent price increase.

Where the 10 percent came from.

Factor	Last year	This year	Change
customers	20,000	20,200	+1.0%
orders per customer	5.0	5.0	0%
average order value	$50.00	$54.46	+8.9%
revenue	$5.00m	$5.50m	+10%

I would not consider it settled without evidence: Show that the product of the factors reproduces revenue in both periods and that the parts sum to the total change.

Split the total before explaining it.

The part of unit economics per order that interviewers probe is the edge case, not the syntax.

Contribution margin per order is revenue minus the variable costs that each order causes, such as product cost, payment fees, shipping, and returns. An order can have positive revenue and negative contribution, and growing those orders loses money faster.

Concretely, list every variable cost per order, compute contribution by segment such as basket size or region, and flag segments with negative contribution.

The reason for that specificity is a failure I have seen: A delivery service promoted small baskets to grow order count, but orders under 15 dollars had a contribution of minus 2.40 dollars each, and losses rose 18 percent in a month.

Contribution by basket size.

Basket size	Revenue	Variable cost	Contribution
under $15	$11.00	$13.40	-$2.40
$15 to $40	$27.00	$22.10	$4.90
over $40	$62.00	$44.30	$17.70

I would not consider it settled without evidence: Compute contribution margin per order by basket-size band and confirm the total matches the finance margin.

Growth in unprofitable orders is growth in losses.

For analyzing a price change, I would state the assumption out loud and then try to break it.

A price increase trades lower volume for higher revenue per unit, so the result depends on how many customers stopped buying. Comparing before and after alone mixes the price effect with seasonality and other launches.

Concretely, compare against a control, such as regions or products without the change, or last year's same period, measure volume, revenue, and margin, and estimate elasticity as the percent change in volume divided by the percent change in price.

The reason for that specificity is a failure I have seen: A before-and-after comparison credited a price rise with 14 percent more revenue, but the same period last year also rose 11 percent seasonally, so the price effect was about 3 percent.

Price effect net of the control.

Group	Revenue change	Volume change
price raised	+14%	-4%
control, no change	+11%	+1%
estimated price effect	+3%	-5%

I would not consider it settled without evidence: Compare the change with a control group or the same period last year and report the difference.

Separate the price effect from everything else that changed.

What separates a good answer on measuring campaign incrementality is a number someone else can check.

Customers chosen for an email are usually already more engaged, so comparing recipients with non-recipients measures selection, not the email's effect. Incrementality is the difference between recipients and a randomly held-out group from the same audience.

Concretely, randomly hold out a share of the eligible audience, such as 10 percent, send to the rest, and compare spend between the two groups over a fixed window.

The reason for that specificity is a failure I have seen: An email program was credited with 2.1 million dollars of extra spend, but a later holdout test showed recipients spent only 3 percent more than the holdout, about 200,000 dollars.

Selection versus incrementality.

Comparison	Spend per customer	Difference
recipients vs non-recipients	$130 vs $100	+30%, biased
recipients vs random holdout	$130 vs $126	+3%, incremental

I would not consider it settled without evidence: Compare the treated group with a random holdout from the same eligible audience.

Compare with who would have been treated, not with who was not.

I would treat forecasting a simple metric as a claim that has to survive a second slice of the data.

A forecast should start with a simple baseline, such as last year's value times recent growth, or a seasonal naive model, and a more complex model is only worth using if it beats that baseline on past data. Every forecast needs a range, not just a point.

Concretely, backtest the baseline and any model on several past quarters, compare their errors, and report the forecast with an interval based on those past errors.

The reason for that specificity is a failure I have seen: A team shipped a machine learning forecast that missed by 18 percent, while a seasonal naive baseline that nobody had tested would have missed by 6 percent.

Backtest errors over four quarters.

Method	Mean absolute error
seasonal naive	6%
linear trend	9%
complex model	18%

I would not consider it settled without evidence: Backtest the forecast against the simple baseline on at least four past periods and report both errors.

Beat the naive forecast before trusting a clever one.

Before I share anything on spotting anomalies in a daily metric, I would write the one-sentence recommendation.

A number is unusual relative to its expected value and normal variation, and both depend on the day of week and season. A fixed threshold, such as any drop over 10 percent, fires constantly on noisy metrics and misses slow drifts on stable ones.

Concretely, compare today with the same weekday over the past several weeks, compute a typical range such as the median plus or minus a few median absolute deviations, and alert only outside that range.

The reason for that specificity is a failure I have seen: A fixed 10 percent alert on daily signups fired 23 times in a quarter, mostly on Sundays, and the team muted it a week before a real 35 percent drop.

Two alert rules, one quarter.

Rule	Alerts per quarter	Real issues caught
fixed 10% drop	23	1 of 2
same weekday, 3 MAD	3	2 of 2

I would not consider it settled without evidence: Backtest the alert rule on past data and count how many alerts would have fired and how many were real problems.

Unusual means outside the normal range for that day.

The first question I would ask about estimating a market size is what decision the answer will change.

A market sizing is judged on its structure and its stated assumptions rather than on hitting the exact number. Break the estimate into factors you can reason about, such as population, share of coffee drinkers, cups per drinker, and share bought rather than made at home.

Concretely, state each assumption with a number, multiply through, sanity check the result against something known, such as the number of cafes and their capacity, and say which assumption the result is most sensitive to.

The reason for that specificity is a failure I have seen: A candidate gave a single figure of 40 million cups with no breakdown, and when the interviewer asked where it came from, there was no way to find which assumption was wrong.

A top-down estimate with every assumption visible.

Factor	Assumption
population	8,000,000
adult coffee drinkers	50%
cups per drinker per day	1.5
share bought out	30%
cups sold per day	1,800,000

I would not consider it settled without evidence: Cross-check the top-down estimate with a bottom-up estimate from cafe counts and daily capacity.

Show the structure, then the number.

I would open handling a take-home analysis case by checking the denominator before the numerator.

Reviewers grade the reasoning path, which means how the data was checked, which question was chosen, and whether the recommendation follows from the evidence. A short answer-first summary with clear assumptions scores better than many charts with no conclusion.

Concretely, start with a one-paragraph recommendation, list the data checks and assumptions, show only the three or four analyses that support the recommendation, and end with limitations and next steps.

The reason for that specificity is a failure I have seen: A candidate submitted 34 charts and no written conclusion, and the panel rejected it because none of the reviewers could tell what the candidate recommended.

A take-home that a panel can grade.

Section	Length
recommendation	1 paragraph
data checks and assumptions	5 bullets
supporting analyses	3 to 4 charts
limitations and next steps	3 bullets

I would not consider it settled without evidence: Ask whether a reviewer could state your recommendation after reading only the first paragraph.

Lead with the answer and let the evidence follow.

My starting point for what a p-value means is the row count before and after every step.

A p-value is the probability of seeing a difference at least as large as the one observed if there were really no difference. It is not the probability that the result is due to chance and not the probability that the change works.

Concretely, explain it with the test's numbers, such as a 0.03 p-value meaning a gap this large would appear about 3 times in 100 experiments where nothing changed, and pair it with the effect size and its confidence interval.

The reason for that specificity is a failure I have seen: A product manager read p = 0.04 as a 96 percent chance the feature worked and shipped it, although the estimated lift was 0.1 percent and the interval ran from almost zero to 0.2 percent.

A significant result that is barely there.

Result	Value
observed lift	+0.1%
95% interval	+0.004% to +0.2%
p-value	0.04

I would not consider it settled without evidence: Report every test result with the effect size, its confidence interval, and the p-value together.

A p-value measures surprise under no effect, not the chance you are right.

With interpreting a confidence interval, the headline figure is where I stop trusting and start checking.

The interval gives the range of lifts that are consistent with the data. If the experiment were repeated many times, 95 percent of intervals built this way would contain the true lift, and a narrow interval means the estimate is precise.

Concretely, read the interval against the decision, checking whether its lower end still justifies shipping and whether it excludes zero, and report it in business units such as extra orders per week.

The reason for that specificity is a failure I have seen: A team reported a 4 percent lift without its interval, and when the interval of minus 1 to 9 percent was later shown, the planned hiring for the expected growth had already been approved.

The interval in business units.

Estimate	Lift	Weekly orders
lower bound	+2%	+400
point estimate	+4%	+800
upper bound	+6%	+1,200

I would not consider it settled without evidence: State the interval's lower and upper bounds in business terms and check whether the decision holds at the lower bound.

Decide on the range, not on the midpoint.

I would scope statistical versus practical significance with the stakeholder before writing a single query.

Statistical significance says the effect is probably not zero, and practical significance says it is large enough to matter. With millions of users, tiny effects become significant, so the decision should weigh the effect against its cost and risk.

Concretely, agree before the test on a minimum effect worth acting on, compare the estimate and its interval with that threshold, and include maintenance and complexity costs in the decision.

The reason for that specificity is a failure I have seen: A team shipped a statistically significant 0.2 percent lift that required a new service costing 90,000 dollars a year, while the extra margin from the lift was 25,000 dollars.

Significant and still a loss.

Item	Value
lift	+0.2%, p < 0.001
extra margin per year	$25,000
cost per year	$90,000
net	-$65,000

I would not consider it settled without evidence: Compare the expected annual value of the lift with the annual cost of shipping and maintaining the change.

Significant is not the same as worth it.

The honest answer on sample size and statistical power begins with what the data cannot tell us.

The required sample size depends on the baseline rate, the smallest effect worth detecting, the significance level, and the desired power. Smaller effects and lower baseline rates need much larger samples, and running for a fixed number of days without this calculation makes results unreliable.

Concretely, compute the sample size per group before launch with a power calculation, convert it into days using real traffic, and round up to whole weeks so every weekday is represented.

The reason for that specificity is a failure I have seen: A test with a 2 percent baseline ran for 3 days to detect a 5 percent relative lift, when a power calculation called for about 310,000 users per group, roughly 5 weeks of traffic.

Users per group at 80 percent power and 5 percent significance.

Baseline	Relative lift to detect	Users per group
2%	5%	about 310,000
2%	10%	about 80,000
10%	5%	about 58,000

I would not consider it settled without evidence: Record the power calculation and planned duration before launch and do not stop early without a planned rule.

Size the test before you start it.

I would test peeking at test results on a ten-row example I can verify by hand.

Each check is another chance for random noise to cross the significance line, so stopping at the first significant result raises the false positive rate well above the 5 percent the test was designed for. With daily checks over a month it can exceed 20 percent.

Concretely, fix the sample size in advance and read the result once, or use a sequential testing method designed for repeated looks, and treat dashboards during the test as health monitoring only.

The reason for that specificity is a failure I have seen: A team stopped 9 of 20 tests early at the first significant reading, and when those features were later re-tested with fixed durations, 6 of the 9 showed no effect.

False winners in A/A simulations.

Stopping rule	False positive rate
one look at the planned end	5%
look daily for 30 days	about 25%
sequential method	5%

I would not consider it settled without evidence: Simulate A/A tests with the same stopping rule and measure how often they declare a false winner.

Look as often as you like, but decide only at the planned end.

The part of multiple comparisons that interviewers probe is the edge case, not the syntax.

With 20 independent metrics at a 5 percent significance level, you expect about one significant result by chance alone. A single significant metric among many is weak evidence unless it was named as the primary metric in advance.

Concretely, declare one primary metric before the test, treat the rest as secondary, and apply a correction such as Bonferroni or Benjamini-Hochberg when several metrics are judged together.

The reason for that specificity is a failure I have seen: A redesign was shipped because time on page improved with p = 0.03, the only significant result among 22 metrics, and the effect disappeared in the next quarter.

Chance of at least one false positive at 5 percent.

Metrics tested	Probability
1	5%
5	23%
20	64%

I would not consider it settled without evidence: Check whether the metric was pre-registered as primary and whether it stays significant after correction.

Test enough metrics and one will win by chance.

For choosing the randomization unit, I would state the assumption out loud and then try to break it.

The randomization unit should match the unit that experiences the change and the unit of the metric. Randomizing by session lets one user see both versions, which contaminates the comparison and understates the variance of user-level metrics.

Concretely, randomize by user for anything a user can notice across visits, analyze at the same unit or use a method that accounts for clustering, and check that each user saw only one variant.

The reason for that specificity is a failure I have seen: A pricing test randomized by session, so 40 percent of buyers saw both prices, and complaints about inconsistent prices forced the test to stop after a week with no usable result.

Randomization units compared.

Unit	Users seeing both variants	Suitable for
page view	high	one page with no memory
session	40% of buyers	short one-visit flows
user	0%	pricing and anything remembered

I would not consider it settled without evidence: Check the share of users exposed to more than one variant and confirm the analysis unit matches the randomization unit.

Randomize the unit that experiences the change.

What separates a good answer on sample ratio mismatch is a number someone else can check.

A sample ratio mismatch means assignment or logging is broken, because a real 50/50 split with large samples should differ by only a fraction of a percent. Any result from such a test is suspect, since the missing users are rarely missing at random.

Concretely, run a chi-square test on the group sizes before reading any outcome, and if it fails, find the cause, such as a redirect that drops users or a bot filter applied to one arm, before trusting results.

The reason for that specificity is a failure I have seen: A test with 51.5 versus 48.5 percent of 200,000 users was read as a 4 percent win, and the cause turned out to be a slow variant that lost impatient users before they were logged.

A split that cannot be chance.

Group	Planned	Observed
control	100,000	103,000
treatment	100,000	97,000
chi-square p-value	none	below 0.0001

I would not consider it settled without evidence: Test the observed split against the planned split with a chi-square test before analyzing any metric.

Check the split before you check the result.

I would treat novelty and primacy effects as a claim that has to survive a second slice of the data.

Existing users often engage with anything new because it is new, which inflates early results, while a change that disrupts habits can show an early dip that recovers. The long-run effect is what matters for the decision.

Concretely, plot the treatment effect by week and by user tenure, compare new users, who have no old habits, with existing users, and run the test long enough for the effect to stabilize.

The reason for that specificity is a failure I have seen: A redesign was shipped on a 9 percent week-one engagement lift, and three months later engagement was 1 percent below the old design.

A lift that fades.

Week	Engagement lift
1	+9%
2	+4%
3	+1%
12	-1%

I would not consider it settled without evidence: Plot the daily or weekly treatment effect and confirm it has stabilized before deciding.

Wait for the effect to settle.

Before I share anything on correlation versus causation, I would write the one-sentence recommendation.

Users who adopt a feature are often already more engaged, so the retention gap may reflect who chooses the feature rather than what the feature does. Only a randomized experiment or a careful quasi-experiment can separate the two.

Concretely, compare users with similar engagement before adoption, look for a natural experiment such as a staggered rollout, and propose an experiment that randomly promotes the feature.

The reason for that specificity is a failure I have seen: A company forced every new user through a feature associated with 40 percent better retention, and retention did not change, because the association came from engaged users choosing it.

The gap shrinks as the comparison improves.

Comparison	Retention gap
adopters vs non-adopters	+40%
matched on prior engagement	+8%
randomized promotion test	+2%

I would not consider it settled without evidence: Run a randomized test that promotes the feature to one group and compare retention with a control group.

Users who choose something differ from users who do not.

The first question I would ask about selection and survivorship bias is what decision the answer will change.

Looking only at customers who survived leaves out the ones who churned, and if churned customers also used onboarding, the pattern says nothing about its effect. Selection bias appears whenever the sample is filtered on the outcome or on something related to it.

Concretely, start from everyone who was eligible at the beginning, compare outcomes by whether they used onboarding, and check what share of each group survived.

The reason for that specificity is a failure I have seen: A report concluded onboarding drove retention from a sample of one-year survivors, but 90 percent of churned customers had also completed onboarding.

Survival rates from the full population.

Group	Started	Survived a year	Survival
used onboarding	9,000	2,700	30%
skipped onboarding	1,000	290	29%

I would not consider it settled without evidence: Rebuild the analysis from the full starting population and compare survival rates between groups.

Analyze the starting population, not the survivors.

I would open interpreting a regression coefficient by checking the denominator before the numerator.

The coefficient is the expected change in sales for a one-unit change in price with advertising held constant. It describes an association in the data, not a causal effect, unless price varied for reasons unrelated to demand.

Concretely, state the coefficient in the units of the variables, check its confidence interval, check whether important variables are missing, and avoid extrapolating beyond the range of prices in the data.

The reason for that specificity is a failure I have seen: An analyst used a coefficient estimated on prices between 8 and 12 dollars to predict sales at 25 dollars, and the forecast of 1,400 units compared with 150 actually sold.

Coefficients with their intervals and data range.

Variable	Coefficient	95% interval
price, per $1	-120 units	-150 to -90
advertising, per $1k	+35 units	+20 to +50
price range in data	$8 to $12	none

I would not consider it settled without evidence: Check the range of the input data and the confidence interval of the coefficient before using it for a prediction.

A coefficient holds only within the data that produced it.

My starting point for difference-in-differences is the row count before and after every step.

Difference-in-differences compares the change in the treated region with the change in a similar untreated region over the same period. Subtracting the control's change removes shared trends, provided the two regions were moving in parallel before the launch.

Concretely, pick a control region with similar pre-launch trends, compute the before-after change in each, subtract, and plot several pre-launch periods to check the parallel-trends assumption.

The reason for that specificity is a failure I have seen: A simple before-and-after comparison credited a delivery policy with a 12 percent rise in orders, but the control region rose 9 percent over the same weeks, so the estimated effect was 3 percent.

The control's change is subtracted.

Region	Before	After	Change
treated	10,000	11,200	+12%
control	8,000	8,720	+9%
difference-in-differences	none	none	+3 points

I would not consider it settled without evidence: Plot both regions for at least six periods before the launch and confirm their trends were parallel.

Subtract what would have happened anyway.

With choosing a statistical test, the headline figure is where I stop trusting and start checking.

A t-test compares means of a continuous measure between two groups, such as average order value, while a chi-square test compares proportions or counts across categories, such as conversion rates. The choice follows the type of the outcome variable.

Concretely, identify whether the outcome is continuous, binary, or categorical, check the test's assumptions such as independence and sample size, and use a non-parametric or bootstrap method for heavily skewed continuous data.

The reason for that specificity is a failure I have seen: An analyst ran a t-test on heavily skewed revenue per user with 800 users per group, and 3 very large purchases produced a significant result that a bootstrap test did not support.

Outcome type decides the test.

Outcome	Example	Test
binary	converted or not	chi-square or z-test for proportions
continuous, roughly normal	trimmed session time	t-test
continuous, skewed	revenue per user	bootstrap or Mann-Whitney

I would not consider it settled without evidence: Check the outcome's type and distribution and confirm the result with a bootstrap when the data is skewed.

Let the outcome's type choose the test.

I would scope type I and type II errors with the stakeholder before writing a single query.

A type I error declares an effect that does not exist, and a type II error misses an effect that does. Which is worse depends on the cost of each mistake, such as shipping a harmful change versus abandoning a good one.

Concretely, set the significance level to control type I errors and the sample size to control type II errors, and adjust both to the cost of each mistake for the specific decision.

The reason for that specificity is a failure I have seen: A team ran tests with only 30 percent power, so 7 of 10 real improvements came out not significant and were abandoned as failures.

The four outcomes of a test.

Truth	Test says effect	Test says no effect
real effect	correct	type II, miss
no effect	type I, false alarm	correct
with 30% power	3 of 10 found	7 of 10 missed

I would not consider it settled without evidence: Record the power of every test and review how many abandoned ideas were underpowered.

Choose error rates by what each mistake costs.

The honest answer on standard deviation versus standard error begins with what the data cannot tell us.

The standard deviation describes how spread out individual values are, and the standard error describes how precisely a sample mean estimates the true mean. The standard error equals the standard deviation divided by the square root of the sample size, so it shrinks as the sample grows.

Concretely, use the standard deviation to describe variation among customers and the standard error to build confidence intervals around an average, and label which one a chart's error bars show.

The reason for that specificity is a failure I have seen: A chart used standard deviation error bars to compare two average order values, the bars overlapped, and a real 7 percent difference based on 40,000 orders per group was dismissed as noise.

Order value with 40,000 orders per group.

Measure	Value
standard deviation of order value	$60
standard error of the mean	$0.30
difference between groups	$3.50

I would not consider it settled without evidence: Label error bars explicitly and compute the confidence interval of the difference between the means.

Spread describes customers, standard error describes the estimate.

I would test the central limit theorem in practice on a ten-row example I can verify by hand.

The central limit theorem says the sampling distribution of a mean approaches a normal distribution as the sample grows, even when individual values are skewed. With very heavy skew, the sample must be much larger before that approximation holds.

Concretely, check the sample size against the degree of skew, compare the normal interval with a bootstrap interval, and use the bootstrap when they disagree.

The reason for that specificity is a failure I have seen: An analyst built a normal interval for average revenue from 60 customers with one very large account, and the interval implied negative revenue was plausible, which no bootstrap interval allowed.

Normal and bootstrap intervals for average revenue.

Sample size	Normal interval	Bootstrap interval
60	-$40 to $520	$35 to $610
5,000	$210 to $250	$211 to $252

I would not consider it settled without evidence: Compare the normal-based interval with a bootstrap interval on the same data.

The mean becomes normal only with enough data.

The part of bootstrapping a confidence interval that interviewers probe is the edge case, not the syntax.

Bootstrapping resamples the observed data with replacement many times, computes the statistic on each resample, and uses the spread of those results as the interval. It works for medians, ratios, and other statistics that lack a simple formula.

Concretely, resample at the level of the independent unit, such as the user rather than the event, repeat 1,000 to 10,000 times, and take the 2.5th and 97.5th percentiles of the results.

The reason for that specificity is a failure I have seen: A bootstrap resampled individual events instead of users, and because heavy users contributed many correlated events, the interval was 3 times narrower than a user-level bootstrap.

95 percent interval for median spend.

Resampling unit	Interval	Width
event	$41.80 to $43.20	$1.40
user	$40.10 to $44.30	$4.20

I would not consider it settled without evidence: Confirm the resampling unit matches the unit of independence and that the interval is stable across repeated runs.

Resample the independent unit.

For regression to the mean, I would state the assumption out loud and then try to break it.

Units selected for extreme results tend to move back toward the average on the next measurement, because part of their extreme result was chance. Improvement among the worst performers is expected even with no intervention.

Concretely, compare the coached stores with similarly low-performing stores that were not coached, or randomize which low performers get coaching.

The reason for that specificity is a failure I have seen: A coaching program was expanded nationwide after the bottom 20 stores improved 15 percent, but uncoached bottom stores in the prior year had improved 13 percent.

Most of the improvement was expected anyway.

Group	Next quarter change
bottom 20, coached	+15%
bottom 20 prior year, not coached	+13%
estimated coaching effect	+2%

I would not consider it settled without evidence: Compare the improvement of the treated extreme group with an untreated group selected the same way.

Extremes drift back toward average on their own.

What separates a good answer on conditional probability and base rates is a number someone else can check.

When the event is rare, most flags come from the large pool of legitimate cases even with a low false positive rate. Bayes' rule combines the base rate with the error rates to give the share of flags that are true.

Concretely, work through a concrete population such as 1,000,000 orders, count true and false flags separately, and report precision alongside recall.

The reason for that specificity is a failure I have seen: An operations team was told the fraud model was 99 percent accurate and staffed for a handful of cases, then received 10,980 flags a day, of which only 990 were fraud.

One million orders.

Orders	Count	Flagged
fraud	1,000	990
legitimate	999,000	9,990
share of flags that are fraud	none	9.0%

I would not consider it settled without evidence: Compute the expected number of true and false flags from the base rate and error rates before deploying the model.

Rare events make most alarms false.

I would treat probability distributions in business data as a claim that has to survive a second slice of the data.

Counts of independent events per period, such as orders per hour, often follow a Poisson distribution, waiting times between events follow an exponential distribution, and positive skewed amounts such as order value are often roughly log-normal. Knowing the shape guides which summary and test to use.

Concretely, plot the data against the candidate distribution, check whether the variance matches the mean for counts, and use the fitted shape to set expected ranges.

The reason for that specificity is a failure I have seen: A capacity plan assumed hourly orders had a variance equal to the mean of 40, but the observed variance was 180, and peak hours exceeded capacity 3 times more often than planned.

Expected shapes and quick checks.

Quantity	Typical shape	Quick check
orders per hour	Poisson or overdispersed	variance 40 expected, 180 seen
days between purchases	exponential	constant hazard
order value	log-normal	log values look normal

I would not consider it settled without evidence: Compare the observed variance with the variance the assumed distribution implies.

Check the shape before using a formula that assumes one.

Before I share anything on expected value in decisions, I would write the one-sentence recommendation.

The decision depends on the expected value, which is the probability the discount changes behaviour times the value saved, minus the discount paid to everyone who receives it, including customers who would have stayed anyway.

Concretely, estimate the uplift from a test, multiply by customer value, subtract the total cost of discounts sent, and compare targeting options on net value.

The reason for that specificity is a failure I have seen: A retention offer went to 20,000 predicted churners, but only 2 percent changed their decision, so the program spent 200,000 dollars to save customers worth 96,000 dollars.

Expected value of the retention offer.

Input	Value
customers targeted	20,000
cost per discount	$10
uplift in retention	2%
value per saved customer	$240
net value	-$104,000

I would not consider it settled without evidence: Estimate the uplift with a holdout and compute net value per customer targeted before scaling.

Pay for the behaviour you change, not for the behaviour you get anyway.

The first question I would ask about confounding variables is what decision the answer will change.

A confounder influences both variables, such as store size or location driving both staffing and sales. Without accounting for it, the relationship between staff and sales overstates or even invents an effect.

Concretely, list plausible confounders, compare stores that are similar on them, include them in a regression, and prefer an experiment or natural variation in staffing when possible.

The reason for that specificity is a failure I have seen: A plan to add 2 staff to every store was based on a raw correlation, and a comparison within similar-sized stores showed an effect about one fifth as large.

The effect after controlling for size.

Analysis	Extra sales per added staff member
all stores, raw	$48,000
within similar-size stores	$9,500

I would not consider it settled without evidence: Compare the relationship within groups of similar stores and check whether it shrinks.

Ask what else drives both numbers.

I would open working with small samples by checking the denominator before the numerator.

With small samples, rates move a lot from a few individuals, so a 20 percent difference may be two or three conversions. Intervals are wide, and the honest statement is usually that the data is not yet enough to decide.

Concretely, report counts as well as rates, compute an interval, and state how much more data would narrow it enough to decide.

The reason for that specificity is a failure I have seen: A leadership update reported a 20 percent conversion lift that rested on 9 conversions versus 7, and the lift vanished after 2,000 more users.

Overlapping intervals on tiny counts.

Group	Users	Conversions	Rate	95% interval
new feature	45	9	20%	10% to 35%
existing	42	7	17%	8% to 31%

I would not consider it settled without evidence: Show the raw counts and the confidence interval next to any rate from a small sample.

Show the counts behind every small-sample rate.

My starting point for A/B testing when users affect each other is the row count before and after every step.

When users interact, treating one user affects others, such as sellers competing for the same buyers, so the control group is changed by the treatment. The measured effect is biased, often overstating the benefit.

Concretely, randomize at a level that contains the interaction, such as city, time period, or social cluster, and accept the larger variance that comes with fewer units.

The reason for that specificity is a failure I have seen: A marketplace ranking test showed treated sellers gaining 8 percent more sales, but total sales did not change, because treated sellers simply took sales from control sellers.

User-level and city-level readouts.

Design	Measured effect	Total sales change
user-level test	+8%	none
city-level test	+1%	+1%

I would not consider it settled without evidence: Compare total outcomes across whole clusters or regions rather than between users within the same market.

When users interact, randomize the market.

With validating a dataset before analysis, the headline figure is where I stop trusting and start checking.

Basic profiling catches most problems before they reach a conclusion, covering row counts against the source, the date range, duplicates on the expected key, NULL rates, value ranges, and category lists. Each check has an expected answer that can be written down in advance.

Concretely, write the expected values first, run the profile, investigate every surprise, and keep the checks as a script so they run again when the data is refreshed.

The reason for that specificity is a failure I have seen: An analysis of 2026 sales used an extract that stopped on June 30, and the second-half decline it reported was simply missing data.

Expected against found.

Check	Expected	Found
date range	Jan 1 to Sep 10	Jan 1 to Jun 30
duplicate order ids	0	0
NULL customer ids	under 1%	0.4%

I would not consider it settled without evidence: Compare the minimum and maximum dates, total rows, and key uniqueness with what the source system reports.

Profile the data before you trust any pattern in it.

I would scope handling missing data with the stakeholder before writing a single query.

The right handling depends on why data is missing. If it is missing at random, dropping rows loses precision but not accuracy, and if it is missing for a reason, such as high earners skipping the question, dropping it biases every result.

Concretely, compare respondents with and without the field on other variables, report results with the missing group shown separately, and impute only with a stated method and a check of how much the conclusion depends on it.

The reason for that specificity is a failure I have seen: A pricing survey dropped the 15 percent who skipped income, who were disproportionately high earners, and the resulting willingness-to-pay estimate was 22 percent too low.

The missing group is not like the rest.

Group	Share	Average basket
income given	85%	$48
income missing	15%	$91

I would not consider it settled without evidence: Compare the characteristics of rows with and without the missing field before choosing a method.

Ask why it is missing before deciding what to do.

The honest answer on duplicate customer records begins with what the data cannot tell us.

Exact duplicates are easy to remove, but real duplicates often differ slightly, such as different capitalization, a missing middle name, or a new email address. Deciding which records represent the same customer is a matching rule that should be explicit.

Concretely, count duplicates on the expected key, normalize fields such as email and phone, define a matching rule, review a sample of matches by hand, and keep the rule versioned.

The reason for that specificity is a failure I have seen: A loyalty report counted 18 percent more members than existed because the same people had signed up with differently capitalized emails, and the program budget was set on the inflated count.

Each normalization step removes duplicates.

Normalization	Distinct members
raw email	118,000
lowercased and trimmed email	102,500
plus phone match	100,000

I would not consider it settled without evidence: Review a random sample of matched and unmatched pairs by hand and report the estimated error rate of the matching rule.

Make the matching rule explicit and test it.

I would test reconciling to a source of truth on a ten-row example I can verify by hand.

Differences usually come from definitions, such as gross versus net of refunds, timing, such as order date versus payment date, or scope, such as including tax or excluding a region. Resolving them means listing each difference and quantifying it until the gap is explained.

Concretely, pick a single day or month, compare totals, then walk through each definitional difference and quantify it, and document the bridge from one figure to the other.

The reason for that specificity is a failure I have seen: Two teams argued for three weeks over a 4 percent revenue gap that turned out to be refunds netted in one report and not the other, plus 0.6 percent of sales tax.

A bridge from analytics to finance.

Step	Amount
analytics gross revenue	$1,000,000
minus refunds	-$34,000
minus sales tax	-$6,000
finance net revenue	$960,000

I would not consider it settled without evidence: Build a written bridge that starts at one number, adds or subtracts each quantified difference, and ends at the other.

Explain the gap line by line.

The part of documenting analysis assumptions that interviewers probe is the edge case, not the syntax.

A reproducible analysis records the data sources and extraction date, the definitions and filters, the assumptions, and the code that produced every number. Without them, nobody can check or update the work, including the original author six months later.

Concretely, keep queries and notebooks in version control, put a short assumptions section at the top, record the extraction date and row counts, and make every chart traceable to the code that produced it.

The reason for that specificity is a failure I have seen: A churn analysis quoted in a board deck could not be reproduced three months later because the query had been edited in place and nobody knew which filters had been used.

What a reproducible analysis records.

Item	Recorded as
data source	warehouse.orders
extracted	2026-09-10, 1.2m rows
filters	paid orders, excluding staff
code	analysis repo, commit reference

I would not consider it settled without evidence: Ask a colleague to regenerate one key number from the repository without your help.

If someone else cannot rerun it, it cannot be checked.

For lookup formulas in spreadsheets, I would state the assumption out loud and then try to break it.

VLOOKUP defaults to approximate match when its last argument is left out, which returns a wrong value from sorted data without any error. It also breaks when columns are inserted, because it refers to a column by number.

Concretely, use XLOOKUP or INDEX with MATCH with exact matching, check the count of lookups that return not found, and trim and standardize keys before matching.

The reason for that specificity is a failure I have seen: A commission sheet used VLOOKUP with the default approximate match, and 214 sales reps were paid another rep's rate, an overpayment of 31,000 dollars.

A missing key, two behaviours.

Formula	Rep ID looked up	Result
VLOOKUP, default match	1045, not in table	rate of rep 1044
XLOOKUP, exact match	1045, not in table	not found

I would not consider it settled without evidence: Count the not-found results and spot-check a sample of looked-up values against the source table.

Always ask for an exact match.

What separates a good answer on pivot tables for quick analysis is a number someone else can check.

Pivot tables are fast for exploring a moderate dataset by a few dimensions and for sharing with non-technical colleagues. They become risky when the data is large, refreshed often, or needs complex logic, because steps are manual and hard to audit.

Concretely, use pivots for one-off exploration, check that the source range covers all rows, and move recurring or large analyses to SQL or Python with version control.

The reason for that specificity is a failure I have seen: A weekly pivot's source range was fixed at 50,000 rows, so when the data grew to 61,000 rows the newest 11,000 orders were silently excluded for a month.

A fixed range falls behind the data.

Week	Source rows	Rows in pivot
1	48,000	48,000
5	61,000	50,000

I would not consider it settled without evidence: Compare the pivot's grand total with the row count and total of the source data every time it is refreshed.

A pivot is a quick answer, not a pipeline.

I would treat pandas merges and row explosions as a claim that has to survive a second slice of the data.

A merge where the key is duplicated on both sides produces every combination of matching rows, so 3 rows on the left and 4 on the right for the same key become 12. The result looks normal until totals are checked.

Concretely, check key uniqueness with duplicated before merging, pass validate='many_to_one' so pandas raises an error when the assumption fails, and compare row counts afterwards.

The reason for that specificity is a failure I have seen: A notebook merged sessions to purchases on user_id when both had many rows per user, which grew 2 million rows into 37 million and reported spend per session 18 times too high.

Output rows per key.

Left rows per key	Right rows per key	Output rows per key
1	4	4
3	4	12
3	1	3

I would not consider it settled without evidence: Use the validate argument in merge and assert the output row count equals the expected count.

State the relationship when you merge.

Before I share anything on vectorized operations in pandas, I would write the one-sentence recommendation.

Calling apply with a Python function runs that function once per row, while vectorized column operations run in compiled code across the whole column at once. The difference is often 50 to 150 times on large data.

Concretely, express the logic as column arithmetic, numpy.where, or map on a dictionary, use groupby with built-in aggregations, and time both versions on a sample.

The reason for that specificity is a failure I have seen: A notebook computed a discount flag with apply over 8 million rows and took 11 minutes, while the same logic with numpy.where took 6 seconds.

Same logic, two speeds.

Method	Rows	Runtime
apply per row	8,000,000	11 min
numpy.where	8,000,000	6 s

I would not consider it settled without evidence: Time the apply version and the vectorized version on the same sample and confirm they give identical results.

Operate on columns, not rows.

The first question I would ask about choosing the right chart is what decision the answer will change.

The chart follows the comparison, so lines show change over time, bars compare categories, scatter plots show relationships, and histograms show distributions. Pie charts make comparisons hard beyond two or three slices.

Concretely, state the one comparison the reader should make, choose the chart that makes it easiest, sort bars by value, and label the key number directly on the chart.

The reason for that specificity is a failure I have seen: A deck compared 12 product categories in a pie chart, and the audience could not tell which of the two largest slices was bigger, although they differed by 4 percentage points.

The question picks the chart.

Question	Chart
how did it change over time	line
which category is largest	sorted bar
are two measures related	scatter
how are values spread	histogram

I would not consider it settled without evidence: Show the chart to someone unfamiliar with the data and ask them to state the main comparison.

Choose the chart for the comparison you want made.

I would open misleading chart axes by checking the denominator before the numerator.

For bar charts, the bar length represents the value, so a truncated axis exaggerates differences. For line charts showing change over time, a non-zero axis is acceptable if it is clearly labelled, because the reader compares slopes rather than lengths.

Concretely, start bar charts at zero, label truncated line axes clearly, avoid dual axes that invite false comparisons, and show the actual values.

The reason for that specificity is a failure I have seen: A bar chart with the axis starting at 95 made a rise from 96.1 to 96.8 percent uptime look like a 64 percent improvement, and the team's bonus review was argued over it.

What the axis start does to the bars.

Axis start	Visual ratio of bars	Actual ratio
95	1.64	1.007
0	1.007	1.007

I would not consider it settled without evidence: Redraw the chart with the axis at zero and check whether the visual impression still matches the size of the change.

Bar length must match the value.

My starting point for presenting findings to executives is the row count before and after every step.

Executives need the recommendation, the evidence that supports it, the size of the impact, and the main risk, in that order. Methodology matters only as far as it affects whether they can trust the answer.

Concretely, open with the recommendation and its expected impact in one sentence, support it with two or three key figures, state the main uncertainty, and keep methods in an appendix.

The reason for that specificity is a failure I have seen: An analyst spent 8 of 10 minutes on data sources and methods, and the meeting ended before the recommendation to cut a 400,000-dollar channel was discussed.

A ten-minute deck.

Slide	Content
1	cut channel X, saves $400k a year
2	incremental sales from X were 2% of spend
3	risk, brand search may dip slightly
appendix	data and method

I would not consider it settled without evidence: Check whether the first slide alone states the recommendation, its impact, and the main risk.

Lead with what to do and why.

With communicating uncertainty, the headline figure is where I stop trusting and start checking.

Uncertainty is information that affects the decision, so it should be stated plainly with a range and what would reduce it. Presenting a single precise number that later proves wrong damages trust more than an honest range.

Concretely, give a range with a most likely value, explain in one sentence what drives the uncertainty, and say what decision is safe now and what extra data would change it.

The reason for that specificity is a failure I have seen: A forecast of exactly 12.4 million dollars was presented without a range, actual revenue came in at 10.9 million, and later forecasts from the team were discounted by leadership.

False precision versus an honest range.

Presentation	Example
false precision	$12.4m
range with most likely	$10.5m to $13.0m, most likely $12.4m
driver	depends mainly on Q4 renewal rate

I would not consider it settled without evidence: Check that every forecast or estimate shared with decision makers has a range and a statement of what drives it.

A range you can defend beats a number you cannot.

I would scope a stakeholder who disagrees with the data with the stakeholder before writing a single query.

The disagreement may point to a real gap, such as a segment or definition the analysis missed, or it may be a preference for a different answer. Treating it as a hypothesis to test is more productive than defending the result.

Concretely, ask what specifically they see that differs, check whether the analysis covered that case, share the data and definitions openly, and agree on a test that would settle the question.

The reason for that specificity is a failure I have seen: An analyst dismissed a sales director's objection, and the director was right, because the analysis excluded channel partners who produced 28 percent of the region's revenue.

The missing scope reversed the trend.

Scope	Regional revenue trend
direct sales only	-8%
including channel partners	+3%

I would not consider it settled without evidence: Rerun the analysis for the case the stakeholder describes and share the result with the definitions used.

Treat an objection as a hypothesis to test.

The honest answer on prioritizing ad hoc requests begins with what the data cannot tell us.

Prioritize by the value of the decision each request informs, the urgency of that decision, and the effort required. A request that feeds a large decision this week beats a curiosity question from a senior person.

Concretely, ask each requester what decision the answer feeds and by when, estimate effort, rank by value and urgency over effort, and share the ranking so the trade-offs are visible.

The reason for that specificity is a failure I have seen: An analytics team answered requests in the order they arrived and spent a week on dashboard tweaks while a pricing decision worth 2 million dollars was made without data.

A ranked request log.

Request	Decision value	Deadline	Effort	Priority
pricing test readout	$2m	Friday	1 day	1
churn by region	$300k	next month	2 days	3
dashboard colour change	none	none	1 hour	8

I would not consider it settled without evidence: Keep a visible request log with the decision, deadline, and estimated effort for each item.

Rank requests by the decisions they feed.

I would test writing an analysis summary on a ten-row example I can verify by hand.

A good summary states the question, the answer, the size of the effect, the confidence in it, and the recommended action. A reader who stops after the paragraph should still make the right decision.

Concretely, write the summary last but place it first, use concrete numbers rather than adjectives, name the main caveat, and keep it under about 100 words.

The reason for that specificity is a failure I have seen: A summary said engagement was significantly impacted without a number or direction, and two readers took opposite actions based on it.

The five parts of a summary.

Element	Example
question	did free shipping raise orders
answer	yes, +6% orders
confidence	95% interval +3% to +9%
action	keep it for orders over $30

I would not consider it settled without evidence: Ask a reader to state the decision after reading only the summary and check that it matches yours.

The summary should be enough to act on.

The part of building a KPI dashboard for a team that interviewers probe is the edge case, not the syntax.

A dashboard should answer a small set of recurring questions that lead to actions, with each metric shown against a target or a comparison. Dashboards that show everything get opened once and ignored.

Concretely, list the decisions the team makes weekly, pick three to six metrics that inform them, show each with a comparison period and target, and check usage after a month.

The reason for that specificity is a failure I have seen: A 40-chart dashboard was viewed 3 times in its first two months, and the team kept asking analysts the same questions by message.

Each metric has a target and an action.

Metric	Comparison	Target	Action if off target
weekly orders	last 4 weeks	25,000	review campaigns
on-time delivery	last week	95%	escalate to ops
refund rate	last month	under 3%	audit top reasons

I would not consider it settled without evidence: Check dashboard view counts after a month and remove charts nobody uses.

Build for the decisions, not for completeness.

For protecting personal data in analysis, I would state the assumption out loud and then try to break it.

Analyses rarely need direct identifiers, so the safest approach is to work with pseudonymous IDs and aggregated results. Exporting raw personal data into spreadsheets or personal drives creates privacy risk and often breaks company policy and regulations such as GDPR.

Concretely, select only the fields needed, use hashed or internal IDs instead of emails, aggregate before exporting, suppress very small groups, and keep data inside approved systems.

The reason for that specificity is a failure I have seen: An analyst exported 250,000 customer emails to a personal spreadsheet to count domains, and the file was later found in a shared folder open to the whole company.

Counting email domains without exporting emails.

Field	Needed	Handling
email	no	derive domain in the warehouse
email domain	yes	aggregate
customer id	no	keep internal
groups under 10	no	suppress

I would not consider it settled without evidence: Review each export for direct identifiers and for groups small enough to identify individuals before sharing it.

Use the least personal data that answers the question.

What separates a good answer on checking an answer for plausibility is a number someone else can check.

Most wrong numbers fail a simple plausibility check, such as being larger than the company's total revenue, implying more customers than the population, or moving far more than usual. A quick comparison with a known figure catches errors that careful code review misses.

Concretely, compare every headline number with a known benchmark, such as last period, the finance total, or a back-of-envelope estimate, and investigate any figure off by more than a stated tolerance.

The reason for that specificity is a failure I have seen: A market report claimed 3.2 million monthly buyers in a country with 2.1 million adults online, and the error was a join that duplicated users, which a plausibility check would have caught.

Headline figures against benchmarks.

Figure	Result	Benchmark	Plausible
monthly buyers	3.2m	2.1m adults online	no
monthly revenue	$4.1m	finance $4.0m	yes

I would not consider it settled without evidence: Write down the expected order of magnitude before running the query and compare the result with it.

Guess the answer before computing it.

I would treat a surprising positive result as a claim that has to survive a second slice of the data.

Very large improvements are more often caused by errors, such as a tracking change, a bot filter, or a definition change, than by real behaviour. The more surprising the result, the more checking it needs before it is shared.

Concretely, check for tracking or code releases on the date of the jump, compare with independent sources such as payment records, and break the change down by segment to see if it is concentrated in one place.

The reason for that specificity is a failure I have seen: A 60 percent conversion jump was announced company-wide, and a day later it was traced to a tag change that stopped counting visits from one browser, shrinking the denominator.

The numerator did not move.

Source	Change
analytics conversion rate	+60%
payment system orders	+1%
analytics visits	-37%

I would not consider it settled without evidence: Confirm the change in an independent data source such as payments or orders before sharing it.

Surprising results need the most checking.

Before I share anything on the data analyst role compared with data science, I would write the one-sentence recommendation.

A data analyst usually answers business questions with descriptive analysis, SQL, experiment readouts, and clear communication, while a data scientist more often builds predictive models and statistical tools. The line varies by company, and interviewers mainly want to hear how you turn data into decisions.

Concretely, describe the work in terms of the questions you answer and the decisions you influence, give an example of an analysis that changed a decision, and ask how the team splits the work.

The reason for that specificity is a failure I have seen: A candidate described the analyst role as making dashboards and building reports, and the interviewer marked them down because the team's analysts ran 40 experiments a quarter and advised product decisions.

How the roles usually differ.

Aspect	Data analyst	Data scientist
main output	analyses, readouts	models, tools
typical tools	SQL, spreadsheets, BI, Python	Python, ML libraries
success measure	better decisions	better predictions

I would not consider it settled without evidence: Prepare one example with the question, your method, the result, and the decision it changed.

Describe the decisions you influence, not the tools you use.

The first question I would ask about an analysis that changed a decision is what decision the answer will change.

Interviewers want a specific situation, the question, what you did with the data, the result with a number, and the decision that followed. The STAR structure of situation, task, action, and result keeps the story short and concrete.

Concretely, pick one example with a measurable outcome, spend most of the time on your actions and reasoning, quantify the result, and mention what you would do differently.

The reason for that specificity is a failure I have seen: A candidate spent 6 minutes describing the company's data stack and never said what decision changed, so the panel could not assess the candidate's impact.

A STAR answer in four lines.

Part	Example
situation	churn rose 2 points in one quarter
task	find the cause within two weeks
action	cohort and segment analysis by plan
result	annual plan price change reversed, churn back to 4%

I would not consider it settled without evidence: Rehearse the story in under 2 minutes and check that it ends with a number and a decision.

End the story with the decision and the number.

I would open working with engineers on event tracking by checking the denominator before the numerator.

Reliable analytics needs a tracking plan that names each event, its properties, and when it fires, agreed before the feature is built. Analysts are responsible for specifying what they need and for testing that events arrive as designed.

Concretely, write a tracking plan with event names and properties for each feature, test events in a staging environment before release, and monitor daily event volumes for sudden changes.

The reason for that specificity is a failure I have seen: A checkout redesign shipped with a renamed purchase event, and conversion reports showed zero purchases for 4 days until someone noticed the event name had changed.

Two rows of a tracking plan.

Event	Fires when	Required properties
checkout_started	cart page submitted	cart_value, items
purchase_completed	payment confirmed	order_id, revenue, currency
daily volume check	alert below	70% of the 4-week median

I would not consider it settled without evidence: Test every new or changed event in staging and add a daily volume check that alerts on large drops.

Specify and test the events before the feature ships.

My starting point for moving an analysis out of a spreadsheet is the row count before and after every step.

Spreadsheets are good for small, one-off work and for sharing with colleagues who do not code. Analyses that are repeated, depend on large data, or feed important decisions belong in SQL or Python, where every step is written down and can be rerun.

Concretely, move work out of the spreadsheet when it is rerun weekly, when it exceeds tens of thousands of rows, or when manual steps such as copy and paste are needed, and keep the spreadsheet only as an output view.

The reason for that specificity is a failure I have seen: A monthly forecast built by copying data between 7 spreadsheet tabs was wrong for 3 months because one paste was shifted by a single row.

When to stay and when to move.

Condition	Stay in spreadsheet	Move to SQL or Python
one-off, small data	yes	no
rerun weekly	no	yes
over 100,000 rows	no	yes
manual copy and paste steps	no	yes

I would not consider it settled without evidence: List the manual steps in the current process and check whether each could silently go wrong.

If it runs every week, it should be code.

With structuring an open-ended business case, the headline figure is where I stop trusting and start checking.

Break revenue into its drivers, such as traffic, conversion, and average transaction value, and ask which one changed before guessing causes. Then consider internal factors such as staffing and pricing and external factors such as competition and local events.

Concretely, state the driver tree, ask for or estimate the data for each branch, narrow to the branch that moved, and propose a way to test the likely cause.

The reason for that specificity is a failure I have seen: A candidate listed 15 possible causes without structure, and when asked which to investigate first had no way to choose.

Traffic is the branch that moved.

Driver	Last year	This year	Change
store visits	200,000	170,000	-15%
conversion	30%	30%	0%
transaction value	$40	$41	+2.5%

I would not consider it settled without evidence: Check whether your answer names the driver that changed and a way to confirm the cause.

Split the problem before guessing the cause.