This scenario probes whether the candidate understands the importance of statistical significance, sample size, and predetermined decision criteria, going beyond surface-level A/B testing knowledge to show analytical maturity.
Acknowledge the temptation to end early, then walk through a disciplined decision process: refer to pre-set goals (e.g., minimum sample size, significance level), consider the risk of a false positive, and explain why you would let the test run until a reliable conclusion. Use terms like 'sample size' and 'statistical significance' in a practical, non-academic way.
Start by acknowledging the pull to declare a winner early, then immediately pivot to discipline. Say plainly that a small difference after two days is almost certainly noise, not signal, and that acting on it risks a false positive that could cost more in the long run than waiting. Explain that your first move is to check the plan you set before the test started, specifically the minimum sample size and the confidence level you agreed on, typically 95 percent. If you have not hit that sample size, the test is not done, regardless of how the numbers look today. Then walk through the practical factors you would weigh: the volume of traffic coming in, the baseline conversion rate, and how much of a lift you actually need to justify changing the page. A tiny uplift might not be worth the effort of a rollout, so you would also consider the business impact, not just the p-value. In the Philippine context, be mindful of BPO or e-commerce campaigns where traffic spikes on paydays or weekends, so two days of data could be skewed by a single high-traffic day. Finally, say you would let the test run to its predetermined end date, then check whether the result holds, and if it does, you would recommend the change with the caveat that you might run a follow-up validation if the lift is marginal.
A typical Filipino-candidate mistake is to say, 'Tinalo na ng bagong version yung luma, so i-stop na natin at yun na gamitin.' (The new version has beaten the old one, so let’s stop and use it already.) Instead, you must explain that a small early difference could be random noise, and you need a large enough sample and statistical confidence before declaring a winner.
Situation
In my last role as a marketing assistant for a SaaS startup, we ran an A/B test on the main headline of our free-trial signup page. After just three days, the new headline was leading by a conversion rate of 11.2% versus the original’s 10.5%, but daily visitor volume was only around 200.
Task
I had to decide whether to declare the new headline a winner and make the change permanent, or to keep the test running and risk losing potential signups if the early lead was just noise.
Action
I first checked our pre-test agreement: we had decided that a winner would be called only after 1,000 visitors per variation, or a 95% statistical significance level using a free online calculator. With only 600 visitors per variation so far and a significance level of 85%, the difference was not yet reliable. I also considered the downside of a false winner: if we changed the headline prematurely and the advantage disappeared, we might lose more signups over time. So I recommended continuing the test, but to minimize risk I set a hard deadline of 10 days total. At that point, if the difference remained directionally positive but still not statistically significant, we would run a follow-up test with a larger sample.
Result
On day 10, with over 1,200 visitors each, the new headline’s conversion rate had stabilized at 11.5% versus 10.2% for the original, with 97% significance. We confidently adopted the new headline, and free-trial signups increased by 12% in the following month.
Relying on pre-set rules and statistical thresholds prevents you from making hasty, gut-based decisions that can backfire.
Write your own answer, then get instant AI feedback graded against:
Get AI feedback on your answer — free.
3 free AI-graded answers + 1 free mock interview, no card needed.
Sign Up FreeAlready have an account? Log in
Sign in to join the conversation.
No answers shared yet — be the first to show how you'd approach this.