Foundation · 2 min
What you need to know
How should ecommerce businesses prioritize optimization opportunities?
Rank opportunities by customer harm, affected volume, expected contribution, evidence, implementation effort, reversibility and risk. Separate certain defect fixes from hypotheses that require experiments.
Should I optimize the biggest drop-off first?
Not automatically. The biggest percentage drop may involve low-value or ineligible visitors, while a smaller high-volume problem causes more loss. Estimate recoverable customers and contribution.
How do I estimate the impact of an ecommerce improvement?
Estimate affected eligible volume × realistic change in behavior × contribution per outcome, then subtract implementation and operating cost. Use a range because baseline and effect estimates are uncertain.
What should I test first?
First fix verified defects and data failures; then test a high-value, evidence-backed customer problem with a clear hypothesis and measurable outcome. Avoid cosmetic changes without a decision rationale.
How do I know whether a change actually caused improvement?
Use randomized controlled experiments when feasible, with a predefined primary metric, guardrails, sample, duration and analysis plan. When randomization is impossible, use stronger comparison designs and state the remaining uncertainty.
Correlation vs causation: what’s the difference in ecommerce analytics?
Correlation means two measures move together; causation means changing one produced a change in the other. Confounding factors can create correlation without a causal effect.
Why can before-and-after comparisons be misleading?
Other changes, seasonality, traffic mix, regression to the mean and measurement drift may occur between the two periods. A control group helps separate the change from background movement.
How should experiment results be documented?
Record the hypothesis, variants, eligibility, allocation, dates, primary metric, guardrails, exclusions, data checks, result, uncertainty, decision and follow-up. Preserve losing and inconclusive tests too.
A quick store check
Write a hypothesis, primary metric, guardrails, eligible population and stopping rule before launching the next test.
Foundation · 2 min
Frequently asked questions
Does a statistically significant result guarantee business value?
No. Check effect size, profit, customer guardrails, data quality and whether the result applies to the intended population.
Should a losing experiment be hidden?
No. Record it so the team avoids repeating the test and can learn where the hypothesis failed.