A product category has its worst conversion week of the quarter. The team changes the page, watches conversion recover, and records a successful optimisation. The recovery is real, but the conclusion may not be: selecting an unusually bad week makes some rebound plausible even if the page change has no effect.
EcomToolkit’s approach is to separate the decision to fix a problem from the evidence used to claim an improvement. This guide explains regression to the mean through hypothetical ecommerce examples and an evaluation worksheet. It does not argue against urgent fixes; it shows how to describe their results without giving an intervention credit for every subsequent movement.
Table of Contents
- Recognise selection on an extreme result
- Work through a hypothetical recovery
- Choose the evaluation before making the claim
- Keep the baseline and window stable
- Distinguish recovery from a verified repair
- Build a review sheet that resists hindsight
- The EcomToolkit view
Recognise selection on an extreme result
Observed performance combines a more persistent underlying level with temporary influences and measurement variation. When teams select the lowest-converting categories or slowest pages from one short window, they are likely to select some units experiencing unusually adverse temporary conditions. Later observations can be less extreme without a causal improvement.
The methodological paper Regression to the mean: what it is and how to deal with it explains why repeated measurements can make natural variation resemble intervention effects. Selection using an extreme baseline and measurement error make the issue more noticeable. Ecommerce is an application of that reasoning, not the population studied in the paper.
This is a tendency across selected observations, not a guarantee that every poor category will recover. A broken checkout can remain broken. A genuine demand shift can persist. The practical lesson is to examine why the unit entered the improvement queue and what evidence would distinguish a repair from normal variation.
Write the selection rule into the project brief. “Lowest five categories last week” is a materially different baseline from “all categories using a component with a reproducible rendering defect.” Both can justify investigation, but only the first selects directly on the extreme outcome being celebrated later.
Work through a hypothetical recovery
Suppose two comparable category groups are selected because both performed poorly in the baseline week. One receives a planned merchandising change; the other continues with its existing treatment during the evaluation period. Assume for this arithmetic example that each group has 10,000 eligible sessions in each period.
| Group | Baseline orders and rate | Follow-up orders and rate | Change |
|---|---|---|---|
| Changed group | 100 orders, 1.0% | 160 orders, 1.6% | +0.6 percentage points |
| Comparison group | 110 orders, 1.1% | 150 orders, 1.5% | +0.4 percentage points |
| Difference in changes | — | — | +0.2 percentage points |
The changed group improved by 60% relative to its own baseline rate. That is correct arithmetic, but it is not automatically a 60% causal lift. Subtracting the comparison group’s 0.4-point improvement leaves a 0.2-point difference in changes. This is a descriptive comparison, not proof of an effect or statistical significance.
The groups could differ in promotions, available stock, acquisition mix or prior trends. A comparison selected after seeing the results is especially vulnerable to convenient storytelling. These figures show why the original claim needs more work; they do not establish that the remaining difference was caused by merchandising.
Choose the evaluation before making the claim
Where feasible, randomise eligible units to treatment and comparison conditions before the intervention. If the team specifically wants to improve weak categories, apply the eligibility rule first and then randomise within that selected population. Both conditions are then exposed to the same selection mechanism in expectation.
Match the randomisation unit to the intervention. A shared navigation change cannot be treated as independent for every product viewed through that navigation. The experiment randomisation unit guide explains why grouping and shared exposure affect the analysis. Do not increase the apparent sample size by counting dependent observations as independent assignments.
When randomisation is impractical, document an observational comparison before reviewing follow-up results. Inspect several pre-intervention periods, explain the choice of comparator and record simultaneous changes. Similar baseline values alone do not establish similar underlying trajectories, and matching on a single unusually poor week can repeat the original problem.
Use language that fits the design. A controlled experiment can support a causal estimate under its assumptions. A simple before-and-after chart usually supports a description of change. Teams can still make operating decisions from imperfect evidence, provided the uncertainty is visible and the claim is proportionate.
Keep the baseline and window stable
Do not begin the baseline at the lowest day and end the follow-up at the highest day. Fix the evaluation dates before inspecting the desired result. Include complete trading cycles where appropriate and document promotions, outages and stock constraints rather than removing awkward days without explanation.
Longer baselines can help reveal whether the selected week was unusual, but more data is not automatically more comparable. A quarter containing a different assortment or tracking implementation may mix incompatible conditions. Explain which periods describe the same operating system and which are only historical context.
The classic BMJ examples of regression towards the mean illustrate how selecting extreme groups affects repeated comparisons. The editorial application here is to preserve the selection window separately from the evaluation window. That separation makes it easier to see how a team arrived at its candidate list.
For performance work, use equivalent test conditions too. A single very slow laboratory run followed by a normal run can create a dramatic apparent gain. Keep device, network, cache state, journey and instrumentation comparable. A reproducible technical explanation strengthens the claim beyond a favourable timing difference.
Distinguish recovery from a verified repair
Some changes have direct correctness evidence. If a reproducible error blocks a payment method and a patch makes the same test succeed, the team has evidence that the specific defect was repaired. It still should not assign every improvement in total store conversion to that patch.
| Claim being considered | Evidence that helps | Limit that remains |
|---|---|---|
| A checkout defect was repaired | Repeatable failing and passing scenario | Storewide sales impact is separate |
| A page became faster | Comparable traces and repeated measurements | Revenue effect is not established |
| A category improved after editing | Stable definitions and complete time windows | Rebound and other changes may explain it |
| A change raised conversion | Appropriate controlled comparison | Estimate still has uncertainty |
| A programme improved weak categories | Consistent eligibility and evaluation rules | Generalisation beyond selected categories needs care |
Keep engineering acceptance and commercial evaluation as separate fields in the release record. This prevents a useful bug fix from being dismissed because the revenue estimate is uncertain, while also preventing a technical success from becoming an unsupported sales claim.
Look for the mechanism. If the proposed benefit is clearer size information, observe whether the relevant content is available and usable. If the intervention is reducing input delay, check the changed work in a trace. Mechanism evidence does not replace an outcome evaluation, but it makes the explanation more credible and actionable.
Build a review sheet that resists hindsight
Before launch, record the candidate selection rule, unit of analysis, primary metric, baseline dates, follow-up dates, comparison design and known concurrent changes. Assign an owner for changes to the plan. The sheet should be short enough to use during normal trading rather than only on major research projects.
After the evaluation, show absolute levels, denominators and uncertainty where the design supports it. Separate percentage-point changes from relative percentages. In the earlier example, 1.0% to 1.6% is a 0.6-point change and a 60% relative increase; writing only “up 0.6%” obscures the meaning.
If many categories were screened, retain the full candidate list rather than publishing only the winners. The multiple-testing guide addresses a related selection problem. Regression to the mean concerns repeated extreme measurements; multiple testing concerns the chance of false discoveries across many comparisons. They can occur together.
For a recurring improvement programme, review whether the selection process keeps chasing yesterday’s noise. Combine a persistent issue, meaningful traffic and a plausible fix when prioritising work. A category that repeatedly exhibits the same customer problem is a more useful candidate than one selected solely because its latest ranking looks alarming.
The EcomToolkit view
A rebound can be good news without being proof that the latest change caused it. Better evaluation protects useful work from exaggerated claims and helps teams repeat interventions that actually have a defensible mechanism. Choose the evidence standard before the chart turns green.
Bring the selection rule and comparison plan to an EcomToolkit measurement review. The aim is a claim your trading, product and finance teams can interpret consistently, including what the evidence does and does not establish.
