A platform report says the average merchant grew this year. Before comparing that result with another platform, ask whether the same merchants were measured in both periods. If weaker stores closed or stopped reporting, the remaining group can look stronger even when none of its members improved.
EcomToolkit treats population continuity as a required part of benchmark interpretation. This guide shows how to separate continuing stores from entrants, exits, and unknown observations. Every numerical example is hypothetical. It is a method for reviewing evidence, not a ranking of named platforms or a claim about current ecommerce market shares.
Table of Contents
- Write down the population behind the percentage
- Reproduce the survivor effect with four stores
- Separate entrants exits and missing records
- Review performance dataset eligibility too
- Add sensitivity instead of invented certainty
- Use the benchmark in a platform decision
- EcomToolkit point of view
Write down the population behind the percentage
Start with the entity being counted. A domain, storefront, merchant account, company, and transaction are different units. One company may operate several storefronts, and a single storefront may use technologies from several providers. A percentage based on detected websites should not silently become a percentage of merchants or revenue.
W3Techs describes its technology survey methodology, including website-level counting and its treatment of subdomains. That is a useful reminder to read the provider’s definition before using a technology statistic in a platform decision. A methodology for technology detection does not automatically support conclusions about commercial success.
Record geography, observation date, active-store rule, minimum traffic requirement, and any industry or size filters. Identify whether the report describes a current cross-section or follows an original group over time. Both can be useful, but they answer different questions and should not share an unqualified growth label.
Reproduce the survivor effect with four stores
Suppose a fictional platform has four measured stores in period one. Their sales are 100, 100, 20, and 20 in arbitrary units. The average is 60. In period two, only the first two stores remain observed, and each still has sales of 100. The observed average is now 100, an increase of approximately 66.7%.
| Fictional store | Period one sales | Period two sales | Observation status |
|---|---|---|---|
| A | 100 | 100 | Observed in both periods |
| B | 100 | 100 | Observed in both periods |
| C | 20 | Unknown | No later observation |
| D | 20 | Unknown | No later observation |
Neither continuing store grew. The current observed group has a higher average because its composition changed. Calling that result average merchant growth would invite a false interpretation unless the definition made the changing population explicit.
Do not fill C and D with zero simply because their later observations are absent. They may have closed, migrated, changed domains, or lost a reporting connection. A confirmed closure and a missing record are different facts. Even when a closure is confirmed, whether zero belongs in a later sales calculation depends on the metric’s stated estimand.
The honest report can show the current-group average, the continuing-store change, and observation coverage separately. It should also state that the later outcomes of two original stores are unknown. This retains useful information without inventing the missing half of the story.
Separate entrants exits and missing records
Create a status bridge between periods. Start with stores observed at baseline. Identify those still observed, those confirmed to have left the defined population, and those whose status cannot be established. Then identify new entrants in the later period. Keep the evidence for each classification.
A change in domain is not necessarily a new business. A technology detector losing a signal is not necessarily a migration. A reporting integration disconnecting is not necessarily a closure. Resolve identity changes where possible and use an unknown category where evidence remains incomplete.
A practical bridge should reconcile counts exactly. If 500 baseline stores become 400 continuing stores, 40 confirmed exits, and 60 unknowns, those categories sum back to 500. If 80 new stores enter, the later observed group contains 480 stores under this simplified definition. A reviewer can then understand what changed before seeing a growth chart.
| Population view | Question it answers | What it cannot establish alone |
|---|---|---|
| Current cross-section | What do observed stores look like now? | Growth of the original group |
| Continuing-store panel | How did the same observed stores change? | Outcomes of departed stores |
| Baseline cohort with status tracking | What happened to the original population? | Unknown outcomes without more evidence |
| Entrant group | What do newly observed stores contribute? | Whether all entrants are new businesses |
| Coverage bridge | How much of the original sample remains visible? | Why every missing record disappeared |
A balanced panel removes the simplest composition change by requiring observations in both periods. It still conditions on remaining observable. If staying in the sample is related to business performance, the continuing-store result should not be generalized automatically to all merchants that started on the platform.
Review performance dataset eligibility too
The same problem appears in site-performance statistics. A group of origins that has enough observations in both periods may differ from the full merchant population. A store disappearing from a performance dataset does not, by itself, prove that its platform failed or that the business closed.
The Chrome UX Report methodology describes eligibility and coverage conditions for CrUX data. Read those conditions when using CrUX-based platform comparisons. Treat data availability as part of the measurement process, rather than interpreting every absence as a measured performance result.
Keep browser population, device category, origin versus URL scope, and reporting window aligned. A continuing-origin comparison can be useful for observing changes in that subset. It does not isolate the effect of the platform from changes in themes, apps, traffic, or merchant behavior.
Add sensitivity instead of invented certainty
When outcomes are missing, a sensitivity analysis can show how conclusions depend on explicit assumptions. For example, calculate alternative results under documented plausible outcomes for missing stores. Label these as scenarios, not recovered observations. Explain why the chosen range is useful and where it remains speculative.
Weighting observed stores to resemble a target population can address some known differences, but it does not magically recover unobserved outcomes. Its credibility depends on measured variables and assumptions about selection. A report should state those assumptions and retain the unweighted counts for inspection.
Separate selection issues from ordinary sampling uncertainty. A narrow confidence interval around the survivor average does not make it representative of the original cohort. The archive’s conversion confidence interval guide explains precision; population continuity is an additional question that must be answered first.
Use the benchmark in a platform decision
Ask a vendor for the population definition, baseline count, continuing count, entrants, exits, unknowns, and merchant size distribution. If only a headline percentage is available, record that limit in the comparison rather than supplying assumed detail. A missing methodology is a reason to reduce the weight of the claim.
Then connect the available evidence to your own requirements: catalog structure, operating team, integration ownership, checkout constraints, and migration effort. A broad platform benchmark can supply context, but your decision should also include a representative workflow demonstration and a cost model tied to your actual operation.
For changing segment weights inside an observed population, use the Simpson’s paradox guide. That is related but distinct: one problem concerns how groups are weighted, while this guide concerns which stores remain visible at all. Investigating both prevents a polished chart from doing more argumentative work than its data supports.
Before circulating a comparison internally, give its chart a precise subtitle. For example, state that it covers stores observed in both named periods and include the retained share of the baseline sample. Place the unknown count near the result rather than in an inaccessible appendix. This makes the chart harder to reuse without its qualifications and helps colleagues distinguish an observed survivor result from a claim about every merchant that adopted the platform.
EcomToolkit point of view
The most useful question about a growth benchmark may be who is no longer in it. Keeping entrants, exits, and unknowns visible turns a promotional percentage into a reviewable comparison. If platform selection is being driven by incompatible statistics, request an EcomToolkit audit to define the populations and connect the evidence to the workflows your team actually needs.
