A conversion report can be precise about the sessions it observes and uncertain about the customers it misses. The usual response is to increase tracking coverage or choose a preferred dashboard. Neither action answers the immediate decision: would a plausible pattern among the missing outcomes reverse the conclusion?
EcomToolkit recommends treating that question explicitly. Sensitivity analysis shows how a result changes under stated assumptions about missing information. It does not reconstruct individual shoppers or make incomplete data complete. The following examples are hypothetical, and the population is deliberately bounded so every denominator can be checked.
Table of Contents
- Define what is missing
- Separate coverage from representativeness
- Build a transparent scenario calculation
- Find the decision tipping point
- Distinguish sensitivity ranges from confidence intervals
- Use external totals without forcing a match
- Turn uncertainty into a practical action
- The EcomToolkit view
Define what is missing
Start by separating a missing outcome from a missing attribute and a missing population member. A known eligible session with an unresolved purchase outcome is one problem. A confirmed order whose marketing channel is unknown is another. Sessions absent from every available record create a third problem because the total population size is unknown too.
The simple arithmetic in this guide applies to a known set of eligible units, each with one binary outcome: converted or did not convert. It assumes that some outcomes are observed accurately and others are unresolved. It does not apply unchanged to an analytics stream that selectively loses only purchase events while incorrectly labelling those sessions as non-converters.
Choose a consistent unit. If a person can make several orders, order count divided by people is not a binary conversion probability. If the platform and analytics system define sessions differently, their counts cannot be joined merely because both columns say sessions.
Write the eligibility window, identity rules and outcome deadline before comparing groups. The open and closed funnel guide addresses a related issue: the rules that admit someone into a funnel change the interpretation of its rates.
Separate coverage from representativeness
High coverage does not guarantee that observed units resemble missing ones. A small missing segment concentrated on a broken mobile checkout can matter more than a larger segment missing evenly across the store. Coverage is an operational measure; representativeness is an assumption about what the observed data can tell you.
Statistical discussions distinguish missingness unrelated to values, missingness explainable by observed information, and missingness depending on unobserved values. In practice, a storefront team rarely knows which description holds perfectly. The National Academies’ discussion of sensitivity analysis explains why conclusions should be assessed under alternative assumptions. Its original context is clinical research; the principle is applied here to a separate ecommerce example.
Do not declare data representative because device proportions look similar. Two groups can share the same device mix while differing in purchase intent or error exposure. Conversely, an observed difference does not tell you exactly how much bias exists. It tells you which assumptions deserve attention.
Build a transparent scenario calculation
Suppose there are 10,000 known eligible sessions. Outcomes are resolved for 8,000, of which 320 converted. The observed rate is 4%. Outcomes for the remaining 2,000 sessions are unknown. Let q represent the assumed conversion rate in that missing group.
The overall rate under that assumption is (320 + 2,000 × q) / 10,000. This is a weighted mixture, not a recovered measurement. If q is 1%, the scenario gives 3.4%. If q equals the observed 4%, it gives 4%. If q is 7%, it gives 4.6%.
| Assumed rate among missing sessions | Implied missing conversions | Implied total conversion rate |
|---|---|---|
| 0% | 0 | 3.2% |
| 1% | 20 | 3.4% |
| 4% | 80 | 4.0% |
| 7% | 140 | 4.6% |
| 100% | 2,000 | 23.2% |
The extreme cases establish logical bounds under the example’s assumptions. They are usually too wide for a commercial decision, but they reveal how little the observed data alone guarantees. Narrower scenarios require a reason: historical validation, a comparable instrumented cohort or another relevant source of evidence.
Avoid picking only scenarios that preserve the preferred decision. Record who selected the range and why. If the business cannot defend a plausible range, say that the result is not identified well enough for the proposed action.
Find the decision tipping point
Suppose a planning rule requires the total conversion rate to exceed 3.8% before increasing capacity. This is an illustrative operational threshold, not a recommended universal target. Solve (320 + 2,000 × q) / 10,000 = 0.038. The missing group’s rate must be 3% to reach the threshold.
That tipping point makes the discussion concrete. The decision depends on whether a rate below 3% among unresolved sessions is plausible. A team can investigate that question more effectively than arguing about whether its dashboard is generally trustworthy.
Now compare two campaign variants. Each has its own observed outcomes, missing count and unknown-group rate. Applying the same missing rate to both can conceal differential failure. A variant with a broken confirmation page might lose outcomes differently from the control.
Use a small grid of assumptions for both groups and mark where the ranking reverses. Label it as a scenario comparison. If plausible combinations support opposite choices, the experiment has not supplied a robust answer for that decision, even if its observed-data result looks decisive.
Distinguish sensitivity ranges from confidence intervals
A confidence interval describes sampling uncertainty under a specified model. A sensitivity range describes how conclusions move when assumptions change. They answer different questions and can both be necessary.
A large observed sample can produce a narrow confidence interval while leaving a substantial missing-data problem. Collecting more of the same selectively observed sessions may narrow the interval further without resolving the selection issue. Precision around the observed population is not automatic accuracy for the target population.
Published methods for missing outcome sensitivity analysis show how assumptions about unobserved outcomes enter inference. The arithmetic here is intentionally simpler: it exposes the decision’s dependence without claiming a fitted missing-data model or formal interval for the entire population.
Keep the visual language separate. Use one panel for observed rate and its sampling interval, another for assumption-based scenarios. The conversion confidence interval guide provides additional context for interpreting the first panel.
Use external totals without forcing a match
An order system can help reconcile completed transactions, but it does not automatically provide the missing session denominator or channel attribution. Reconcile the same time window, currency, cancellation policy and order scope before using that total as a constraint.
If all orders can be linked reliably to the known eligible population, the unresolved outcome count may shrink substantially. That is new evidence, not a sensitivity assumption. Preserve the distinction in the report: show what became observed and what remains inferred.
| Additional evidence | What it may resolve | What it does not automatically resolve |
|---|---|---|
| Confirmed order identifiers | Duplicate or absent purchase records | Number of missing sessions |
| Validated session-order linkage | Outcomes for known eligible sessions | Unobserved visitors outside the population |
| Device-specific coverage checks | Where instrumentation differs | Missing group’s true conversion rate |
| Controlled validation cohort | Evidence for a scenario range | Transportability to every channel |
Do not distribute unattributed orders across channels in proportion to observed sales and then describe the result as measured attribution. That is an allocation model. It may be useful for planning if clearly labelled, but its assumptions should remain inspectable.
Turn uncertainty into a practical action
Match the next step to the decision’s sensitivity. If every defensible scenario supports a small reversible change, proceed with monitoring. If the result reverses near the centre of the plausible range, prioritise validation before committing substantial resources.
Assign a specific evidence task: test outcome delivery on the affected browser, reconcile a sample of known orders or examine the release that changed coverage. Avoid an open-ended instruction to fix analytics. The task should reduce a named uncertainty enough to change or stabilise the decision.
Store the scenario assumptions, observed counts and extraction time with the decision record. Revisit them when outcomes resolve or tracking changes. A sensitivity worksheet becomes valuable organisational memory when it explains why a team acted despite uncertainty, rather than merely showing a range of numbers.
The EcomToolkit view
The most useful missing-data question is whether the unknown part could change the action. A transparent tipping point is often more practical than a falsely complete dashboard. Keep measured outcomes, assumptions and operational judgement visibly separate.
If incomplete measurement is blocking growth decisions, request an EcomToolkit audit to identify the smallest validation task that would make the decision defensible.
