A checkout experiment can record thousands of sessions while containing far fewer independent treatment assignments. Returning shoppers create repeated visits; employees share business accounts; local stores fulfil overlapping delivery areas. If the analysis ignores those connections, a precise-looking result may rest on the wrong description of how the experiment was actually run.
EcomToolkit’s view is that an experiment begins with the unit that receives the change. This article focuses on assignment and dependence, rather than sample-ratio checks or repeated significance testing. The examples are hypothetical planning exercises, not evidence that a particular checkout treatment improves revenue.
Table of Contents
- Separate assignment from observation
- Recognise when groups share treatment effects
- Count the independent assignments honestly
- Choose weighting before looking at the result
- Preserve assignment through the data pipeline
- Review interference and operational feasibility
- The EcomToolkit view
Separate assignment from observation
The randomisation unit is the entity assigned to a variant. The observation unit is the row recorded in the data. A customer can be assigned once and generate several sessions, orders and line items. Those rows do not become independent assignments simply because the analytics table stores them separately.
Write down a third concept: the outcome you want to estimate. Revenue per assigned customer, orders per business account and conversion per eligible session are different quantities. Choose the outcome because it answers the commercial question, then choose an analysis that respects the design.
Microsoft Research’s pre-experiment guidance discusses selecting randomisation units and interference between experimental units. The relevant lesson for a merchant is to make assignment behaviour explicit before relying on a dashboard’s default statistical calculation.
For a normal storefront interface test, persistent customer assignment may reduce the confusion caused by showing a returning shopper two experiences. Whether the available identifier actually persists across devices, logins and consent changes needs validation. A label called user does not guarantee a stable person-level unit.
Recognise when groups share treatment effects
Consider a wholesale portal in which several employees purchase for one company. If each employee receives a different discount presentation, they may discuss the offer or share a cart. Assigning the whole business account can better match the way the feature is experienced.
Physical operations create similar constraints. A warehouse picking policy may be implemented by shift or site, making order-level randomisation impractical. A delivery-slot change can influence capacity available to other shoppers. The design must reflect these shared resources rather than assuming every order lives in isolation.
| Candidate unit | Plausible ecommerce use | Main design concern |
|---|---|---|
| Session | Short, isolated interface exposure | Returning shoppers can cross variants |
| Customer identifier | Persistent storefront experience | Identity changes and cross-device exposure |
| Business account | Shared B2B purchasing workflow | Fewer independent accounts than users |
| Store or fulfilment site | Operational process change | Site differences and small cluster count |
| Geographic group | Local service or capacity policy | Spillovers across boundaries |
Cluster assignment means assigning groups together. It can reduce some forms of contamination, but it does not guarantee independence between groups. Customers can shop across regions, teams can share operating practices, and marketplaces can connect treated and untreated participants.
Count the independent assignments honestly
Imagine a fictional experiment with 40 business accounts, each generating 100 sessions. The event table contains 4,000 sessions, but treatment was assigned only 40 times. Reporting the session count without the account count hides a central constraint on the experiment’s precision.
The sessions within one account may resemble each other because they share purchasing needs, negotiated prices and an approval process. Treating all 4,000 rows as independent can understate uncertainty. The appropriate remedy depends on the estimator and design, such as account-level aggregation or an analysis that accounts for clustering.
The CONSORT cluster-trial methods paper documents a common planning approximation for equally sized clusters. It uses a design effect of 1 + (m − 1) × rho, where m is observations per cluster and rho is within-cluster correlation. This is a simplified variance-inflation illustration, not a universal sample-size calculator for ratio metrics or unequal account sizes.
| Hypothetical within-account correlation | Cluster size | Illustrative design effect |
|---|---|---|
| 0.00 | 100 | 1.00 |
| 0.01 | 100 | 1.99 |
| 0.05 | 100 | 5.95 |
| 0.10 | 100 | 10.90 |
The arithmetic shows why even modest dependence matters when groups are large. Do not divide a row count by this factor and present the result as the exact number of independent customers. Use a design-specific power analysis that accounts for cluster sizes, outcome variability and the intended estimator.
Choose weighting before looking at the result
A business-account experiment can estimate the average effect per account or an effect weighted toward accounts with more activity. These are not interchangeable. Giving each account equal weight answers a different question from pooling all sessions and allowing the largest account to dominate.
For example, suppose one wholesale account creates half of the recorded orders. Its response can drive a pooled order-level measure even if most accounts respond differently. Report the account-size distribution and specify the weighting policy before examining treatment results.
Be careful with denominators that treatment can change. If a new interface increases repeat visits, revenue per session can fall while revenue per assigned customer rises. That does not automatically invalidate either metric, but it changes the interpretation. Define a primary decision metric and supporting diagnostics in advance.
The metric grain guide helps define these numerators and denominators. The experiment plan must go further by linking the metric to the assignment table and preserving eligible units with zero outcomes.
Preserve assignment through the data pipeline
Keep an assignment record separate from event records. It should contain the experiment identifier, assigned unit, variant and assignment time. Join outcomes back to that record using a documented eligibility window. Avoid inferring assignment from whichever variant happens to appear in the latest event.
Log exposure separately when the feature is actually encountered. Assignment and exposure answer different questions. Restricting analysis to exposed users can be useful under a justified design, but treatment-related exposure differences can introduce selection. Do not switch populations after seeing which produces the preferred result.
Test identity transitions before launch. A guest who signs in, an account that merges and a store that changes region can all create ambiguous mappings. Decide whether the assignment remains fixed, becomes ineligible or follows another prespecified rule. Record exceptions rather than silently moving observations between arms.
Run an A/A exercise through the same pipeline when feasible. It can reveal assignment inconsistencies, broken joins and unexpected imbalance. Passing it does not prove the future treatment will be free from interference, but it verifies some of the mechanics before a commercial decision depends on them.
Review interference and operational feasibility
Ask who can influence whom. If treated accounts receive priority fulfilment from a shared warehouse, untreated accounts may experience slower service. The experiment then measures more than the isolated account experience. A rollout to every account may not reproduce the same effect because the shared capacity constraint changes.
Marketplace research provides concrete examples of this problem. The paper on experimental design in two-sided platforms examines how design choices relate to interference. Its setting is not a universal recipe for a retailer, but it supports asking whether treatment changes the environment experienced by the comparison group.
When few clusters are available, adding more sessions within each cluster may not solve the core precision problem. Consider a longer observation window, a better matched operational design or a narrower decision. Seek statistical review before promising that a handful of stores can reliably detect a small effect.
Connect this work with the sample-ratio mismatch guide. Balanced counts do not make an inappropriate unit correct, and an appropriate unit does not excuse broken assignment. Both design and implementation need to support the claim.
The EcomToolkit view
The number of analytics rows is not the number of independent opportunities to learn. Choose assignment, outcomes and weighting together, then preserve those decisions through implementation. If experiment reports disagree with the way customers actually use the store, request an EcomToolkit analytics audit to review the measurement design before the next rollout decision.
