A theme update makes one journey twice as fast and another journey twice as slow. Someone averages the two speedup ratios and announces an improvement. The calculation may be arithmetically correct for the chosen operation while answering a question that the team never intended to ask.
EcomToolkit recommends deciding what a benchmark summary represents before calculating it. This guide explains geometric means of positive speed ratios using original ecommerce examples. The numbers are hypothetical and do not rank platforms or themes. The goal is a report that preserves both relative change and the absolute waiting time experienced by shoppers.
Table of Contents
- Define the ratio consistently
- See why ratios need care
- Keep absolute milliseconds beside the ratio
- Decide what each test is allowed to represent
- Handle invalid inputs explicitly
- Preserve variability and pairing
- Write a release conclusion that can be checked
- The EcomToolkit view
Define the ratio consistently
For this guide, speedup equals baseline duration divided by candidate duration. A value above one means the candidate is faster, one means unchanged, and below one means slower. Some reports use the inverse convention, so the definition must appear beside the chart.
Choose a journey with the same start and end condition in both versions. Comparing one implementation’s first response with another implementation’s fully usable cart is not a valid speed comparison. The instrumentation contract matters before the averaging method does.
Pair equivalent runs or well-defined summary values under comparable conditions. Record device class, network condition, data volume, and cache state. A candidate tested with a warm cache against a cold baseline can produce a persuasive ratio that mostly describes the test setup.
See why ratios need care
Imagine two equally weighted test journeys. The first improves from 1000 milliseconds to 500 milliseconds, giving a speedup of 2. The second worsens from 1000 milliseconds to 2000 milliseconds, giving a speedup of 0.5. The arithmetic average of those ratios is 1.25.
The geometric mean is the square root of their product: the square root of 1, which equals 1. That summary reflects the reciprocal nature of the two changes under equal weighting. It does not say that customers experienced no additional waiting or that the release is acceptable.
NIST defines the geometric mean as the nth root of the product of the observations. For positive ratios, it can also be calculated by averaging their logarithms and exponentiating the result. The choice of observations and weights remains an analytical decision.
| Hypothetical journey | Baseline duration | Candidate duration | Baseline divided by candidate |
|---|---|---|---|
| Product selection | 1000 ms | 500 ms | 2.0 |
| Cart update | 1000 ms | 2000 ms | 0.5 |
| Equal-weight geometric mean | Not an absolute duration | Not an absolute duration | 1.0 |
| Combined time for one of each | 2000 ms | 2500 ms | 0.8 |
The combined-time row answers a different question: how long one execution of each journey takes in total. It shows a slower combined sequence even though the geometric mean of the ratios is one. Neither summary is a substitute for defining the decision.
Keep absolute milliseconds beside the ratio
Relative gains can look dramatic on small tasks. Reducing a 20 millisecond operation to 10 milliseconds produces the same speedup ratio as reducing a 2000 millisecond operation to 1000 milliseconds. The absolute savings are very different.
A release report should therefore include baseline duration, candidate duration, ratio, and absolute change for each important journey. If checkout deteriorates substantially, a collection of small improvements elsewhere should not conceal that fact inside a headline score.
Set critical-path acceptance rules separately from the benchmark index. For example, a team can require that its selected checkout interaction meet a locally defined latency and correctness condition. That requirement should not disappear when the average across unrelated tests improves.
For broader monitoring, connect the result to the performance observability framework. Controlled benchmarks help isolate an implementation change; production observations reveal which conditions customers actually encounter.
Decide what each test is allowed to represent
Equal weighting gives each test equal influence. That may be suitable for a stable engineering suite intended to cover distinct capabilities. It does not automatically represent customer traffic, because a rarely used configuration screen and a frequently used cart action would count equally.
Traffic weighting makes the summary depend on the chosen traffic distribution. Document the period and population supplying those weights. If a promotion changes the mix of journeys, distinguish the effect of the implementation from the effect of changing the benchmark weights.
A weighted geometric mean can be written as the exponential of the sum of each normalized weight times the logarithm of its positive ratio. Weights must sum to one. This creates a defined index; it does not convert the result into expected total milliseconds saved.
Suppose product selection has weight 0.8 and cart update has weight 0.2 in the earlier example. The index becomes approximately 1.516. Under the same frequencies, expected duration changes from 1000 milliseconds to 800 milliseconds. The ratio of those expected durations is 1.25, illustrating another distinct summary.
Handle invalid inputs explicitly
A logarithmic calculation requires positive values. A recorded zero duration may reflect rounding, inadequate timer resolution, or a measurement error. A timeout is not a zero and should not silently disappear from the benchmark because no successful duration was produced.
Do not add an arbitrary constant merely to make a formula run. Changing every duration by a constant changes the ratios, with a larger effect on short tasks. Investigate the measurement and decide how failures should be reported before constructing the index.
| Input problem | Honest handling | Misleading shortcut |
|---|---|---|
| Rounded zero duration | Improve resolution or flag invalid | Add one without explanation |
| Timed-out journey | Report failure rate and timeout rule | Drop unsuccessful runs silently |
| Missing baseline pair | Mark comparison unavailable | Borrow an unrelated baseline |
| Changed test definition | Version the benchmark contract | Treat old and new timings as identical |
| Different cache conditions | Separate or standardize conditions | Blend them into one ratio |
A benchmark with fewer successful candidate runs may appear faster because its slowest attempts failed. Keep completion counts and failure outcomes visible. A speed index over successful observations alone needs that scope stated explicitly.
Preserve variability and pairing
One run per journey is vulnerable to noise. Repeat the exercise under controlled conditions and inspect the distribution, not just the best recording. Where the design supports pairing, keep the relationship between corresponding baseline and candidate conditions available for analysis.
Decide whether the index uses ratios of per-journey summaries or a set of paired run ratios. Those constructions can differ. Document the exact calculation and avoid switching methods between releases just because one produces a more favorable headline.
The pooled percentile guide addresses a related aggregation problem. Percentiles, means, and ratios have different mathematical behavior. Reusing a familiar dashboard aggregation without checking that behavior can create confident but misleading comparisons.
If the result is close to the acceptance boundary, collect enough evidence to understand uncertainty. A geometric mean is a point summary, not proof that every observed difference is stable or caused by the release. The test design and variation still matter.
Write a release conclusion that can be checked
A useful conclusion names the suite, environment, weighting policy, completion count, and index definition. It highlights any important regressions even when the overall score improves. Keep the per-journey table close enough that the reader can verify what drives the headline.
Separate the engineering conclusion from the commercial conclusion. Faster controlled interactions may support a release decision, but they do not establish a conversion lift. Claims about revenue require evidence from an appropriate business analysis with its own population and uncertainty.
Store the raw timings, benchmark version, and calculation alongside the report. When someone reruns the suite next month, they should be able to explain whether a change came from code, devices, traffic weights, or the definition of the test itself.
The EcomToolkit view
A geometric mean can summarize proportional speed changes well when its inputs and purpose are explicit. It should sit beside absolute durations, failures, and critical journey results. A single elegant number cannot decide whether the shopping experience improved where it matters.
If benchmark reports disagree about whether a storefront release is faster, request an EcomToolkit audit. We can help define comparable journeys and a reporting method that makes the underlying tradeoffs visible before the team commits to a rollout.
For a related practical question, read our guide to paired platform benchmark design.
