A platform evaluation can contain accurate measurements and still produce an inaccurate ranking. One common cause is averaging the p95 latency from several regions, stores, or test runs and calling the result an overall p95. The spreadsheet looks tidy, but it no longer describes the underlying request population.
EcomToolkit’s approach is to ask how a statistic was assembled before using it to choose infrastructure. This guide focuses on percentile aggregation, a specific problem that broad platform scorecards often leave unexplained. The request distributions below are invented to make the arithmetic reproducible. They are not measurements of Shopify, WooCommerce, BigCommerce, or any other provider.
Table of Contents
- Decide what population the benchmark represents
- Reproduce the averaging error with ten requests
- Preserve distributions when aggregating
- Require an evidence packet from each candidate
- Compare candidates under your expected mix
- Repair an existing executive dashboard
- EcomToolkit point of view
Decide what population the benchmark represents
A request percentile and a merchant percentile answer different questions. A request-weighted p95 describes latency across a population of requests. A median merchant result describes the middle merchant after calculating one value for each store. Large stores have more influence on the first; each included store can have equal influence on the second.
Neither is automatically the correct platform-selection measure. A buyer may want the experience of a typical implementation, the experience of a typical request, or performance under their own traffic mix. Name that decision explicitly. A label such as platform speed conceals too much to guide an expensive migration.
Also define the operation. A cached product document, a personalized pricing request, and a completed cart update should not be combined as though they were interchangeable. Establish response boundaries, payload expectations, errors, geography, concurrency, and cache state before comparing numbers. Use the platform total cost model for the broader decision beyond latency.
Reproduce the averaging error with ten requests
Consider two regions in a small diagnostic dataset. Region A has nine requests, each taking 100 milliseconds. Region B has one request taking 1,000 milliseconds. Use the nearest-rank percentile definition for this example: sort the observations and select position ceiling(p × n), with positions starting at one.
Region A’s p95 is 100 milliseconds. Region B’s p95 is 1,000 milliseconds. The arithmetic average of those two percentiles is 550 milliseconds. Weighting each regional percentile by its request count gives 190 milliseconds. Neither calculation gives the p95 of the pooled requests.
| Illustrative calculation | Method | Result |
|---|---|---|
| Region A p95 | Ninth of nine identical values | 100 ms |
| Region B p95 | Only observation | 1,000 ms |
| Mean of regional p95 values | (100 + 1,000) / 2 | 550 ms |
| Request-weighted mean of p95 values | (9 × 100 + 1 × 1,000) / 10 | 190 ms |
| Pooled nearest-rank p95 | Tenth value of all ten requests | 1,000 ms |
The pooled position is ceiling(0.95 × 10), which is ten. That observation is 1,000 milliseconds. Different percentile conventions can interpolate and produce another value for a tiny dataset, which is why the method belongs in the benchmark documentation. The central issue remains: regional percentile summaries do not retain the complete distribution.
This example is intentionally too small for a purchasing decision. Its purpose is to expose a calculation error with numbers that anyone can check. In a real benchmark, retain enough observations to characterize the tail and publish the sample count. A precise-looking p99 from a handful of requests should prompt questions about evidence quality.
Preserve distributions when aggregating
The Prometheus histogram guidance explains why precomputed quantiles cannot generally be aggregated into a combined quantile. Histograms preserve counts across value ranges, allowing compatible distributions to be combined before estimating a percentile. Raw observations also permit recomputation where retention and cost allow.
That does not make every histogram result exact. Bucket width or histogram resolution affects the estimate. A wide bucket around an important acceptance threshold may conceal a meaningful difference between two implementations. Select precision according to the decision, and avoid presenting more decimal places than the underlying representation supports.
Ask the benchmark provider whether it retains raw measurements, compatible histograms, or a documented mergeable sketch. If only p50, p95, and p99 summaries remain, request a fresh combined measurement or keep the segments separate. Reconstructing an exact distribution from three summary points is not a valid substitute.
For recurring reports, agree on aggregation before collection begins. Retrofitting the correct method after the raw data has expired can be impossible. Keep units consistent and record changes to bucket boundaries, collection libraries, and query definitions with the benchmark version.
Require an evidence packet from each candidate
A useful evidence packet is short enough to read and detailed enough to reproduce. It should explain who generated the traffic, where requests originated, what operations ran, and which outcomes were excluded. An unusually fast result may simply reflect a narrower workload than the competing test.
| Evidence field | What to request | Why it changes interpretation |
|---|---|---|
| Observation unit | Request, page view, session, or merchant | Prevents mixed populations |
| Workload | Routes, payloads, cache mix, authentication | Establishes comparable work |
| Distribution method | Raw data, histogram, sketch, percentile convention | Determines aggregation validity |
| Coverage | Counts by region and operation | Reveals uneven representation |
| Failure treatment | Errors, timeouts, dropped work | Exposes missing slow outcomes |
| Environment | Version, resources, network, warmup | Makes the result reproducible |
Separate completed-request latency from failure rate. A timeout should not quietly disappear from the evaluation, but assigning an arbitrary duration and treating it as an observed completion can also mislead. Report the timeout boundary and count alongside successful-request distributions.
If the load generator could not deliver the intended workload, record offered and achieved rates. The arrival-rate load testing guide explains that related measurement problem. Correct percentile aggregation cannot rescue a test that failed to exercise the intended demand.
Compare candidates under your expected mix
Build a workload from the shopping journeys the business actually serves. If most sessions browse cached pages but revenue-critical customers use negotiated prices, preserve both groups. The aggregate can help estimate overall exposure, while the personalized segment protects an important minority from being hidden by volume.
Use the same traffic mixture for each candidate when testing the effect of architecture. Also show the observed mixture if it differs. This separates a platform difference from a change in the people or requests included in the calculation. State which implementation choices are native, customized, or supplied by an integration.
Pair tail latency with correctness and operating effort. A faster cart result that omits a required promotion is not an equivalent result. Similarly, a benchmark that requires continuous manual cache preparation may have a different operating cost from the production design the team can maintain.
Avoid converting a technical percentile improvement directly into a revenue forecast. The benchmark demonstrates behavior under its test conditions. Commercial impact needs evidence from the relevant shopping population, with attention to stock, pricing, acquisition mix, and seasonality.
Repair an existing executive dashboard
Begin by tracing one headline value back to its source rows. If the dashboard averages regional p95 fields, relabel the existing value accurately while the calculation is repaired. Preserve the original export so the team can understand why the historical series changes.
Next, determine whether raw samples or compatible distribution data still exist. Recompute the overall percentile from those inputs and retain regional views for diagnosis. If they do not exist, publish the separate regional statistics and explain the gap. Do not invent a corrected historical aggregate.
Add a compact methodology note beside the chart: population, time window, percentile method, sample count, and failure treatment. Give one person ownership of that definition. An automated chart can change meaning when an upstream report changes, even if no dashboard code is edited.
Can a median of store-level p95 values still be useful? Yes, if it is explicitly presented as the median of those store summaries. It can describe a typical included store’s tail result. It must not be renamed the p95 of all customer requests, because the observation unit and weighting are different.
Should a platform with the lowest pooled p95 always win? No. Check the critical segments and functional requirements first. A pooled result can obscure a weak region or an unsupported business workflow even when its aggregation is mathematically correct.
EcomToolkit point of view
The strongest platform comparison lets another team reproduce the conclusion. Keep distributions long enough to aggregate them correctly, separate the shopping operations that matter, and make missing outcomes visible. Platform statistics should reduce uncertainty rather than hide it behind a familiar p95 label. For an independent review of the evidence, request a platform analytics audit.
