Two reports place the same ecommerce platforms near the top, yet their scores and recommendations look different. One team says the reports agree; another highlights the changed winner. Both observations may be valid because agreement in ordering is not the same as agreement in score magnitude or suitability for a particular store.
EcomToolkit recommends using rank correlation only after making the compared lists genuinely comparable. Spearman correlation offers a compact description of how two orderings move together. This guide uses fictional platforms and invented scores to explain the method without presenting an unverified ranking of real vendors.
Table of Contents
- Establish what each ranking measures
- Calculate the correlation on matched platforms
- Handle tied scores before assigning positions
- Do not give missing vendors an invented last place
- Keep score magnitude beside the ordering
- Avoid turning a small list into a certainty claim
- The EcomToolkit view
Establish what each ranking measures
Before calculating anything, record the underlying measure. A list ordered by detected store count is not measuring the same property as a list ordered by enterprise revenue, developer satisfaction or observed page speed. Correlating those lists may answer a relationship question, but it does not validate either as a universal platform ranking.
Match geography, date and eligible population. A global installed-base report and a survey of a small professional community can produce different orders for legitimate reasons. Differences in coverage should be described before the coefficient appears, because the number cannot explain those design choices for the reader.
Check ranking direction as well. In one dataset, a high score may mean greater satisfaction; in another, a high value may mean more downtime. Decide what rank one means and orient the comparison consistently. Otherwise, a negative association may reflect reversed conventions rather than substantive disagreement.
Our guide to platform share denominators is a useful companion when the rankings come from market statistics. Store count, traffic and transaction value provide different perspectives, and a correlation does not turn those perspectives into the same thing.
Calculate the correlation on matched platforms
The SciPy Spearman reference defines the statistic as a measure of monotonic association. For ranking comparisons, pair each platform’s rank in one list with that same platform’s rank in the other. Matching by row position without matching identities creates a meaningless calculation.
In the fictional example below, only the second and third positions swap. There are five platforms, no ties and complete coverage. The squared rank differences sum to two. The no-tie shortcut gives 1 minus 6 times 2 divided by 5 times 24, which equals 0.9.
| Fictional platform | Rank in report A | Rank in report B | Squared rank difference |
|---|---|---|---|
| Cedar | 1 | 1 | 0 |
| Maple | 2 | 3 | 1 |
| Oak | 3 | 2 | 1 |
| Pine | 4 | 4 | 0 |
| Willow | 5 | 5 | 0 |
The coefficient indicates strong positive ordering association in this specific illustrative pair of lists. It does not mean the reports are 90% correct, that 90% of buyers would make the same decision, or that the platform scores differ by only 10%. Those interpretations attach meanings the calculation does not contain.
Handle tied scores before assigning positions
If two platforms share the same underlying score, assigning them different ranks alphabetically introduces an arbitrary distinction. A conventional approach assigns tied observations the average of the ranks they occupy. Platforms tied across second and third place would each receive rank 2.5.
Use an implementation that handles ties rather than applying the simple no-tie shortcut to every dataset. Spearman correlation can be calculated as Pearson correlation on the rank values, with the chosen tie policy. Keep that policy visible when comparing results from different spreadsheets or analytics libraries.
Rounded published scores can create apparent ties that do not exist in the underlying values. If more precise data are unavailable, acknowledge that limitation instead of inventing extra precision. The published information supports a tied or uncertain ordering, not an exact secret hierarchy.
If all values in a list are identical, the rank variation needed for an ordinary correlation disappears. Do not replace an undefined result with zero. Zero and undefined describe different situations, and silently substituting one makes an uninformative ranking look like evidence of no relationship.
Do not give missing vendors an invented last place
A platform absent from a top-ten article may rank eleventh, may be ineligible, or may never have been evaluated. Giving every missing vendor rank eleven asserts information the publisher did not provide. That shortcut can manufacture agreement or disagreement depending on which vendors each source covers.
One option is to compare the shared set and clearly report its size. If doing so, rerank the common platforms within that set and state that the result describes their relative ordering, not agreement across the complete original lists. Preserve the original ranks in the working file for inspection.
Also report coverage separately: how many platforms appear in both lists and how many occur in only one. A high correlation among three shared vendors is a narrow result when each report evaluates twenty. Readers need both the association and the extent of the comparison.
For truncated lists, the decision may be better served by a simple membership table and a written explanation. There is no requirement to force every comparison into one coefficient. If inclusion rules dominate the disagreement, investigate those rules before applying more elaborate mathematics.
Keep score magnitude beside the ordering
Rank correlation discards the size of the gaps. Two reports can give exactly the same ordering while one places all platforms within a narrow score range and the other separates them dramatically. The ordering agrees, but the commercial interpretation of choosing second instead of first may differ.
For a performance benchmark, retain milliseconds, error rates and workload definitions alongside ranks. A tiny timing difference can move a platform several places in a tightly clustered group. A rank chart alone cannot establish that the shopper would experience a meaningful difference.
Our paired platform benchmark guide explains how to compare equivalent journeys before ordering platforms. Correlation between poorly controlled benchmark lists cannot repair their measurement design. It can only describe the association between the values that were supplied.
| Comparison issue | What to report alongside correlation | Decision it protects |
|---|---|---|
| Different populations | Eligibility, geography and date | Whether comparison is meaningful |
| Tied or rounded scores | Tie policy and score precision | Whether positions are distinguishable |
| Missing vendors | Shared count and coverage table | How narrow the result is |
| Small score gaps | Original units and uncertainty | Whether rank changes matter |
| Different buyer needs | Required workflows and constraints | Whether the ranking is relevant |
Avoid turning a small list into a certainty claim
Five platform names are five paired observations, not thousands, even if the underlying studies measured many stores. The example’s coefficient is a descriptive summary of those five pairs. Treat any inferential claim as a separate statistical question with explicit assumptions about how the observations arose.
The SciPy documentation cautions about the accuracy of its asymptotic p-value for small samples and discusses permutation approaches. A permutation calculation still requires a meaningful null model and appropriate exchangeability assumptions. It is not a button that makes a handpicked vendor shortlist representative of the market.
Ask whether an inferential test is even needed. A buyer comparing two published lists may benefit more from knowing which platforms changed position and why. An exact descriptive table can be more transparent than an impressive-looking p-value attached to a population that was never randomly sampled.
Retain the source snapshots and calculation date. Rankings can change when providers alter weights or add vendors, even if no platform changes its product. A reproducible comparison should make it possible to distinguish methodology changes from changes in observed platform behaviour.
The EcomToolkit view
Agreement between rankings is useful evidence about ordering, not a substitute for platform evaluation. Use Spearman correlation to summarise a clearly matched comparison, then show the changed positions, score gaps and coverage limits that matter to the buying team. A shared winner is still only a starting point.
Bring the two source tables and your store’s required workflows to an EcomToolkit platform review. The goal is to determine which evidence informs your decision, rather than average several league tables into an apparently objective answer.
For a related analytical step, read our guide to testing how platform scorecard weights change the recommendation.
