A platform report says that most merchants are satisfied. The chart is clear, the sample looks large, and a purchasing team adds the result to its shortlist. Yet the report may say little about merchants who were too busy, too frustrated, or no longer connected to the survey’s distribution channel to answer.
EcomToolkit’s position is that a platform statistic needs an account of who could answer and who actually did. This guide provides an original review framework for merchant surveys, with deliberately hypothetical arithmetic. It does not rank platforms or present the examples as current market shares, satisfaction rates, or migration forecasts.
For a related diagnostic, see our guide to weight concentration and effective survey sample size.
Table of Contents
- Define the population behind the headline
- Distinguish nonresponse from detection errors
- Calculate what the observed answers say
- Use sensitivity scenarios to expose the gap
- Compare the merchants who responded
- Read individual questions separately
- Turn the report into evaluation tasks
- The EcomToolkit view
Define the population behind the headline
Start by translating the headline into a precise question. Does the survey describe all active merchants on a platform, newsletter subscribers, agency clients, or visitors who chose to complete a public form? Those groups can have very different experiences and operating constraints.
The population is the group the report wants to describe. The sampling frame is the route or list used to reach potential participants. The respondents are the people who supplied usable answers. A large respondent group does not automatically connect those three layers.
AAPOR’s survey best practices emphasize sample design, clear questions, and transparent methods. For a platform buyer, the practical application is to ask for the recruitment channel, eligibility rules, collection dates, questionnaire, and treatment of incomplete answers before interpreting the ranking.
Distinguish nonresponse from detection errors
Nonresponse concerns eligible people who do not provide the relevant answer. Technology detection errors concern whether a website’s platform was classified correctly. A merchant can be correctly identified as a platform user and still be missing from the satisfaction survey.
The archive’s platform detection guide addresses classification quality. Keep it separate from survey participation. Improving the crawler’s accuracy will not tell you whether nonresponding merchants have different opinions about support, operating cost, or reliability.
Also distinguish nonresponse from survivorship. A survey restricted to active merchants may intentionally exclude former users. That limitation is important, but it is different from an active merchant being invited and declining. State both boundaries rather than combining every missing voice into one vague caveat.
| Question for the report | Why it changes interpretation | Useful disclosure |
|---|---|---|
| Who was eligible? | Defines the intended population | Active merchant criteria |
| How were invitations distributed? | Shows recruitment coverage | Lists, panels, or public channels |
| Who answered the key item? | Establishes its denominator | Valid answer count per question |
| Were former users included? | Changes the experience represented | Inclusion and exclusion policy |
| Were answers weighted? | Changes each response’s influence | Variables, targets, and method |
Treat this as a purchasing checklist. Missing disclosure is a reason to reduce confidence or request clarification, not proof that the published result is false. The report may still be useful for generating questions to test during a platform evaluation.
Calculate what the observed answers say
Consider a hypothetical invitation list containing 1000 eligible merchants. Suppose 200 answer a satisfaction question and 160 report being satisfied. The observed satisfaction rate among respondents is 80 percent. The simple participation fraction for this fully known example is 20 percent.
Those figures do not establish that 80 percent of all 1000 merchants are satisfied. The 800 nonrespondents have not supplied that answer. Calling the respondent result a population estimate requires a sampling and adjustment argument, not just a large font on the chart.
AAPOR’s standard definitions explain why response rates alone cannot establish the presence or size of nonresponse error. Formal outcome-rate calculations also distinguish dispositions and eligibility. The simple fraction above should not be substituted for an appropriate formal rate in a more complex real survey.
Use sensitivity scenarios to expose the gap
Sensitivity analysis asks how the conclusion changes under explicit assumptions about missing answers. It does not discover those answers. For the same hypothetical list, vary the assumed satisfaction rate among the 800 nonrespondents while keeping the 160 observed satisfied respondents fixed.
| Assumed nonrespondent satisfaction | Implied satisfied nonrespondents | Implied satisfaction across 1000 |
|---|---|---|
| 40 percent | 320 | 48 percent |
| 60 percent | 480 | 64 percent |
| 80 percent | 640 | 80 percent |
| 90 percent | 720 | 88 percent |
Every row is a scenario, not a confidence interval or a probability statement. The exercise shows how much the result depends on an unobserved group. It can guide a follow-up study, but it cannot determine which scenario is correct.
For a procurement decision, ask whether the platform recommendation changes across plausible scenarios. If it does, give the survey less weight and collect more direct evidence about the capabilities that matter to your business. Avoid disguising an assumption as an adjustment that has solved the missing-data problem.
Compare the merchants who responded
Where lawful and appropriate data exist, compare respondents with the eligible invitation list on known business characteristics. Revenue band, region, merchant tenure, and operating model may reveal participation differences. Use definitions that are available consistently for both respondents and nonrespondents.
Suppose enterprise merchants answer more often than small retailers. An unweighted result will give them more influence than their share of the invitation list. That may matter if support experience differs by contract, but it does not establish the direction or size of bias without further evidence.
Weighting can align measured characteristics with a justified target population. It cannot guarantee correction for unmeasured differences, such as frustrated merchants being less willing to participate within every revenue band. Show both weighted and unweighted results and explain the assumptions behind the adjustment.
A weighting scheme built from the respondent sample itself also needs scrutiny. Population targets should have an independent rationale. Giving a small subgroup enormous weights can make the estimate sensitive to a few answers, so ask for the weight distribution and the method used to quantify uncertainty.
Read individual questions separately
A merchant may complete the general satisfaction item but skip a question about checkout failures. The denominator for that item is therefore different. A report that repeats the total survey sample beside every chart may obscure important variation in who answered each question.
Check whether “not applicable,” “do not know,” and missing answers are separated. A merchant without international sales should not automatically count as dissatisfied with cross-border tools. Equally, removing everyone who cannot evaluate a feature changes the population described by the resulting percentage.
Question wording also matters. “How satisfied are you with your current platform?” differs from “Would you choose it again?” A merchant can be satisfied with daily operations yet prefer another system for a future expansion. Do not merge these measures into a single score without explaining the decision.
Turn the report into evaluation tasks
Use a survey finding to identify what to test. If merchants describe slow catalog operations, run a representative import and publishing exercise. If support is the concern, examine the available support process and contractual scope. If integration maintenance is the issue, model the ownership of your specific connections.
Keep the survey’s role explicit in the decision record. It may help discover risks, inform interview questions, or describe a relevant peer group. It should not substitute for testing your checkout, catalog, localization, and reporting requirements on the shortlisted platform.
Ask the publisher for methodological details when the statistic materially influences the purchase. Preserve the response alongside the report date. If details remain unavailable, record the limitation and continue with evidence that can be verified directly rather than inventing a certainty score.
The EcomToolkit view
The most useful platform survey is one whose boundaries are visible. A respondent percentage can be informative when it is labelled honestly and connected to a relevant operating context. It becomes misleading when the missing merchants disappear from the interpretation.
Before a satisfaction ranking drives a platform decision, request an EcomToolkit audit. We can help turn broad claims into a concrete evaluation brief focused on the workflows, costs, and failure modes that your own store needs to handle.
For the next analytical step, read our guide to reading the full distribution behind platform satisfaction scores.
