A pre-sale load test reports acceptable response times and few errors. During the campaign, shoppers queue behind slow cart requests. One possible explanation is that the test quietly reduced the rate of new work when the application slowed down. The tested workload no longer represented the traffic the business expected to receive.
EcomToolkit’s approach is to ask how much work was offered, how much started, and how much completed before reading the latency chart. This article explains why workload scheduling matters and gives a practical review method for ecommerce teams. All traffic numbers are simplified teaching examples, not capacity estimates for Shopify, WooCommerce, or any other platform.
Table of Contents
- Separate concurrency from arrivals
- Understand coordinated omission
- Compare two workload models
- Account for work that never started
- Model a real shopping workload
- Define evidence for a release decision
- Questions about load-test statistics
- EcomToolkit point of view
Separate concurrency from arrivals
Concurrency describes how many activities are in progress at the same time. Arrival rate describes how frequently new activities begin. A test with 100 active virtual users is not automatically a test of 100 new shoppers per second. Those users may be waiting for responses, pausing between actions, or executing long journeys.
Define the unit of work explicitly. A page request, a cart operation, and a complete shopping session have different durations and resource demands. If one iteration represents a session containing six requests, an iteration rate of 20 per second does not imply a constant rate of 20 HTTP requests per second.
For a simple closed loop with no think time, approximate throughput is concurrency divided by average iteration duration. With 100 active iterations taking one second each, that is about 100 iterations per second. If the duration becomes five seconds, the same concurrency supports only about 20 per second. This arithmetic is a useful warning, not a full queueing model.
Understand coordinated omission
Grafana’s open and closed workload documentation explains that a closed model ties new iteration starts to the completion of earlier iterations. When the system slows, the test can start less work. An open model schedules arrivals independently of completion, subject to the generator’s ability to execute them.
The related problem is often called coordinated omission: the measurement process omits demand or waiting that would have existed under the intended arrival pattern. A latency distribution can look reassuring because work that should have arrived during the slowdown never entered the measurement population.
This does not make closed models inherently invalid. They can represent a bounded population that waits before doing more work, such as a fixed group of back-office operators. The problem is choosing that model unintentionally when the business question concerns externally arriving campaign traffic. The model must match the question being asked.
Write that question at the top of the test plan. “Can these 200 operators complete their workflow?” and “What happens when 80 new journeys arrive every second?” are different tests. A result should not be reused across those questions without explaining the workload change.
Compare two workload models
Consider an invented service slowdown from one to five seconds per iteration. Ignore think time, networking differences, and warmup for this simple comparison. One test keeps 100 iterations active. The other attempts to start 100 iterations each second regardless of completion.
| Workload condition | Healthy service | Slower service |
|---|---|---|
| Closed model: active iterations | 100 | 100 |
| Closed model: approximate starts per second | 100 | 20 |
| Open model: scheduled starts per second | 100 | 100 |
| Open model: approximate concurrency needed | 100 | 500 |
The closed test reduces offered work as duration increases. The open test needs more concurrent execution capacity to sustain its schedule. If the load generator cannot supply that capacity, the intended arrivals may not start. The resulting report must expose that shortfall.
The 500-concurrency figure is an approximation based on rate multiplied by duration under steady conditions. Real shopping workloads include variable latency, retries, pauses, and limits. Use the calculation to plan generator headroom, then inspect observed behavior rather than assuming the estimate is sufficient.
A platform comparison also needs the same workload model on both sides. Testing one system with fixed concurrency and another with fixed arrivals does not produce a clean latency ranking. Preserve the scenario, data, caching state, and test location as part of the comparison evidence.
Account for work that never started
Track scheduled starts, actual starts, completions, application failures, and unfinished work over the same interval. In k6, arrival-rate executors can report dropped iterations when scheduled work cannot start. Investigate generator resources and virtual-user allocation before treating that metric as a direct application failure count.
| Example one-minute accounting | Count | Share of scheduled work |
|---|---|---|
| Scheduled iterations | 6,000 | 100% |
| Started iterations | 5,400 | 90% |
| Completed successfully by cutoff | 5,130 | 85.5% |
| Completed with application failure | 170 | 2.8% |
| Still running at cutoff | 100 | 1.7% |
| Never started | 600 | 10% |
The rows reconcile: successful, failed, and still-running iterations total 5,400 starts; adding 600 never-started iterations returns the 6,000 scheduled total. Successful completion among starts is 95%, while successful completion against the schedule is 85.5%. Both denominators matter.
Do not label the 600 never-started iterations as HTTP errors: no request may have been sent. Do not hide them either. They mean the experiment failed to deliver its complete intended workload. Repeat the run after resolving the generator issue, or report the achieved workload as the actual test boundary.
Define a completion grace period separately from the arrival window. A journey beginning near the final second may reasonably finish later. Reporting all unfinished iterations as failures without that convention can exaggerate problems; allowing unlimited drain time can conceal an unacceptable backlog.
Model a real shopping workload
Build the scenario from observed journey proportions where available: browsing collections, opening products, searching, changing variants, and adding to cart. Keep purchase execution within an authorized test environment, using test payment behavior and isolated inventory. A read-heavy catalog test does not establish payment or order-creation capacity.
Include realistic product diversity. Repeatedly requesting one cached product can make a catalog look easier to serve than a campaign visiting thousands of variants. Conversely, forcing every request to bypass cache may describe a failure scenario rather than normal traffic. Label both scenarios and explain why each matters.
Model pauses intentionally. Human shoppers read and compare products, but random sleep values are not a substitute for an observed journey. Describe whether arrivals represent people entering the site or individual API operations. Keep retries bounded and record them separately so retry amplification is visible during a slowdown.
Use the performance observability guide to align the generated workload with application traces and service metrics. Load-test timing explains the experiment; server evidence helps identify why a specific stage degraded.
Define evidence for a release decision
A useful report includes the test purpose, workload schedule, achieved rate, generator health, application errors, latency distribution, and completion accounting. Break latency down by operation. A fast cached collection response should not dilute a slow cart mutation into an acceptable overall average.
Choose pass criteria before the run. They might include sustaining the target arrival schedule, keeping a defined tail-latency budget, completing journeys within a deadline, and preserving order correctness. The actual thresholds must come from business requirements and the platform’s authorized testing constraints, not a generic benchmark table.
Inspect saturation and recovery as well as peak behavior. After arrivals fall, queues should drain within the agreed window. Check for duplicate operations, stale stock, and downstream backlog. The payment retry and idempotency guide covers a related correctness concern that latency alone cannot establish.
When a run fails, change one major variable at a time. Increasing generator capacity, changing caching, and rewriting the application together makes it difficult to explain which action resolved the issue. Preserve the failed run as evidence of the previous limit.
Questions about load-test statistics
Is an open model always better? No. Choose the model that represents the population and arrival behavior relevant to the decision.
Can dropped iterations prove the store is overloaded? They require investigation; generator limits can prevent work from starting.
Does good average latency prove campaign readiness? No. Inspect tail latency, achieved load, completion, and correctness.
Can results predict exact sales capacity? Only within the tested workload and assumptions. Shopper behavior and downstream dependencies still matter.
EcomToolkit point of view
A load test earns trust by accounting for the work it did and the work it failed to start. Confirm the arrival schedule before celebrating the latency chart. For help turning test results into a launch decision, request a performance readiness audit.
