An export job finishes without an HTTP error and produces a plausible sales table. That does not prove it retrieved every eligible order. The job may have stopped after the first page, repeated a cursor, omitted nested line items, or traversed records while their ordering changed. Every chart built from that extract inherits the gap.
EcomToolkit’s position is that platform statistics need a completeness contract before a dashboard. This guide focuses on paginated commerce data extraction, rather than storefront pagination or disaster recovery. The sample counts are hypothetical. The platform documentation establishes API behavior; the reconciliation workflow is an operational proposal to validate against your own integration.
Table of Contents
- Define what a complete extraction means
- Follow the platform continuation contract
- Count unique records as well as returned rows
- Check nested connections independently
- Make retries and checkpoints safe
- Handle changes during active trading
- EcomToolkit point of view
Define what a complete extraction means
Write down the resource, store, permissions, filter, time window, sort order, API version, and required fields. “All orders” is not a useful acceptance definition if the access token can retrieve only a subset or the query includes only one financial status. A complete response to a narrow query can still be incomplete for the business question.
Choose whether the output represents current state or a historical reporting period. An order created last month can be updated today. Filtering only by creation time may suit one question while missing changes required by another. Document the extraction window and the downstream treatment of refunds, edits, and deleted records.
Do not assume the dataset stays fixed while the job runs. A multi-page traversal may span minutes during active trading. Unless the API explicitly guarantees a consistent snapshot, a cursor should be treated as a continuation mechanism rather than proof of a frozen dataset.
Follow the platform continuation contract
Shopify’s GraphQL pagination documentation describes connections, cursors, and page information. A forward traversal uses the returned end cursor for the next request and checks whether a next page exists. Preserve the same logical query while advancing through the result set.
Do not calculate a cursor or infer completion from a familiar-looking record count. A response with fewer records can reflect the API’s behavior or query constraints. Follow the documented completion indicator, while also detecting repeated cursors and an absence of progress so a bad loop cannot run indefinitely.
Other commerce APIs expose page numbers or continuation tokens. WooCommerce’s Store API pagination advisory is a concrete reminder that callers must respect documented batch constraints. A connector abstraction should preserve each API’s semantics rather than force every provider into an assumed identical loop.
| Contract field | Why it matters | Verification |
|---|---|---|
| Query filters | Defines eligible records | Compare with reporting requirement |
| Sort order | Controls traversal sequence | Preserve across requests |
| Continuation value | Identifies the next request | Store returned value unchanged |
| Completion indicator | Distinguishes end from interruption | Verify final response state |
| Access scope | Limits visible data | Reconcile with permitted coverage |
Count unique records as well as returned rows
Suppose a test export returns 250 rows on page one, 250 on page two, and 100 on page three. The job reports 600 rows. If 20 identifiers occur on both the first and second pages, the extract contains 580 unique orders. Summing revenue over raw rows could also double-count those repeated orders.
Now suppose a controlled fixture contains 620 eligible orders. The extract’s unique-record coverage is 580 divided by 620, approximately 93.55%. The raw row count would misleadingly suggest 96.77% coverage. Neither the completed HTTP requests nor the plausible file size reveals the missing 40 orders by itself.
| Illustrative extraction measure | Value | Meaning |
|---|---|---|
| Raw rows across three pages | 600 | Includes duplicate occurrences |
| Duplicate row occurrences | 20 | Rows beyond the first instance per identifier |
| Unique order identifiers | 580 | Candidate order coverage |
| Eligible orders in controlled fixture | 620 | Known comparison population |
| Unique-record coverage | 93.55% | 580 divided by 620 |
These values are a test fixture, not a platform failure rate. In production, the true eligible count may be unavailable. Report “coverage unknown” when no reliable denominator exists. Use reconciliation samples and gap detection without presenting those checks as mathematical proof of complete extraction.
Check nested connections independently
A complete list of parent orders does not establish complete order details. Line items, refunds, fulfillment records, or other nested collections may have their own pagination boundaries. An export can contain every order identifier while omitting the products needed for category revenue analysis.
Inspect the query shape and the returned page information for every required connection. Compare child counts against a known fixture with unusually large orders. A typical order containing two products will not expose a line-item limit that truncates a wholesale order containing many more.
Shopify’s bulk query documentation explains an asynchronous alternative for larger extracts. Bulk retrieval changes how the work is executed and delivered; it still requires checking operation status, output integrity, relationships, and the documented restrictions. A downloaded file is not automatically a validated reporting table.
Keep parent and child identifiers in the staged dataset until joins have been verified. Count orphaned child rows, missing required parents, and unexpectedly empty collections. This complements the metric grain guide, which explains why combining incompatible row levels distorts ecommerce measures.
Make retries and checkpoints safe
Persist a checkpoint only after the page has been durably staged. If the process records the next cursor before saving the current response, a crash can skip that response on restart. If it saves data but crashes before the checkpoint, the page may be fetched again. The staging design should tolerate that replay.
Use stable source identifiers and an explicit update rule. Deduplicating by order number alone can be unsafe across stores, and deduplicating only by identifier can erase meaningful versions in a historical change log. Include the store boundary and decide whether the destination stores current records or versioned events.
Record query fingerprints with checkpoints so a resumed job cannot quietly use a cursor from different filters. If the provider rejects an old cursor, restart according to a documented recovery strategy and reconcile overlap. Do not silently continue from an arbitrary page number that appears close to the previous position.
Rate limits and transient failures deserve bounded retries with observable outcomes. The platform API rate-limit guide covers capacity management. Here the acceptance condition is that retry behavior preserves completeness and does not inflate revenue through duplicate ingestion.
Handle changes during active trading
For recurring current-state synchronization, an overlapping update-time window can help catch records changed near a checkpoint boundary when the API supports the necessary filters. Deduplicate or upsert the overlap deliberately. Choose its duration from observed delivery and indexing behavior, not an unexplained constant copied between platforms.
An overlap does not recreate an exact historical snapshot. A record that changes after the cutoff may no longer be returned under an older update-time filter, depending on API semantics. If the business requires state as it existed at a specific instant, evaluate event history, a documented snapshot export, or another source that actually supports that requirement.
Run periodic broader reconciliation alongside incremental jobs. Compare identifiers and selected totals under matching definitions, and investigate differences by record rather than adjusting aggregates until they agree. Maintain a visible freshness timestamp so a complete old extract is not confused with a current one.
Before approving the extract, inspect a small set of deliberately difficult records: an order with many lines, an order updated during the run, a record near each time boundary, and a record retrieved after a retry. Keep their identifiers in a protected validation report, not in the public dashboard. These checks do not prove complete coverage, but they expose predictable failure modes that a grand total can hide.
For platform selection, ask the integration vendor to demonstrate this process on a representative dataset. Record how long reconciliation takes and which gaps require manual intervention. An export that finishes quickly but needs hours of unexplained cleanup has a different operating cost from one with explicit, reproducible completeness checks.
EcomToolkit point of view
The most useful platform comparison asks whether your team can reproduce a complete, explainable dataset within its operating constraints. Measure unique-record coverage, nested-record integrity, restart behavior, and freshness before trusting downstream charts. To review the extraction behind your reporting, request an ecommerce data pipeline audit with one representative export and its query contract.
