An integration can return a successful response, display a convincing list of records, and still import only a fraction of the available data. Pagination is a common reason.
For a startup building a customer-facing integration, this is more than a programming detail. An incomplete import can distort onboarding, reports, and customer expectations while every individual request appears healthy.
The integration needs a definition of completion that extends beyond the first successful response.
Find the API’s continuation contract
Paginated APIs divide a collection into multiple responses. The mechanism varies: page numbers, cursors, continuation tokens, or links may be used.
GitHub’s REST API documentation provides a concrete example. Paginated responses can include a Link header with navigation links, and the available relations depend on the result. GitHub recommends following the provided links rather than assuming every endpoint behaves identically.
For any integration, read the endpoint’s own documentation. Establish how the next page is identified and what indicates that there is no next page.
Do not infer completeness from a conveniently round record count. A response containing 100 items may represent a complete collection or merely a page. The number alone does not answer the question.
Model progress explicitly
A practical import design distinguishes “started,” “partially processed,” “completed,” and “failed.” These are product states, not just log messages.
Suppose an import processes two pages and encounters an error on the third. Marking the entire job as complete would conceal missing data. Throwing away all progress may also be unnecessary.
Instead, define how progress is recorded and how the operation resumes. A checkpoint should represent work that has been successfully processed, not simply a request that was sent.
When retries can revisit records, use a deliberate strategy for recognizing records already handled. Stable external identifiers are often useful inputs to that design, but the exact behavior must fit the API and application.
Respect errors and rate limits
Pagination can turn one user action into many requests. GitHub’s integration guidance recommends avoiding concurrent request patterns that trigger limits and following rate-limit response instructions, including relevant waiting information.
Build those responses into the import’s behavior. A temporary limit should not silently become “no more records.”
Also distinguish a valid empty page from an error response. If the client converts every unexpected response into an empty list, the pagination loop may stop early while reporting success.
Record enough operational context to investigate a failure: the import identifier, the last completed step, and the relevant response status. Keep credentials and unnecessary personal data out of diagnostic logs.
Test beyond the happy path
Use test cases with no records, one page, and several pages. Include a final page containing fewer records than earlier pages.
Then simulate failure between pages, an expired credential, a rate-limit response, and a retry that sees previously processed records. Check the customer-visible status as well as the database result.
Finally, consider whether the source collection can change during the import. Do not assume a stable snapshot unless the API documents one. Your reconciliation strategy should reflect that uncertainty.
A complete import is a verified outcome: the continuation mechanism has been followed, processing has succeeded, and the application can explain what happened. A successful first request is only the beginning.
Sources: GitHub: Using pagination in the REST API · GitHub: REST API best practices