Pagination

Pagination allows you to retrieve large datasets from APIs by breaking them into smaller chunks. This system supports multiple pagination strategies to work with different API designs.

When to Use Pagination

• APIs that return data in pages (e.g., 100 records per page)

• APIs with rate limits that require controlled data retrieval

• Large datasets that cannot be retrieved in a single request

• APIs that use cursor tokens or next URL links

Pagination Types Overview

Type	Use Case	Example
Page-based	APIs using page numbers	?page=1, ?page=2
Offset-based	APIs using skip/offset	?offset=0&limit=20, ?offset=20&limit=20
Cursor-based	APIs using cursor tokens	?cursor=abc123
Next URL	APIs providing next page links	Uses full URL from response
Computed	Custom calculation needed	skip=iteration*20

---

Getting Started

Step 1: Access Pagination Configuration

1. Navigate to your API connection configuration

2. Look for the Pagination section

3. Click Add Pagination Field to start configuring

Step 2: Understand Field Properties

Each pagination field has these key properties:

• Name: A descriptive name for this field

• Parameter Name: The API parameter name (e.g., page, offset, cursor)

• Parameter Type: The type of pagination (page, offset, cursor, next_url, custom)

• Field Type: How the value is determined (static, jsonpath, computed)

• Parameter Location: Where to place the parameter (query, body, header)

• Start Value: Initial value (for static pagination)

• End Value: Final value (for static pagination)

• Step: Increment value (for static pagination)

• Max Iterations: Maximum number of pages to fetch

Step 3: Test Your Configuration

1. Click Test Next Iteration to test a single page

2. Review the response to verify data is correct

3. Click Verify Pagination Configuration to test iteration 2

4. Once verified, save your configuration

---

Scenario 1: Page-Based Pagination

Use Case: API uses page numbers like ?page=1, ?page=2, etc.

Example API

GET https://api.example.com/users?page=1
GET https://api.example.com/users?page=2

Configuration Steps

1. Add Pagination Field

   • Click Add Pagination Field

2. Configure Field Properties

   • Name: Page Number

   • Parameter Name: page

   • Parameter Type: page

   • Field Type: static

   • Parameter Location: query

   • Start Value: 1

   • End Value: 100 (or leave empty for no limit)

   • Step: 1

   • Max Iterations: 100

3. Test Configuration

   • Click Test Next Iteration - should show page 1

   • Click again - should show page 2

   • Verify the URL shows ?page=1, then ?page=2

4. Verify and Save

   • Click Verify Pagination Configuration

   • Once verified, save your connection

Expected Behavior

• Iteration 1: GET /users?page=1

• Iteration 2: GET /users?page=2

• Iteration 3: GET /users?page=3

• Continues until end value or max iterations reached

---

Scenario 2: Offset-Based Pagination

Use Case: API uses offset/limit like ?offset=0&limit=20, ?offset=20&limit=20

Example API

GET https://api.example.com/products?offset=0&limit=20
GET https://api.example.com/products?offset=20&limit=20

Configuration Steps

1. Add Pagination Field for Offset

   • Name: Offset

   • Parameter Name: offset

   • Parameter Type: offset

   • Field Type: static

   • Parameter Location: query

   • Start Value: 0

   • End Value: 1000 (or leave empty)

   • Step: 20 (must match your limit)

   • Max Iterations: 50

2. Add Limit Field (if API requires it)

   • Name: Limit

   • Parameter Name: limit

   • Parameter Type: custom

   • Field Type: static

   • Parameter Location: query

   • Static Value: 20

   • Max Iterations: 50

3. Test Configuration

   • Iteration 1 should show ?offset=0&limit=20

   • Iteration 2 should show ?offset=20&limit=20

   • Iteration 3 should show ?offset=40&limit=20

Expected Behavior

• Iteration 1: GET /products?offset=0&limit=20

• Iteration 2: GET /products?offset=20&limit=20

• Iteration 3: GET /products?offset=40&limit=20

• Offset increases by step (20) each iteration

Important Notes

• Step value must match your limit if the API uses offset/limit pattern

• If API only uses offset (no limit parameter), you only need the offset field

---

Scenario 3: Cursor-Based Pagination

Use Case: API returns a cursor token in the response that you use for the next request

Example API Response

{
  "data": [...],
  "next_cursor": "eyJpZCI6IjEyMzQ1In0",
  "has_more": true
}

Configuration Steps

1. Add Cursor Field

   • Name: Cursor Token

   • Parameter Name: cursor

   • Parameter Type: cursor

   • Field Type: jsonpath

   • Parameter Location: query

   • JSONPath: $.body.next_cursor (adjust based on your response structure)

   • Max Iterations: 100

   • Is Dynamic Pagination: true

2. Optional: Add Stop Condition

   • Name: No More Data

   • Source: stop_condition

   • Logic Operator: OR

   • Rules:

     • Rule 1: Field $.body.next_cursor, Operator isNull

     • Rule 2: Field $.body.has_more, Operator equals, Value false

3. Test Configuration

   • First iteration: No cursor parameter (initial request)

   • Second iteration: Should extract cursor from first response

   • Verify cursor value appears in query params

Expected Behavior

• Iteration 1: GET /items (no cursor)

• Iteration 2: GET /items?cursor=eyJpZCI6IjEyMzQ1In0 (from iteration 1 response)

• Iteration 3: GET /items?cursor=eyJpZCI6IjY3ODkwIn0 (from iteration 2 response)

• Stops when cursor is null or stop condition met

Finding the Correct JSONPath

1. Make a test API call and view the response

2. Identify where the cursor appears (e.g., response.next_cursor, response.pagination.cursor)

3. Use JSONPath format:

   • Root level: $.next_cursor

   • Nested: $.pagination.next_cursor

   • In body: $.body.next_cursor

---

Scenario 4: Next URL Pagination

Use Case: API provides a complete next page URL in the response

Example API Response

{
  "data": [...],
  "pagination": {
    "next": "https://api.example.com/users?page=2&token=abc123"
  }
}

Configuration Steps

1. Add Next URL Field

   • Name: Next Page URL

   • Parameter Type: next_url

   • Field Type: jsonpath

   • Parameter Location: query (or leave default)

   • JSONPath: $.body.pagination.next (adjust based on your response)

   • Max Iterations: 100

   • Is Dynamic Pagination: true

2. Test Configuration

   • First iteration: Uses your base URL

   • Second iteration: Should use the next URL from first response

   • Verify the URL changes correctly

Expected Behavior

• Iteration 1: GET https://api.example.com/users?page=1

• Iteration 2: GET https://api.example.com/users?page=2&token=abc123 (from response)

• Iteration 3: GET https://api.example.com/users?page=3&token=def456 (from response)

• Stops when next URL is null or empty

Handling Relative URLs

If the API returns relative URLs like /users?page=2:

• The system automatically converts them to absolute URLs

• Uses your base URL as the origin

• No additional configuration needed

Handling Query Parameters in Next URL

If the next URL contains query parameters:

• The system extracts them automatically

• Merges with your existing query parameters

• Next URL parameters take priority

---

Scenario 5: Computed Pagination

Use Case: Need to calculate pagination value using a formula

Example: Skip Parameter

GET /items?skip=0
GET /items?skip=20
GET /items?skip=40

Configuration Steps

1. Add Computed Field

   • Name: Skip Records

   • Parameter Name: skip

   • Parameter Type: custom

   • Field Type: computed

   • Parameter Location: query

   • Computed Rule: iteration * 20

   • Max Iterations: 50

2. Test Configuration

   • Iteration 1: skip=20 (1 * 20)

   • Iteration 2: skip=40 (2 * 20)

   • Iteration 3: skip=60 (3 * 20)

Common Computed Rules

Rule	Result (iteration 1, 2, 3)
iteration * 20	20, 40, 60
(iteration - 1) * 50	0, 50, 100
iteration * 10 + 5	15, 25, 35
iteration * 100	100, 200, 300

Expected Behavior

• Iteration 1: GET /items?skip=20

• Iteration 2: GET /items?skip=40

• Iteration 3: GET /items?skip=60

• Value calculated using the formula each iteration

---

Scenario 6: Pagination with Stop Conditions

Use Case: Stop pagination when certain conditions are met (e.g., empty data, null next cursor)

Example: Stop When Data is Empty

Configuration Steps

1. Add Pagination Field (e.g., page-based)

   • Configure your pagination as usual

2. Add Stop Condition

   • Name: Empty Data Check

   • Source: stop_condition

   • Logic Operator: OR

   • Rules:

     • Rule 1:

       • Field: $.body.data

       • Operator: isEmpty

     • Rule 2:

       • Field: $.body.items

       • Operator: isEmpty

3. Test Configuration

   • Test iterations until you reach an empty response

   • System should stop automatically when condition is met

   • Status should show "Stop condition met"

Common Stop Conditions

Stop When Next Cursor is Null

{
  "name": "No More Pages",
  "logicOperator": "OR",
  "rules": [
    {
      "field": "$.body.next_cursor",
      "operator": "isNull"
    }
  ]
}

Stop When Has More is False

{
  "name": "Last Page",
  "logicOperator": "AND",
  "rules": [
    {
      "field": "$.body.has_more",
      "operator": "equals",
      "value": "false"
    }
  ]
}

Stop When Data Array is Empty

{
  "name": "No More Records",
  "logicOperator": "OR",
  "rules": [
    {
      "field": "$.body.data",
      "operator": "isEmpty"
    },
    {
      "field": "$.body.results",
      "operator": "isEmpty"
    }
  ]
}

Available Operators

• Equality: equals, notEquals

• Null Checks: isNull, isNotNull

• Empty Checks: isEmpty, isNotEmpty

• Comparison: greaterThan, lessThan, greaterThanOrEqual, lessThanOrEqual

• String: contains, notContains, startsWith, endsWith

Logic Operators

• AND: All rules must be true

• OR: At least one rule must be true

---

Scenario 7: Pagination in Request Body

Use Case: API requires pagination parameters in the request body (POST/PUT requests)

Example API

POST /api/search
{
  "query": "test",
  "page": 1,
  "pageSize": 20
}

Configuration Steps

1. Add Pagination Field

   • Name: Page Number

   • Parameter Name: page

   • Parameter Type: page

   • Field Type: static

   • Parameter Location: body ⚠️ Important: Select "body"

   • Start Value: 1

   • Max Iterations: 50

2. Add Page Size Field (if needed)

   • Name: Page Size

   • Parameter Name: pageSize

   • Parameter Type: custom

   • Field Type: static

   • Parameter Location: body

   • Static Value: 20

3. Test Configuration

   • Verify the request body includes pagination parameters

   • Check that page number increments correctly

Expected Behavior

• Iteration 1: Body contains {"page": 1, "pageSize": 20}

• Iteration 2: Body contains {"page": 2, "pageSize": 20}

• Iteration 3: Body contains {"page": 3, "pageSize": 20}

Important Notes

• Body pagination works with JSON request bodies

• Existing body parameters are preserved

• Pagination parameters are merged into the JSON body

---

Scenario 8: Pagination in Headers

Use Case: API requires pagination token in request headers

Example API

GET /api/data
Headers:
  X-Page-Token: abc123

Configuration Steps

1. Add Pagination Field

   • Name: Page Token

   • Parameter Name: X-Page-Token

   • Parameter Type: cursor (if token comes from response)

   • Field Type: jsonpath

   • Parameter Location: header ⚠️ Important: Select "header"

   • JSONPath: $.body.next_token

   • Max Iterations: 100

2. Test Configuration

   • Verify headers include the pagination token

   • Check that token updates from response

Expected Behavior

• Iteration 1: Header X-Page-Token: (none or initial value)

• Iteration 2: Header X-Page-Token: abc123 (from iteration 1 response)

• Iteration 3: Header X-Page-Token: def456 (from iteration 2 response)