Skip to main content
Feedback

Pagination options in REST API

Pagination is a technique used to split large amounts of data into smaller, more manageable chunks. This technique is useful when dealing with large data sets that could slow down or crash the application if loaded all at once.

Data Integration provides the ability to paginate data when you work with large data sets in a REST API. You implement pagination in Data Integration using Action Rivers.

Setting up the data source

Procedure

  1. Navigate to the Data Integration Account.
  2. Click Create River and select Action River.
  3. Select Rest Action.
  4. Click Test connection to verify the connection. Refer to your API documentation for details on pagination.

Defining the pagination parameters

  1. Click the Results tab and ensure you set the response type to Data.

  2. After clicking Pagination details, you can select the drop-down arrow to view four different options.

The available options include:

No pagination

The No Pagination option indicates that the API request only provides the initial response, without additional pages. However, if the response size is large, using this option decreases the speed of your River. Select this only when the API call returns information that fits on a single page.

Paginate by Offset

The simplest form of pagination is Paginate by Offset. In the GET request, it can be represented as: image.png

To specify the pages to return, Paginate by Offset uses the Limit and Offset variables in the GET call.

  • Offset Start - If you want to fetch only pages 30-40 from a 100-page API request, you specify the Offset Start as 30. This value determines the starting point of the offset, and the API returns pages 30 to 40 based on the limit set in the GET request.

  • Offset Key -is the JSON data element that you want to offset by. Usually, it is named offset, but there are cases where it may be identified by another name, such as "start_page" or "begin_page," depending on the nature of the response.

  • Page_Offset variable - lets you update the value of the next offset during each iteration using a pre-configured variable named “Page_Offset”. To enable this feature, create a Page_Offset variable in the Variables tab, and select the Update offset value into a page_offset variable in every page iteration checkbox.

  • Stop according to is the criteria at which the Action River stops paginating and pulls data from the API. This drop-down list provides two options:

    • The Page Size option enables you to determine the number of data rows included on a single page of the response and stops the pagination once a page has fewer than the full number of entries, which means it is the last page. This option is valuable when the number of pages is variable or unknown.

    • Key in Response option lets you set a threshold for a specific key to restrict the number of pages returned. For instance, consider you are fetching pages 30-40 from a 100-page API request. Here, you would specify the "Key in Response" as page_number and assign it a value of 30 to limit the number of pages returned.

  • The Sleep After Each Page option lets you set a time delay (in seconds) between each API pagination.

Paginate by Page

Paginate by Page lets you iterate through all the pages without an offset.

  • The Page Key option lets you indicate the key in the JSON response that corresponds to the page number. This key might be named page_number, page_num, or page.
  • Page_Value Variable lets you update the value of the next page during each iteration using a pre-configured variable named page_value. To enable this feature, create a **page_value** variable in the Variables tab, select the Paginate By Page option under Results → Data, and select the “Update the value of the next page into a page_value variable in every page iteration” checkbox.

  • Stop according to is the criteria at which the Action River stops paginating and pulling data from the API. This drop-down list provides two options:

    • The Page Size option lets you determine the number of data rows included on a single page of the response and to stop the pagination once a page has fewer than the full number of entries (that is, the last page). This option is valuable when the number of pages is variable or unknown.
    • Key in Response option lets you set a threshold for a specific key to restrict the number of pages returned. For instance, consider you are fetching pages 30-40 from a 100-page API request. Here, you would specify the Key in Response as **page_number** and assign it a value of 30 to limit the number of pages returned.

  • The Sleep After Each Page option lets you set a time delay (in seconds) between each API pagination.

Next page in response

Use this option where the pagination does not follow a standard cadence established by a variable in the GET call. Instead, it relies on information obtained from a previous GET response.

For instance, consider an API that takes an "id" as an input parameter and returns customer information, such as name, age, height, weight, income, and a value named "id_behind", which corresponds to the ID of the person waiting behind you in line. To obtain the information for the next customer in line, you must make another API call by setting the id parameter to the "id_behind" value of the previous customer. Data Integration provides three options to accomplish this task.

  1. The Static URL With Param From Response feature lets you set a new URL for the next page of the response and map a variable from the previous response to a variable that you can use in the GET call for the next page of the response.
    • The Next Page Key holds the parameter you can use to retrieve the next page of the response. In the example, the "Next Page Key" is "id_behind".
    • Send next page in key: lets you map the “Next Page Key” to the Key in the GET request for the next page. In the example, this can be set to "id".
    • The Send next page using a new URL lets you specify a different URL for the next page's GET request. This is useful when you want to retrieve information from a different database using a different REST URL and pass in the "id_behind" parameter from the previous GET request.

  • Stop according to - is the criteria at which the Action River stops paginating and pulls data from the API. This drop-down list provides two options:

    • The Page Size option lets you determine the number of data rows included on a single page of the response and to stop the pagination once a page has fewer than the full number of entries (that is, the last page). This option is valuable when the number of pages is variable or unknown.
    • Key in Response option lets you set a threshold for a specific key to restrict the number of pages returned. For instance, consider you are fetching pages 30-40 from a 100-page API request. You must specify the "Key in Response" as "page_number" and assign it a value of 30 to limit the number of pages returned.

  • The Sleep After Each Page option lets you set a time delay (in seconds) between each API pagination.

  1. The URL In Response option is used when the URL for the next page is stored within a key of the previous response.

    • Next page key - Specify the key in the previous response that holds the full URL for the next page.

  2. Next Page in Body - Use this option when you want to store the information needed in a two-layer next page call (the ‘next to next page’) where the data for the next page (page ‘b’) is stored in the previous response (page ‘a’), but the page with the information to derive (page ‘c’) can be accessed only via a key in page ‘b’.

    • Page a → contains information to access → Page b → contains information to access→ Page c
    • “Next page key” - Specify the key in the previous response (page ‘’a’) where the page should navigate to is located in (page ‘b’).
    • “Send next page in key” - Specify the key in the page ‘b’ API call to access the page (page ‘c’).

Dynamic changing of parameters

The parameters in many of these settings can be modified dynamically using Variables, which can be created by clicking the button on the top right and identified using the {variable_name}s notation. By inserting these variables into the parameter boxes, you can adjust the pagination of the API call according to the requirements of your Logic River workflow.

On this Page