Skip to main content
Feedback

Pagination options in Rest API

Pagination is a common technique used to split large amounts of data into smaller, more manageable chunks. This technique is especially useful when dealing with large data sets that could slow down or crash the application if loaded all at once. Data Integration provides users with the ability to paginate data when working with large data sets in REST API. This document will discuss how to implement pagination in Data Integration using Action Rivers.

Implementation procedure

To implement pagination in REST API, we will use the following steps:

Set up the data source

  1. Click the Create River and select Action River.
  2. Select Rest Action.
  3. To establish your Custom API connection, follow to these guidelines. Test the connection, and refer to your API documentation for information on pagination.

Define the pagination parameters

  1. Click the Results tab and ensure that the response type is set to Data.

  2. After clicking on Pagination details, you can select the dropdown arrow to reveal 4 different options.

What do these 4 options mean?

No pagination

The No Pagination option means that the API request will only provide the initial response without any additional pages. However, if the response size is large, using this option could potentially decrease the speed of your River. It is a suitable choice only when the API call returns information that fits on a single page.

Paginate by Offset

The simplest form of pagination is Paginate by Offset. In the GET request, it can be represented as: image.png

To specify the pages to return, Paginate by Offset uses the Limit and Offset variables in the GET call. In the example mentioned above, the response will retrieve pages 100 to 120.

  • Offset Start - If you want to fetch only pages 30-40 from a 100-page API request, you need to specify the Offset Start as 30. This value determines the starting point of the offset, which means that the API will return pages 30 to 40 based on the limit set in the GET request.

  • Offset Key - refers to the JSON data element that you wish to offset by. Usually, it is named offset, but there are cases where it may be identified by another name such as "start_page" or "begin_page," depending on the nature of the response.

  • Page_Offset variable - allows you to update the value of the next offset during each iteration by using a preconfigured variable named “Page_Offset”. To enable this feature, create a Page_Offset variable in the Variables tab, and check the Update offset value into a page_offset variable in every page iteration box.

  • Stop according to is the criteria at which the Action River will stop paginating and thus pulling data from the API. This dropdown list gives you 2 options:

    • The Page Size option enables the user to determine the number of data rows included on a single page of the response and to stop the pagination once a page has less than the full number of entries (i.e., the last page). This option is particularly valuable when the number of pages is variable or unknown.
    • Key in Response option enables you to set a threshold for a specific key to restrict the number of pages returned. For instance, let's consider the example of fetching pages 30-40 from a 100-page API request. Here, you would specify the "Key in Response" as "page_number" and assign it a value of 30 to limit the number of pages returned.

  • The Sleep After Each Page option is available universally, allowing you to set a time delay (in seconds) between each API pagination.

Paginate by Page

Paginate by Page allows you to iterate through all of the pages without an offset.

  • The Page Key option lets you indicate the key in the JSON response that corresponds to the page number. This key might be named page_number, page_num,or simply page.
  • Page_Value Variable allows you to update the value of the next page during each iteration by using a preconfigured variable named page_value. To enable this feature, create a page_value variable in the Variables tab, select the Paginate By Page option under Results → Data, and check the “Update the value of the next page into a page_value variable in every page iteration” box.

  • Stop according to is the criteria at which the Action River will stop paginating and thus pulling data from the API. This dropdown list gives you 2 options:

    • The Page Size option enables the user to determine the number of data rows included on a single page of the response and to stop the pagination once a page has less than the full number of entries (i.e., the last page). This option is particularly valuable when the number of pages is variable or unknown.
    • Key in Response option enables you to set a threshold for a specific key to restrict the number of pages returned. For instance, let's consider the example of fetching pages 30-40 from a 100-page API request. Here, you would specify the Key in Response as page_number and assign it a value of 30 to limit the number of pages returned.

  • The Sleep After Each Page option is available universally, allowing you to set a time delay (in seconds) between each API pagination.

Next page in response

This option is designed for situations where the pagination does not follow a standard cadence established by a variable in the GET call. Instead, it relies on information obtained from a previous GET response. For instance, consider an example API that takes an "id" as an input parameter and returns customer information such as name, age, height, weight, income, and a value named "id_behind" that corresponds to the id of the person waiting behind you in the line. To obtain the information for the next customer in line, you must make another API call by setting the id parameter to the "id_behind" value of the previous customer. Data Integration provides 3 options to accomplish this task.

  1. The Static Url With Param From Response feature lets you set a new URL for the next page of the response and map a variable from the previous response to a variable that you will use in the GET call for the next page of the response.
    • The Next Page Key refers to the key that holds the parameter you will use to retrieve the next page of the response. In the example mentioned earlier, the "Next Page Key" would be "id_behind".
    • Send next page in key: - This allows us to map the “Next Page Key” to the Key in the GET request for the next page. In the example above, this would be set to “id”.
    • The option Send next page using a new url enables you to specify a different URL for the next page's GET request. This can be useful if you want to retrieve information from a different database using a different REST URL and pass in the "id_behind" parameter from the previous GET request.

  • Stop according to is the criteria at which the Action River will stop paginating and thus pulling data from the API. This dropdown list gives you 2 options:

    • The Page Size option enables the user to determine the number of data rows included on a single page of the response and to stop the pagination once a page has less than the full number of entries (i.e., the last page). This option is particularly valuable when the number of pages is variable or unknown.
    • Key in Response option enables you to set a threshold for a specific key to restrict the number of pages returned. For instance, let's consider the example of fetching pages 30-40 from a 100-page API request. Here, you would specify the "Key in Response" as "page_number" and assign it a value of 30 to limit the number of pages returned.

  • The Sleep After Each Page option is available universally, allowing you to set a time delay (in seconds) between each API pagination.

  1. The Url In Response option is used when the URL for the next page is stored within a key of the previous response.

    • Next page key- Here, all we have to do is specify the key in the previous response that holds the full Url for the next page.

  2. Next Page in Body - This option is used for when the information needed is stored in a two layer next page call (the ‘next to next page’) where the information for the next page (call this, page ‘b’) is stored in the previous response (call this, page ‘a’), but the page with the information we want to derive (call this, page ‘c’) can only be accessed via a key in page ‘b’.

    • Page a → contains information to access → Page b → contains information to access-> Page c
    • “Next page key” - This allows us to specify the key in the previous response (page ‘’a’) where the page we want to navigate to is located in (page ‘b’).
    • “Send next page in key” - This allows us to specify the key in the page ‘b’ api call in order to access the page that we want (page ‘c’).

Dynamic changing of parameters

The parameters in many of these settings can be modified dynamically by using Variables, which can be created by clicking the button on the top right and identified using the {variable_name}s notation. By inserting these variables into the parameter boxes, you can alter the pagination of the API call based on the requirements of your Logic River workflow.

On this Page