Skip to main content
Feedback

Connecting to REST API using action rivers

You can use the Data Integration "Action River" feature to establish a connection with any REST endpoint and incorporate it as a new data source. While Data Integration provides built-in support for data sources; you might sometimes need to connect to a source not on the supported list. You can integrate it efficiently using the available tools and configurations—without writing additional code.

For example, you can connect to an API endpoint using the GET method to fetch a user list.

Authentication

To create a connection to an API, you must provide the API URL. When you create a Source, you can make a GET call to the API.

You can use authentication methods such as username and password, OAuth2, or Multi-Bearer authentication.

This API endpoint is public, so authentication is not required.

Testing the connection

You can access an API endpoint that returns a list of users. To verify the connection, select Test REST Action. A successful connection returns a 200 response code and a sample dataset.

Request and results

When working with a REST API, you can configure standard options in the Request and Results menus:

  • Adding parameters
  • Pagination methods
  • Sending a request in a loop.

Adding parameters

You can apply Additional parameters (for example, last modified date) to filter the retrieved data. Setting the last modified date to "1 day" retrieves only added or modified users within the past 24 hours.

note

  • You cannot use the same parameter more than once.

  • When you click Add under REST URL Params, the specified parameter appends automatically to the URL.

**https://reqres.in/api/users?last_mod_date={time_range.start_date}**

  • Do not use the same name in both Key and Value fields. For example, using the "StartDate" in both fields will prevent the parameter from being added.

Pagination methods

To handle large data sets, Data Integration lets you retrieve and process data in smaller chunks using pagination.

Pagination methods include:

  • Pagination by Offset
  • Pagination by Page
  • Next Page in Response.

The method you choose depends on the structure of the data source and the task requirements. For more information, refer to Pagination options.

Sending a request in a loop

You can configure Data Integration to send repeated API requests in a loop until it reaches a specific stop value.. Use this when data updates continuously and you need to retrieve it regularly.

image.png

Creating a custom connector

After establishing a connection to a REST endpoint, you can save it as a custom connector using the REST API Source.

Procedure

  1. Create a new "Source to Target River".
  2. Select the REST API Source you created.
  3. Configure the "Target" source to complete the integration.

Automating table structure mapping

Select Source to Target River, choose the "REST API" as the "Source", and 'Snowflake' as the "Target". Data Integration handles the task of using the JSON payload from the API endpoint to define the target schema.

You can use Auto Mapping to automatically map the table structure and data types based on the metadata retrieved from a sample test.

Ingesting data

After generating the table structure, you can ingest data by running the "River." The River retrieves data from the API using the specified parameters (such as the last modified date). It pushes it into the target (Snowflake) using the specified primary key for merging.

On this Page