Action river overview

The Data Integration Action River feature lets you connect to any REST endpoint and incorporate it as a new data source. While Data Integration provides built-in support for data sources; you might need to connect to one not on the supported list. You can integrate it efficiently using available tools and configurations—without writing extra code.

Use Action Rivers when a native connector is unavailable. You can create an "Action River" to establish the connection and retrieve data effectively.

Action rivers types

• REST Action: Used to perform single actions such as fetching data from or sending data to a REST API endpoint.
• Multi Actions: Let you orchestrate complex processes by combining multiple actions sequentially, including REST Actions and others.

Creating a custom connector

After establishing a connection to a REST endpoint, you can save it as a Custom Connector using the REST API Source. Create a new "Source to Target River", select the REST API Source you created, and configure your "Target" for any other "Source to Target River".

REST action

The REST Action in Data Integration simplifies the process of interacting with web services. It lets you make API calls to retrieve, update, or delete data, making it easy to integrate external data sources into your workflows. It performs single actions such as fetching data from or sending it to a REST API endpoint.

Authentication

To create a connection to an API, you must provide the API URL. If you are creating a Source that makes a GET call to the API, you can use a username and password, OAuth2, or Multi-Bearer authentication. This API endpoint is public, so authentication is not required.

Testing the connection

For example, if you can access an API endpoint that returns a list of users, you can select the Test REST Action to verify the connection. This action provides a 200 (connection succeeded) success code and a sample dataset.

Automating table structure mapping

Choose a "Source to Target River" and select the "REST API" as the "Source" and 'Snowflake' as the "Target". Data Integration handles the task of using the JSON payload from the API endpoint to define the target schema. You can use the Auto Mapping and Data Integration automatically maps the table structure and data types based on the metadata retrieved from a sample test.

Ingesting data

After generating the table structure, you can ingest data by running the "River". Data will be retrieved from the API based on the specified parameters (such as last modified date) and pushed into the target (Snowflake) using the specified primary key for merging.

Multi actions

Multi Actions in Data Integration lets you orchestrate complex processes by combining multiple actions sequentially. You can create, configure, and execute Multi actions within Data Integration.

Many REST APIs require multiple steps to connect to the service, as well as GET/POST requests. For example, you can submit a request to generate a token and then use that token to send a GET or POST request to the service. The first action is a REST Action to create the token, and the second action is the request to the service to make a GET or POST call.

Action variables

Action Variables in Data Integration serves as an interface between actions and other rivers, letting dynamic values within Rest Actions and Multi Actions.

Variable looping in action rivers

The Variable Looping in Action Rivers feature transforms how variables are managed in the REST Action River, providing a smooth and effective iteration method. Instead of iterating through "Source to Target Rivers" for each endpoint separately, you can loop within the Action by passing a list of parameter values. You can use this for scenarios with varying parameter sets.

For example, when an API endpoint accepts diverse values for a specific parameter, Variable Looping enables users to handle all potential cases efficiently. This enhancement enhances load time efficiency and minimizes resource consumption.