REST API
A REST API flow extracts data directly from an external API endpoint and loads it into a target such as a data warehouse or storage system. Use this flow type when your data source exposes a REST API. For example, a payment processor, a SaaS platform without a native connector, or an internal service.
Example: A company processes payments through Stripe and wants to centralize transaction data in Snowflake for reporting and fraud analysis.
Flow: Stripe REST API > Source to Target > Snowflake
If your source is already available as a supported Data Integration connector (for example, Salesforce or HubSpot), use an Application source flow instead. REST API source is best suited for custom or unsupported endpoints.
Before you begin
Before creating a REST API flow, ensure you have the following:
- An active Data Integration account.
- A REST API connection configured in Data Integration, with valid credentials such as an API key, OAuth token, or Basic Auth. Refer to Creating a connection for details.
- A REST API Action that defines how to call the endpoint. Refer to REST API in the Sources section for guidance on creating actions.
- Access to a supported cloud data warehouse or storage target.
Step 1: Create a Data Flow
- Navigate to the Data Integration Console.
- Click Data Flows in the left-hand menu.
- Click Create Data Flow and select Source to Target Data Flow, or open an existing data flow.
Step 2: Set up the source
- In the Source tab, select REST API as the source type.
- Select an existing REST API Action from the list, or create a new one.
- Select a connection from the Connections drop-down menu, or click New Connection to add one.
Source configuration options
The options available in the Source tab depend on whether the selected REST API Action pulls data from an endpoint or sends data to one:
| Action type | Available options |
|---|---|
| Pull (GET): retrieves data from the API | Start date, End date, Last days back (for retroactive changes), Interval chunk size (optional, to split large requests into smaller time-windowed calls) |
| Push (POST/PUT): sends data to the API | Request parameters defined in the REST Action configuration |
Date range and incremental options (pull actions)
For API actions that retrieve data, you can control the time range and incremental behavior:
- Start date / End date: Define the date range for the data pull. Leave the end date empty to retrieve data up to the current time.
- Last days back: Extends the start date by the specified number of days before the selected period. Use this to capture retroactive updates or late-arriving data. For example, setting Last days back to
2with a daily run retrieves the last two days in addition to the current day's data. - Interval chunk size: Splits the overall date range into smaller chunks per API call. Use this when the API enforces rate limits or returns too many records in a single response. For example, setting the interval to
7 dayson a 30-day pull makes four separate API calls, each covering one week.
Step 3: Select a target
In the Target tab, select the cloud destination where Data Integration loads the extracted data.
Data Integration supports all major cloud data warehouses and cloud file storage systems as targets.
Select or create a connection for your target. Then define the Database, Schema, and Target Table where the data will be stored. Data Integration automatically detects available databases and schemas.
Loading modes
Choose how Data Integration writes data into the target table:
| Mode | Behavior | When to use |
|---|---|---|
| Overwrite | Replaces all existing data in the target table with data from the current run. Creates the table automatically if it does not exist. | Full refreshes, or small tables where only the latest state matters. |
| Append only | Adds new records to the target table without modifying existing rows. | Event logs, audit trails, or any dataset where every record must be preserved. |
| Upsert-Merge | Inserts new records and updates existing ones based on a defined merge key. | Incremental syncs where the target must stay in sync with the source. |
If you select Upsert-Merge, define your merge key in the Schema tab.
Upsert-Merge also supports de-duplication within incoming source records. Add an expression in target database SQL syntax to the Filter Order Expression field. This acts as an ORDER BY clause, only the first matching record per key is loaded. Existing records in the target are not de-duplicated.
For file storage targets, specify the bucket and file path where Data Integration stores the data. The connection's default bucket is used if you do not specify one.
Step 4: Configure the schema
The Schema tab lets you review and adjust the target table structure before loading data. When you open the tab, Data Integration automatically detects the schema from the API response.
Click Auto-Mapping to populate target field names and data types automatically. You can then edit any field before running the flow.
Column mapping
| Field | Description |
|---|---|
| Target field name | Rename the column as it appears in the target table. |
| Data type | Change or cast the data type during loading. |
| Mode | Controls how the column handles values. Options: Nullable, Required, or Repeated. |
| Primary key | Mark one or more fields as the merge key for Upsert-Merge flows. |
| Cluster key | Define a partition or cluster field, if supported by the target warehouse. |
| Expression | Write SQL to create a new field or transform an existing one using target warehouse syntax. |
Column modes
| Mode | Description |
|---|---|
| Nullable | The column can contain null values. Not every row needs a value for this field. |
| Required | The column must always contain a value. Null values are not allowed. Use for IDs, timestamps, and mandatory fields. |
| Repeated | The column stores multiple values per row as an array or list. Use for nested or multi-value API response fields such as tags or line items. |
Defining primary keys (upsert-merge only)
If you selected Upsert-Merge as the loading mode, click the key icon next to a field to mark it as the primary key. To create a composite key, click multiple fields. The combination of those field values acts as the unique identifier for each row.
- Data Integration does not include column descriptions in the target metadata. Document field descriptions separately if your data governance process requires them.
- Column names cannot contain dots (.). Dots cause errors during data processing. Rename any affected fields before running the flow.
Step 5: Schedule and run the Flow
Click Run at the bottom of the page to run immediately.
Schedule the Flow
Click Schedule Me in the Settings tab to set up automatic runs.
Scheduling intervals by plan:
- Starter: minimum 60-minute interval
- Professional: minimum 15-minute interval; includes custom CRON expressions
- Enterprise: minimum 5-minute interval; includes custom CRON expressions
Using a CRON expression
Professional and Enterprise plans support custom CRON scheduling using Quartz format. The pattern has seven space-separated fields:
<second> <minute> <hour> <day of month> <month> <day of week> <year>
Enter your expression under the Custom tab. Use last instead of L for last-day-of-month syntax. For example, to run on the last day of every month at 22:30:
0 30 22 last * ? *
Extended execution time
For multi-table flows, Data Integration automatically extends execution time up to 48 hours for large RDBMS tables and predefined large API reports. You can set a custom timeout limit in the Settings tab to override this default.
Set up notifications
In the Settings tab, enter your email address to receive alerts for run failures, warnings, or runtime threshold breaches.
- Each table in a multi-table flow runs as a background process. You receive a separate alert for each failed table.
- Enable On Warning to also receive alerts when a run returns no data.
- To receive alerts in Slack, create a dedicated Slack channel and connect it to Data Integration's email alert system.
Monitor the Flow
After the flow runs, open the Activities tab in the right-hand menu to review run status and logs. Data Integration provides a detailed log for each run showing which operations completed, how many records were processed, and any errors encountered.
If the API returns an error code (such as 429 rate limit or 5xx server error), the run appears as failed in the Activities log with the API response details. Review the error message and adjust the interval chunk size, authentication, or endpoint configuration as needed.
