Storage and files

A Storage and files source flow extracts data from file storage systems, such as cloud or on-premise, and loads it into a destination system. Use this flow type when your source data lives in files such as CSV or JSON stored in a cloud storage bucket.

Example: A retail company stores daily sales files (CSV/JSON) in Amazon S3 and wants to ingest them automatically into Snowflake for reporting and dashboards.

Flow: Amazon S3 > Source to Target > Snowflake

Before you begin

You have a configured connection to your cloud storage provider (Amazon S3, Azure Blob Storage, Google Cloud Storage, SFTP, or similar) in the Data Integration Console.

For the full list of supported connectors, refer to Storage & Files in the Sources section.

Your files are in a supported format: CSV, JSON, Excel, Parquet, or similar structured format.
You know the bucket name, file pattern, and any authentication credentials required to access the files.

Step 1: Create a Data Flow

Navigate to the Data Integration Console.
Click Create Data Flow and select Source to Target Data Flow, or open an existing data flow.

Step 2: Set up the source

In the Source tab, select the storage connector you want to ingest data from. For example, select Amazon S3.
Select or create a connection. Existing connections appear in the Connections drop-down menu. To add a new one, click New Connection.
Choose a container name from the drop-down list.
Select an extraction method.

Extraction methods

Option	Description	When to use
All	Loads all available files every time the flow runs.	Full refreshes or initial data loads.
Incremental load by file modified timestamp	Loads only files added or modified since the last run, based on the file modification date.	When files are updated over time and you want to avoid reloading everything.
Incremental run: by template	Loads files that match a dynamic naming pattern with date or time placeholders.	When files follow a structured naming format such as `sales_YYYY_MM_DD.csv` and you want to load data for specific time periods.

Configure the extraction method

Depending on the extraction method you select, fill in the corresponding settings:

If you select All:

Enter a File Pattern (for example, *.csv) to filter specific file types, or enter a Prefix (for example, sales_) to load only files whose names start with that text.
Leave both fields empty to load all files in the container.

If you select Incremental load by file modified timestamp:

Choose a Start Date and Time.
Optionally choose an End Date and Time.

Data Integration loads only files modified within the specified time range, ensuring that already-processed files are not reloaded.

If you select Incremental run: by template:

Select the Insert Template Folder that contains the template-based files.
Enter the Start Value (for example, 2026-01-01).
Enter the End Value (for example, 2026-01-31).

Data Integration loads files whose names match the template within the specified range. For example, if your template is sales_YYYY_MM_DD.csv, the flow loads all matching files between the start and end dates.

Step 3: Configure file settings

File action and limit

Select what happens to source files after the data is pulled:

Remain in original place — The file stays in its original location after processing.
Move to archive path or delete — Some connectors support moving processed files to an archive folder or deleting them.

To limit how many files are processed per run, enter a number in the Number of files to pull field. Leave the field empty to pull all available files.

File type and parsing options

Select the file type: CSV, Excel, JSON Lines, or another supported format. Then configure the parsing options for your file:

Setting	Description	Example
Delimiter	Character used to separate columns in the file.	`,` (comma), `;` (semicolon), `\t` (tab)
Quote char	Character used to wrap text values that contain the delimiter.	`"` (double quote)
Header rows to skip	Number of rows to skip before reading column headers or data.	Enter `1` if the first row contains notes or metadata.
Newline delimiter	Defines how rows are separated in the file.	Standard line break (`\n`)

Additional file options

Compressed file — Turn on this option if your file is compressed (.zip or .gz). Data Integration decompresses the file before loading.
Handling special characters — Turn on this option if your file contains special characters such as é, ñ, or currency symbols.

File manipulations (optional)

Use these settings to trim unwanted rows from the file before loading:

Start loading rows from row number — Enter the row number where data begins. Rows before this number are skipped. For example, if data starts on row 3, enter 3.
Number of rows to remove from bottom — Enter how many rows to remove from the end of the file. Use this when the file contains totals or notes in the last few rows.

Step 4: Select a target

In the Target tab, select the cloud destination where Data Integration loads the extracted data.

Data Integration supports all major cloud data warehouses and cloud file storage systems as targets.

Select or create a connection for your target. Then define the Database, Schema, and Target Table where the data will be stored. Data Integration automatically detects available databases and schemas.

Loading modes

Choose how Data Integration writes data into the target table:

Mode	Behavior	When to use
Overwrite	Replaces all existing data in the target table with data from the current run. Creates the table automatically if it does not exist.	Full refreshes, or small tables where only the latest state matters.
Append only	Adds new records to the target table without modifying existing rows.	Event logs, audit trails, or any dataset where every record must be preserved.
Upsert-Merge	Inserts new records and updates existing ones based on a defined merge key.	Incremental syncs where the target must stay in sync with the source.

If you select Upsert-Merge, define your merge key in the Schema tab.

Upsert-Merge also supports de-duplication within incoming source records. Add an expression in target database SQL syntax to the Filter Order Expression field. This acts as an ORDER BY clause, only the first matching record per key is loaded. Existing records in the target are not de-duplicated.

note

For file storage targets, specify the bucket and file path where Data Integration stores the data. The connection's default bucket is used if you do not specify one.

Step 5: Configure the schema

The Schema tab lets you review and adjust the target table structure before loading data. When you open the tab, Data Integration automatically detects the schema from your source configuration.

Click Auto-Mapping to populate target field names and data types automatically. You can then edit any field before running the flow.

Column mapping

Field	Description
Source field name	The original column name from the source file.
Target field name	Rename the column as it appears in the target table.
Data type	Change or cast the data type during loading.
Mode	Controls how the column handles values. Refer to Column modes below.
Primary key	Mark one or more fields as the merge key for Upsert-Merge flows.
Cluster key	Define a partition or cluster field, if supported by the target warehouse.
Expression	Write SQL to create a new field or transform an existing one using target warehouse syntax.

Column modes

Nullable: The column can contain null values. Not every row needs a value for this field. Use for optional attributes.
Required: The column must always contain a value. Null values are not allowed. Use for IDs, timestamps, and mandatory fields.
Repeated: The column stores multiple values per row as an array or list. Use for nested or multi-value data such as tags or categories. Supported in warehouses such as BigQuery.

Defining primary keys (upsert-merge only)

If you selected Upsert-Merge as the loading mode, click the key icon next to a field to mark it as the primary key. To create a composite key, click multiple fields — the combination of those field values acts as the unique identifier for each row.

note

Data Integration does not include column descriptions in the target metadata. Document field descriptions separately if your data governance process requires them.
Column names cannot contain dots (.). Dots cause errors during data processing. Rename any affected fields before running the flow.

Step 6: Schedule and run the Flow

Click Run at the bottom of the page to run immediately.

Schedule the Flow

Click Schedule Me in the Settings tab to set up automatic runs.

note

Scheduling intervals by plan:

Starter: minimum 60-minute interval
Professional: minimum 15-minute interval; includes custom CRON expressions
Enterprise: minimum 5-minute interval; includes custom CRON expressions

Using a CRON expression

Professional and Enterprise plans support custom CRON scheduling using Quartz format. The pattern has seven space-separated fields:

<second> <minute> <hour> <day of month> <month> <day of week> <year>

Enter your expression under the Custom tab. Use last instead of L for last-day-of-month syntax. For example, to run on the last day of every month at 22:30:

0 30 22 last * ? *

Extended execution time

For multi-table flows, Data Integration automatically extends execution time up to 48 hours for large RDBMS tables and predefined large API reports. You can set a custom timeout limit in the Settings tab to override this default.

Set up notifications

In the Settings tab, enter your email address to receive alerts for run failures, warnings, or runtime threshold breaches.

note

Each table in a multi-table flow runs as a background process. You receive a separate alert for each failed table.
Enable On Warning to also receive alerts when a run returns no data.
To receive alerts in Slack, create a dedicated Slack channel and connect it to Data Integration's email alert system.

Monitor the Flow

After the Flow runs, open the Activities tab in the right-hand menu to review run status and logs. Data Integration provides a detailed log for each run showing which tables were processed, how many records were extracted and loaded, and any errors encountered.

note

A Data Flow supports up to 500 tables.