Application
An Application source flow extracts data from a SaaS application and loads it into a cloud data warehouse or storage target. Use this flow type when your source is a business application, such as a CRM, marketing platform, or e-commerce system.
Supported application categories include Analytics, Marketing, CRM and support, Finance and Accounting, Business Operations, Human Resources, and E-commerce platforms.
For the full list of supported connectors, refer to Application connectors in the Sources section.
Example: A marketing team wants to centralize Facebook Ads campaign performance data in BigQuery for cross-channel reporting.
Flow: Facebook Ads > Source to Target > BigQuery
If your source is not listed as a supported connector, use a REST API flow to build a custom connection.
Before you begin
Before creating an Application source flow, ensure you have the following:
- An active Data Integration account.
- Valid credentials for the SaaS application you want to connect to (for example, API keys or OAuth tokens).
- Access to a supported cloud data warehouse target such as Snowflake, BigQuery, Amazon Redshift, or Azure Synapse Analytics.
Step 1: Create a Data Flow
- Navigate to the Data Integration Console.
- Click Data Flows in the left-hand menu.
- Click Create Data Flow and select Source to Target Data Flow, or open an existing data flow.
Step 2: Set up the source
In the Source tab, select the application you want to ingest data from and configure the ingestion parameters.
Select or create a connection for your source. Existing connections appear in the Connections drop-down menu. To add a new one, click New Connection.
Source data options
For Analytics connectors such as Google Analytics, the Source tab shows a Report drop-down instead of Predefined or Custom report options. Select the report type from the list, for example, Events Report. Additional parameters such as a Time or List input toggle, entity filters, and field selectors may appear below depending on the connector.
- Predefined Reports
- Custom Report
Select one or more reports with a predefined set of fields. The field structure is already defined, so you can start loading data immediately. You can select multiple predefined reports simultaneously. When working with many reports, use Bulk Actions in the Tables View to configure extract method, loading mode, and calculated columns across all selected tables at once.
Refer to the documentation for your connector for a full list of available reports, fields, and schema definitions. For example, Facebook Ads predefined reports.
After selecting a predefined report, the following configuration options appear:
Auto Complete or Advanced
Use this toggle to control how you specify which accounts to extract data from. In Auto Complete mode, Data Integration loads available accounts from your connection as a searchable drop-down. Click to select or leave empty for all accounts.
In Advanced mode, enter account names or IDs manually as a comma-separated list (for example, name1,name2,name3) and press enter after each value.
When you select Advanced, an Ad Account Type option appears. Choose whether the values you entered are account IDs (Ad Accounts id's list), exact account names (Ad Accounts name list), or partial name matches (All Ad Accounts names that contain the name).
Include Business Accounts
Select this checkbox to fetch Business Accounts associated with the selected Ad Accounts in addition to the Ad Accounts themselves. Leave it unchecked to pull Ad Account data only.
Build a single report by selecting the report type and the specific fields you need. This gives you full control over what data is extracted. Custom Report supports one target table per flow.
For example, when connecting to Facebook Ads, select the ad report and parameters the API supports, such as campaign, ad set, or ad-level metrics.
After selecting a report type, the following configuration options appear:
Data Level
Select the granularity at which data is reported. For example, in Facebook Ads you can select ad_account, campaign, adset, or ad. The available options depend on what the connector's API supports.
Auto Complete or Advanced
Use this toggle to control how you specify which accounts to extract data from. In Auto Complete mode, Data Integration loads available accounts from your connection as a searchable drop-down. Click to select or leave empty for all accounts.
In Advanced mode, enter account names or IDs manually as a comma-separated list (for example, name1,name2,name3) and press Enter after each value.
When you select Advanced, an Ad Account Type option appears. Choose whether the values you entered are account IDs (Ad Accounts id's list), exact account names (Ad Accounts name list), or partial name matches (All Ad Accounts names that contain the name).
Include Business Accounts
Select this checkbox to fetch Business Accounts associated with the selected Ad Accounts in addition to the Ad Accounts themselves.
Breakdown
Select one or more dimensions to break down the report data. For example, by device, platform, or region. Use Select All to include all available breakdowns, or Clear All to remove selections.
For multiple breakdowns, only certain combinations are valid. Refer to your connector's documentation for supported breakdown combinations.
Select a time period
For most application connectors, you can define the date range for data extraction. Available options depend on the source connector and its API.
| Option | Description |
|---|---|
| Custom date range | Set a specific start and end date. Use for historical backfills or initial data migrations. |
| Day — Yesterday | Retrieves data from yesterday up to the current moment. Use for daily incremental syncs. |
| Week (Monday–Sunday) | Retrieves data for a full calendar week starting Monday. |
| Week to date | Retrieves data from the start of the current week through today. |
| Previous week | Retrieves data for the full previous calendar week. |
| Previous week to date | Retrieves data from the start of the previous week through today. |
| Month | Retrieves data for the entire current month. |
| Month to date | Retrieves data from the start of the current month through today. |
| Previous month | Retrieves data for the full previous month. |
| Previous month to date | Retrieves data from the start of the previous month through today. |
| Year to date | Retrieves data from the start of the current year through today. |
- Available time period options vary by source connector and API limitations.
- When you run the flow for the first time with a Custom date range, Data Integration retrieves data from the start date to the end date. If you leave the end date empty, it retrieves data up to the current time.
- On subsequent runs, Data Integration automatically advances the start date to the timestamp of the last successful run, keeping the end date empty. This ensures the flow picks up exactly where it stopped.
Last days back
The Last days back option extends the data pull by including additional retroactive days before the selected time period. Use this when source data is delayed or back-filled after the initial sync.
For example, if you set the time period to Yesterday and Last days back to 1, Data Integration retrieves data for the current date, yesterday, and the day before yesterday.
Step 3: Select a target
In the Target tab, select the cloud destination where Data Integration loads the extracted data.
Data Integration supports all major cloud data warehouses and cloud file storage systems as targets.
Select or create a connection for your target.
Define the Database, Schema, and Target Table where the data will be stored. Data Integration automatically detects available databases and schemas.
In Snowflake, the connection form includes a Default Database/Schema field. Setting this lets the connection remember and apply those values as the default across flows.
Loading modes
Choose how Data Integration writes data into the target table:
- Overwrite: Replaces all existing data with data from the current run. Creates the table automatically if it does not exist.
- Append only: Adds new records without modifying existing rows.
- Upsert-Merge: Inserts new records and updates existing ones based on a merge key you define in the Schema tab. Optionally, add a Filter Order Expression to de-duplicate incoming source records before loading.
For file storage targets, specify the bucket and file path where Data Integration stores the data. The connection's default bucket is used if you do not specify one.
Step 4: Configure the schema
The Schema tab lets you review and adjust the target table structure before loading data. When you open the tab, Data Integration automatically detects the schema from your source configuration.
Click Auto-Mapping to populate target field names and data types automatically. You can then edit any field before running the flow.
Column mapping
| Field | Description |
|---|---|
| Target field name | Rename the column as it appears in the target table. |
| Data type | Change or cast the data type during loading. |
| Mode | Controls how the column handles values. Refer to Column modes below. |
| Primary key | Mark one or more fields as the merge key for Upsert-Merge flows. |
| Cluster key | Define a partition or cluster field, if supported by the target warehouse. |
| Expression | Write SQL to create a new field or transform an existing one using target warehouse syntax. |
Column modes
- Nullable: The column can contain null values. Not every row needs a value for this field. Use for optional attributes.
- Required: The column must always contain a value. Null values are not allowed. Use for IDs, timestamps, and mandatory fields.
- Repeated: The column stores multiple values per row as an array or list. Use for nested or multi-value data such as tags or categories. Supported in warehouses such as BigQuery.
Defining primary keys (upsert-merge only)
If you selected Upsert-Merge as the loading mode, click the key icon next to a field to mark it as the primary key. To create a composite key, select multiple fields. The combination of these field values will act as the unique identifier for each row.
- Data Integration does not include column descriptions in the target metadata. Document field descriptions separately if your data governance process requires them.
- Column names cannot contain dots (.). Dots cause errors during data processing. Rename any affected fields before running the flow.
Step 5: Schedule and run the Flow
Click Run at the bottom of the page to run immediately.
Schedule the Flow
Click Schedule Me in the Settings tab to set up automatic runs.
Scheduling intervals by plan:
- Starter: minimum 60-minute interval
- Professional: minimum 15-minute interval; includes custom CRON expressions
- Enterprise: minimum 5-minute interval; includes custom CRON expressions
Using a CRON expression
Professional and Enterprise plans support custom CRON scheduling using Quartz format. The pattern has seven space-separated fields:
<second> <minute> <hour> <day of month> <month> <day of week> <year>
Enter your expression under the Custom tab. Use last instead of L for last-day-of-month syntax. For example, to run on the last day of every month at 22:30:
0 30 22 last * ? *
Extended execution time
For multi-table flows, Data Integration automatically extends execution time up to 48 hours for large RDBMS tables and predefined large API reports. You can set a custom timeout limit in the Settings tab to override this default.
Set up notifications
In the Settings tab, enter your email address to receive alerts for run failures, warnings, or runtime threshold breaches.
- Each table in a multi-table flow runs as a background process. You receive a separate alert for each failed table.
- Enable On Warning to also receive alerts when a run returns no data.
- To receive alerts in Slack, create a dedicated Slack channel and connect it to Data Integration's email alert system.
Step 6: Monitor the Flow
After the flow runs, open the Activities tab in the right-hand menu to review run status and logs. Data Integration provides a detailed log for each run showing which operations completed, how many records were processed, and any errors encountered.
Use the Activities view to troubleshoot failed runs, verify data was loaded correctly, or review historical run performance.
A Data Flow supports up to 500 tables.
