Skip to main content
Feedback

Working with settings tab

The Settings tab lets you set and manage configurations in Data Flows. It provides a centralized location for all settings-related tasks, making it easier to manage Data Flows.

Schedule me

Once you configure your Data Flow's Column Mapping step, you can run it on demand by clicking Run or schedule it.

Click Schedule Me under the Settings tab to access the scheduling options.

note

Data Flow scheduling options by plan:

  • Starter Plan: Schedule Data Flows with a minimum interval of 60 minutes.
  • Professional Plan: Schedule Data Flows with a minimum interval of 15 minutes.
  • Enterprise Plan: Schedule Data Flows with a minimum interval of 5 minutes.

Professional and higher plans provide advanced scheduling options using custom CRON expressions.

CRON expression

You can use CRON expression to set custom scheduling frequencies. Data Integration uses the quartz style format of cron expressions. The CRON pattern has seven space-separated fields:

`<second>` `<minute>` `<hour>` <day of the month> `<month>` <day of the week> `<year>`
  • Enter your CRON Expression under the Custom tab.
  • Use last instead of L for last day of month syntax. For example, to schedule the Data Flow to run on the last day of every month at 22:30 PM, use:
0 30 22 last * ? *

Extended execution time for large tables and API reports

When you use a Source-to-Target Data Flow in Multi tables mode, Data Integration extends the execution time for handling large datasets, including RDBMS tables and API reports. This feature lets your Data Flow process and load extensive data, automatically extending the processing time up to 48 hours, if necessary, to ensure successful data loading.

Key details

  • RDBMS tables:
    Data Integration automatically adjusts the execution time based on the table size or the row count. For large tables, Data Integration automatically switches to long-duration mode to complete the process without manual intervention.

  • API reports: Data Integration pre-defines certain API reports that return large datasets. It automatically configures these reports to run with extended execution time as needed. You can adjust the default timeout value in your Data Flow's settings to limit the runtime based on your preference.

User control

You can adjust the default execution time in the Settings tab to set a custom timeout limit. If you set a custom timeout value, the process terminates after exceeding the selected time.

Automatic update to reports

Data Integration updates the list of API reports requiring extended execution as new extensive reports are identified. These reports are set to use the extended time.

note

For Source-to-Target Flow in Multi tables mode, Data Integration processes large tables and reports for up to 48 hours, unless you modify the default timeout setting.

Notifications

To receive notifications after a failure, warning, or when a run-time threshold is reached, turn on the notifications and enter your email address.

note

Each table functions as a background run so that you can receive an alert for every unsuccessful run (table).

  • To configure alerts for no data events, enable the On Warning checkbox.
  • To receive Data Flow failure notifications in Slack, create a dedicated Slack channel and link it to Data Integration's email alert system.

Sub Data Flows

Sub Data Flows let you embed smaller and specialized processes within the larger Data Flow. These sub-processes can inherit the main Data Flow settings or have unique parameters, enhancing flexibility and customization in data processing. Sub Data Flows are supported in specific Sources. For more information, refer to Sub Data Flows.

On this Page