Targets overview
The Data Integration Targets include data warehouses and data storage. You can configure as many targets as you would like.
Refer to the sections below for guidance on connecting to each target type:
Flows and concepts when loading into DWH targets
Key concepts
ELT
Data Integration loading and transformation methods run based on the ELT (Extract, Load, Transform) approach. This minimizes transformation bottlenecks during loading and ensures the high performance of the target databases by performing transformations directly within them.
Running over data images
To ensure the data represents the accurate and complete truth, Data Integration loads and transforms the data over the pipeline with the Target tables. Data Integration never drops, changes, alters, or modifies the Target table, or its data, in case of failures.
Data Integration platform ensures that the data in the table, and the metadata of the table, remain complete as defined in the River (pipeline). This guarantees no data loss, incomplete data, or changes in the table structure, even if the River fails to complete its run.
File zone
To prevent data loss and reduce dependency on the source when loading data into the target, Data Integration stores the data in a cloud storage service—such as AWS S3, Google Cloud Storage, or Azure Blob Storage—referred to as the File Zone.
This ensures that data is preserved even if a load fails for any reason. Any failure during the loading step does not result in data loss. The Data Integration platform retries only the loading step without re-pulling the same data from the source.
Data Warehouse (DWH) river types
Source to target rivers
Source to Target rivers are pipelines that pull data from sources and load it into the target databases via the File Zone. The Source to Target river handles "Extract and Load" phases in the ELT process. Each pipeline configuration can load the data using Overwrite, Append Only, or Upsert Merge modes.
Logic river
A smart engine handles the Transform phase in the ELT process. This river type lets you run SQL queries steps over the Database, manage and select query results into tables in the DWH or into file(s) in the File Zone.
The Logic river steps can run in parallel, via a loop over a list, by condition, or step by step, and can also use smart variables.
Loading flowcharts
Source to target river: loading data into the target table flowchart
Logic river: selecting into the target table flowchart
