Targets overview
Overview
Data Integration targets include data warehouses and data storage. You can configure as many targets as you would like.
See below for specific guidance on connecting to each type of target:
Flows and concepts when loading into DWH targets
Some key concepts
ELT
Data Integration loading and transforming methods are running based on the concepts of ELT (Extract, Load, Transform).
Thus, in order to reduce the bottleneck in transforming the data during the load, and ensure using the target DBs high performances in order to transform the data in the DB itself.
Running over data images
In order to ensure the data represents the last complete truth, Data Integration loads and transforms the data over the pipeline in think of the target tables.
That means, Data Integration never drops/changes/alter/modify the target table, or its data, in case of failures.
Data Integration platform ensure the data in the table, and the metadata of the table, will be complete as it defined in the River (pipeline), so there will be no data loss, incomplete data, or changes in the table structure in cases of the river did not complete its mission.
File zone
In order to prevent data loss, and reduce the dependence in the source when loading data into the target is failed for some reason, Data Integration stores the data in a cloud storage service such as AWS S3, Google Cloud Storage, or Azure Blob Storage - termed as File Zone.
Therefore any failures in loading data into the target databases won’t cause a data loss and in case of failures.
Data Integration platform knows to make the retry in the loading step only, without pulling again the same data from the source.
DWH river types
Source to target rivers
Source to Target rivers are pipelines that pull data from sources, and load it, by the key concepts described above, into the target databases, via the File Zone. Source to Target river is responsible for the _Extract _and _Load _phases in the ELT process. Each pipeline configuration can load the data using Overwrite , Append Only or Upsert Merge mode.
Logic river
A smart engine that is responsible for the Transform phase in the ELT process. This river type gives the ability to the user running SQL queries steps over the DB he chooses, and therefore manage and select queries results into tables in the DWH or into file(s) in the File Zone.
The steps in the logic river can run in parallel, by a loop over a list, by condition or step by step, and also available using smart variables.
Loading flowcharts
Source to target river: loading data into target table flowchart
Logic river: selecting into target table flowchart
