Creating a Data Flow

In Data Integration, a Data Pipeline refers to a Data Flow. Data Flows are the primary building blocks, with three Data Flow types:

• Source to Target Flow
• Logic Flow
• REST Action

Source to Target Flow

Source to Target Flow build simple data pipelines that take data from a Source and push it to a Target destination. Data Integration simplifies the process by recognizing the structure of incoming data and creating target tables and columns as needed.

Data Integration supports over 180 Sources and Targets, including Facebook Ads, Google Analytics, Oracle, and popular data warehouses.

Logic Flow

Logic Flow use a logical data model to define the structure of data elements and the relationships between the Data Flows. These Data Flows handle data transformations and workflow orchestration, and can contain other Data Flows.

For data transformations, use SQL for in-warehouse transformations or Python for complex scenarios. Data Integration Orchestration supports branching, multi-step processes, conditional logic, loops, and so on, making it simple to create complex workflows.

REST Action

REST Action connect to any REST API as a data source, custom data ingestion, or data target. For example, an Action Data Flow can pull data from a custom API endpoint or push data into a CRM such as HubSpot.

Creating your first Data Flow

When you log in for the first time after creating a new Data Integration account, the onboarding process helps you familiarize yourself with the console. Use Create Your First Data Flow to create a Source-to-Target Data Flow.