Selecting the source type
Data Integration supports five source types in a Source to Target Data Flow. Each is designed for a different kind of data origin. Use this page to identify which one applies to your source, then follow the link to the setup.
Before you begin
- You have access to the Data Integration Console.
- You have credentials for the source system you want to connect to.
Source type comparison
| Source type | Connect to | Best for |
|---|---|---|
| Application | SaaS business applications, such as Salesforce, HubSpot, Google Ads, Facebook Ads, Shopify, and 180+ others | Marketing, CRM, finance, HR, and e-commerce data on a scheduled or incremental basis |
| Database | Relational databases, such as MySQL, PostgreSQL, Oracle, SQL Server, BigQuery, Snowflake, Redshift, MongoDB, and more | Full table replication, incremental extraction, Change Data Capture (CDC), or custom SQL queries from your own databases |
| Storage and files | Cloud and on-premise file storage, such as Amazon S3, Azure Blob Storage, Google Cloud Storage, SFTP, and more | CSV, Excel, JSON, or other structured files that are deposited on a schedule or generated by batch processes |
| REST API | Any external REST API endpoint with supported authentication (API key, OAuth, Basic Auth) | Systems without a native Data Integration connector, or when you need full control over the API call, pagination, and parameters |
| Events | Webhook-based event push. Any system that can send HTTP POST requests with a JSON body | Real-time, event-driven ingestion where the source pushes data the moment an event occurs, rather than polled on a schedule |
Reference for selecting source
-
My data is in Salesforce, HubSpot, Google Analytics, or another business SaaS tool. Use Application. These connectors are pre-built and maintained. Select your connector, choose predefined or custom reports, and configure the time period.
-
My data is in a MySQL, PostgreSQL, Oracle, or other database I manage. Use Database. You can replicate full tables, load incrementally, or use Change Data Capture to stream only changes with minimal impact on the source.
-
My data arrives as CSV or JSON files dropped into Amazon S3, Azure Blob, or a similar storage bucket. Use Storage and files. Configure the file pattern, parsing options, and whether to load all files or only new ones.
-
My source is a custom internal API, a third-party API without a native connector, or a Stripe/Twilio-style API I need to call directly. Use REST API. Define the endpoint, authentication, and date range parameters. Use interval chunking if the API enforces rate limits.
-
My source system fires events in real time, such as order placed, user signed up, status changed, and I want those captured immediately. Use Events. Data Integration generates a webhook URL. Register it in your source system, and each event payload is loaded automatically without polling.
Regardless of which source type you choose, every Source to Target Data Flow follows the same structure:
- Set up the source: select the connector and configure source-specific settings
- Select a target: choose the destination warehouse or storage and set the loading mode
- Configure the schema: review column mapping, set data types and modes
- Schedule and run: run immediately or set a recurring schedule
The target configuration, loading modes (Overwrite, Append only, Upsert-Merge), and schema settings are consistent across all source types. Only the source setup steps differ.
Browse all source connectors
For the complete catalog of individual connectors across all five source types, refer to Sources overview.
