Skip to main content
Feedback

BigQuery as a source walkthrough

You can establish a new River within Data Integration, proceeding to the selection of extraction modes and the administration of your data storage. You can effectively harness Data Integration capabilities to extract data from BigQuery and seamlessly merge it into your data ecosystem.

Prerequisite

Ensure to create a connection for BigQuery source within Data Integration.

River modes

When using BigQuery as a source, you can select between two River modes:

  • Multi-Tables (Standard extraction)
  • Custom Query

Multi-tables (Standard extraction)

This mode in Data Integration combines data from various tables into a single schema before transferring it to the destination. It establishes table relationships to ensure consistent linking and loading. Data Integration Multi-Tables River mode mainly employs SQL queries for transformations, with scheduling or manual triggering options.

To obtain further details about using Multi-tables (Standard extraction), refer to Databases River Modes topic.

Custom query

Data Integration Custom Query River mode empowers you to input data into the platform via personalized SQL queries, offering control over data loading and transformations. You can specify data and transformations precisely, using SQL, pulling from databases or data warehouses. Data is then scheduled for automatic or on-demand loading into Data Integration, ensuring real-time data access.

note

  • When using a custom query, specific data types are considered as strings. The data types labeled as "TO_JSON_STRING" are enclosed in double quotes when converted into strings.

  • When opting for the Incremental Extract mode, you have only 2 options: Datetime and Running number.

To obtain further details about using Custom Query, refer to Databases River Modes topic.

Limitations

  • Data export bucket region - The region designated in the 'Region' input for the connection must align with the region of the Data export bucket (Custom Filezone).

  • Project-based connection - Data Integration enables connections on a per-region basis. When dealing with multiple BigQuery projects, establish a distinct connection for each project.

  • Mapping for primary and foreign keys - Automatic detection of mapping match keys is restricted to native BigQuery Primary and Foreign key fields only. Custom keys may not be recognized during the mapping process.

  • Partition and cluster fields - The mapping process within Data Integration does not allocate Partition and Cluster Fields.

  • Absence of deleted records indication - Data that is removed is not flagged, resulting in its absence not being indicated within the Target.

On this Page