Skip to main content
Feedback

Datastore walkthrough

There are various methods available in Data Integration for extracting data from Datastore, a NoSQL document database provided by Google Cloud Platform. Data Integration facilitates the extraction process through two primary methods:

  • Query by Kind
  • Query by GQL (Google Query Language). It also provides mapping attributes to control how data is pulled from the Datastore.

Query by Kind

The "Query by Kind" method resembles the Datastore console, letting you browse and extract data from a specified kind in Datastore. It offers options for extracting all available data or using an incremental approach.

Extraction steps

  1. Kind(case sensitive): Insert the exact name of the kind from which data is to be extracted.
  1. Extract method You can choose between:
  • All: Pulls all available data from the selected kind.
  • Incremental: Pulls data based on specified increments.
  • If selecting Incremental:
    • Timestamp: If the field is a standard timestamp field in Datastore.
    • Epoch: If the field is an Integer field representing epoch time.
  1. Incremental Field: Filter data by this field. Ensure to index the field in the selected Datastore kind to enable filtering.
  2. Incremental Type: Choose between Timestamp or Epoch.
  3. Start Date & End Date: Select the datetime range. Leaving it empty pulls data until the current moment. Use "Include end value" if pulling data until a specific end datetime is necessary.
  4. Interval Chunks Size: Use this to pull large amounts of data to reduce the load on Datastore in each request.
  5. Filters: Add filters based on indexed columns to further refine the data extraction.
  6. Get Ancestors: Optionally retrieve ancestors' information for each entity in the results.

Query by GQL

The "Query by GQL" method employs Google Query Language for retrieving and manipulating data in Datastore. You can specify criteria for data selection and manipulation through GQL queries.

Extraction procedure

  1. Include query parameters for criteria specification.
  2. Enter the query in the designated text box.
  3. Optionally toggle to retrieve data ancestors for each entity.
note

Incremental extraction is not supported with GQL.

Mapping attributes

Mapping attributes control how Data Integration pulls data from the selected Datastore kind, offering flexibility in data handling.

Attribute mapping procedure

  1. Get Attributes Mapping of Data:
  • Click on auto-mapping to retrieve column details.
  • Ensure to include required columns in the mapping results.
  1. Convert Data Types:
  • Change data types of selected columns as needed.
  • Useful for resolving mapping errors or mismatches.
  1. Select Columns:
  • Modify column selection to pull only required data.
  • Ensure selected columns have appropriate indexes in Datastore.
note

After making any changes to attribute mapping, refresh the mapping of the Target table to synchronize the changes.

On this Page