Google Cloud Storage (GCS) walkthrough

Prerequisites

Google Cloud Storage connection.

Pulling data from GCS using Data Integration

Procedure

1. Navigate to the Data Integration Account.

2. Select Create New River.

3. Choose Data Source to Target as your river type.

4. In the General Info tab, name your river, describe it, and choose a group.

5. Click the Source tab and find Google Cloud Storage under Storage in the list of data sources.

6. Under Source Connection, select the connection you created, or create a new one.

7. Select the desired Bucket Name from the list.

8. Choose an Extract Method.

9. Define the Prefix.

10. Define the File Pattern.

11. Define the File type.

12. Define the action After pulling - files will be

13. (Optional): Configure the Number of files to pull.

Extract methods

• Run all:

  • Returns data from all times. Given the provided Bucket, Prefix, and File Pattern.

• Incremental run: by modified timestamp

  • Pulls data in the date range between the start and end date provided, including the end date. Given the provided Bucket, Prefix, and File Pattern.
  • You must select a start date.
  • Leaving the end date empty pulls data according to the current time of the River's run.
  • Select a timezone offset to send dates considering the offset.
  • Use the Last Days Back For Each Run to expand the start date and pull data a given number of days back before the chosen start date.

• Incremental run: by template

  • Templates give you the option to run over folders and load the files according to the folder order. You need to choose your template type and write your template structure.
  • Timestamp Template - Use {} and the proper timestamp part to define the folder format.
  • Epoc time template - Use {e} (for an epoc) or {ee} (for an epoc in milliseconds) to define the folder to run on.
  • epoc time.
  • Enter the desired Start Value (required) and End Value (optional)
  note

  • This method is valid for the whole library and not available for files.
  • The Start Date does not advance if a River run is unsuccessful. If you want to remove this default setting, click More Options and select the checkbox to advance the start date even if the River run is unsuccessful (Not recommended).

The time-dependent report includes Campaigns, Email templates, and Events.

Results after the pull

Options:

• Move to archive path
• Remain in original place
• Deleted

note

The chosen action applies to both successful and unsuccessful rivers. For example, if something fails during the upload to the target, the system moves to the archive path from the source.