Skip to main content
Feedback

GitHub walkthrough

info

This connector is currently in Early Access. Contact support for details.

Prerequisite

An established GitHub connection.

Step 1: Set up the data source

  1. Navigate to the Data Integration console.

  2. Select Create, then select Data Flow.

  3. Select GitHub as the data source.

  4. Under Source Connection, select an existing GitHub connection from the dropdown, or select Add New to create one. See GitHub connection for setup steps.

  5. Select Test Connection to confirm Data Integration can authenticate and reach your GitHub account.

  6. Under Source Settings, configure the connector settings that apply to every report:

    • Organization: select the GitHub organization to sync data from.
    • Repositories: select which repositories within the selected organization to sync (issues, pull requests, commits, releases, branches, workflow runs, and deployments). This field populates only after you select an organization.

    Select the refresh icon next to either field to reload the list of organizations or repositories.

  7. Select Next.

note

Repository-level reports, such as issues and pull requests, sync once for each repository you select. If a report can't sync for one repository (for example, issues are disabled on that repository), Data Integration skips it and still syncs the report for your other selected repositories.

Step 2: Select a data target

  1. Select the destination you want to load GitHub data into, then select Next.

    note

    Some destinations are labeled Alpha or Beta. Review the labeled destination's documentation before you use it in a production data flow.

  2. Under Target Connection, select an existing connection from the dropdown, or select Add New to create one.

  3. Select Test Connection to confirm Data Integration can authenticate and reach your destination.

  4. Under Data Loading Settings, select the values required for your destination, such as the target Database and Schema. Expand Advanced Settings for additional options specific to your destination.

  5. Select Next.

Step 3: Configure the schema

  1. On the Tables Definitions tab, Data Integration lists every GitHub report available to sync. See GitHub predefined reports for what each report returns.
  2. Select the checkbox next to each report you want to sync. Use Search Reports to find a specific report, or use the Selected and All toggles to switch between views.
  3. For each selected report, review its Target Table name and Extract Method. For incremental reports, such as issues and commits, set the Incremental Field, Incremental Type, Start Value, and End Value.
  4. Optional: select Tables Definitions, then expand Advanced Target Definitions to:
    • Add a Table Prefix to the beginning of every target table name.
    • Turn on Convert File To Parquet Format to write output as Parquet instead of CSV or JSON.
  5. If you change your organization or repository selection after this step, select Reload Reports Metadata to refresh the list of available reports.
  6. Select Next.

Step 4: Set the schedule and settings

  1. Turn on Schedule Data Flow and set how often you want the data flow to run. Data Integration displays and runs schedules in UTC.
  2. Optional: turn on Set Custom Timeout to set a fixed run timeout. By default, Data Integration sets the timeout automatically per run, based on table size, capped between 12 hours and 7 days.
  3. Optional: under Notifications, turn on Failure, Warning, or Run Threshold, and enter one or more email addresses, separated by commas, to receive alerts.
  4. Under Data Flow Info, enter a Data Flow Name. Optionally select a Group and enter a Description.
  5. Select Activate to save the data flow and start running it on its schedule, or select Save & Exit to save it without activating.

Sync behavior

  • Issues and commits sync incrementally, based on the incremental values you set in step 3. All other reports run a full refresh on every sync.
  • Every row returned by a repository-level report includes a repo_full_name field, so you can join data across reports without an extra lookup.
  • Because GitHub doesn't return tombstones for deleted issues, pull requests, branches, or releases, Data Integration doesn't track deletes explicitly. A full refresh naturally drops rows that GitHub no longer returns.
On this Page