Bitbucket walkthrough
Prerequisites
An established Bitbucket connection.
Step 1: Set up the data source
- Navigate to the Data Integration console.
- Select Create, then select Data Flow.
- Select Bitbucket as the data source.
- Under Source Connection, select an existing Bitbucket connection from the dropdown, or select Add New to create one. See Bitbucket connection for setup steps.
- Select Test Connection to confirm Data Integration can authenticate and reach your Bitbucket workspace.
- Once a connection is selected, the Repositories picker populates automatically with the repositories the connection can see. Select the repositories you want to sync. Every per-repository report reads only these.
- Select Next.
Keep Repositories to the repositories you need. Bitbucket allows 1,000 API calls per hour on an API token, and Documentation Files costs about 2 calls per file on every run. See Rate limits.
Step 2: Select a data target
-
Select the target you want to load Bitbucket data into, then select Next.
- Content data (Documentation Files): select Knowledge Hub.
- Engineering data: select a data warehouse, such as Snowflake.
A data flow has one target, so to load both, build one data flow for each.
noteData Integration labels some targets Alpha or Beta. Review that target's documentation before you use it in a production data flow.
-
Under Target Connection, select an existing connection from the dropdown, or select Add New to create one.
-
Select Test Connection to confirm Data Integration can authenticate and reach your target.
-
Under Data Loading Settings, select the required fields for your target, such as Schema. Select the refresh icon next to a field to reload its list of values. Expand Advanced Settings for additional options specific to your target.
-
Select Next.
Step 3: Configure the schema
- On the Tables Definitions screen, Data Integration lists all 12 Bitbucket reports: the content data report, Documentation Files, and 11 engineering data reports for repositories, projects, members, pull requests and their activity, commits, branches, pipelines and their steps, deployments, and environments. See Bitbucket predefined reports for what each report returns.
- Select the checkbox next to each report you want to sync. For Knowledge Hub, select Documentation Files. Use Search Reports to find a specific report, or use the Selected and All toggles to switch between views.
- For each selected report, open its Table Source Settings to narrow what it reads, for example a Folder such as
docs/for Documentation Files, or a Branch and Status for Pipelines. Bitbucket applies these filters itself, so they also cut API calls. - Review each report's Target Table, Extract Method, and Loading Mode. In a data warehouse, use Upsert-Merge with the merge keys listed in Bitbucket predefined reports. Repositories, Projects, Pull Requests, and Pull Request Activity can load incrementally: select Incremental as the Extract Method and set a start date. Every other report, Documentation Files included, is a full load.
- Optional: select Extraction Mode, then look for additional options such as a Table Prefix for target table names or a Convert File To Parquet Format toggle.
- If you change Repositories after this step, or a report is missing from the list, select Reload Reports Metadata to refresh the list of available reports and their columns.
- Select Next.
Columns are detected from a sample of your data. A report with nothing to sample in the selected repositories, for example Deployments on a repository that doesn't use Bitbucket Pipelines deployments, shows no columns until you reload metadata with a repository that has that data.
Step 4: Set the schedule and settings
- Turn on Schedule Data Flow and set how often you want the data flow to run. Data Integration displays and runs schedules in UTC. For Documentation Files, a daily schedule is usually enough, since every run reloads every matching file.
- Optional: turn on Set Custom Timeout to set a fixed run timeout. By default, Data Integration handles the run automatically per run, based on table size, capped between 12 hours and up to 7 days.
- Optional: under Notifications, turn on Failure, Warning, or Run Threshold, and enter one or more email addresses, separated by commas, to receive alerts.
- Under Data Flow Info, enter a Data Flow Name. Data Integration suggests a default name based on your source and target connections and the current date and time. Optionally select a Group and enter a Description.
- Select Activate to save the data flow and start running it on its schedule.