Using Data Connector Agent

The Data Connector Agent is an engine that automatically builds connections to REST APIs by reading their official documentation. Instead of manually coding API calls, you simply provide the API documentation URL. The agent uses this to generate a valid YAML blueprint that defines endpoints, parameters, pagination, and schema for your pipeline.

Prerequisites

Before creating a connection with the Data Connector Agent, ensure the following:

• API documentation URL: The agent depends on the content of the API docs to understand endpoints and generate a valid YAML file.

• API Credentials: If the API requires authentication.

Create and activate a river with AI blueprint

Follow these steps to build and activate a River using the AI-based Blueprint generator.

Step 1. Logging in

• Sign in to the Data Integration console.

Step 2. Creating a new River

1. In the left-hand menu, click + Create River > Build Your Own.
2. Click Generate with our AI.

Step 3. Providing API details

1. Enter the following details:

   • Blueprint Name: Provide a descriptive name.
   • API-based URL: Enter the public API documentation URL.
   • Selected Report: Choose the resource to extract.

2. Click Generate.

   A processing message appears while the agent generates the YAML configuration.

   note

   This process may take a few minutes. You can leave the page and return later using the email link provided, or remain on the page while the agent generates the configuration.

The AI agent creates a draft Blueprint, including endpoints, authentication model, and proposed schema.

Step 4. Configuring the data source

1. Review the Selected Data Source to confirm your Blueprint.

2. Under Interface Parameters, enter the required values.

   note

   If parameters are missing or invalid, a warning appears. This may result in incomplete or incorrect data retrieval.

3. Configure Connection:

   • Select an existing connection, or
   • Create a new one (API Key, OAuth2, or other supported methods).

4. Click Next to continue.

Step 5. Selecting the data target

1. On the Select Data Target screen:
   • Choose your target integration from the list.
   • Use filters (Storage & Files or Databases) or the search bar to find your integration.
2. Click Next.

Step 6. Configuring the target

1. Selected Data Target: Confirm your chosen destination.

   • To change, click Change and select another target.

2. Target Connection:

   • From the dropdown, select an existing connection or click + Add New Connection.
   • Enter the required credentials.
   • Click Test Connection to test access.

3. Data Loading Settings:

   • Database: Select the target database or assign a variable.
   • Schema: Select the schema to load the data into.
   • Expand Advanced Settings for custom loading options.

4. Click Next to Configure Schema.

Step 7. Configuring the schema

1. The Blueprint loads schema definitions from the selected source.
2. Click Tables Definitions to view and manage tables and fields.
3. (Optional) Click Reload Blueprint Metadata to refresh schema information.
4. Review the table structure and field mappings.

Step 8. Configuring schedule & settings

1. Schedule River:

   • Toggle the switch to enable scheduling.
   • Define how often the River should run.
   • Set the maximum execution time (hours and minutes).

2. Notifications:

   • Enter email addresses to receive alerts for:
     • Failure
     • Warning
     • Run Threshold

3. River info:

   • Enter a River Name (auto-generated, but editable).
   • Assign a Group.
   • Add a Description.

4. Click Activate to save and activate the River.

Step 9. Verifying River activation

1. After activation, click Run River.
2. A confirmation message appears. River Was Successfully Activated!

This confirms that:

• The system saved the River successfully.
• The target (database and schema) was validated.
• The system updated and activated the River settings.
  Your River is now active and ready for use.