Skip to main content
Feedback

Using variable looping to analyze Data Flow activity statistics

You can collect and analyze Data Flows Activity Statistics using variable looping in REST Actions. The primary objective is to gather data from the Data Flow Activities and present the required statistics through a comprehensive dataset.

To learn more about the fundamental concepts, refer to the Overview of Variable Looping.

Example use case

Retrieving specific Data Flows activity statistics through API integration

In this use case, the primary goal is to collect Data Flows Activity Statistics using APIs. The process includes the following steps:

  • Retrieving data from Activities API: Accessing the Activities API to gather relevant information. This API provides data on Data Flow Activities.

  • Loading data into Snowflake: After retrieving the data from the Activities API, Data Integration loads the information into a data warehouse like Snowflake.

  • Populating data into a variable: Once the data is in Snowflake, Data Integration extracts it and stores it in a variable. This variable holds the relevant information used later in the process.

  • Creating a GET call for Data Flow activity statistics: In this step, you make a new GET call to another API specifically dedicated to Data Flow Activity Statistics. The API URL contains a crucial parameter for retrieving the desired data.

  • Iterating the variable in a loop: Use the variable populated to iterate through a loop. The loop repeatedly requests the Data Flow Activity Statistics API, using different values from the variable as the parameter in the API URL.

  • Loading data into Snowflake: As the loop progresses, Data Integration retrieves data from the Data Flow Activity Statistics API using the variable's values as parameters. Data Integration then loads each retrieved record into Snowflake, building a comprehensive dataset.

  • Presenting the results: You derive the required data from the looped variable values. With all the data stored in Snowflake, you can derive the relevant data by analyzing and processing the gathered information.

Use case benefits

  • Efficient data analysis: By leveraging API integration and looping in Data Integration, you can effectively analyze a bulk dataset of activities and extract essential statistics, thereby eliminating the need for manual processing.
  • Scalability: The process can easily scale to handle a growing volume of activities, making it suitable for applications with expanding datasets.
  • Customizability: You can tailor the REST Action to meet specific requirements, letting you define parameters to focus on and activity statistics relevant to your needs.

Variable looping configuration steps

The process covers six primary steps.

Step 1: Create an API token

Before initiating any API calls, ensure proper authentication and authorization mechanisms are in place to access the required data.

Step 2: Establish the API connection (REST Action)

In this step, you must access the Activities API to collect relevant information. The API offers data about Data Flow Activities.

  • Providing API URL
  • Configuring the setup to retrieve an explicit field from the API response.
note

To retrieve the Account and Environment IDs from the URL, follow the instructions: image.png

  • For more information about the API Tokens, refer to the API Tokens.

Step 3: Push activities API data to Snowflake (Source to Target Flow)

  • Get data from the REST Action created.
  • Load the retrieved information into a data warehouse, such as Snowflake.

Step 4: Establish the loop (REST Action)

Set up a loop to iterate over the retrieved values from the Activities API response. Within the loop, construct secondary API calls, using each value as a parameter to fetch the relevant activity statistics.

note

To retrieve the Account and Environment IDs from the URL, follow the instructions: image.png

Step 5: Load cross ID data into Snowflake (Source to Target Flow)

  • Gather statistics from the secondary API calls.
  • Store the retrieved statistics in a suitable data structure.

Step 6: Trigger all Data Flows and get the results you need

This step concludes the entire process, during which you activate all preceding steps. Once you complete this step, you can view the API call's outcomes and the iterated values in your Snowflake warehouse.

  • Using a query to extract and store information from the Target in a variable.
select CROSS_ID from {Your Database} where is_deleted=false
  • Using the loop created in an REST Action to iterate over the variable containing all relevant information.
On this Page