Handling bulk data using master loops

When working with bulk data, you need to loop through the data into manageable chunks. For example, you can loop over more than 1000 objects using a master loop approach in a Logic River.

Determine the number of master loops

1. Get the number of parent loops needed:

   • Calculate the total number of rows in your data set. For example, if **my_data** has 3335 rows, the following steps store 4 in the variable **loop_count**.

note

my_data is the result of the SELECT ID FROM table statement. These IDs are then passed into the API endpoint to be looped over.

2. Generate the parent loop variable:

   • Generate a list of offsets based on loop_count. This helps parameterize the offset needed to retrieve the correct data. For example, the list generated would be [0, 1000, 2000, 3000].

Implementing the loop

Procedure

1. Master loop:

   • Loop over the list [0, 1000, 2000, 3000].

2. Query the Data:

   • In each iteration of the master loop, query the data using a limit of 1000 and the appropriate offset. This ensures that each iteration retrieves a subset of the data:

Rows 1 to 1000 Rows 1001 to 2000 Rows 2001 to 3000 Rows 3001 to 3335

3. Loop on the rows and apply action:

After querying the data in each iteration, loop through the rows and perform the necessary actions.