Installing a cloud cluster with elastic scaling in Kubernetes

About this task

There are several prerequisites and steps for using Elastic Executions to scale your Boomi runtimes. This topic covers the prerequisites, procedure, and configuration for creating a brand new cloud with elastic executions enabled.

Prerequisites

Boomi Platform

In order to install a cloud runtime, you should have the appropriate permissions to manage clouds. You can manage access to the Cloud Management screen by granting or revoking the “Cloud Management” privilege to user roles in your account’s setup.

Kubernetes cluster

You can use any Kubernetes cluster, as long as the you (or the authorized user) have permissions to create the following resources:

• Namespace
• Deployment
• HorizontalPodAutoscaler
• Role
• RoleBinding
• Service
• ServiceAccount
• StatefulSet

Optionally, managed Kubernetes solutionfor these capabilities are available from various providers.

Kubectl

kubectl is a command line tool which is used for running commands against a Kubernetes cluster. Several steps in this document rely on this utility. Instructions for installing kubectl can be found in the Kubernetes documentation-installing kubectl.

Helm

Helm is a package manager for Kubernetes which lets us create, distribute, and deploy applications to a cluster via helm charts. Boomi provides two helm charts to set up elastic executions: runtime-cloud and runtime-elastic-controller. Both of these helm charts are hosted on Docker Hub.

To install Helm, refer to the Helm documentation.

KEDA

Kubernetes Event Driven Autoscaler (KEDA) is a tool that can scale the workers based on custom application metrics. Additionally, it lets us scale the workers to 0. Refer to the Scaling elastic cloud clusters topic for more details.

KEDA needs to be deployed to our cluster in order to scale the workers efficiently. To install via Helm, refer to the Keda documentation.

Installation

There are several installation steps to follow for installing your new cloud. The below section details the process for each of the following steps.

1. Namespace
2. Persistent Volume
3. Persistent Volume Claim
4. Elastic Controller
   1. Elastic Controller configuration
5. Boomi runtime cloud and cloud cluster
   1. Runtime cloud configuration
      1. Enable workers for the cloud cluster
      2. Attachment Quotas
      3. Helm

1. Namespace

All of the Boomi related resources in this guide are created in the Boomi namespace. Create this namespace by running the following command:

kubectl create namespace boomi

note

If you install multiple runtime clouds, each should be in its own namespace. Use unique namespaces accordingly.

2. Persistent Volume

A Persistent Volume is needed for the Persistent Volume Claim which is used for the shared file system required for the runtime. The user is responsible for creating the Persistent Volume since it is heavily dependent on the storage technology.

For customers deploying in AWS, the following links provide more information for using EFS in EKS:

• Use elastic file system storage with Amazon EFS
• Amazon EFS

3. Persistent Volume Claim

A Persistent Volume Claim with the name of runtime-claim should be created in the corresponding namespace. This should point to the shared file system required for the runtime cloud.

If configured correctly, the created Persistent Volume Claim will have a status of Bound.

4. Elastic Controller

The elastic controller is a new Boomi runtime service which is responsible for creating the appropriate Kubernetes resources for forked executions and workers. The helm chart and the image for this application are hosted in Docker Hub.

1. Create a separate file for the helm values of the elastic controller. Let’s call this file elastic-controller-helm-values.yaml. The contents of this file should be as follows:

fullnameOverride: runtime-elastic-controller

2. Deploy the elastic controller via the helm chart:

helm upgrade --install --wait --namespace boomi --create-namespace -f elastic-controller-helm-values.yaml runtime-elastic-controller-release oci://docker.io/boomi/runtime-elastic-controller --version <version>

3. Verify the state of the elastic controller pods in the Boomi namespace:

kubectl get pods -n boomi -l app=runtime-elastic-controller

A. Elastic Controller Configuration

Since the elastic controller manages the Kubernetes resources required for the integration processes, it provides several hooks to customize how these resources are created.

Run the command below to list the helm values.

helm show values oci://docker.io/boomi/runtime-elastic-controller --version <version>

Pay special attention to these fields:

• boomi.runner: This customizes the forked execution pods.
• boomi.worker: This customizes the worker pods.
• boomi.browser: This customizes the browser pods.
• boomi.execution: This customizes the forked execution pods, worker pods, and the browser pods.

You can add these fields to the elastic-controller-helm-values.yaml file; then run the helm upgrade command for these changes to be reflected.
Only new Kubernetes resources created by the elastic controller will be affected, existing resources will remain as they are.

5. Boomi Runtime Cloud and Cloud Cluster

This example details how to deploy a new runtime cloud using the helm chart. For further instructions on creating new runtime clouds, refer to the Runtime Clouds topic.

1. If you have not created a Cloud yet, within the Boomi Platform, go to Integration > Manage > Cloud Management and create a new Cloud. Refer to Adding a runtime cloud for more details.

2. Click Install cloud cluster and generate a new install token for that cloud. Create a secret in the cluster by running the following command:

kubectl create secret -n boomi generic runtime-token --from-literal=token=<cloud install token>

3. Create the “runtime-cloud-helm-values.yaml” file for the helm values of the runtime cloud. The contents of this file are as follows:

fullnameOverride: runtime-cloud
boomi:
  installTokenSecretRef:
    name: runtime-token
    key: token
  atomName: ElasticCloud
  containerPropertiesOverrides: 'com.boomi.container.processExecMode=ELASTIC'

Note: These helm values (via the containerPropertiesOverrides field) will also set the following container properties:

• com.Boomi.container.processExecMode=ELASTIC
  • Enables elastic executions.

4. Deploy the runtime cloud via the helm chart:

helm upgrade --install --wait --namespace boomi --create-namespace -f runtime-cloud-helm-values.yaml runtime-cloud-release oci://docker.io/boomi/runtime-cloud --version <version>

5. Verify the state of the runtime cloud pods in the Boomi namespace::

kubectl get pods -n boomi -l app=runtime-cloud

A. Runtime Cloud Configuration

I. Enable Workers for the cloud cluster

1. In the platform UI, go to Manage > Runtime Management > select the cloud cluster > Properties panel > Advanced tab, and add the “Execution Workers Enabled” property, set to true. Refer to Enabling execution workers on a Private runtime cloud for more details.

2. Configure worker attachment quotas in the below section.

Ii. Cloud Attachment Quotas

Cloud quotas are set per cloud attachment via the Manage > Cloud Management > select cloud > Attachment Quotas screen. Default quotas for all attachments can be set at the cloud cluster level. Additionally quotas can be overridden for a specific attachment.

Quota	Description
Execution Workers (Required)	Sets the maximum number of workers allowed for an attachment. Must be set to at least 1 to have a worker start. The cluster can dynamically scale the number of workers between this maximum and the optional minimum set below.
Minimum Execution Workers (Optional)	Sets the minimum number of workers. The default is 0. If not set, a worker will start only upon receiving the first execution request configured to run on a worker. In most practical cases, this should be set to at least 1 to ensure event-based listeners (e.g. messaging connectors) are running and to have capacity ready for inbound real time requests.
Worker Elastic Scaling Threshold	This quota is the percentage value of current executions relative to the Execution Worker Maximum Running Executions that will trigger elastic scaling of additional workers. Refer to Scaling Elastic cloud clusters for more details.

Iii. Helm

Similar to the elastic controller’s helm chart, customizations are possible via the helm chart of the runtime cloud as well.

Helm values can be listed by running the following command:

helm show values oci://docker.io/boomi/runtime-cloud --version <version>

These fields can then be added to the elastic-cloud-helm-values.yaml file, and the helm upgrade command should be run again for these changes to be reflected.

Installation is now complete.