Enabling Amazon Bedrock agent metrics

You can view your agents' metrics in the Monitor Agents dashboard. Ensure the Bedrock agents have the CloudWatch-PutMetricData permission attached to the execution role.

note

For regions with a small user base, invoking agents in the Monitoring Agent dashboard results in a lag of 10 minutes or less.

Prerequisites

To add the Cloudwatch-PutMetricData policy, you must deploy a CloudFormation template for that particular region. To deploy the template, ensure you have the IAMFullAccess permission or the following permissions enabled:

• cloudformation:CreateStack: Create the CloudFormation stack
• cloudformation:DescribeStacks: Monitor stack creation progress
• iam:CreateRole: Create the Lambda execution role
• iam:PutRolePolicy: Configure role policies
• lambda:CreateFunction: Create the Lambda function
• events:PutRule: Create the EventBridge rule
• events:PutTargets: Configure EventBridge targets
• lambda:AddPermission: Grant EventBridge permission to invoke Lambda

To check if your account has the following permissions:

1. Navigate to the IAM dashboard in your AWS account.
2. Select User.
3. Search for your account user name and select it.
4. Under Permission policies, check if you have the above permissions. To add permissions refer to adding and removing IAM identity permissions.

Adding the Cloudwatch-PutMetricData policy

The CloudWatch-PutMetricData permission enables sending Agent invocation metrics into CloudWatch to show up in the Monitor Agents dashboard in near real-time. The following CloudFormation template(CFT) can be deployed for adding the Cloudwatch-PutMetricData policy to all the agents in the region in which CFT is running. This task must be done for every new region of an AWS account added to the Control Tower.

1. Review and copy the following CloudFormation template and save it as a .yaml file in your local desktop

CloudFormation template

AWSTemplateFormatVersion: '2010-09-09'
Description: 'The template creates an Eventbridge scheduler that triggers a lambda function, creating a CloudWatch:PutMetricData permission policy. At an agent's invocation, this policy monitors metrics data in CloudWatch. The lambda function is triggered hourly to ensure new agents have the CloudWatch:PutMetricData permission attached to them.'

Resources:
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    DeletionPolicy: Delete
    Properties:
      RoleName: !Sub 'bedrock-policy-role-${AWS::Region}-${AWS::AccountId}'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
      Policies:
        - PolicyName: !Sub 'bedrock-permissions-${AWS::Region}'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - iam:AttachRolePolicy
                  - iam:ListAttachedRolePolicies
                  - iam:GetPolicy
                  - iam:CreatePolicy
                  - sts:GetCallerIdentity
                  - bedrock:ListAgents
                  - bedrock:GetAgent
                Resource: '*'

  AttachBedrockAgentPolicyFunction:
    Type: AWS::Lambda::Function
    DependsOn: LambdaExecutionRole
    Properties:
      FunctionName: !Sub 'bedrock-policy-function-${AWS::Region}'
      Runtime: python3.11
      Handler: index.lambda_handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: |
          import json
          import boto3
          from botocore.exceptions import ClientError

          # Policy config to create and attach
          POLICY_NAME = 'PutMetricDataPolicyForBedrockAgents'
          POLICY_DESCRIPTION = 'Allows CloudWatch PutMetricData for Bedrock Agents'
          POLICY_DOCUMENT = {
              "Version": "2012-10-17",
              "Statement": [
                  {
                      "Sid": "VisualEditor0",
                      "Effect": "Allow",
                      "Action": "cloudwatch:PutMetricData",
                      "Resource": "*"
                  }
              ]
          }

          session = boto3.Session()
          AWS_REGION = session.region_name
          iam_client = session.client('iam')
          bedrock_client = session.client('bedrock-agent')

          def get_or_create_policy():
              sts_client = session.client('sts')
              account_id = sts_client.get_caller_identity()['Account']
              policy_arn = f'arn:aws:iam::{account_id}:policy/{POLICY_NAME}'

              try:
                  iam_client.get_policy(PolicyArn=policy_arn)
                  print(f"Policy already exists: {POLICY_NAME}")
                  return policy_arn
              except ClientError as e:
                  if e.response['Error']['Code'] == 'NoSuchEntity':
                      print(f"Policy {POLICY_NAME} does not exist. Creating it...")
                      response = iam_client.create_policy(
                          PolicyName=POLICY_NAME,
                          PolicyDocument=json.dumps(POLICY_DOCUMENT),
                          Description=POLICY_DESCRIPTION
                      )
                      print(f"Policy created: {POLICY_NAME}")
                      return response['Policy']['Arn']
                  else:
                      raise

          def extract_role_name(role_arn):
              return role_arn.split('/')[-1]

          def get_all_agents():
              agents = []
              next_token = None

              try:
                  while True:
                      params = {'maxResults': 10}
                      if next_token:
                          params['nextToken'] = next_token

                      response = bedrock_client.list_agents(**params)

                      if response.get('agentSummaries'):
                          agents.extend(response['agentSummaries'])

                      next_token = response.get('nextToken')
                      if not next_token:
                          break

                  return agents

              except ClientError as e:
                  print(f"Error listing agents: {e}")
                  return []

          def process_agent(agent_id, policy_arn):
              """Process individual agent and attach policy to its role"""
              try:
                  # Get detailed agent information
                  print(f"\nProcessing agent ID: {agent_id}")
                  agent_response = bedrock_client.get_agent(agentId=agent_id)

                  # Check if agentResourceRoleArn exists and is not empty
                  if 'agentResourceRoleArn' not in agent_response['agent'] or not agent_response['agent']['agentResourceRoleArn']:
                      print(f"Skipping agent {agent_id}: No resource role ARN found")
                      return False

                  # If we get here, we know the role ARN exists
                  role_arn = agent_response['agent']['agentResourceRoleArn']
                  role_name = extract_role_name(role_arn)

                  print(f"Found agent role: {role_name}")

                  # Attach policy to the role
                  attach_policy_to_role(policy_arn, role_name)
                  return True

              except ClientError as e:
                  print(f"Error processing agent {agent_id}: {e}")
                  return False

          def attach_policy_to_role(policy_arn, role_name):
              try:
                  response = iam_client.list_attached_role_policies(RoleName=role_name)
                  for policy in response['AttachedPolicies']:
                      if policy['PolicyArn'] == policy_arn:
                          print(f"Policy already attached to role {role_name}")
                          return

                  print(f"Attaching policy to role: {role_name}")
                  iam_client.attach_role_policy(RoleName=role_name, PolicyArn=policy_arn)
                  print(f"Successfully attached policy to role {role_name}")
              except ClientError as e:
                  print(f"Error attaching policy to role {role_name}: {e}")

          def main():
              try:
                  print("Starting policy attachment process...")
                  print(f"Region : {AWS_REGION}")

                  policy_arn = get_or_create_policy()
                  if not policy_arn:
                      print("Failed to get or create policy")
                      return

                  print("Fetching all Bedrock agents...")
                  agents = get_all_agents()

                  if not agents:
                      print("No agents found")
                      return

                  print(f"Found {len(agents)} agents")

                  successful_attachments = 0
                  for agent in agents:
                      agent_id = agent['agentId']
                      agent_name = agent.get('agentName', 'Unknown')
                      print(f"\nProcessing agent: {agent_name} (ID: {agent_id})")

                      if process_agent(agent_id, policy_arn):
                          successful_attachments += 1

                  print(f"\nProcess completed. Successfully processed {successful_attachments} out of {len(agents)} agents")

              except Exception as e:
                  print(f"An unexpected error occurred: {e}")

          def lambda_handler(event, context):
              main()
              return {
                  'statusCode': 200,
                  'body': 'Script execution completed'
              }
      Timeout: 300
      MemorySize: 256

  EventBridgeRule:
    Type: AWS::Events::Rule
    DependsOn: AttachBedrockAgentPolicyFunction
    Properties:
      Name: !Sub 'bedrock-policy-rule-${AWS::Region}'
      Description: 'Hourly trigger for Lambda function'
      ScheduleExpression: 'rate(1 hour)'
      State: ENABLED
      Targets:
        - Arn: !GetAtt AttachBedrockAgentPolicyFunction.Arn
          Id: "TargetLambda"

  LambdaPermissionForEventBridge:
    Type: AWS::Lambda::Permission
    DependsOn:
      - EventBridgeRule
      - AttachBedrockAgentPolicyFunction
    Properties:
      FunctionName: !Ref AttachBedrockAgentPolicyFunction
      Action: 'lambda:InvokeFunction'
      Principal: 'events.amazonaws.com'
      SourceArn: !GetAtt EventBridgeRule.Arn

Outputs:
  LambdaFunctionArn:
    Description: ARN of the Lambda function
    Value: !GetAtt AttachBedrockAgentPolicyFunction.Arn
  EventBridgeRuleArn:
    Description: ARN of the EventBridge rule
    Value: !GetAtt EventBridgeRule.Arn

2. Log in to your AWS account.

3. Select the region you used from the AWS dashboard when adding an account.

4. Navigate to the CloudFormation page.

5. Click Create stack > With new resources (standard) from the Stacks page.

6. Select Choosing an existing template from the Prerequisites section.

7. Select Upload a template file from the Specify template section.

8. Upload the CloudFormation template you saved as a .yaml file earlier.

9. Click Next.

10. Create a stack name and click Next.

11. Keep the details as default, select the checkbox at the end of the page, and click Next.

12. Click Submit.

This task creates an eventBridge function that triggers the lambda function hourly. The lambda function attaches a metrics policy to newly created agents or agents without a metrics policy.