Custom metrics collection policy

The Custom Metrics Collector Gateway policy tracks user-defined metrics for API requests and responses, such as request size or rate limits. This policy helps monitor API performance and usage patterns in real-time, offering flexibility with histograms, gauges, and other metric types.

Once this policy is running, metrics are written to Prometheus, which you can access directly at localhost:9090 or through Grafana at localhost:3000.

Prerequisites

• Active LLM Gateway instance. Refer to Installation.
• Familiarity with Gateway policy structure and configuration template.
• Access to Prometheus (localhost:9090) or Grafana (localhost:3000) to view exported metrics.

Scenarios

1. Track API Call Sizes: Monitor API payload sizes to identify performance-impacting requests.
2. Monitor Rate Limit Usage: Track remaining rate limits from responses to manage API consumption.
3. Analyze Request Trends: Use custom labels to analyze traffic based on HTTP methods, URLs, or headers.
4. Performance Metrics: Measure performance of API endpoints to optimize response times and reduce overhead.

Gateway policy components

• UserDefinedMetrics Processor

Gateway policy example

In this configuration:

• API Request Metrics: The policy captures the size of each API call (api_call_size) and categorizes the data into histogram buckets. Metrics are labeled with http_method, url, and consumer_tag for detailed tracking.
• API Response Metrics: The policy collects the remaining rate limits from API responses using the X-Ratelimit-Remaining header, stored as a gauge metric with the url as a label.

/etc/lunar-proxy/flows/flow.yaml

name: CustomMetricsCollector

filter:
  url: "*" # Capture metrics for all URLs

processors:
  ResponseMetrics: # Collect custom metrics for API responses
    processor: UserDefinedMetrics
    parameters:
      - key: metric_name
        value: "rate_limit_remaining" # Metric to track remaining rate limits
      - key: metric_type
        value: "gauge" # Metric type (e.g., gauge)
      - key: metric_value
        value: "$[\"headers\"][\"X-Ratelimit-Remaining\"]" # JSON path for rate limit from headers
      - key: labels
        value:
          - "url" # Label for URL
      - key: custom_metric_labels
        value:
          "x-lunar-context": "$[\"headers\"][\"x-lunar-temp-context\"]"
          "x-lunar-special-context": "$[\"headers\"][\"x-lunar-special-context\"]" 

flow:
  request:
    - from:
        stream:
          name: globalStream
          at: start 
      to:
        stream:
          name: globalStream
          at: end 

  response:
    - from:
        stream:
          name: globalStream
          at: start 
      to:
        processor:
          name: ResponseMetrics 

    - from:
        processor:
          name: ResponseMetrics 
      to:
        stream:
          name: globalStream
          at: end