Prometheus integration

LLM Gateway integrates with Prometheus to provide comprehensive metrics on traffic and system performance. Exporting these metrics allows you to leverage visualization tools such as Grafana and Datadog to monitor API performance and track internal flow execution. Configure LLM Gateway to expose metrics for Prometheus scraping using the steps below.

Scraping metrics from Prometheus

To scrape metrics from LLM Gateway with Prometheus, add the following to your prometheus.yml configuration:

scrape_configs:
  - job_name: "lunar-proxy"
    scrape_interval: 5s
    static_configs:
      - targets: ["lunar-proxy-address:3000"]
        labels:
          group: "production"

Adjust the address and interval as per your system setup.

Scraping metrics in Kubernetes

To scrape metrics in Kubernetes environments, use the Prometheus Operator’s serviceMonitor resource. Enable it in the Helm chart’s values.yaml file:

serviceMonitor:
  enabled: true