Skip to main content
Feedback

HAR collector policy

The HAR Collector Gateway policy captures detailed HTTP transaction data, including headers, query parameters, and request/response bodies. It stores the data in a log file using a File Exporter, making it easy to trace API interactions for troubleshooting and auditing.

The HAR Metrics Collector Processor includes robust obfuscation capabilities to protect sensitive data. By default, if obfuscate_enabled is set to true, the processor will mask:

  • query_params
  • path_params
  • request_headers
  • response_headers
  • request_body_paths
  • response_body_paths

The HAR Metrics Collector Processor only writes structured logs; it does not create a .har file by itself. To generate a valid .har file, download the har-convert.js script and run it with the following arguments:

HAR Convert Script
node har-convert.js <input_log_path> <output_har_path>

The input_log_path is defined in your gateway_config.yaml. Running this script will generate a .har file that can be read in your preferred browser. Make sure to run this script in the folder where it is saved.

note
  • Obfuscation is disabled by default. To enable it, set obfuscate_enabled: true
  • When enabled, the processor obfuscates values, not names, in query/path parameters, request/response headers, and bodies.
  • Obfuscation replaces values with consistent hashes to protect privacy while preserving traceability (for example, 123 always hashes to the same value).
  • If no log output path is set in gateway_config.yaml, logs are written to the fluent-bit folder in your local Docker container.

Prerequisites

  • Active LLM Gateway instance. Refer to Installation.
  • Familiarity with Gateway policy structure and configuration template.
  • A configured File Exporter in your gateway_config.yaml.

Scenarios

  1. Granular HTTP Data Capture: Record complete HTTP transactions for diagnostics.
  2. Data Obfuscation: Mask sensitive information with configurable obfuscation rules.
  3. Local Log Export: Store logs locally with customizable file paths and names.
  4. Metrics Integration: Monitor log collection and export using Prometheus-compatible metrics.

Gateway policy components

Gateway policy example

In this example, the policy captures HTTP transaction logs for a specific API endpoint and exports them using the configured File Exporter. Obfuscation is enabled for all data except for the specified exclusions (id, user.name, Retry-After), letting specific data points to remain visible for diagnostics.

/etc/lunar-proxy/flows/flow.yaml
name: HARCollectorFlow

filter:
  url: "api.com/resource/{id}"
processors:
  HARCollectorResponse:
    processor: HARCollector 
    parameters:
      - key: exporter_id
        value: "file_exporter_01"
      - key: transaction_max_size_bytes
        value: 5000 
      - key: obfuscate_enabled
        value: true
      - key: obfuscate_exclusions
        value:
          - '$.request.query_param.id' # Exclude specific query parameters from obfuscation
          - '$.request.body.user.name' # Exclude specific paths in the request body from obfuscation
          - '$.response.headers["Retry-after"]'
    metrics:
      enabled: true  # Enables metric collection for this processor
      labels:
        - flow_name
        - processor_key
        - http_method
        - url
        - status_code
flow:
  request:
    - from:
        stream:
          name: globalStream 
          at: start  
      to:
        stream:
          name: globalStream
          at: end
  response:
    - from:
        stream:
          name: globalStream
          at: start 
      to:
        processor:
          name: HARCollectorResponse 
    - from:
        processor:
          name: HARCollectorResponse 
      to:
        stream:
          name: globalStream
          at: end

/etc/lunar-proxy/gateway_config.yaml
exporters:
  file:
    exporter_id: my_file_exporter
    file_dir: "/var/log/lunar-proxy"
    file_name: "transaction.log"
har-convert.js
node har-convert.js /Users/Lunar/Downloads/transaction.log /Users/Lunar/Downloads/log.har

Troubleshooting

  • Log File Not Created:
    • Ensure exporter_id in flows.yaml matches the File Exporter ID in gateway_config.yaml.
    • Verify that the specified file_dir exists and has appropriate write permissions.
  • Large Log Files:
    • Use transaction_max_size_bytes to limit the size of each logged transaction.
    • Set max_file_size in gateway_config.yaml to manage log file growth.
  • Obfuscation Not Working:
    • Check that obfuscate_enabled is set to true and verify the exclusions list.
    • Ensure that the fields specified for exclusion exist in the captured HTTP data.
  • HAR File Not Generated:
    • Remember that the policy does not automatically generate a .har file.
    • Run the har-convert.js script manually, in the directory where the file is saved, providing the log file full path as input and the destination for the output HAR file.
On this Page