Client-side limiting policy

The Client-side Limiting Gateway policy regulates API traffic by enforcing predefined quotas. This ensures API usage stays within limits, preventing overconsumption and maintaining stable integration. The policy dynamically manages requests and provides appropriate responses when quotas are exceeded using the Limiter and GenerateResponse processors.

Prerequisites

• Active LLM Gateway instance. Refer to Installation.
• Familiarity with Gateway policy structure and configuration template.
• A configured Quota, such as a Fixed Window Quota.

Scenarios

1. Quota Enforcement: Applies traffic limits based on configured quotas, ensuring that API requests adhere to predefined thresholds for visibility and management.
2. Custom Responses: Configures specific messages or status codes when limits are reached, giving clear feedback on quota breaches.
3. Quota Segmentation: Enforces quotas based on attributes like headers or URLs, providing granular traffic control, such as setting different limits for production and staging environments.
4. Noisy Neighbor Prevention: Ensures that high-traffic clients do not impact others by enforcing client-specific quotas within the policy.
5. Proactive Client-Side Throttling: Protects API provider resources by enforcing traffic limits directly on the client side, ensuring responsible and efficient API consumption.

Gateway policy component

Fixed Window Quota

Gateway policy example

The following configuration example restricts requests to https://httpbin.com/ to 100 requests per minute and returns HTTP 429 when traffic exceeds the quota:

/etc/lunar-proxy/flows/flow.yaml

name: ClientSideLimitingFlow

filter:
  url: httpbin.com/*
processors:
  RateLimiter:
    processor: Limiter
    parameters:
      - key: quota_id
        value: MyQuota
  GenerateResponseLimitExceeded:
    processor: GenerateResponse
    parameters:
      - key: status
        value: 429
      - key: body
        value: "Quota Exceeded. Please try again later."
      - key: Content-Type
        value: text/plain
flow:
  request:
    - from:
        stream:
          name: globalStream
          at: start
      to:
        processor:
          name: RateLimiter

    - from:
        processor:
          name: RateLimiter
          condition: above_limit
      to:
        processor:
          name: GenerateResponseLimitExceeded

    - from:
        processor:
          name: RateLimiter
          condition: below_limit
      to:
        stream:
          name: globalStream
          at: end

  response:
    - from:
        processor:
          name: GenerateResponseLimitExceeded
      to:
        stream:
          name: globalStream
          at: end

The following quota configuration defines the target primary quota referenced by the client-side limiting policy:

/etc/lunar-proxy/quota/quota.yaml

quotas:
  - id: MyQuota
    filter:
      url: httpbin.com/*
    strategy:
      fixed_window:
        static:
          max: 100
          interval: 1
          interval_unit: minute