Quota strategies

LLM Gateway supports quota strategies to align consumer-side traffic execution with provider-enforced constraints.

Strategy options include:

• Fixed window strategy
• Fixed window custom counter strategy
• Concurrent strategy

Selecting a quota strategy

When selecting a quota strategy, consider that as an API consumer, your applications must align with the quota configuration defined by the API provider. For example, if the provider uses a Fixed Window rate-limiting strategy, your applications must align with these limits. Internal limits nested under a parent quota offer flexibility to structure them based on your specific traffic management needs.

The following matrix outlines recommended scenarios for each strategy:

Use Case	Recommended Strategy	Description
Steady traffic control	Fixed Window	Suitable for consistent rate-limiting needs, such as capping requests within a fixed time frame.
Burst handling	Fixed Window	Helps manage and contain traffic spikes within a defined window, preventing overload.
High concurrency	Concurrent	Limits the number of active requests, ideal for environments with high real-time demand.
Token-based API usage	Fixed Window Custom Counter	Controls requests and tokens to ensure efficient use of token-based APIs.
User-based quotas	Fixed Window	Groups requests by user-level headers, for example, x-lunar-consumer-tag, for differentiated user or subscription-tier quotas.
Service load balancing	Concurrent	Controls simultaneous connections to balance load across your resources dynamically.

Combining strategies with internal limits

In complex scenarios, such as managing quotas across different user levels or services, configure internal limits that apply both Fixed Window and Concurrent strategies within the same configuration file.

/etc/lunar-proxy/quotas/{fileName}.yaml

quotas:
  - id: CombinedQuota # Unique identifier for the main quota
    filter:
      url: api.website.com/* # URL pattern to apply the quota
    strategy:
      fixed_window: # Main quota strategy using Fixed Window
        max: 5000 # Maximum requests allowed in the main quota window
        interval: 1
        interval_unit: day
      group_by_header: x-lunar-consumer-tag # Optional grouping

# Nested concurrent limit for premium users within the main quota
internal_limits:
  - id: PremiumConcurrentLimit
    parent_id: CombinedQuota
    filter:
      headers:
        - key: x-lunar-consumer-tag
          value: premium
    strategy:
      concurrent:
        max_request_count: 100 # Max concurrent requests for premium users

The following configuration assigns a 5,000 daily request quota (CombinedQuota) to api.website.com/ grouped by the x-lunar-consumer-tag header via the Fixed Window strategy. An internal limit restricts concurrent execution for requests containing the x-lunar-consumer-tag: premium header, capping premium users at a maximum threshold of 100 simultaneous requests to the provider endpoint.