Priority queue policy

The Priority Queue policy prioritizes API requests, handling high-priority traffic while maintaining smooth processing for lower-priority requests. The Queue processor controls the number of requests in the queue and assigns priorities using the x-lunar-consumer-tag header. Requests remain queued until slot availability opens or the time-to-live (TTL) expires. If the queue is full or limits are exceeded, the policy generates a 429 Too Many Requests response.

This policy optimizes traffic load, prioritizes critical requests, and balances resource consumption.

Prerequisites

• Active LLM Gateway instance. Refer to Installation.
• Familiarity with Gateway policy structure and configuration template.
• Configured Fixed Window Quota for the traffic you want to queue.

Scenarios

1. Critical Request Prioritization: Processes high-priority requests, such as production traffic, ahead of lower-priority requests in mixed traffic environments.
2. Preventing Overload with Queuing: Delays lower-priority requests in a queue to manage traffic without immediately generating Too Many Requests responses, unless the queue is full or TTL expires.
3. Handling High-Traffic APIs: Manages APIs with mixed-priority requests, balancing both staging and production traffic efficiently.
4. Fair Resource Allocation: Assigns priority levels to different requests to balance resource usage and prevent low-priority traffic from impacting high-priority operations.
5. Custom Error Handling: Configures responses for exceeding limits to ensure users are informed of delays or retries in a controlled manner.

Gateway policy components

Fixed Window Quota

Priority queue policy configuration scenario

The following configuration example queues API requests to https://api.website.com/ up to 10 requests per second based on the x-lunar-consumer-tag header, where production traffic gets higher priority (1) over staging traffic (2). If the queue is full or a request is delayed beyond the defined TTL (12 seconds), the system returns a 429 HTTP status code with the message: Too many requests. Please try again later.

/etc/lunar-proxy/flows/flow.yaml

name: PriorityQueueFlow

filter:
  url: api.website.com/*

processors:
  QueueLimiter:
    processor: Queue
    parameters:
      - key: quota_id
        value: MyQuota
      - key: ttl_seconds
        value: 12
      - key: queue_size
        value: 10
      - key: priority_group_by_header
        value: x-lunar-consumer-tag
      - key: priority_groups
        value:
          production: 1
          staging: 2
  GenerateResponseTooManyRequests:
    processor: GenerateResponse
    parameters:
      - key: status
        value: 429
      - key: body
        value: "Too many requests. Please try again later."
      - key: Content-Type
        value: text/plain
flow:
  request:
    - from:
        stream:
          name: globalStream
          at: start
      to:
        processor:
          name: QueueLimiter
    - from:
        processor:
          name: QueueLimiter
          condition: blocked
      to:
        processor:
          name: GenerateResponseTooManyRequests
    - from:
        processor:
          name: QueueLimiter
          condition: allowed
      to:
        stream:
          name: globalStream
          at: end
  response:
    - from:
        processor:
          name: GenerateResponseTooManyRequests
      to:
        stream:
          name: globalStream
          at: end

The following quota configuration defines the target primary quota referenced by the priority queue:

/etc/lunar-proxy/quotas/quota.yaml

quotas:
  - id: MyQuota
    filter:
      url: api.website.com/*
    strategy:
      fixed_window:
        static:
          max: 10
          interval: 1
          interval_unit: second