Async queue policy

The Async Queue Gateway policy provides robust asynchronous communication using LLM Gateway. LLM Gateway handles requests asynchronously and delivers responses once they are ready, decoupling the client from the timing of the response.

note

• Dedicated Port: The Async Queue Gateway policy exposes port 8010 for submission and retrieval, leaving the standard port 8000 available.
• Submission Response: Upon submission, you receive an immediate 202 Accepted response. This response includes a Location header for retrieving the result, and the x-lunar-enqueued: true header, which confirms the job is queued.
• Retrieval Processing: The retrieval process fetches a response from lunar::steve::response::{seq-id}, and materializes it as a real HTTP response. Redis sets an expiration time (TTL) to automatically delete saved responses.
• Pending Status: If the response is pending (key not found), LLM Gateway returns a 202 Accepted with the x-lunar-enqueued: true header again.

Prerequisites

• Active LLM Gateway instance. Refer to Installation.
• Familiarity with Gateway policy structure and configuration template.
• A configured Fixed Window Quota for the traffic you want to queue.

Scenarios

• Long queue execution: Instead of keeping connections open during long queue waits, LLM Gateway closes the request and notifies the client when the response is ready.

Gateway policy components

• Fixed Window Quota
• Async Retry Processor
• Async Queue Processor

Async queue policy configuration scenario

The following configuration example demonstrates an async queue policy with retry and status filtering processors:

/etc/lunar-proxy/flows/flow.yaml

name: AsyncPriorityQueueFlow

filter:
  url: test.org/*

processors:
  Queue:
    processor: AsyncQueue
    parameters:
      - key: quota_id
        value: test.org_PrimaryQuota

  RetryProcessor:
    processor: AsyncRetry
    parameters:
      - key: attempts
        value: 3
      - key: cooldown_between_attempts_seconds
        value: 10
    metrics:
      enabled: true
      labels:
        - flow_name
        - processor_key
        - http_method
        - host
        - status_code
        - consumer_tag
  FilterProcessor:
    processor: Filter
    parameters:
      - key: status_code_range
        value: "400-500"
    metrics:
      enabled: true
      labels:
        - flow_name
        - processor_key
        - http_method
        - host
        - status_code
        - consumer_tag

flow:
  request:
    - from:
        stream:
          name: globalStream
          at: start
      to:
        processor:
          name: Queue

    - from:
        processor:
          name: Queue
      to:
        stream:
          name: globalStream
          at: end

  response:
    - from:
        stream:
          name: globalStream
          at: start
      to:
        processor:
          name: FilterProcessor
    - from: 
        processor:
          name: FilterProcessor
          condition: miss
      to:
        stream:
          name: globalStream
          at: end

    - from:
        processor:
          name: FilterProcessor
          condition: hit
      to:
        processor:
          name: RetryProcessor

    - from:
        processor:
          name: RetryProcessor
          condition: retry
        
      to:
        stream:
          name: globalStream
          at: end

    - from:
        processor:
          name: RetryProcessor
          condition: failed
      to:
        stream:
          name: globalStream
          at: end

The following quota configuration defines the target primary quota referenced by the async queue policy:

/etc/lunar-proxy/quota/quota.yaml

quotas:
  - id: test.org_PrimaryQuota
    filter:
      url: test.org/*
    strategy:
      fixed_window:
        static:
          max: 100
          interval: 1
          interval_unit: minute

Troubleshooting

• Pending response calls: Calling /retrieve before response completion returns the initial 202 Accepted payload.
• Port verification: 8010 returns 202 Accepted with x-lunar-enqueued: true to confirm processing by LLM Gateway rather than the upstream provider. Retrieve response payloads via the Location header.