Fallback policy

The Fallback Gateway policy enables users to retry requests to an LLM provider and dynamically switch models based on user-defined criteria. These conditions are evaluated using the Filter Processor, which provides granular control over when a fallback should be triggered. For example, if a request to GPT-4 times out or returns an error, the system can automatically retry the request using GPT-3.5.

Prerequisites

• Active LLM Gateway instance. Refer to Installation.
• Familiarity with Gateway policy structure and configuration template.

Scenarios

1. LLM Rate Limit Reached: If a request returns a specific response code, the Fallback Gateway policy retries it using a different LLM model instead of waiting for the original model to become available again.

Gateway policy components

• Filter Processor
• Retry Processor
• Transform API Call Processor

Gateway policy example

Here is an example of a fully configured Fallback Flow. In this case, if a request to OpenAI Model GPT-4.5 returns a status code in the range 429–504, the request will be transformed and resent to OpenAI Model GPT-4.1.

/etc/lunar-proxy/flows/flow.yaml

name: OpenAIModelFallbackFlow
filter: 
  url: api.openai.com/*

processors: 
  DidRetryFilter: 
    processor: Filter
    parameters: 
      - key: headers
        value: 
          x-lunar-retry-attempt: "true"
  RetryProcessor: 
    processor: Retry
    parameters: 
      - key: attempts
        value: 1
  ShouldRetryFilter: 
    processor: Filter
    parameters: 
      - key: status_code_range
        value: "429-504"
      - key: expressions
        value: 
          - $.request[?(@.body.model == "gpt-4.5")]
  TransformProcessor: 
    processor: TransformAPICall
    parameters: 
      - key: set
        value: 
          $.request.body.model: gpt-4.1
flow: 
  request: 
    - from: 
        stream: 
          name: globalStream
          at: start
      to: 
        processor: 
          name: DidRetryFilter
    - from: 
        processor: 
          name: DidRetryFilter
          condition: miss
      to: 
        stream: 
          name: globalStream
          at: end
    - from: 
        processor: 
          name: DidRetryFilter
          condition: hit
      to: 
        processor: 
          name: TransformProcessor
    - from: 
        processor: 
          name: TransformProcessor
      to: 
        stream: 
          name: globalStream
          at: end
  response: 
    - from: 
        stream: 
          name: globalStream
          at: start
      to: 
        processor: 
          name: ShouldRetryFilter
    - from: 
        processor: 
          name: ShouldRetryFilter
          condition: miss
      to: 
        stream: 
          name: globalStream
          at: end
    - from: 
        processor: 
          name: ShouldRetryFilter
          condition: hit
      to: 
        processor: 
          name: RetryProcessor
    - from: 
        processor: 
          name: RetryProcessor
          condition: retry
      to: 
        stream: 
          name: globalStream
          at: end
    - from: 
        processor: 
          name: RetryProcessor
          condition: failed
      to: 
        stream: 
          name: globalStream
          at: end