Retry policy

The Retry Gateway policy automatically handles retries for failed API requests based on configurable criteria, such as status codes and retry strategies. It is designed to enhance API reliability by managing temporary failures without requiring client-side retry logic.

Prerequisites

• Active LLM Gateway instance. Refer to Installation.
• Familiarity with Gateway policy structure and configuration templates.

Scenarios

1. Handling Temporary API Failures: Automatically retry failed requests, reducing manual error handling.
2. Improving Reliability: Manage network issues and server downtime with exponential backoff retries.
3. Enhanced Monitoring: Track retry attempts, cooldowns, and related performance metrics.

Gateway policy components

• Retry Processor
• Filter Processor

Retry policy configuration scenario

The following configuration example is a fully configured Retry Gateway policy that retries failed API requests based on status codes, with exponential backoff.

/etc/lunar-proxy/flows/flow.yaml

name: RetryFlow

filter:
  url: "api.com/resource/{id}"
processors:
  RetryProcessor:
    processor: Retry
    parameters:
      - key: attempts
        value: 5
      - key: cooldown_between_attempts_seconds
        value: 5
      - key: cooldown_multiplier
        value: 1.5
      - key: maximum_cooldown_seconds
        value: 3600
  FilterProcessor:
    processor: Filter
    parameters:
      - key: status_code_range
        value: "400-500"
flow:
  request:
    - from:
        stream:
          name: globalStream
          at: start
      to:
        stream:
          name: globalStream
          at: end
  response:
    - from:
        stream:
          name: globalStream
          at: start
      to:
        processor:
          name: FilterProcessor
    - from: 
        processor:
          name: FilterProcessor
          condition: miss
      to:
        stream:
          name: globalStream
          at: end
    - from:
        processor:
          name: FilterProcessor
          condition: hit
      to:
        processor:
          name: RetryProcessor
    - from:
        processor:
          name: RetryProcessor
          condition: retry
      to:
        stream:
          name: globalStream
          at: end
    - from:
        processor:
          name: RetryProcessor
          condition: failed
      to:
        stream:
          name: globalStream
          at: end

Troubleshooting

1. Retries Not Triggering:
   • Verify that status_code_range is correctly configured.
   • Ensure the API response status code falls within the specified range.
2. Excessive Retry Attempts:
   • Check the attempts parameter and reduce it if necessary.
   • Adjust the cooldown_multiplier to increase wait times between retries.
   • Make sure that in case of long cooldowns the value of LUNAR_RETRY_REQUEST_TIMEOUT_SEC is also increased. The default is 100.