Caching policy

The Caching Gateway policy enhances API performance and reliability by caching responses to frequently accessed API requests. This policy reduces the load on API providers and improves response times for consumers. The Caching Gateway policy supports caching based on URL, query parameters, and specific headers.

Prerequisites

• Active LLM Gateway instance. Refer to Installation.
• Familiarity with Gateway policy structure and configuration template.

Scenarios

1. Reduced Latency: Serve responses faster by avoiding redundant API calls for frequently accessed data.
2. Cost Optimization: Reduce API call costs by minimizing requests to third-party providers.
3. Failover Support: Provide cached data when APIs are unavailable or experience downtime.
4. Header-based Differentiation: Enable cache rules based on headers for advanced scenarios like user-specific data or subscription tiers.

Gateway policy components

The Caching policy relies on two core processors:

• Read Cache Processor
• Write Cache Processor

Caching policy configuration scenario

The following configuration example demonstrates Read Cache and Write Cache processor definitions:

/etc/lunar-proxy/flows/caching-flow.yaml

name: CachingFlow

filter:
  url: httpbin.com/*
processors:
  ReadCache:
    processor: ReadCache 
    parameters:
      - key: caching_key_parts
        value:
          - $.request.headers.api_key
          - $.request.query_param.resource_id
  WriteCache:
    processor: WriteCache 
    parameters:
      - key: ttl_seconds
        value: 600 
      - key: record_max_size_bytes
        value: 8192 
      - key: max_cache_size_mb
        value: 200 
      - key: caching_key_parts
        value:
          - $.request.headers.api_key
          - $.request.query_param.resource_id
flow:
  request:
    - from:
        stream:
          name: globalStream
          at: start
      to:
        processor:
          name: ReadCache
    - from:
        processor:
          name: ReadCache
          condition: cache_miss
      to:
        stream:
          name: globalStream
          at: end
  response:
    - from:
        stream:
          name: globalStream
          at: start
      to:
        processor:
          name: WriteCache
    - from:
        processor:
          name: WriteCache
      to:
        stream:
          name: globalStream
          at: end
    - from:
        processor:
          name: ReadCache
          condition: cache_hit
      to:
        stream:
          name: globalStream
          at: end

Troubleshooting

Verify that caching_key_parts parameter values remain consistent between the Read Cache and Write Cache processors.