Custom API quota
Custom API quotas enable dynamic quota allocation based on specific API call properties. Rather than counting raw HTTP requests to an API provider, the fixed_window_custom_counter strategy defines custom metrics for quota limits.
Applying the fixed_window_custom_counter strategy enables LLM Gateway to calculate the quota budget based on a custom attribute instead of per request. The counter_value_path specifies the attribute in your API response that determines quota usage.
You can use JSONPath to define where in the request the counter_value_path extracts the relevant quota value.
Scenarios
1. Quota management based on LLM tokens
Rather than limiting the number of requests to a Large Language Model (LLM) API, you can define a quota based on the total number of tokens used. This enables more granular control over consumption, especially when requests vary significantly in token count. LLM Gateway can track specific fields in the API response and enforce limits based on their sum, maintaining consumption within allocated token budgets.
2. Quota management based on Google Maps elements
For the Google Maps Distance Matrix API, you can manage quotas based on the number of elements in the request (combinations of origins and destinations). LLM Gateway extracts the number of origins and destinations from the request body and applies a custom quota that multiplies these values to represent the elements. This approach controls spending based on the complexity of routing queries, preventing unexpected cost overruns from large matrix calculations.
3. Quota management based on data transfer size
For APIs where data transfer volume is a significant cost factor, you can configure a custom quota that monitors the Content-Length header in the response. By setting limits on total bytes downloaded within a specific timeframe, you can optimize bandwidth usage and prevent exceeding data transfer limits. This is particularly useful for cloud storage or data analytics services where large data payloads are common.
Example: Custom quota based on tokens
quotas:
- id: GPT-4o-mini-quota
filter:
url: api.openai.com/v1/chat/completions
headers:
- key: x-lunar-custom-model
value: gpt-4o-mini
strategy:
fixed_window_custom_counter:
max: 40000
interval: 1
interval_unit: minute
counter_value_path: |
$.request.headers["x-lunar-used-tokens"]
OpenAI configuration scenario
The following configuration permits 10,000 requests per minute. Using the JSONPath expression $.response.body.usage["total_tokens"], LLM Gateway extracts 20 as the number of tokens used in a request. This means there are 9,987 tokens remaining in the quota.
quotas:
- id: GPT-4o-mini-quota
filter:
url: api.openai.com/v1/chat/completions
headers:
- key: x-lunar-custom-model
value: gpt-4o-mini
strategy:
fixed_window_custom_counter:
max: 10000
interval: 1
interval_unit: minute
counter_value_path: |
$.response.body.usage["total_tokens"]
{
"id": "chatcmpl-abc123",
"object": "chat.completion",
"created": 1677858242,
"model": "gpt-4o-mini",
"usage": {
"prompt_tokens": 13,
"completion_tokens": 7,
"total_tokens": 20,
"completion_tokens_details": {
"reasoning_tokens": 0,
"accepted_prediction_tokens": 0,
"rejected_prediction_tokens": 0
}
},
"choices": [
{
"message": {
"role": "assistant",
"content": "\n\nThis is a test!"
},
"logprobs": null,
"finish_reason": "stop",
"index": 0
}
]
}
With each response from OpenAI, the fixed_window_custom_counter calculates usage based on total_tokens and updates the remaining quota accordingly.
