Quotas overview
API providers enforce quotas and rate limits to optimize resource distribution across customer applications. Provider-side API gateways enforce these limits. LLM Gateway applies consumer-side rate limits matching provider specifications, maximizing throughput efficiency within established constraints.
LLM Gateway quotas deliver centralized method for managing and configuring API quotas for the consumed APIs. The API provider defines these quotas, and LLM Gateway internal limits divide them into sub-quotas, enabling precise API consumption limits. Define the quota rules in YAML files under the quotas folder, shared across all Gateway policies for each API provider within an LLM Gateway, providing a scalable method to allocate and monitor API usage.
The quota system defines limits based on specific provider thresholds and internal limits that reflect the API consumption profile, such as monthly, daily, or other intervals. LLM Gateway updates quotas dynamically as consumption occurs, maintaining real-time management across Gateway policies.
Quotas track usage based on common provider limitation strategies. Combining quota rules with Gateway policies controls API access. Operating independently, quotas generate telemetry metrics only.
Quota file structure
The quota section defines the primary constraints on API usage by specifying rate-limiting strategies. This primary quota acts as the overarching rule for API requests, applying to all requests that match specified filters, for example, URLs or headers. The quota filter applies quotas conditionally based on endpoint patterns or request headers, adapting to different usage scenarios and customer types.
The internal_limits section sets more specific constraints within the main quota by defining nested internal limits that inherit from the main parent quota. Each internal limit can have its own filters and strategies, enabling precise control over distinct user groups or endpoints.
Benefits of LLM Gateway quota management
Key benefits of quota management include:
- Real-time API Usage Monitoring: Automatically updated as quotas are consumed, ensuring accurate real-time usage statistics across all consumers.
- Granular Control: Ability to set quotas based on URLs, headers, and user groups. Nested quotas allow more detailed control over specific API resources.
- Scalability: Easily manage and track quotas across multiple API Gateway policies and user environments.
Quota example
The following example shows a quota configuration defined in a quota.yaml file. In this example, api.website.com/* has a quota, MyQuota of 1000 requests per hour as set by the provider. You can choose to further limit this quota by implementing the LimitPerMinute internal limit, which allows only 50 requests per minute.
quotas:
- id: MyQuota
filter:
url: api.website.com/*
strategy:
fixed_window:
max: 1000
interval: 1
interval_unit: hour
internal_limits:
- id: LimitPerMinute
parent_id: MyQuota
strategy:
fixed_window:
max: 50
interval: 1
interval_unit: minute
For step-by-step guidance and technical specifications, refer to the following topics:
