LLM Gateway FAQ

How is LLM Gateway different from API Gateways?

While both LLM Gateway and traditional API gateways serve as intermediaries in managing API traffic, they cater to different types of traffic and functionalities. API gateways primarily handle ingress traffic, which involves incoming requests directed to various backend services. They offer features like load balancing, error handling, throttling, caching, authentication, authorization, access control, and monitoring.

In contrast, LLM Gateway is specifically designed to manage egress traffic, which is the outgoing API requests from your infrastructure to external services. The Egress API-Gateway by LLM Gateway focuses on optimizing, monitoring, and securing these outbound requests. It allows clients to manage API integrations without requiring code changes, thanks to its distributed architecture that uses micro-proxies.

LLM Gateway provides full visibility into egress traffic, including encrypted traffic, by intercepting it before encryption and securing it with up-to-date TLS versions. This approach contrasts with the centralized architecture of traditional API gateways, making LLM Gateway an innovative solution for managing external API consumption effectively.

How is data protected from exposure in LLM Gateway?

LLM Gateway takes multiple steps to ensure data protection:

Data Protection: LLM Gateway ensures that no individual API requests, responses, or sensitive Personally Identifiable Information (PII) are routed to our SaaS infrastructure. All data processing happens within your cloud environment, protecting your sensitive information from exposure.

Security of the LLM Gateway Control Plane: The LLM Gateway Control Plane collects summary information such as the number of API requests, error rates, and total API calls for monitoring and analysis. This information helps in understanding usage patterns, bug identification, and troubleshooting while ensuring that no sensitive data is collected. Metadata, policy configurations, and system telemetry are securely transmitted to LLM Gateway's backend for support and debugging purposes.

Logs of Activity: LLM Gateway does not collect logs of individual requests and responses or their content. Instead, we collect logs from the LLM Gateway, such as error logs, system metrics, and configuration metadata, to help with monitoring and troubleshooting. This approach ensures that detailed activity logs containing sensitive data are not stored or processed by LLM Gateway.

What is the traffic capacity of each LLM Gateway?

LLM Gateway s demonstrate impressive handling of substantial workloads, achieving high requests per second (RPS) depending on the concurrency and core limits. Here are the benchmark results for different scenarios:

• With 32 concurrent connections and 1 core: 9,050.42 RPS for 2,500,000 total requests.
• With 64 concurrent connections and 2 cores: 16,550.48 RPS for 5,000,000 total requests.
• With 128 concurrent connections and 4 cores: 35,896.17 RPS for 10,000,000 total requests.
• With 256 concurrent connections and 8 cores: 84,867.79 RPS for 20,000,000 total requests.

These results indicate that LLM Gateway s can efficiently manage high traffic volumes, scaling effectively with increased concurrency and core allocation.

Is LLM Gateway built on an external solution?

Yes, the LLM Gateway is built on an external solution, specifically, HAProxy Project serves as the foundation for the LLM Gateway . The LLM Gateway enhances this foundation using Golang for best performance, providing sophisticated API intermediation, policy enforcement, and optimization for interactions with external API providers.

How much latency does LLM Gateway introduce?

The LLM Gateway introduces a latency impact of up to 4ms at the 95th percentile and up to 9 ms at the 99th percentile. This minimal latency is considered negligible, especially when balanced against the benefits of using LLM Gateway for API calls.

What happens if we cannot access LLM Gateway? Will the API calls still go through?

If access to the LLM Gateway is not possible, the API calls will still go through. The LLM Gateway includes a configurable fail-safe mechanism that ensures uninterrupted traffic flow. If the Gateway fails or exceeds timeouts, LLM Gateway will directly make the API call to the API provider, thereby preventing any data loss. This fail-safe mechanism includes a cooldown period after a set number of failed connection attempts, during which all traffic is directed to the original provider. This process ensures that there is no downtime or single point of failure and efficiently managing high traffic volumes, scaling effectively with increased concurrency and core allocation.

How will I know which responses were generated by LLM Gateway?

Any responses generated by LLM Gateway using the GenerateResponse Processor will have the header x-lunar-generated: true added to them. This is a custom header that LLM Gateway generates and can used to differentiate between responses generated by LLM Gateway and your API Provider.