Latency footprint

Our benchmarking aimed to reveal the latency footprint of LLM Gateway on response time percentiles compared to direct API calls made without LLM Gateway. We selected a provider with a constant response time of 150ms, which is relatively fast for a web-based API.

To accomplish this, we explore the following scenarios:

• Traffic passes directly to the API provider without any intermediary. (referred to as "direct")
• Traffic passes through LLM Gateway without any flows enabled, acting as a simple proxy. (referred to as "passthrough")
• Traffic passes through LLM Gateway while employing a simple flow that doesn't short circuit requests. (referred to as "with-active-flow")

We allocated three different AWS EC2 instances of type c4a.large for this purpose - one for the client application making API requests, one dedicated to LLM Gateway, and another one dedicated to the API provider.

Setup

We used Apache AB, to simulate client-side behavior and gather necessary metrics. To replicate how client applications interact with LLM Gateway, we directed Apache AB to call LLM Gateway, which, in turn, forwarded the requests to the Provider.

On the provider side, we leveraged go-httpbin, a Docker image that serves as a Go version of httpbin.org.

Benchmark results

In our performance analysis, we conducted three experiments to compare the following scenarios:

• Calls directly to the provider (referred to as "direct").
• Calls to the provider via LLM Gateway Proxy without any active flow (referred to as "passthrough").
• Calls to the provider via LLM Gateway Proxy with an active flow (referred to as "with-active-flow").

The visualization below presents percentile values on the X-axis and runtime in milliseconds on the Y-axis, highlighting the differences between each experiment and the baseline direct experiment, which are relatively small across percentiles.

To provide a clearer understanding, let's examine the same graph with the Y-axis representing the delta from the baseline for each experiment and percentile. In the direct experiment, the delta remains 0 for all percentiles, as it is compared against itself.

• At the 99th percentile, LLM Gateway Proxy adds around 4ms (with passthrough and with an active flow) to the overall response time of HTTP calls.