Capacity benchmark
In our benchmarking sessions, We evaluated how LLM Gateway handles requests per second under various loads. Using an EKS cluster with a reliable hardware configuration, we generated traffic to measure performance. By adjusting parameters like concurrent connections and request rates, we assessed LLM Gateway's efficiency, identified bottlenecks, and found areas for improvement in handling large workloads.
Benchmark results
The results demonstrate the requests per second achieved by LLM Gateway in each scenario, providing insights into its performance characteristics. With multiple requests ranging from 32 to 256, and capacity limits from 1 to 8 cores, we observed a clear correlation between these factors and the resulting requests per second. These findings contribute to a deeper understanding of LLM Gateway's performance and can assist in optimizing its configuration for enhanced capacity and throughput.
The table below summarizes the results across each combination of concurrency, capacity limit, request volume, and the resulting performance:
| Concurrency | Capacity Limit (Cores) | Number of Requests | Requests per Second |
|---|---|---|---|
| 32 | 1 | 2,500,000 | 9,050.42 |
| 64 | 2 | 5,000,000 | 16,550.48 |
| 128 | 4 | 10,000,000 | 35,896.17 |
| 256 | 8 | 20,000,000 | 84,867.79 |
- Concurrency: The number of requests sent at the same time.
- Capacity Limit (Cores): The number of cores allocated to LLM Gateway's capacity limit.
- Number of Requests: The total number of requests performed in the benchmark.
- Requests per Second: The average number of requests LLM Gateway's Proxy handles per second.
