System requirements

Supported operating systems

The LLM Gateway is compatible with both Linux and macOS, supporting CPU architectures including x86_64 and ARM64.

Supported OS distributions

LLM Gateway is compatible with the following Linux distributions:

• AlmaLinux 8, 9
• Debian 11, 12
• Oracle Linux 8, 9
• RHEL 8, 9
• Rocky Linux 8, 9
• SUSE 15.5
• Ubuntu 20.04, 22.04, 24.04

Hardware recommendations

The hardware requirements for the LLM Gateway vary based on the workload it manages. Only CPU and memory are factored in, while disk size depends on your operating system and log retention needs. The recommendations below are for guidance only; please contact us for help in accurately sizing your servers.

Low-Level workload

• Traffic Type: TCP or HTTP
• Connections per Second: Up to 1,000
• SSL Traffic: Very low
• Requirements:
  • 1 CPU core
  • 1 GB of RAM

Mid-Level workload

• Traffic Type: TCP or HTTP (including HTTP manipulation)
• Connections per Second: Up to 4,000
• SSL Traffic: Low
• Requirements:
  • 2 CPU cores
  • 2 GB of RAM

High-Level workload

• Traffic Type: TCP or HTTP (including HTTP manipulation)
• Connections per Second: Up to 20,000
• SSL Traffic: 10% of traffic ciphered
• Requirements:
  • 2 CPU cores (as fast as possible)
  • 4 GB of RAM
  • A powerful network card

Memory and CPU recommendations

Based on our testing with Apache Benchmark, we recommend the following resource limits for optimal performance during high traffic scenarios:

• Memory Limit: 4 GB (recommended)
• CPU Cores: 2 (recommended)

This aligns with HAProxy's guidance for high-traffic environments, which calls for a minimum of 1 GB and up to 4 GB of memory to support high connection rates.

A lower limit of 2 GB and 1 CPU core may be adequate, but the higher limits ensure stability and performance during traffic spikes.