Failsafe mechanisms in LLM Gateway
LLM Gateway Failsafe Mechanisms help guarantee that your application remains operational, ensuring uninterrupted API traffic even when unexpected issues arise.
In the event that the LLM Gateway fails to respond the Failsafe Mechanism automatically reroutes traffic directly to the API provider. This minimizes potential delays and avoids the risk of downtime, keeping your system running smoothly under all conditions.
By default, the Failsafe Mechanism is enabled, providing a safety net that ensures any failure between the Interceptor or Gateway, doesn’t result in traffic loss or service disruption.
Additionally, the mechanism includes customizable settings like cooldown periods and retry attempts, allowing you to adjust the behavior to match your system's resilience needs.
This approach ensures that traffic continues flowing smoothly, even in the event of system issues, making it an essential feature for high-reliability environments where consistent uptime and performance are critical.
Gateway failsafe mechanism
LLM Gateway's Failsafe Mechanism is designed to ensure continuous traffic flow, even if something goes wrong with the LLM Gateway Engine. In the unlikely event that the LLM Gateway Engine crashes, all traffic will be automatically routed directly to the Provider, ensuring that your application's operation remains unaffected.
This approach guarantees uninterrupted traffic flow, minimizing any potential delays or disruptions.
By default, the Failsafe Mechanism is enabled in the LLM Gateway Engine.
To disable it, set the following environment variable:
LUNAR_ENGINE_FAILSAFE_ENABLED="false"
If not configured the Failsafe Mechanism will load with the following default values
LUNAR_ENTER_COOLDOWN_AFTER_ATTEMPTS="5"
LUNAR_EXIT_COOLDOWN_AFTER_SEC="10"
