AI Gateway
Open-Source Multi-Provider AI Orchestration Service
Developers integrating AI face single-point-of-failure risks, frequent provider rate limits (429), regional outages, and fragmented API schemas requiring custom adapters for each AI company.
Must support multiple providers seamlessly under a single endpoint, execute automatic failovers with zero downtime, enforce per-IP rate limiting, and maintain OpenAI API format compatibility.
Architected an open-source AI orchestration gateway providing one unified endpoint for 9 AI providers (Gemini, Groq, Mistral, OpenRouter, OpenCodeZen, Zydit, Zai, Agnes, Nvidia). Features automatic failover cascades, provider load balancing, API token authentication, and a 15 requests/minute per-IP rate limiter.
AI Gateway: Multi-Provider Failover Cascading
Unified OpenAI-compatible proxy routing across 9 LLM providers with automatic 429 failover
Incoming inference requests hit the Fastify dynamic proxy, verified against token auth and a 15 req/min rate limiter. The engine attempts the primary model provider; on HTTP 429 rate limits, timeouts, or quota blocks, it cascades seamlessly to alternative models (Gemini, Groq, Mistral, OpenRouter, Nvidia) with zero client disruption.
Standardized heterogeneous request/response formats across 9 AI providers behind a single OpenAI-compatible /v1/chat/completions endpoint.
When an upstream model encounters timeouts, 429 rate limits, or network errors, the engine cascades dynamically to alternative active providers.
Protected the gateway with secret token validation and an in-memory 15 req/min per-IP rate limiter to prevent abuse and API quota exhaustion.