Logo
Deploy Now

Categories

Developer Tools Machine Learning AI Gateway API Management AI Infrastructure Cost Optimization

Stars

56,856

Forks

10,764

Watchers

225

Developer links

Last Updated: 08-20-2026 09:57 PM (UTC)

LiteLLM

Backed by 56,000+ GitHub stars and over 240 million Docker pulls, LiteLLM delivers the open-source AI gateway trusted by Netflix, Lemonade, Rocket Money, and thousands of engineering teams to route every LLM request through one unified API. The Rust-core gateway adds sub-millisecond overhead per request with 8ms P95 latency at 1,000 RPS, 15x throughput improvement and 11x lower memory footprint compared to Python-only proxies. A single OpenAI-compatible endpoint connects to 100+ providers and 1,800+ models spanning OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Vertex AI, Hugging Face, vLLM, Nvidia NIM, Ollama, and Mistral with day-zero support for new model releases. The Auto Router V2 classifies request complexity across four tiers using rule-based scoring, semantic keyword matching, and adaptive Thompson sampling to route each request to the most cost-effective model without API calls or training data. Virtual API keys enable multi-tenant governance with per-team, per-user, and per-project cost tracking, budget caps with automatic fallback rerouting, and role-based access control. Built-in guardrails provide PII masking, prompt injection detection, and model-graded evaluation before requests reach providers. The Agent Gateway extends routing from model calls to agent workflows with MCP server integration. Observability integrates with Langfuse, Arize Phoenix, OpenTelemetry, and MLflow for complete request tracing. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

LiteLLM
LiteLLM
LiteLLM
LiteLLM
LiteLLM

Benefits

  • One API for 100+ Providers
  • Single OpenAI-compatible endpoint connects to OpenAI, Anthropic, Gemini, Bedrock, Azure, Vertex AI, vLLM, and 100+ more providers without changing application code or managing multiple SDKs.
  • Sub-Millisecond Gateway Overhead
  • Rust-core architecture delivers 8ms P95 latency at 1,000 RPS with 15x throughput and 11x less memory than Python alternatives, adding negligible overhead to model response times.
  • Multi-Tenant Cost Governance
  • Virtual API keys with per-team, per-user, and per-project budget tracking enable automatic fallback rerouting when spend caps are reached, preventing cost overruns across the organization.
  • Intelligent Auto-Routing Engine
  • Auto Router V2 classifies request complexity into four tiers using rule-based scoring, semantic matching, and adaptive Thompson sampling, routing to the cheapest capable model automatically.

Features

  • OpenAI-Compatible API
  • Drop-in replacement accepting OpenAI format requests with automatic translation to Anthropic, Gemini, Bedrock, Azure, Vertex AI, and 100+ provider-native formats.
  • Built-In Guardrails
  • PII masking, prompt injection detection, and model-graded evaluation run before requests reach providers, with configurable policies per virtual key and project.
  • Load Balancing and Failover
  • Automatic retry with fallback logic across multiple deployments, weighted routing, rate limit handling, and cooldown periods ensure continuous availability during provider outages.
  • Admin Dashboard UI
  • Web interface for managing virtual keys, monitoring spend per team and user, configuring models and deployments, viewing request logs, and testing routing configurations.
  • Agent and MCP Gateway
  • Extends LLM routing to agent workflows with A2A protocol support and MCP server integration, connecting any MCP tool to any model through the unified gateway.