If your application talks to more than one LLM provider, you already know the pain: OpenAI wants messages, some SDKs want prompt, error types differ, streaming shapes differ, and every provider swap turns into a small refactor. LiteLLM exists to make that problem disappear. It gives you a single, unified interface to call 100-plus LLM providers — OpenAI, Anthropic, Vertex AI, Bedrock, Azure, Ollama, and more — all in the OpenAI format.
This guide walks through both ways to use it: the Python SDK for dropping straight into your code, and the Proxy (LLM Gateway) for teams that need one controlled endpoint in front of every model. It's an intermediate-level setup — you should be comfortable with Python, environment variables, and a terminal.
Why a gateway instead of raw SDKs
The pitch is simple: write your code once, swap models with a string. Instead of importing five vendor SDKs and maintaining five code paths, you call completion() and change the model argument. LiteLLM normalizes the request going out and the response coming back, so response.choices[0].message.content works the same whether you hit GPT-4o or Claude.
That normalization buys you three things that matter in production: portability (no lock-in to one vendor's SDK), resilience (automatic retries and fallbacks across deployments), and observability (cost tracking and logging hooks in one place). The project is open source and, as of 2026, sits around 40,000 GitHub stars with weekly releases.
Install it
The official docs now recommend uv, but pip works exactly the same. For the SDK alone:
pip install litellm
For the full Proxy Server (LLM Gateway), pull the proxy extras, which include FastAPI and its dependencies:
pip install 'litellm[proxy]'
If you use uv, the equivalents are uv add litellm and uv tool install 'litellm[proxy]'.
The SDK: one function for every model
Here's the entire integration. Set an API key, call completion(), read the response in OpenAI format:
from litellm import completion
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"
response = completion(
model="openai/gpt-4o",
messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.choices[0].message.content)
Now switch providers. The only thing that changes is the model string and the relevant API key — the rest of your code is untouched:
os.environ["ANTHROPIC_API_KEY"] = "your-key"
response = completion(
model="anthropic/claude-sonnet-4-20250514",
messages=[{"role": "user", "content": "Hello, how are you?"}]
)
The provider/model convention is the whole routing system. openai/gpt-4o, anthropic/..., vertex_ai/..., bedrock/..., ollama/... — LiteLLM reads the prefix and routes accordingly.
Streaming works identically
Add stream=True and iterate over chunks. No provider-specific streaming code:
for chunk in completion(
model="openai/gpt-4o",
messages=[{"role": "user", "content": "Write a short poem"}],
stream=True,
):
print(chunk.choices[0].delta.content or "", end="")
Errors map to OpenAI exceptions
This is the underrated feature. LiteLLM maps every provider's errors onto the standard OpenAI exception types, so one try/except block covers all of them:
import litellm
try:
litellm.completion(
model="anthropic/claude-instant-1",
messages=[{"role": "user", "content": "Hey!"}]
)
except litellm.AuthenticationError as e:
print(f"Bad API key: {e}")
except litellm.RateLimitError as e:
print(f"Rate limited: {e}")
except litellm.APIError as e:
print(f"API error: {e}")
Cost tracking and observability in one line
Wire success callbacks to send input/output to Langfuse, MLflow, Helicone, and others, or capture cost per response yourself:
import litellm
litellm.success_callback = ["langfuse", "mlflow"]
def track_cost(kwargs, completion_response, start_time, end_time):
print("Cost:", kwargs.get("response_cost", 0))
litellm.success_callback.append(track_cost)
The Proxy: one endpoint for a whole team
The SDK is perfect for a single app. But when a platform team needs to manage LLM access across an organization — budgets, keys, logging, guardrails — you want the Proxy Server, a self-hosted, OpenAI-compatible gateway. Any client that already speaks OpenAI works against it with zero code changes.
Start it against a model directly:
litellm --model huggingface/bigcode/starcoder
# Proxy running on http://0.0.0.0:4000
Then point any OpenAI client at it. Note the base_url — that's the only change from a normal OpenAI call:
import openai
client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Write a short poem"}]
)
print(response.choices[0].message.content)
For anything beyond a demo, use a config file to declare your models and pull secrets from the environment:
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: azure/your-deployment
api_base: os.environ/AZURE_API_BASE
api_key: os.environ/AZURE_API_KEY
api_version: "2023-07-01-preview"
On top of that, the proxy layers the features that make it a real gateway: virtual keys with per-key, per-team, and per-user budgets; centralized spend tracking across every provider; guardrails for content filtering and PII masking; caching; and an admin UI. For production, the docs recommend running the Docker image with Docker Compose and PostgreSQL.
Fallbacks and load balancing with the Router
The single biggest reason to standardize on LiteLLM isn't convenience — it's resilience. Providers rate-limit you, have outages, and deprecate models on their own schedule. The LiteLLM Router turns those failures into non-events by spreading traffic across multiple deployments and automatically retrying elsewhere when one fails.
You define several entries under the same model_name — say, the same logical model deployed in two Azure regions plus an OpenAI backup — and the Router load-balances across them:
from litellm import Router
router = Router(model_list=[
{"model_name": "gpt-4o",
"litellm_params": {"model": "azure/gpt-4o-east",
"api_base": "https://east.openai.azure.com/",
"api_key": "os.environ/AZURE_EAST_KEY"}},
{"model_name": "gpt-4o",
"litellm_params": {"model": "azure/gpt-4o-west",
"api_base": "https://west.openai.azure.com/",
"api_key": "os.environ/AZURE_WEST_KEY"}},
])
response = router.completion(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}]
)
Callers just ask for gpt-4o; the Router picks a healthy deployment. Layer fallbacks on top and a failure on one provider silently reroutes to another — for example, falling back from an Azure deployment to Anthropic when OpenAI is down — so a single provider's bad afternoon doesn't take your app down with it. The same behavior is available declaratively in the Proxy through its router settings, which means platform teams get failover without every service reimplementing retry logic.
Common pitfalls
A few things trip up newcomers. Provider prefixes are mandatory for anything ambiguous — gpt-4o may resolve, but openai/gpt-4o and azure/gpt-4o are unambiguous, so prefer the explicit form. Each provider still needs its own environment variable (OPENAI_API_KEY, ANTHROPIC_API_KEY, and so on); LiteLLM unifies the interface, not the credentials. And when you move to the Proxy, remember the client api_key is the virtual key you issue from the gateway, not the underlying provider key — that indirection is the entire point of centralized budgets and access control.
When to use which
Reach for the SDK when you're building an application and want a clean, portable abstraction over model providers inside your own process. Reach for the Proxy when multiple apps or teams need to share access through one governed endpoint, and you care about budgets, keys, and centralized logging. Many teams run both — the SDK in services, the proxy as the org-wide control plane. Notably, LiteLLM also now doubles as an MCP and A2A gateway, so the same endpoint fronts models, agents, and tools.
The Bottom Line
LiteLLM's value is that it turns "which provider are we on?" from an architectural decision into a configuration detail. The SDK gets you provider-agnostic code in about five lines; the proxy turns that into a governed, observable gateway for a whole team. If you're calling more than one model — or expect to — standardizing on the OpenAI format through LiteLLM is one of the cheapest pieces of future-proofing you can buy. Start with pip install litellm, swap a model string, and you'll see the point immediately.
