LiteLLM gives applications one gateway in front of supported model providers. It is useful when you want a stable model alias instead of changing every client when a backend changes. This deliberately small, database-free example authenticates requests with a master key and forwards them to a private Ollama model.
Target: Ubuntu Server 24.04 LTS, x86-64, SSH and a sudo account. Documentation reviewed 15 November 2026. Commands have not been tested end to end on Ubuntu.
Before you start
Allow about 2 vCPUs, 4 GB RAM and 10 GB disk for a tiny local-model evaluation; increase resources for larger models. A gateway using external inference can need less model memory. This recipe does not include database-backed virtual keys, budgets or spend tracking, and must not be described as enforcing spending caps.
Resource figures are planning estimates for a small evaluation, not performance guarantees or plan specifications. A VPS/VDS does not imply a GPU. Check available memory, storage and resources before ordering.
On a fresh Ubuntu installation, use the packages below. If Docker is already installed from another repository, keep that installation and check compatibility first: do not mix Docker CE and docker.io packages. Use Compose v2; if the package is missing, check that Ubuntu Universe is enabled.
sudo apt update
sudo apt install -y docker.io docker-compose-v2 curl openssl
sudo systemctl enable --now docker
sudo docker compose version
All examples publish ports only on the server’s 127.0.0.1. Access them from your computer through an SSH tunnel; a domain is unnecessary. Do not open these ports to the Internet. An internal container listener on 0.0.0.0 does not change the host’s loopback-only published port.
Each project below is a new installation. Do not rerun secret-generation steps against an existing deployment.
1. Define the gateway and model alias
Create a private key file, save config.yaml and compose.yaml as shown. The backend is reachable only within this Compose network. The gateway exposes only local port 4000.
umask 077
mkdir -p ~/ai-litellm
cd ~/ai-litellm
printf 'LITELLM_MASTER_KEY=sk-%s\n' "$(openssl rand -hex 32)" > .env
model_list:
- model_name: local-small
litellm_params:
model: ollama_chat/qwen2.5:0.5b
api_base: http://ollama:11434
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
services:
ollama:
image: ollama/ollama:latest
restart: unless-stopped
environment:
OLLAMA_NUM_PARALLEL: "1"
OLLAMA_CONTEXT_LENGTH: "2048"
volumes:
- models:/root/.ollama
gateway:
image: docker.litellm.ai/berriai/litellm:latest
restart: unless-stopped
env_file: .env
command: ["--config", "/app/config.yaml", "--port", "4000"]
ports:
- "127.0.0.1:4000:4000"
volumes:
- ./config.yaml:/app/config.yaml:ro
depends_on:
- ollama
volumes:
models:
2. Start and verify authorization
Run these commands on the VPS. The first request intentionally has no key and should return HTTP 401. The authorized request should return JSON with a completion after the model loads. If the unauthenticated request succeeds, stop and correct the gateway configuration before using real prompts.
sudo docker compose up -d
sudo docker compose exec ollama ollama pull qwen2.5:0.5b
curl -s -o /dev/null -w '%{http_code}
' http://127.0.0.1:4000/v1/models
set -a
. ./.env
set +a
curl --fail-with-body --max-time 180 http://127.0.0.1:4000/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' -d '{"model":"local-small","messages":[{"role":"user","content":"Say hello in five words."}],"max_tokens":80}'
unset LITELLM_MASTER_KEY
3. Connect a client and plan the next stage
Use an SSH tunnel and the base URL http://127.0.0.1:4000/v1 on your computer, with model local-small. Keep the master key with the operator; do not distribute it to multiple untrusted users. For teams, add the supported database-backed virtual-key setup, review per-key scope and protect admin endpoints separately. External provider limits must be set at that provider; database-free gateway settings are not a reliable budget cap. Back up config and .env, plus a consistent database backup if one is added later.
ssh -N -L 127.0.0.1:4000:127.0.0.1:4000 YOUR_USER@YOUR_SERVER_IP
Updates and recovery
Before upgrading, stop the application for a consistent backup, keep data and secrets off the VPS, and test recovery on a separate instance. Moving image tags in these examples are for initial evaluation; pin a reviewed release tag or digest before ongoing use. Record the current digest with sudo docker image inspect IMAGE_NAME --format '{{json .RepoDigests}}'. Avoid unattended upgrades without compatibility checks, and do not delete persistent volumes with down -v.
If startup fails, inspect sudo docker compose ps, recent logs, free disk and memory. Do not share logs containing keys, prompts or personal data. For directory-permission errors, check the expected UID/GID and ownership instead of granting 777. Set up HTTPS, authentication and request limits before sharing access.