llama.cpp is a model runtime with a lightweight server and support for quantized GGUF weights. It is a useful alternative when you want explicit control over the model file, CPU threads and context size. Here we expose an authenticated API through SSH, using a model file you have independently selected and verified.
Target: Ubuntu Server 24.04 LTS, x86-64, SSH and a sudo account. Documentation reviewed 15 November 2026. Commands have not been tested end to end on Ubuntu.
Before you start
For a small 0.5–1.5B quantized text model, budget about 2–4 vCPUs, 4 GB RAM and 10 GB free disk, then measure. Larger weights and contexts require more. A compatible instruction-tuned GGUF file and its publisher’s checksum/license are prerequisites; do not treat arbitrary renamed model files as compatible.
Resource figures are planning estimates for a small evaluation, not performance guarantees or plan specifications. A VPS/VDS does not imply a GPU. Check available memory, storage and resources before ordering.
On a fresh Ubuntu installation, use the packages below. If Docker is already installed from another repository, keep that installation and check compatibility first: do not mix Docker CE and docker.io packages. Use Compose v2; if the package is missing, check that Ubuntu Universe is enabled.
sudo apt update
sudo apt install -y docker.io docker-compose-v2 curl openssl
sudo systemctl enable --now docker
sudo docker compose version
All examples publish ports only on the server’s 127.0.0.1. Access them from your computer through an SSH tunnel; a domain is unnecessary. Do not open these ports to the Internet. An internal container listener on 0.0.0.0 does not change the host’s loopback-only published port.
Each project below is a new installation. Do not rerun secret-generation steps against an existing deployment.
1. Prepare a verified model file
Create the project. On your computer, copy the selected GGUF to the VPS as shown, replacing every placeholder. On the VPS compare sha256sum models/model.gguf with the trusted publisher’s value before starting it. Do not run remote model-repository code to load a GGUF.
umask 077
mkdir -p ~/ai-llama/models
cd ~/ai-llama
openssl rand -hex 32 > api-key.txt
scp /PATH/TO/SELECTED.gguf YOUR_USER@YOUR_SERVER_IP:~/ai-llama/models/model.gguf
2. Define a bounded CPU server
Save compose.yaml. The model and API-key files are read-only in the container. --alias supplies a stable client-visible name; two threads and one parallel slot are conservative evaluation choices, not tuning advice for every plan.
services:
llama:
image: ghcr.io/ggml-org/llama.cpp:server
restart: unless-stopped
ports:
- "127.0.0.1:8080:8080"
volumes:
- ./models:/models:ro
- ./api-key.txt:/run/llama-api-key:ro
command: ["-m", "/models/model.gguf", "--alias", "local-gguf", "--host", "0.0.0.0", "--port", "8080", "--ctx-size", "2048", "--threads", "2", "--parallel", "1", "--api-key-file", "/run/llama-api-key"]
3. Verify loading, authentication and a response
Run on the VPS after the checksum check. /health can be public and return 503 while loading; that endpoint is not an authentication test. The /v1/models request without a key should return 401. Keep prompts short initially and inspect the logs if the model’s chat template is unsupported.
sha256sum models/model.gguf
sudo docker compose up -d
curl -i http://127.0.0.1:8080/health
curl -s -o /dev/null -w '%{http_code}
' http://127.0.0.1:8080/v1/models
LLAMA_KEY=$(cat api-key.txt)
curl --fail-with-body --max-time 180 http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $LLAMA_KEY" -H 'Content-Type: application/json' -d '{"model":"local-gguf","messages":[{"role":"user","content":"Explain SSH in one sentence."}],"max_tokens":80}'
unset LLAMA_KEY
4. Connect and preserve reproducibility
Tunnel from your computer and use http://127.0.0.1:8080/v1 with the same API key and alias. Record model provenance, checksum, quantization, license and container digest. A model that fits in RAM can still be slow on shared CPU; compare measured latency before increasing concurrency. Copy the exact weights or retain a verified way to obtain them again, and back up the key/configuration.
ssh -N -L 127.0.0.1:8080:127.0.0.1:8080 YOUR_USER@YOUR_SERVER_IP
Updates and recovery
Before upgrading, stop the application for a consistent backup, keep data and secrets off the VPS, and test recovery on a separate instance. Moving image tags in these examples are for initial evaluation; pin a reviewed release tag or digest before ongoing use. Record the current digest with sudo docker image inspect IMAGE_NAME --format '{{json .RepoDigests}}'. Avoid unattended upgrades without compatibility checks, and do not delete persistent volumes with down -v.
If startup fails, inspect sudo docker compose ps, recent logs, free disk and memory. Do not share logs containing keys, prompts or personal data. For directory-permission errors, check the expected UID/GID and ownership instead of granting 777. Set up HTTPS, authentication and request limits before sharing access.