Customer portal
Website
← Knowledge Base

Run a small language model on a VPS with Ollama

Ollama downloads and serves language models behind a simple local API. A CPU VPS can be useful for learning, lightweight text experiments and private prototypes. It will not make a large model fast simply because the service starts. We use Qwen2.5 0.5B as a small reproducible smoke test, not as the best current model for every task.

Target: Ubuntu Server 24.04 LTS, x86-64, SSH and a sudo account. Documentation reviewed 15 November 2026. Commands have not been tested end to end on Ubuntu.

Before you start

Start an evaluation with about 2 vCPUs, 4 GB RAM and 10 GB free disk. Model weights, context cache and concurrent requests consume additional memory. Check each model’s license, language quality and required RAM; larger models often need substantially more memory or a GPU.

Resource figures are planning estimates for a small evaluation, not performance guarantees or plan specifications. A VPS/VDS does not imply a GPU. Check available memory, storage and resources before ordering.

On a fresh Ubuntu installation, use the packages below. If Docker is already installed from another repository, keep that installation and check compatibility first: do not mix Docker CE and docker.io packages. Use Compose v2; if the package is missing, check that Ubuntu Universe is enabled.

sudo apt update
sudo apt install -y docker.io docker-compose-v2 curl openssl
sudo systemctl enable --now docker
sudo docker compose version

All examples publish ports only on the server’s 127.0.0.1. Access them from your computer through an SSH tunnel; a domain is unnecessary. Do not open these ports to the Internet. An internal container listener on 0.0.0.0 does not change the host’s loopback-only published port.

Each project below is a new installation. Do not rerun secret-generation steps against an existing deployment.

1. Define the model service

Create the project, then save compose.yaml. The model store is persistent. No GPU device is requested, so this is explicitly a CPU example.

umask 077
mkdir -p ~/ai-ollama
cd ~/ai-ollama
services:
  ollama:
    image: ollama/ollama:latest
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    environment:
      OLLAMA_NUM_PARALLEL: "1"
      OLLAMA_CONTEXT_LENGTH: "2048"
    volumes:
      - models:/root/.ollama
volumes:
  models:

2. Download and query a small model

Pulling the image does not download a model. Run these commands on the VPS; the first generation may take time while weights load. A successful JSON response contains the generated message and done: true.

sudo docker compose up -d
sudo docker compose exec ollama ollama pull qwen2.5:0.5b
curl --fail-with-body --max-time 180 http://127.0.0.1:11434/api/chat   -H 'Content-Type: application/json'   -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"Explain what a VPS is in two sentences."}],"stream":false,"options":{"num_ctx":2048,"num_predict":100}}'
sudo docker compose exec ollama ollama ps

3. Access from your computer

Use an SSH tunnel for an application on your laptop. Connect that application to http://127.0.0.1:11434. Ollama’s local API is not a public authenticated gateway; a UI’s login does not protect an independently exposed model port.

ssh -N -L 127.0.0.1:11434:127.0.0.1:11434 YOUR_USER@YOUR_SERVER_IP

4. Measure before scaling

Use sudo docker stats --no-stream during a request. If the container is killed, inspect host out-of-memory events, reduce model/context size and add RAM; swap is not a substitute for adequate memory. Keep one concurrent request initially. Downloaded local models run inference on this server; selecting cloud models or external integrations changes where prompts go. Keep a list of exact model names and digests as well as any custom Modelfiles.

Updates and recovery

Before upgrading, stop the application for a consistent backup, keep data and secrets off the VPS, and test recovery on a separate instance. Moving image tags in these examples are for initial evaluation; pin a reviewed release tag or digest before ongoing use. Record the current digest with sudo docker image inspect IMAGE_NAME --format '{{json .RepoDigests}}'. Avoid unattended upgrades without compatibility checks, and do not delete persistent volumes with down -v.

If startup fails, inspect sudo docker compose ps, recent logs, free disk and memory. Do not share logs containing keys, prompts or personal data. For directory-permission errors, check the expected UID/GID and ownership instead of granting 777. Set up HTTPS, authentication and request limits before sharing access.

Related deployments