Running a Local LLM on a Linux Server: systemd Units, GPU Checks, and the vLLM vs llama.cpp Call

The model answered fine in an SSH session last week. After the reboot there is no process, or the service is alive but answers arrive at four tokens per second, and nobody can say whether the weights are on the card or half in system RAM.

💡 What You Will Learn

The model answered fine in an SSH session last week. After the reboot there is no process, or the service is alive but answers arrive at four tokens per second, and nobody can say whether the weights

📜 Table of Contents

• Three different things people mean by "run it on Linux"
• Step 1: prove the GPU is visible before installing anything
• Step 2: install Ollama and read the unit it created
• Step 3: the systemd override that fixes most production complaints
• Step 4: prove the weights are on the card
• Step 5: measure tokens per second from the API, not from a first impression
• Step 6: know when Ollama is the wrong tool and vLLM is the right one
• llama.cpp, Ollama, vLLM: pick by shape of traffic, not by benchmark chart
• A restart-safe checklist
• FAQ
• Does Ollama use the GPU automatically on Linux?
• How much VRAM does context length actually cost?
• Can several people share one Ollama endpoint?
• How do I tell whether a slow answer is the model or the server?
• What to do before you buy a bigger card
• Related on this site

Last Tuesday the model answered a 900-token question in about two seconds. This Tuesday the same prompt takes eleven, the fans stay quiet, and nvidia-smi shows a card sitting at 3 percent utilization. Nothing in the logs says "error". That is the shape of most "local LLM on Linux" problems: not a crash, but a quiet demotion from GPU to something worse, wrapped in a service that nobody configured on purpose.

This guide is the sequence we use on a fresh Ubuntu box, in the order that surfaces problems early: verify the card, install the server, pin its behaviour in a systemd unit, prove the weights are where you think they are, then measure throughput from the API instead of trusting a first impression. Every version number and default below was checked on 2026-10-23 against the vendors' own docs and PyPI.

Three different things people mean by "run it on Linux"

The phrase covers three deployments that fail in unrelated ways, and mixing them up is why advice online contradicts itself:

• Interactive CLI — ollama run or llama-cli in a terminal. Great for evaluating a model, useless as a service. It dies with the SSH session and has no restart policy.
• Long-lived server process — an HTTP endpoint your apps call. Needs a systemd unit, an explicit model directory, memory limits that match the card, and a health check.
• Container — the same server inside Docker. Adds GPU passthrough (nvidia-container-toolkit) as a new failure point between the driver and the model.

The rest of this article covers the middle one, because it is what "we want to self-host it for the team" actually means, and because both the CLI and the container are thin wrappers around it.

Step 1: prove the GPU is visible before installing anything

Skipping this step is how people spend an hour debugging a model when the real problem is a missing driver. On a server with an NVIDIA card, three commands tell you where you stand:

nvidia-smi
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv
ls /dev/nvidia* 2>/dev/null | head

If nvidia-smi is missing entirely, stop. Install the driver first, reboot, and confirm the card reports a memory total. If it reports a card but inference later runs on CPU, the usual causes are a driver/kernel mismatch after an unattended upgrade, or running inside a container without the NVIDIA runtime registered.

Write down the memory total. Every capacity decision later — context length, batch size, how many models can stay resident — is a subtraction from that number, not from the marketing figure on the box.

Step 2: install Ollama and read the unit it created

The official installer is a shell script that also registers a systemd service. Download it, read it, then run it — the reading takes two minutes and tells you where your models and your service will live:

curl -fsSL https://ollama.com/install.sh -o /tmp/ollama-install.sh
less /tmp/ollama-install.sh
sh /tmp/ollama-install.sh
systemctl status ollama --no-pager
systemctl cat ollama | head -30
id ollama

Two facts about that unit matter operationally. First, it runs as a dedicated ollama user, not root, so anything the service must read needs to be readable by that account. Second, on a standard Linux install the models live under /usr/share/ollama/.ollama/models. On a server where / is a 40 GB system volume — a very common cloud default — a few 8B models will fill it, and the failure looks like a disk error mid-download rather than a capacity problem.

Move the model directory before you pull anything large, set ownership to the service account, and put the new path in the unit (Step 3 covers how):

mkdir -p /data/ollama/models
chown -R ollama:ollama /data/ollama

Step 3: the systemd override that fixes most production complaints

Do not edit the vendor unit file — an upgrade will overwrite it. Use a drop-in with systemctl edit ollama, and set the behaviour you actually want in the service section it opens. This is a working baseline for a single 24 GB card serving one team:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_MODELS=/data/ollama/models"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
Environment="OLLAMA_MAX_QUEUE=128"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Then apply it and confirm the environment actually reached the process, rather than assuming the file was picked up:

systemctl daemon-reload
systemctl restart ollama
systemctl show ollama -p Environment
journalctl -u ollama -n 30 --no-pager

Each variable exists to prevent a specific, documented behaviour. The right-hand column is why the default is not good enough for a shared box. Defaults below come from Ollama's FAQ documentation:

The parallelism rule deserves emphasis because it explains most "it worked yesterday" reports: 4K context with 4 parallel slots reserves KV cache for 16K, not 4K. Concurrency doubles memory without touching the model file.

Step 4: prove the weights are on the card

Load a model, run one prompt, then read what the server says about it:

ollama pull qwen3:8b
ollama run qwen3:8b "reply with the single word ok"
ollama ps

The PROCESSOR column is the honest answer: either a single percentage on GPU, or a split such as 60/40 CPU/GPU. A split is your explanation for slow tokens, and it has nothing to do with the model's reasoning ability. Watch memory while a request is in flight:

nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv -l 1

Three patterns, three fixes. Memory near zero during generation means the weights never reached the card. Memory spiking then dropping after every request means KEEP_ALIVE is shorter than your traffic pattern. Memory pinned at the ceiling plus one silent restart means the requested context does not fit — cut context or quantize the cache before buying hardware.

Step 5: measure tokens per second from the API, not from a first impression

"Feels fast" is not evidence, and the first response after a cold load always looks terrible. Ollama's generation endpoint returns the counters you need, so no external benchmarking tool is required. Ask for a single non-streamed completion and read the timings out of the JSON:

curl -s http://127.0.0.1:11434/api/generate -d '{"model":"qwen3:8b","prompt":"List five symptoms of a partially offloaded model, one line each.","stream":false}' -o /tmp/gen.json
python3 -c "import json; d=json.load(open('/tmp/gen.json')); ec=d['eval_count']; ed=d['eval_duration']; pc=d['prompt_eval_count']; pd=d['prompt_eval_duration']; print('generation %.1f tok/s over %d tokens' % (ec/(ed/1e9), ec)); print('prefill %.1f tok/s over %d tokens' % (pc/(pd/1e9), pc)); print('cold load %.2f s' % (d.get('load_duration',0)/1e9)); print('wall clock %.2f s' % (d.get('total_duration',0)/1e9))"

Four numbers, four diagnoses. Slow prefill with healthy generation means the prompt is heavy and the cache is cold. Healthy prefill with slow generation points at the model, the quantization, or an offloaded layer. A large load_duration on every call means the model is evicted between requests. Run the same command twice: if the second run is dramatically faster, you were measuring disk, not GPU.

Then measure under the concurrency your team actually generates, because single-stream speed is the number vendors quote and the number your users never see. Fire three requests at once and watch whether total time triples or stays roughly flat. Flat is batching. Tripling is a one-slot queue.

Step 6: know when Ollama is the wrong tool and vLLM is the right one

Ollama is a model manager plus a single-model inference server. It is very good at "one team, a handful of models, no Kubernetes". It is not built for throughput: when twenty people share an endpoint and each waits for a full generation before the next begins, the bottleneck is scheduling, not the card.

vLLM targets exactly that problem with continuous batching and PagedAttention, and it exposes an OpenAI-compatible API. As of 2026-10-23 it is on PyPI at 0.32.0, Apache-2.0, with roughly 95k GitHub stars. A minimal serving command, using flags that exist in the current vllm serve CLI:

python3 -m venv /opt/vllm && source /opt/vllm/bin/activate
pip install vllm
vllm serve Qwen/Qwen3-8B \
  --served-model-name qwen3-8b \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 16

Verify it with the endpoint your client will use, not with a browser tab:

curl -s http://127.0.0.1:8000/v1/models | python3 -m json.tool | head -20

Three flags deserve explanation because they are where people get bitten. --gpu-memory-utilization is the share of card memory vLLM may claim for weights plus KV cache; pushing it to 0.95 buys longer context and makes the machine useless for anything else. --max-model-len is the longest sequence accepted, and setting it beyond the model's trained range yields degraded output rather than a clean error. --tensor-parallel-size spreads one model over N GPUs and must match the hardware you actually have; a value above the GPU count fails at startup.

vLLM needs a CUDA-capable GPU to be worth its setup cost. On CPU-only hardware, or where the model already fits and concurrency is one or two people, the extra layer buys nothing.

llama.cpp, Ollama, vLLM: pick by shape of traffic, not by benchmark chart

The migration trigger is measurable rather than philosophical: when queue depth or p95 latency grows faster than request rate, the engine is the problem. Before that, moving to vLLM is spending a week on a limit you have not reached.

A restart-safe checklist

• systemctl enable ollama — a service that does not survive a reboot is a demo, not a deployment
• Models on a data volume, not the system disk, and free space checked before each large pull
• A firewall rule in front of the port plus auth at the reverse proxy; the API itself has no user accounts
• A size cap on the journal; a chatty endpoint fills /var/log and takes the box down in a way that looks unrelated
• Model tags pinned to a fixed version rather than a moving alias, so an update does not change behaviour mid-week
• One command the uptime monitor can run: a two-token completion with a timeout
• The four numbers from Step 5 written down with the date; that note is the baseline the next "it got slower" complaint gets measured against

FAQ

Does Ollama use the GPU automatically on Linux?

It uses whatever it can see. With a working driver and a model that fits, weights go to VRAM; if they do not fit, it splits across GPU and CPU and keeps serving — which is why the slowdown carries no error. ollama ps during a request settles the question in one line.

How much VRAM does context length actually cost?

KV cache per token is 2 x layers x KV heads x head dimension x bytes per element. For an 8B model with 32 layers, 8 KV heads and head dimension 128 in f16 that is 131,072 bytes per token: 1 GiB at 8K context and 4 GiB at 32K, before any batching. A q8_0 cache roughly halves it. Weights are separate — the same 8B model is about 4.5 GB at Q4_K_M and about 16 GB at f16.

Can several people share one Ollama endpoint?

Yes, within memory limits. Every parallel slot multiplies the context footprint, so two slots at 8K reserve the equivalent of 16K. Past a handful of regular users the scheduler, not the card, is the constraint, and that is where vLLM's batching starts to pay for its setup cost.

How do I tell whether a slow answer is the model or the server?

Compare prefill and generation rates from the API response. Slow prefill with fast generation points at prompt handling and a cold cache. Fast prefill with slow generation points at the model, quantization or an offloaded layer. Both slow with a large load_duration means the model is reloaded for every request.

What to do before you buy a bigger card

Measure the four numbers, look at ollama ps during a real request, and decide whether the ceiling you hit is memory, scheduling, or a five-minute keep-alive expiring between conversations. The cheapest fix is rarely hardware: it is a longer KEEP_ALIVE, a quantized KV cache, or admitting the box should be running vLLM because twenty people share it. Hardware answers exactly one of those three.

Related on this site

• Local LLM Setup on Linux: From Environment to Running
• Ollama in Docker With a GPU: Compose Block and VRAM Math
• LLM Inference Server Comparison: vLLM vs Ollama vs Triton vs TGI