OLLAMA

Last reviewed: 2026-06-16

Purpose: Comprehensive reference guide for the Ollama CLI — installation, model management, Modelfiles, GPU acceleration, API serving, and troubleshooting.

Overview

Ollama is the most popular tool for running large language models (LLMs) locally on Linux, macOS, and Windows. It wraps model weights, tokenizers, and inference engines into a single ollama binary and exposes both a CLI and a REST API. Models are distributed via the Ollama Library and include Llama, Mistral, Gemma, Qwen, DeepSeek, Phi, CodeGemma, and hundreds more.

Ollama supports GPU acceleration (NVIDIA CUDA, AMD ROCm, Apple Metal) and runs on CPU-only hardware for smaller models.

Installation (Linux)

Automatic install script

curl -fsSL https://ollama.com/install.sh | sh

This script detects your OS, architecture, and GPU driver availability, then installs Ollama as a systemd service.

Manual install (standalone binary)

# Download the latest binary
curl -L https://ollama.com/download/ollama-linux-amd64.tgz -o ollama-linux-amd64.tgz

# Extract to /usr/local
sudo tar -C /usr/local -xzf ollama-linux-amd64.tgz

# Clean up
rm ollama-linux-amd64.tgz

Package manager (Debian / Ubuntu)

# Add Ollama repository
curl -fsSL https://ollama.com/install.sh | sudo bash

Verify installation

ollama --version

Basic CLI Commands

Download (pull) a model

ollama pull <model-name>

Downloads model layers from the Ollama registry. Models are stored in ~/.ollama/models/ by default.

Examples:

ollama pull llama3.2:3b        # Default variant, ~2 GB
ollama pull llama3.2:1b        # Smaller 1B parameter variant
ollama pull mistral:7b-instruct
ollama pull qwen2.5:7b
ollama pull deepseek-r1:8b

Run a model (interactive chat)

ollama run <model-name>

Starts an interactive chat session. Type your prompts and receive streaming responses. Exit with /bye or Ctrl+D.

ollama run llama3.2:3b
>>> What is the capital of France?
The capital of France is Paris.
>>> /bye

To pass a single prompt without entering interactive mode:

ollama run llama3.2:3b "Explain quantum computing in one sentence"

List downloaded models

ollama list

Output shows model name, size, and modification date:

NAME                    ID              SIZE      MODIFIED
llama3.2:3b             a9b474cbe86e    2.0 GB    2 days ago
mistral:7b-instruct     f97403f7b34b    4.1 GB    5 days ago

Remove (delete) a model

ollama rm <model-name>

Deletes all model layers from disk.

ollama rm llama3.2:1b

Show model details

ollama show <model-name>

Displays the model's architecture, parameters, Modelfile template, system prompt, and license.

Copy a model

ollama cp <source-model> <target-name>

Creates a local copy under a new name.

Pull model info

ollama info <model-name>

Displays metadata about a model from the registry without downloading it.

Modelfile & Custom Models

A Modelfile is a configuration file (similar to a Dockerfile) that specifies how to build or customize an Ollama model.

Modelfile structure

# Base model (required)
FROM llama3.2:3b

# System prompt / system message
SYSTEM "You are a helpful assistant specialized in Python programming."

# Temperature setting (default: 0.8)
PARAMETER temperature 0.7

# Top-p sampling (default: 0.9)
PARAMETER top_p 0.9

# Context window size (tokens)
PARAMETER num_ctx 4096

# Stop sequences
TEMPLATE "{{ .Prompt }}"

Create a custom model

Write a Modelfile and build:

ollama create my-custom-model -f ./Modelfile

The model is saved locally and appears in ollama list.

One-liner without a file

You can pipe a Modelfile directly:

echo "FROM llama3.2:3b\nSYSTEM 'You are a Python expert.'" | ollama create python-helper

Common Modelfile parameters

Parameter	Description	Default
`temperature`	Randomness of output (0.0 = deterministic)	0.8
`top_p`	Nucleus sampling threshold	0.9
`top_k`	Limit next token selection to top K tokens	40
`num_ctx`	Context window size (in tokens)	2048
`repeat_penalty`	Penalize repetition	1.1
`seed`	Random seed for reproducibility	0
`stop`	Stop sequences (can be specified multiple times)	[]

Using a GGUF file directly

You can also build from a custom GGUF quantized model:

FROM ./path/to/my-model.Q4_K_M.gguf

ollama create my-gguf-model -f ./Modelfile

GPU Acceleration

NVIDIA CUDA (Linux)

Prerequisites: NVIDIA drivers and CUDA toolkit.

# Check driver version
nvidia-smi

# Verify CUDA
nvcc --version

Ollama automatically detects CUDA and uses the GPU for inference. To force GPU-only:

ollama serve &
ollama run llama3.2:3b

Monitor GPU usage:

watch -n 1 nvidia-smi

Set the number of GPU layers to offload:

# Default: all layers offloaded to GPU
# Set explicitly via environment variable
export OLLAMA_GPU_LAYERS=99
ollama run llama3.2:3b

AMD ROCm (Linux)

Prerequisites: ROCm drivers installed and rocm-smi working.

# Verify ROCm
rocm-smi

# Check Ollama logs for ROCm detection
journalctl -u ollama --no-pager | grep -i rocm

Ollama ships with ROCm support for supported AMD GPUs (RX 7000 series, MI series).

CPU-only mode

If no GPU is detected, Ollama falls back to CPU inference. To explicitly force CPU:

export OLLAMA_GPU_OVERHEAD=0
export OLLAMA_GPU_LAYERS=0
ollama run llama3.2:3b

Serving via API

Ollama includes an HTTP server that exposes a REST API, enabled by default when you run ollama serve or when the systemd service is active.

Start the server

# As a foreground process
ollama serve

# As a background daemon (if not using systemd)
ollama serve &

The server listens on http://localhost:11434 by default.

Check server status

curl http://localhost:11434/

Expected response: Ollama is running

List available models via API

curl http://localhost:11434/api/tags

Generate a completion (non-streaming)

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "What is the capital of France?",
  "stream": false
}'

Generate with streaming

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Tell me a short story."
}'

Streaming responses are newline-delimited JSON (NDJSON). Each line contains a response field and a final done: true line with stats.

Chat completion API

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2:3b",
  "messages": [
    {"role": "user", "content": "Hello, how are you?"}
  ],
  "stream": false
}'

Pull a model via API

curl http://localhost:11434/api/pull -d '{
  "model": "llama3.2:3b"
}'

Change listen address / port

# Bind to all interfaces on port 11434
export OLLAMA_HOST=0.0.0.0:11434
ollama serve

Configuration environment variables

Variable	Purpose	Default
`OLLAMA_HOST`	Listen address and port	`127.0.0.1:11434`
`OLLAMA_MODELS`	Model storage directory	`~/.ollama/models`
`OLLAMA_KEEP_ALIVE`	Duration to keep models loaded in memory	`5m`
`OLLAMA_NUM_PARALLEL`	Number of parallel requests	`1`
`OLLAMA_MAX_LOADED_MODELS`	Max concurrently loaded models	`3`
`OLLAMA_DEBUG`	Enable debug logging	unset

Python client example

import requests
import json

response = requests.post(
    "http://localhost:11434/api/generate",
    json={"model": "llama3.2:3b", "prompt": "Say hello", "stream": False}
)
print(response.json()["response"])

Common Troubleshooting

"ollama: command not found"

Install Ollama first:

curl -fsSL https://ollama.com/install.sh | sh

Or add /usr/local/bin to your PATH:

export PATH=$PATH:/usr/local/bin

Out of memory / model crashes

Large models require significant RAM/VRAM. Check memory usage:

free -h                     # System RAM
nvidia-smi                  # GPU VRAM

Tips: - Use smaller quantized model variants (e.g., llama3.2:1b, qwen2.5:0.5b) - Set num_ctx lower (e.g., 2048 instead of 8192) - Close other GPU-using applications - If using CPU, lower num_thread in Modelfile

"Error: pull access denied" or model not found

Verify the model name in the Ollama Library. Common issues: - Typo in model name or tag - Misspelled tag separator — use model:tag, not model/tag - Network connectivity problems

GPU not detected

# Verify NVIDIA drivers
nvidia-smi

# Check Ollama logs
journalctl -u ollama --no-pager | tail -30

# Restart Ollama service
sudo systemctl restart ollama

If using AMD ROCm:

# Verify ROCm installation
rocm-smi

# Check for ROCm in Ollama logs
journalctl -u ollama --no-pager | grep -iE "rocm|hip"

"Unauthorized" when pulling a model

Some models require authentication (e.g., Llama 3 from Meta). Log in first:

ollama login

Or use the Ollama web login flow at https://ollama.com/settings/keys.

Model loads slowly every time

Increase the keep-alive duration:

export OLLAMA_KEEP_ALIVE=30m
ollama serve

Server refuses connection

# Verify Ollama is running
ps aux | grep ollama
sudo systemctl status ollama

# Check port binding
ss -tlnp | grep 11434

# If bound only to 127.0.0.1, set OLLAMA_HOST=0.0.0.0:11434 for remote access

"Too many requests" or rate limiting

The Ollama API (not the local server) may rate-limit pulls. Wait and retry, or use a local mirror.

Reset or reinstall Ollama

# Stop service
sudo systemctl stop ollama

# Remove Ollama (installed via install script)
sudo rm -rf /usr/local/bin/ollama /usr/local/lib/ollama

# Remove model cache (backup first if needed)
rm -rf ~/.ollama

# Reinstall
curl -fsSL https://ollama.com/install.sh | sh