How to run an LLM locally: Ollama, LM Studio and llama.cpp compared

To run an LLM locally, install a runner such as Ollama, LM Studio or llama.cpp, download an open-weight model that fits in your computer's memory, and chat with it or call it through a local API. This guide covers hardware needs, model choice, setup for all three tools and the trade-offs against cloud models.

What does it mean to run an LLM locally?

Running an LLM locally means the model's weights sit on your own disk and every token is generated by your own CPU or GPU, with no request sent to a cloud API. You download an open-weight model file once, load it into memory with a runner program, and talk to it through a chat window or a local HTTP endpoint.

Three runners cover almost everyone:

• Ollama: a background service plus command-line tool. One command downloads and runs a model.
• LM Studio: a desktop app with a model browser, chat UI and a local server, plus an lms command-line tool.
• llama.cpp: the open-source inference engine many other tools build on. Most control, fewest guardrails.

If you want a refresher on what the model itself is doing, read what is a large language model first. The rest of this guide assumes you know a model predicts tokens and has a context window.

Why run a local LLM instead of using ChatGPT or Claude?

Run a local LLM when privacy, offline use, cost at volume or control matters more than having the strongest model. For most everyday knowledge work and hard coding tasks, the top cloud models are still more capable.

Good reasons to go local:

• Sensitive data you are not allowed to send to a third party, such as client documents or health records.
• High-volume, simple jobs like classifying thousands of short texts, where per-token pricing adds up.
• Offline or air-gapped work, including travel and restricted networks.
• Learning and experimenting with prompts, embeddings and fine-tuning without a bill.

If your concern is company data in cloud tools, the guide on whether it is safe to use AI at work covers enterprise plans and training opt-outs, which may solve the problem without new hardware.

What hardware do you need to run an LLM locally?

The deciding factor is memory: the whole model, plus space for its context, must fit in your GPU's VRAM or, on Apple Silicon, in unified memory. Speed comes second and depends on memory bandwidth more than on raw compute.

Models are usually run quantized, meaning their weights are stored at 4 or 8 bits instead of 16. A 4-bit model is about a quarter of the full-precision size with a modest quality loss. A practical rule of thumb for 4-bit models is about 0.7 GB per billion parameters, plus 1 to 4 GB for context and overhead.

Real download sizes from the Ollama library as of October 2026 show the pattern:

The gpt-oss model page states the 20B model runs on systems with as little as 16 GB of memory and the 120B model fits on a single 80 GB GPU.

What that means for common machines:

• 8 GB laptop: 1B to 4B models. Fine for summaries, rewriting and simple questions.
• 16 GB laptop or Mac: 7B to 20B models. The sweet spot for most people.
• 32 to 64 GB Mac, or a 24 GB GPU: 26B to 35B models, which are noticeably smarter.
• Workstation with 80 GB or more of fast memory: 70B to 120B models.

How do you run an LLM locally with Ollama?

Ollama is the quickest route: install it, run one command, and you are chatting with a downloaded model. It also starts a local API on port 11434 that other apps can call.

1. Install Ollama. On macOS and Windows, download the app from ollama.com/download. On Linux, run the official script:

   curl -fsSL https://ollama.com/install.sh | sh

2. Run a model. The first run downloads it, then opens a chat. Type /bye to exit.

   ollama run gemma4:e4b

3. Manage models. ollama pull <model> downloads without chatting, ollama list shows what you have, and ollama rm <model> frees disk space.

4. Call the local API. The Ollama quickstart documents a native API at http://localhost:11434/api/chat plus OpenAI-compatible routes such as /v1/chat/completions and an Anthropic-compatible /v1/messages route.

   curl http://localhost:11434/v1/chat/completions \  -H "Content-Type: application/json" \  -d '{"model": "gemma4:e4b", "messages": [{"role": "user", "content": "Explain quantization in two sentences."}]}'

How do you run an LLM locally with LM Studio?

LM Studio is the easiest option if you prefer a graphical app. You search for a model, see whether it fits your hardware, download it and chat, all in one window.

1. Download LM Studio for macOS, Windows or Linux from lmstudio.ai and open it.

2. Use the model search to find a model. LM Studio shows quantization options and flags files that are likely too large for your machine.

3. Download a 4-bit build (often labeled Q4_K_M) of a model that fits your memory, and load it in the chat tab.

4. To use it from code, start the local server from the Developer tab or from the command line:

   lms server start --port 1234

The LM Studio developer docs list OpenAI-compatible endpoints for chat completions, embeddings, structured output and tool use, plus an Anthropic-compatible messages endpoint. The lms get <model> and lms chat commands let you download and chat from a terminal without opening the app.

How do you run an LLM locally with llama.cpp?

llama.cpp is the engine underneath much of the local LLM world. Use it directly when you want the newest model support, precise control over memory and threads, or a minimal server with no extra services.

The llama.cpp README currently recommends an install script, and then a single command to fetch a GGUF model from Hugging Face and run it:

curl -LsSf https://llama.app/install.sh | shllama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

To serve the model with a built-in web UI and an OpenAI-compatible API:

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Prebuilt binaries, Docker images and building from source are also supported. llama.cpp runs on CPU (with AVX and similar instruction sets), Apple Silicon through Metal, NVIDIA GPUs through CUDA, AMD through HIP, and other GPUs through Vulkan and SYCL. Swap the Hugging Face repository name for a larger GGUF model once the small one works.

Ollama vs LM Studio vs llama.cpp: which should you pick?

Pick Ollama if you live in the terminal or want a local API for other apps, LM Studio if you want a polished desktop app, and llama.cpp if you need maximum control or the newest models first.

You can install more than one. A common setup is LM Studio for trying models and Ollama as the always-running local API.

Which open model should you run locally?

Choose the largest model that fits comfortably in your memory, from a family that matches your task. As of October 2026, the most downloaded families in the Ollama library include Gemma 4, Qwen 3.5, gpt-oss, Llama and DeepSeek-R1.

A starting point by job:

• General chat and writing: Gemma 4 or Qwen 3.5 at the biggest size your memory allows.
• Coding help: a coding-tuned model such as qwen3-coder or qwen2.5-coder, or gpt-oss.
• Reasoning-heavy questions: a model with a thinking mode, such as gpt-oss or DeepSeek-R1 distills.
• Embeddings for search and RAG: a dedicated embedding model such as nomic-embed-text, not a chat model. See what are embeddings.
• Images as input: a vision-capable model; Gemma 4 accepts text and images.

Test candidates on your own tasks. Write five prompts you care about, run each model on them, and compare. Leaderboards measure general skills, not your documents.

Check licenses before commercial use. The gpt-oss models use the Apache 2.0 license according to their model page, while other families have their own terms.

What are the limits and trade-offs of local LLMs?

Local LLMs give you privacy and zero marginal cost, but you trade away some quality, speed and convenience. Knowing the limits up front avoids disappointment.

• Quality gap. Open models in the 7B to 35B range are good at summaries, rewriting and focused questions, and weaker at multi-step reasoning and long, multi-file coding tasks.
• Context costs memory. A 128K or 256K context window is supported on paper, but filling it needs far more memory than the model weights alone.
• Speed varies. Token speed depends on memory bandwidth. A model that barely fits will be slow, and one that spills from GPU to CPU memory can slow down sharply.
• Maintenance is yours. Updates, disk space (models are many gigabytes each) and compatibility are your job.
• Tools and agents need care. Local models can call tools through the OpenAI-compatible API, but smaller models follow complex tool instructions less reliably.

A practical pattern is hybrid: a local model for private, simple or high-volume work, and a cloud model for the hard problems. If you want to run more of the stack yourself, including chat UIs and automation, see the guide on how to self-host AI tools.

Next steps: your first local LLM in 15 minutes

1. Check your memory: 8 GB, 16 GB, 32 GB or more decides your model size.
2. Install Ollama (or LM Studio if you prefer a GUI).
3. Run a model that fits: ollama run gemma4:e4b on 16 GB, a 4B model on 8 GB.
4. Try five prompts from your real work and note where it falls short.
5. Call it from code through the local OpenAI-compatible endpoint.
6. Move up a size if your hardware allows, and compare.

Local models are a poor fit for one job in particular: long autonomous coding sessions, which work best with frontier models and a machine that stays on. For that, Orkio runs agents such as Claude Code on an always-on cloud box using your own Claude account, so your laptop does not have to stay awake.

Frequently asked questions