How to run AI locally: a beginner's guide
Why bother? When AI runs on your own device, your chats never leave it. There are no accounts and no logs, and nobody else reads them.
Last checked: 16 October 2026. Rules of thumb, not guarantees. Always check the model's download page for its exact size.
What is running AI locally?
Chat tools like ChatGPT run on someone else's computers. Running AI locally means the model (the AI's "brain", a large file) runs on your own computer.
- Private: your chats never leave your machine.
- No subscription: the apps and many models are free.
- Works offline once the model is downloaded.
The catch: models are big. How big a model you can run depends mostly on how much memory your graphics card (or Mac) has.
📱 Even easier: on your phone (zero setup)
You can run a small AI right on your phone. Install a free app, tap to download a model, and chat. It works offline, and your chats stay on the phone.
- Google AI Edge Gallery (made by Google, open source): the simplest to start. Pick a model from its built-in list (mainly Google's Gemma models) and chat. It can also describe photos and transcribe audio. iPhone (App Store) · Android (Google Play)
- PocketPal AI (open source): lets you search and download hundreds of models from Hugging Face (the main website where AI models are shared). More choice, a few more settings. iPhone (App Store) · Android (Google Play)
Which phones work? PocketPal's store page says you need 6GB+ of RAM (the phone's working memory) for smaller models and 8GB+ for better ones, and that flagship or recent mid-range phones from the last 2–3 years run them well. Older or budget phones may struggle. Both apps have a built-in test (benchmark) so you can see how fast your phone is.
What to expect: phones run small models, around 1–4 billion parameters ("parameters" = model size, explained below). They're fine for quick questions, summaries and drafting, but noticeably less capable than ChatGPT. Downloads are often 1–3GB, so use Wi-Fi. Long chats can warm the phone and use battery.
Want bigger, smarter models? Use a computer: follow the easiest path below.
⭐ Easiest way to get started (about 15 minutes)
No commands, no accounts, free. Works on Windows, Mac and Linux.
- Download LM Studio from lmstudio.ai (a free desktop app for chatting with AI on your own computer) and install it like any other app.
- Open it and find a model. A "model" is the AI itself, a big file you download once. Use the search in LM Studio and type a small, popular model name, for example Gemma 3 4B or Llama 3.1 8B.
- Pick a download around 3–5GB. Smaller downloads start faster and fit almost anywhere. If you're unsure, choose the version marked Q4 (a compressed version, explained below). LM Studio also warns you if a download looks too big for your computer.
- Load it and chat. Open the chat screen, choose the model you downloaded, and type a question. Everything runs on your computer. You can even switch off the internet.
- Happy? Try a bigger model. Check the table below for what works easily on your hardware.
Prefer typing commands? Ollama is the next easiest (see the list below).
A few words you'll see
- Parameters (e.g. "8B", "27B"): how big a model is. 8B = 8 billion. Bigger is usually smarter, but needs more memory and runs slower.
- VRAM: the memory on your graphics card (GPU). The model needs to fit in it to run fast.
- Unified memory: on Apple Silicon Macs and AMD "Strix Halo" mini PCs, the processor and graphics share one pool of memory. So a 64GB Mac can use much of that 64GB for AI.
- Quantisation ("quant", e.g. Q4, Q8): shrinking a model by storing its numbers with fewer bits. Q4 is about a quarter the size of the original and is the usual choice. Lower than Q4 saves more memory but the answers get noticeably worse.
- GGUF: the most common file format for downloadable models. LM Studio, Ollama, llama.cpp, Jan and KoboldCpp all use it.
- Context: how much text the model can "keep in mind" at once: your chat so far, plus any documents you paste in.
- KV cache: the extra memory the model uses to hold that context. Longer chats need more of it.
- Tokens per second (tok/s): how fast it writes. A token is roughly ¾ of a word. About 10 tok/s or more feels comfortable to read along with.
- MoE ("mixture of experts", e.g. "35B-A3B"): a big model that only uses a small part of itself for each word. "35B-A3B" means 35B in total, about 3B active. It needs memory for all 35B but runs nearly as fast as a 3B model.
What works easily on what you have
Start with the "easy pick". It loads first time without changing any settings. "Stretch" works too, but you may need short chats or a smaller compressed version.
The memory rule behind the table (optional)
The simple memory rule: model size in GB ≈ parameters (in billions) × bits ÷ 8.
- Q4 (about 4.5–5 bits) ≈ 0.6 GB per billion parameters. A 27B model at Q4 is about 16GB.
- Q8 (about 8.5 bits) ≈ 1.1 GB per billion.
- Then leave 1.5–3GB spare for the KV cache and the app. Leave more if you use long documents.
Example models below are open models people benchmark today (Oct 2026). Exact file sizes vary, so check the download page. "Tight" means it fits only with short chats.
| Your hardware | Easy pick: model size (Q4) | Example easy models |
|---|---|---|
| 8GB graphics card | Up to ~8B | Llama 3.1 8B, Gemma 3 4B |
| 12GB graphics card | Up to ~12–14B | Gemma 4 12B, Qwen2.5 Coder 14B |
| 16GB graphics card | ~14B at Q6–Q8, or ~20B at Q4 | gpt-oss-20b (MoE), Qwen2.5 Coder 14B at Q8 |
| 24GB graphics card (e.g. RTX 3090, RX 7900 XTX) | 27–32B | Qwen3.6 27B, Gemma 4 31B, Qwen3 32B; Qwen3.6 35B-A3B (MoE) tight |
| 32GB graphics card (RTX 5090) | 27–32B with room for long context, or at Q5–Q6 | Gemma 4 31B, Qwen3.6 35B-A3B (MoE) |
| Two 24GB cards (48GB total) | Up to ~70B, tight | 70B-class models split across both cards |
| Mac, 16GB unified memory | ~4–8B (macOS and apps need memory too) | Gemma 3 4B, Llama 3.1 8B |
| Mac, 32GB | ~14–20B; 27B tight | Gemma 4 12B, gpt-oss-20b (MoE) |
| Mac, 64GB | 27–35B comfortably | Qwen3.6 27B, Gemma 4 31B, Qwen3.6 35B-A3B |
| Mac, 128GB | ~70B at Q4–Q6, or large MoE models | 70B-class dense models; MoE models around 100B total |
| AMD Strix Halo mini PC, 128GB (Ryzen AI Max+ 395) | Similar to a 128GB Mac in what fits, but slower per token | 70B-class at Q4; large MoE models |
| No graphics card, 32GB RAM | 7–14B, slowly | Llama 3.1 8B; MoE models with few active parameters run best |
| No graphics card, 64GB RAM | Up to ~30B, slowly; MoE models are much faster | Qwen3.6 35B-A3B (MoE) |
Two cards (e.g. 2× 24GB) can run bigger models, but splitting a model across cards needs extra settings, so it is not a beginner setup.
Thinking about buying hardware? The leaderboard compares UK prices against measured speeds. See the leaderboard. Pick a size class (7–9B, 12–14B, 27–35B, 70B+) to match the table above.
Free apps, easiest first
All free. Difficulty is about setup only: Easiest = install and click; Easy = one or two typed commands; Moderate = some setup steps; Advanced = for tinkerers.
| # | App | Setup | What it is, in plain words |
|---|---|---|---|
| 1 | LM Studio | Easiest | Desktop app: search, download and chat in one window. Windows, Mac, Linux. Free, not open source. |
| 2 | Jan | Easiest | Open-source desktop chat app, similar to LM Studio, with a focus on privacy. |
| 3 | Ollama | Easy | Install it, then type one command to download and chat with a model. Many other apps can connect to it. |
| 4 | KoboldCpp | Easy | One download, no install. You point it at a model file. Popular for stories and role-play. |
| 5 | MLX (mlx-lm) | Moderate | Apple's toolkit for Apple Silicon Macs, used from the terminal. Often faster on a Mac. (Easier route: LM Studio on a Mac can use MLX for you.) |
| 6 | Open WebUI | Moderate | A ChatGPT-style web page you run yourself, usually on top of Ollama, so you can chat from your phone or another PC at home. Usually installed with Docker (a tool for running apps in containers). Docs. |
| 7 | llama.cpp | Advanced | The engine inside many of these apps. Gives full control and the newest features, but it is run from the terminal with many options. |
| 8 | vLLM | Advanced | A server for many users at once on Nvidia/AMD cards, mostly on Linux. Overkill for one person. |
Where models come from: Hugging Face is the main website where models are published. LM Studio and Ollama search it (or their own library) for you, so beginners rarely need to visit it. If you do: browse GGUF models (GGUF is the usual file type for these apps; what GGUF is).
Things that trip up beginners (and how to avoid them)
- Starting too big. A huge model is the most common reason setup "doesn't work". Start small (3–5GB download), confirm it runs, then step up.
- Forgetting about context memory. A model that "fits" can still run out of memory when you paste in a long document. Leave a few GB spare, or lower the context length in the app's settings.
- Running a model slightly too big. If it doesn't fully fit in VRAM, the app puts part of it in normal system RAM ("partial offload"). It still works, but speed can fall sharply, often to a fraction of the full-fit speed. A smaller model or quant that fits fully is usually better.
- Squeezing too hard. Q4 is the usual sweet spot. Q3, Q2 and below lose noticeable quality. A smaller model at Q4 often beats a bigger model at Q2.
- Misreading MoE models. MoE models are fast for their size, but you still need memory for the whole model, not just the active part.
- Underpowered PSU (power supply) or poor cooling. Big cards draw a lot of power: an RTX 3090 is rated at 350W and an RTX 5090 at 575W. Check the card maker's recommended PSU and that your case has good airflow.
- Buying a used RTX 3090 without checks. Many were used for crypto mining or heavy gaming. Ask for a stress-test video, check memory temperatures with a monitoring tool, listen for fan noise, check the power connectors aren't scorched, and prefer eBay listings with buyer protection and returns.
- AMD and Intel cards need more setup. Nvidia's CUDA software is the most widely supported. AMD uses ROCm, its version of CUDA. Not every AMD card or operating system is officially supported, so check AMD's ROCm compatibility list. The Vulkan option in llama.cpp and LM Studio is a simpler fallback that works on most AMD and Intel cards.
- Mac memory can't be upgraded later. Apple Silicon memory is fixed when you buy. Get as much as you can afford: it decides which models you can run. macOS also keeps part of it for the system, so you can't use all of it for AI.
- Laptop GPUs aren't desktop GPUs. A "laptop RTX 4090" has less memory and power than the desktop card of the same name.
- Downloading from random places. Stick to Hugging Face pages from the model's maker or a well-known uploader, and read the model's licence if you plan to use it for work.
Spotted something out of date? Email rigvalue@gmail.com.