The complete cookbook for running open-source AI models on your own hardware – from a budget laptop to a workstation. No cloud bills. No data leaving your machine. Real engineers. Real setups. No fluff.
Commands last verified September 2026 against each tool's documentation. Model tags: see the dated note in Section 01 (updated November 2026).
Each episode is a standalone guide. Together they take you from zero – never run a local model – to being the person in your team who actually understands how local AI works.
Five tiers. Every common hardware profile covered. Pick your tier and see exactly which models run, what use cases unlock, and what you can't do yet.
August 2026 outdated The tier cards were built on Gemma 3, because no Gemma 4 tag existed in the Ollama library when this page was verified. The mapping then: 8 GB tier gemma3:4b (3.3 GB); 16 GB tier gemma3:4b; 32 GB tier gemma3:27b (17 GB, 128K context); 8–16 GB GPU tier gemma3:27b and gemma3:12b (8.1 GB, 128K); the Qwen cards were qwen3:1.7b (1.4 GB), qwen3:8b (5.2 GB), qwen3:14b (9.3 GB), qwen3:30b (19 GB), qwen3:32b (20 GB) and qwen3:235b; the Mistral cards were mistral:7b (4.4 GB, 32K context) and mistral-small3.1:24b (15 GB, 128K); the speed column in the use-case table at the end of this section was estimated on gemma3:27b, gemma3:4b and qwen3:8b. Every one of those tags still pulls and runs; they are no longer the newest in their family.
November 2026 current The tier cards now name gemma4 (tags e2b 4.6 GB, e4b 6.6 GB, 12b 8.0 GB, 26b 16–19 GB, 31b 19–20 GB; 128K context for the e-sizes, 256K above; sizes as the Ollama library lists them, which show a range where builds differ). The speed column is not re-measured and still carries the older model names. The Qwen cards now name qwen3.5 (tags 2b, 9b, 27b, 122b; 256K context) and qwen3.6 (27b, with a -coding tag). There is no 14b or 32b in those families, so the 14b card became the 9b and the 30b and 32b cards became the 27b. A newer qwen3.8 exists too (27b, 18 GB), and there is no Qwen4 yet, only an experimental preview of its architecture. mistral-large-4 exists, but only as a cloud-hosted tag with no local download. The Mistral cards now name ministral-3:8b (6.0 GB, 256K context, vision and tools) in place of mistral:7b, and mistral-small3.2:24b (15 GB, 128K, vision and tools) in place of 3.1, at the same size; the library describes 3.2 as an update with better function calling and instruction following. Neither Mistral family has changed in the library for about a year, and no speed figures were re-measured. Check ollama.com/library before you pull, since a list frozen in a page ages fast.
Every use case in this series builds on the same two tools. Install this once. Every playbook below assumes you have these running.
localhost:11434. Every other tool connects to it.# Linux / Mac curl -fsSL https://ollama.com/install.sh | sh # Windows → download installer from ollama.com # Pull your first model (T2 recommendation) ollama pull gemma4:26b ollama pull phi4 ollama pull qwen3.5:9b # List available models ollama list
# Ollama app: Settings, then the context-length slider. # Starting the server yourself? Quit the running Ollama first: OLLAMA_CONTEXT_LENGTH=64000 ollama serve # In a second terminal, show the context length in use: ollama ps
docker run -d -p 3000:8080 \ -v open-webui:/app/backend/data \ --add-host=host.docker.internal:host-gateway \ -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \ --name open-webui \ ghcr.io/open-webui/open-webui:main # Open browser at: # http://localhost:3000
pip install open-webui && open-webui serve. It serves on http://localhost:8080, not 3000.# In the Open WebUI sidebar: 1. Go to Workspace → Knowledge and create a knowledge base 2. Upload PDFs, DOCX, TXT, MD files into it 3. In chat, type # and pick the knowledge base or a file 4. Or attach it to a model (Workspace → Models → Edit) and chat with that model # Advanced: AnythingLLM for full local RAG export STORAGE_LOCATION=$HOME/anythingllm && \ mkdir -p $STORAGE_LOCATION && \ touch "$STORAGE_LOCATION/.env" && \ docker run -d --rm -p 3001:3001 \ --cap-add SYS_ADMIN \ --add-host=host.docker.internal:host-gateway \ -v ${STORAGE_LOCATION}:/app/server/storage \ -v ${STORAGE_LOCATION}/.env:/app/server/.env \ -e STORAGE_DIR="/app/server/storage" \ --name anythingllm \ mintplexlabs/anythingllm # Open http://localhost:3001, choose Ollama as the LLM and set its base URL to: # http://host.docker.internal:11434
docker run -d -p 5678:5678 \ --add-host=host.docker.internal:host-gateway \ -v n8n_data:/home/node/.n8n \ --name n8n \ n8nio/n8n # Open n8n at http://localhost:5678 # Add an Ollama credential and set its Base URL to: # http://host.docker.internal:11434 # Inside the container, localhost is n8n itself, not your machine. # --add-host is what Linux needs; Docker Desktop already provides host.docker.internal.
OLLAMA_HOST=0.0.0.0). That opens Ollama to your whole network, so use it only on a network you trust.Same foundation every time: Ollama + Open WebUI. What changes is the model you choose, the documents you upload, and how you wire the tools together.
phi4Continue.devollama, model to phi4Tab for inline completion, Ctrl+I for chat in editor# and pick the knowledge base or a filegemma4:26b, then raise Ollama's context window (Section 02, step 01): the model reaches 256K, Ollama starts at 4Kgemma4:26b or mistral-small3.2docker run -d --name searxng -p 8080:8080 -v ./searxng:/etc/searxng:rw searxng/searxng./searxng/settings.yml appears; add - json to the formats: list in the search: block, below - html; then run docker restart searxng (without json, Open WebUI gets 403 Forbidden)http://host.docker.internal:8080/search--add-host flag lets it reach Ollama)http://host.docker.internal:11434| Use Case | Min Tier | Recommended Model | Speed on T2 | Extra Tool? |
|---|---|---|---|---|
| Personal Chat | T1 | Phi 4 | 4–6 t/s | None |
| Coding (VS Code) | T1 | phi4 / qwen3:8b | 4–8 t/s | Continue.dev |
| Knowledge Base / RAG | T1 | Phi 4 | 3–5 t/s | Built-in WebUI |
| Document Analysis | T2 | gemma3:27b | 3–5 t/s | None |
| Vision / Images | T0 | gemma3:4b+ | 2–4 t/s | None |
| Research + Web | T2 | gemma3:27b | 3–5 t/s | SearXNG |
| Agentic Workflows | T2 | gemma3:27b | 3–5 t/s | n8n |
A general model knows everything broadly but nothing specifically about your domain, your writing style, or your audience. Fine-tuning teaches it. Here's the complete picture – tools, techniques, process, and what hardware you actually need.
{
"messages": [
{"role": "system", "content": "You are a senior cloud architect who explains complex systems to junior engineers in plain Urdu-influenced English."},
{"role": "user", "content": "Explain microservices in simple terms."},
{"role": "assistant", "content": "Think of it like a dhaba kitchen..."}
]
}
# Repeat this pattern for each training example (200+ recommended)
from unsloth import FastLanguageModel from trl import SFTTrainer # Load model + apply LoRA model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Phi-4", max_seq_length = 2048, load_in_4bit = True, # QLoRA ) model = FastLanguageModel.get_peft_model( model, r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"], ) # Train with your JSONL dataset trainer = SFTTrainer(model=model, tokenizer=tokenizer, train_dataset=your_dataset, ...) trainer.train() # Export to GGUF for Ollama model.save_pretrained_gguf("my_model", tokenizer, quantization_method="q4_k_m")
Local AI is a leverage skill. Every stage below unlocks a new tier of things you can build, automate, or offer. The timeline is real – these are hours-to-weeks, not years.
| What You Can Build | Who Needs It | Why It Matters |
|---|---|---|
| Company Knowledge Chatbot | Any business with internal docs | Replace hours of searching with seconds of querying |
| Private Code Review Bot | Dev teams with compliance constraints | No proprietary code sent to OpenAI |
| Domain-tuned Writing Assistant | Law firms, consultancies, media | Model that speaks in their exact voice and terminology |
| Local Customer Support AI | SMBs that can't afford cloud AI at scale | Cost: no per-token bill. Quality: good enough once grounded in their own domain |
| Research Synthesis Pipeline | Researchers, analysts, journalists | Ingest 100 PDFs, extract structured insights automatically |
| Agentic Task Automation | Operations teams, founders | AI that monitors, decides, and acts – not just chats |
The whole stack is free and runs on your machine. Here is every project this series is built on, with its source. Star the ones you rely on; that is how open source stays alive.
Two companions to this one, same spirit: practical, hands-on, no fluff.
179 commands consolidated from three community cheat sheets, 8 proven workflow recipes, and a four-level mastery roadmap. The practical playbook for working with Claude day to day.
Open the cookbook → Prompt kitThe anatomy of a good prompt, five you can steal today, and the one rule that turns the model into a teacher, not a crutch.
Open the kit →// continue exploring