There's a simple appeal to local AI: your prompts stay on your hardware, you can work offline, and once you've paid for the machine there's no meter running on every question. For researchers, privacy-minded professionals, or devs who just want to experiment without a subscription, that's a big deal.
Ollama made the setup almost boringly easy — one command and you're running Llama, Mistral, DeepSeek, or Gemma. Prefer a GUI? LM Studio and llama.cpp have you covered. Most models come quantized now, trading a bit of quality for RAM you might actually have.
Hardware still matters, despite what some tutorials imply. Sixteen gigs of RAM is a realistic floor for smaller 7B models. If you want 32B or 70B to feel snappy, you'll want a proper GPU — NVIDIA boxes or Apple Silicon with unified memory are the usual picks. CPU-only works for light stuff but don't expect ChatGPT-speed replies.
People run local models for all sorts of reasons: analyzing documents that can't leave the building, coding help on a plane, drafting without sending client data to the cloud. Lawyers, journalists, and clinic staff often have clearer constraints than the average SaaS user. Devs also use local setups to test prompts before pointing production traffic at an API.
Trade-offs are part of the deal. Your laptop isn't a datacenter, so responses take longer. Open models can lag the big cloud players on hard reasoning or coding tasks. You'll manually pull updates when new weights drop — nobody's auto-patching that for you.
The gap is closing faster than skeptics expected. Fine-tuning, RAG, and better quantization improve every few months. For summarizing internal docs, drafting emails, or learning something new, local models are already "good enough" for plenty of people — with privacy and zero usage bills as the cherry on top.