Running Models Locally
What local models are good for, what they are bad at, and why RAM suddenly becomes a personality trait.
Running a model locally means the model executes on your machine or your own server instead of behind someone else's API. The appeal is obvious: privacy, offline use, no per-token bill, and the satisfying feeling of owning the stack.
The pain is also obvious: hardware.
What you need
Small quantized models can run on normal laptops. Larger models want serious RAM, VRAM, or both. The model name often contains clues: size like 7B or 70B, format like GGUF, and quantization like Q4 or Q8.
Tools people actually use
Ollama is the fast path for CLI and local API usage.
LM Studio is a friendly GUI for downloading and chatting with models.
Jan is an open-source local chat app.
llama.cpp is the lower-level engine many local tools build on.
vLLM is for higher-throughput serving, usually on servers with GPUs.
When local wins
Local wins for privacy, prototyping, offline work, and learning the mechanics. Hosted APIs often still win for top-tier quality, managed scaling, multimodal features, and not turning your workstation into infrastructure.