Running an AI model locally means the weights, the maths and your prompts all live on hardware you own. Nothing is sent to a hosted service, nothing is metered, and the whole thing keeps working with the network cable pulled out. LM Studio's documentation says as much about its own app: it can operate entirely offline, provided you have downloaded the model files first.
That is the appeal. The cost is that you become the infrastructure team. You choose the model, you choose the runtime, and you find out the hard way whether your machine has enough memory. This guide is the map for that first week.
What you are actually running
Three things come together on a local setup.
The model is a file, or a handful of files, of learned weights. Open-weight families you will meet include Qwen, Gemma, Llama, Mistral and gpt-oss; LM Studio's own introduction lists several of these as typical downloads.
The runtime is the program that loads those weights and turns a prompt into text. It decides which hardware gets used and how well.
The front end is whatever you type into: a terminal, a chat window, or another application talking to a local server. Most runtimes expose an OpenAI-style HTTP API, which means plenty of existing tools can be pointed at your own machine instead of a cloud endpoint. llama.cpp, LM Studio and vLLM all say they offer one.
What your machine needs
The honest answer is "memory, then more memory".
A model has to fit somewhere the processor can read it quickly. On a PC with a discrete graphics card that is the card's own memory, usually called VRAM. On an Apple silicon Mac, or on a machine built around a shared pool, the processor and the graphics hardware use the same memory, so the whole pool is available. MLX's documentation describes this directly: its arrays live in shared memory, so operations can run on either the CPU or GPU without copying data between them.
If the model does not fit, you are not necessarily stuck. llama.cpp advertises CPU+GPU hybrid inference to partially accelerate models larger than the available VRAM. It works, but the layers left in ordinary system RAM drag the speed down, a point we pick up in our piece on CPU, GPU and NPU inference.
For a floor, LM Studio publishes its minimums. On a Mac it asks for Apple silicon and macOS 14 or newer, and recommends 16GB or more of RAM. On Windows it needs an x64 CPU with AVX2 (or an ARM Snapdragon X Elite machine), recommends at least 16GB of RAM and suggests at least 4GB of dedicated VRAM. On Linux it ships as an AppImage for x64 and ARM64. These are the vendor's recommendations for a pleasant experience with small models, not a guarantee that a given model will run.
To work out whether a specific model fits, read our companion piece on VRAM and large language models. It comes with a script so you can check the arithmetic yourself.
Quantisation in two paragraphs
Model weights are trained at high precision, typically 16 bits each. Quantisation rounds them to fewer bits, so the file shrinks and there is less data to move for each token. llama.cpp's README lists support for 1.5-bit through 8-bit integer quantisation, and its quantisation guide shows the effect on Llama 3.1: the 8B model drops from 32.1 GB at full size to 4.9 GB at Q4_K_M, which is the size that makes it comfortable on a modest graphics card or a 16GB laptop.
There is a quality price, and it grows as you go lower. The people who maintain these tools are open about that trade-off, which is why they publish many variants rather than one. A sensible beginner's habit is to start at a 4-bit variant such as Q4_K_M, and move up a step if the answers feel off and you have the memory to spare.
The runtimes, one at a time
llama.cpp
The foundation of much of the local scene. It is LLM inference in plain C/C++ with no dependencies, and it runs on an unusually wide spread of hardware: Apple silicon through Metal, NVIDIA through CUDA, AMD through HIP, Intel graphics through SYCL, plus Vulkan and a CPU path that uses AVX, AVX2, AVX512 and AMX on x86. Its backend table also lists entries for Snapdragon (Hexagon), Huawei Ascend and an in-progress OpenVINO route for Intel CPUs, GPUs and NPUs.
The project's quick start is two lines: llama cli -hf <repo> downloads a model from Hugging Face and runs it, and llama serve -hf <repo> launches an OpenAI-compatible server. Choose it when you want control and the widest hardware support of the five.
Ollama
Ollama is the easy front door. Install it with a one-line script on macOS and Linux, a PowerShell line on Windows, or use the official Docker image. Then ollama run gemma4 pulls a model and starts a chat, as its README shows, and a REST API listens on port 11434 for other programs. The README names the llama.cpp project under its supported backends, which tells you where the actual number crunching comes from.
Choose it when you want something that works in a few minutes and has a long list of community integrations (its README catalogues dozens).
LM Studio
LM Studio is a desktop app with a proper interface for finding, downloading and chatting with models. Underneath, its documentation says it runs llama.cpp (GGUF) models on Mac, Windows and Linux, and on Apple silicon it can also run models through Apple's MLX. It can serve models on OpenAI-like endpoints locally or across your network.
Choose it when you would rather click than type, or when you want to compare the same model across the two engines on a Mac.
vLLM
vLLM is a different animal. It describes itself as a library for LLM inference and serving with PagedAttention for managing attention memory, continuous batching of incoming requests, and a long list of quantisation formats. It supports NVIDIA, AMD and Intel GPUs and several CPU architectures, and exposes an OpenAI-compatible API.
Its design aim is throughput for many simultaneous users. For one person chatting to one model, it is usually more machinery than you need. Choose it when you are serving a household, a team or an application with several requests in flight at once, and you have a server-style setup to put it on.
MLX
MLX is not a chat app but an array framework for Apple silicon from Apple's machine learning research group, with Python, C++, C and Swift APIs. Programs built on it, and the model conversions that come with it, are how many Mac users get models running natively. You will usually meet MLX indirectly, through LM Studio or a Python package, rather than installing it by name. Choose it if you are on a Mac and want to script or experiment.
A sensible first week
- Check how much memory you have, and whether it is a discrete GPU's VRAM or a shared pool.
- Install Ollama or LM Studio, whichever fits your temperament.
- Pull a small model at a 4-bit quantisation, around 8 billion parameters, and talk to it.
- Watch your memory usage while it runs. This is the single most educational thing you can do.
- Only then try something bigger, and read how much context length you are asking for. Long contexts cost memory on top of the weights.
If that goes well, the next questions are about hardware: whether to buy for capacity or for speed, and how a graphics card compares with a machine that shares one big pool of memory. Those are covered in how to choose a mini PC for home AI.
What we could not tell you
We have not benchmarked any of these tools and quote no speeds here. Which runtime is fastest depends on your hardware, the model and the version, and it changes month to month. Test on your own machine with a model you intend to use.
