Local LLM: Setup Steps, Hardware Needs and Privacy Limits

In brief

The easiest way to run a local LLM is Ollama or LM Studio: install the app, download a model, chat offline. An 8B model is a 4.9 GB download on Ollama, and LM Studio recommends 16 GB of RAM, as of November 2026. Prompts stay on your machine, but small models trail cloud ones.

What is a local LLM?

A local LLM is a large language model that runs on hardware you own, so your prompts never leave your machine. Four apps do most of the work, and each states its own first step.

A small brushed silver laptop on a black plinth with a glowing rainbow glass cube resting on its keyboard and a fine thread running to an unplugged cable lying coiled beside it, in a charcoal studio under white rim light.
ToolWhat it isHow you startFirst run
OllamaAn app and server for open modelsDownload from ollama.com/downloadollama run gemma4:e2b, a 7.2 GB download per its quickstart
LM StudioA desktop chat app with a model search tabInstall the app for Mac, Windows or LinuxSearch the Discover tab, download, chat
llama.cppThe open source engine many apps build onAn install script, prebuilt binaries or a source buildllama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
JanA desktop app with local and cloud modelsDownload the app for your systemIts default model downloads on first launch

The model itself is a file of weights. The runner loads it into memory and does the math on your CPU or graphics card. That is all "local" means.

What is Ollama, and how does it compare with LM Studio?

Ollama is a runner that downloads open models and starts them with one command. Its quickstart for Windows and macOS is to download the app and open it. On Linux you download it and start the server with ollama serve if it isn't already running. Then ollama run pulls a model and opens a chat, and typing /bye leaves it.

LM Studio covers the same ground with windows instead of commands. You search for a model in the app, download it and chat. Jan works the same way and loads a default model on first launch. Neither needs a terminal.

Pick Ollama if you want a command line and a local server that other tools can call. Pick LM Studio or Jan if you want a chat window and a model browser. Both run the same open weights, so the answers depend on the model you load, not the app. Two top ranking guides skip this choice. One walks through Mac only, and the other covers LM Studio alone.

How do you run an LLM locally on Windows, Mac or Linux?

You install a runner, download a model and chat. The steps below come from each tool's own docs, read on 22 November 2026.

  1. Ollama: download it from ollama.com/download and open it. On Linux, run ollama serve if the server isn't up. Then run ollama run gemma4:e2b and wait for the download.
  2. LM Studio: install the app. On Mac it needs an Apple Silicon chip and macOS 14 or newer, and Intel Macs aren't supported. On Windows it needs a CPU with AVX2. On Linux it ships as an AppImage and needs Ubuntu 20.04 or newer. Then search for a model and chat.
  3. llama.cpp: run the install script (curl -LsSf https://llama.app/install.sh | sh on Linux and Mac, irm https://llama.app/install.ps1 | iex in PowerShell), or use a prebuilt binary. Then llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF downloads and runs a small model from Hugging Face. llama serve with the same flag starts a local API server.
  4. Jan: download the app and launch it. Its quickstart says Jan downloads its default model on first launch, and you can chat once that finishes.

Phones are a different story. The four tools above document desktops and servers, so this page doesn't claim a phone route.

How much RAM or VRAM does a local LLM need?

The file you download has to fit in memory, with room left for the chat itself. Ollama's model pages list the file sizes, and the table adds what the same model would weigh at full precision.

Model on OllamaDownloadFull precision (about)Where it fits (my reading)
qwen3:1.7b1.4 GB3.4 GBAny 8 GB machine
qwen3:4b2.5 GB8 GB8 GB with a short chat
llama3.1:8b4.9 GB16 GBTight at 8 GB, easy at 16 GB
qwen3:14b9.3 GB28 GB16 GB
qwen3:32b20 GB64 GBA 24 GB GPU or 32 GB of RAM
llama3.1:70b43 GB140 GBA 64 GB machine
qwen3:235b142 GB470 GBA workstation or server

The download column is what each Ollama page lists. The full precision column is my arithmetic: two bytes for each parameter. The last column is my reading of file size plus headroom, not a vendor claim. Ollama's pages don't state the quantization. The 8B file works out to about 5 bits for each parameter, which fits a 4 bit class format.

One model family spans 1.4 GB to 142 GB, so the size you pick sets the machine you need
1 GB10 GB100 GBQwen3 1.7BQwen3 4BQwen3 8BQwen3 14BQwen3 30BQwen3 32BQwen3 235B

LM Studio's own page sets the floor. It recommends 16 GB of RAM on Mac and on Windows, and at least 4 GB of dedicated VRAM on Windows. It says an 8 GB Mac may still work with smaller models and modest context sizes. Overchat's guide, the current number one result, gives a looser rule of thumb of 0.5 to 1 GB of RAM per billion parameters for quantized models. It states no source for it, and the table above is built from published file sizes instead.

What does quantization change?

Quantization stores each weight in fewer bits, so the file shrinks and runs faster. A 16 bit file spends 16 bits on every parameter and a 4 bit file spends 4, so by my arithmetic the 4 bit file is about a quarter of the size. That is why llama3.1 8B is 4.9 GB on Ollama instead of about 16 GB.

The cost is quality. Overchat's guide calls 4 bit the sweet spot for most people and says the lowest settings lose noticeably more on hard reasoning. Treat that as one site's opinion, not a measured result. The llama.cpp thread below does measure speed, and it shows the 4 bit file running about 3 times faster on an M2 Pro.

When a model is too big for your graphics card, runners can split it, with some layers on the card and the rest in system memory. It works, but the layers in system memory set the pace, which is why a file bigger than your VRAM feels slow.

How fast is a local LLM?

Speed depends on your chip and on the quantization. The best public dataset is a llama.cpp community thread that times one model across Apple chips. It uses the original LLaMA 7B, so read it as a comparison between chips, not a forecast for newer models.

Apple chip4 bit speed (tokens a second)16 bit speed (tokens a second)
M1, 7 GPU cores14.19not listed
M2 Pro, 16 cores37.8712.47
M2 Max, 30 cores60.9924.16
M3 Max, 40 cores66.3125.09
M2 Ultra, 60 cores88.6439.86

Two things stand out. The M2 Pro ran the 4 bit file about 3 times faster than the 16 bit one (37.87 against 12.47). So shrinking a model saves memory and time. And an M1 still managed 14 tokens a second on the small model, which is enough to read as it writes.

No source I read gives a fair speed for a 30B or 70B model on a laptop, so this page doesn't invent one. Run your own test with step 4 above.

Is a local LLM private?

Chats with a local model stay on your machine, and the vendors say so. LM Studio's offline page says "Nothing you enter into LM Studio when chatting with LLMs leaves your device." It says document processing is local too. Jan's quickstart says local models run entirely on your machine with no API key.

The limits are real, though. LM Studio needs the internet to search for models, to download them and to check for runtimes and app updates. Jan also offers cloud models, which its page says run on a provider's servers and need a key, and Ollama's quickstart offers cloud models too. A cloud model in a local app is still a cloud model.

Privacy also doesn't mean safety. A local model can still invent facts, and you are the one who checks the license and where the file came from. How the major chatbots handle your data compares what the hosted apps do with chats.

Can you self host an LLM for a team?

Yes. Self hosting an LLM means running the same engine on a server and letting other people connect to it. llama.cpp's llama serve starts an OpenAI compatible API server, and Ollama runs as a local server as well. The catch is memory. A 32B model is a 20 GB download and a 70B model is 43 GB before any chat history, per Ollama's pages.

That is a server purchase, not a laptop. For which open model to pick once you have the hardware, the open source LLM guide has the shortlist.

Three machines, three realistic setups

These cases use the vendor figures above. The model choices are my arithmetic, not a test I ran.

An 8 GB MacBook Air with an M1. LM Studio says 8 GB may work with smaller models and modest context sizes, and the M1 is supported on macOS 14 or newer. Start with a 1.7B or 4B model, 1.4 GB or 2.5 GB on Ollama. A good result is a clear three paragraph summary of a pasted article.

A Windows laptop with 16 GB of RAM and 4 GB of VRAM. That matches LM Studio's recommendation exactly. An 8B file at 4.9 GB is bigger than 4 GB of VRAM, so part of it will run from system memory and slow down. A 4B model at 2.5 GB fits the card. A good result is working code for a small function you described.

A small team wanting a private assistant. One 64 GB machine can hold a 70B file at 43 GB, with room for chats. Start llama serve or ollama serve on it and point the team's tools at that address. A good result is every teammate getting an answer without any prompt leaving the building.

What can you ask an AI about running a local LLM?

Paste any of these into Claude, ChatGPT, Gemini or Whizi, and fill the brackets. They ask for questions to settle, not for a fake benchmark.

  1. Hardware check: "I have [laptop or desktop model] with [RAM] and [graphics card and VRAM]. List the three largest models from the Ollama library I can run, give each download size, and say which numbers you are sure of."
  2. Runner choice: "I'm [a developer or a non coder] on [Windows, Mac or Linux]. Compare Ollama and LM Studio for me in a table and pick one, using only their own docs."
  3. License check: "Here is a model card: [paste]. Can I use this model in a paid product? Quote the license lines, and say what to ask a lawyer."
  4. Slow answers: "My local model makes [number] tokens a second on [chip]. List what I can change in order of effort: a smaller model, a smaller quantization or a shorter context."
  5. Privacy audit: "Here is my setup: [apps and models]. List every point where data could leave my machine, such as downloads, updates and cloud models."
  6. Team server: "We are [number] people who need a private assistant. Draft a plan for one server running llama.cpp, with the memory I need for a [size] model and what could go wrong."

Prompt 1 is the one to check. Models often invent exact file sizes, so compare its numbers against the model page.

What mistakes ruin a local LLM setup?

Five mistakes cause most bad first runs.

  1. Choosing by parameter count alone. The fix: read the download size, because a 32B file is 20 GB on Ollama and a 4B file is 2.5 GB.
  2. Forgetting the chat itself needs memory. The fix: leave headroom, as LM Studio's note about modest context sizes on 8 GB Macs implies.
  3. Expecting cloud quality from a small model. The fix: test your real tasks before you rely on it.
  4. Thinking a cloud model in a local app is private. The fix: check which model you picked.
  5. Downloading a file from an unknown uploader. The fix: take models from the publisher's own page and read the license first.

Where does Whizi fit if you want an answer today?

Whizi runs in the cloud, so it isn't a local LLM, and nothing here changes that. If you want one app instead of a pile of runners, it helps with the questions you ask before and after you install one. The free plan gives you 10 messages a day on Whizi AI, Auto and Claude Haiku, with no card.

Pro carries Claude, GPT and Gemini in one app, and its first week is $1.99. Side by side Compare, on Pro and Powerhouse, puts two models on one prompt, which suits prompt 1 above. Web search is on by default in a new chat. A local model has no live search unless you add a tool for it.

What can a local LLM not do?

It can't match the biggest hosted models on a normal computer. Ollama's own pages show why: the largest Qwen3 file is 142 GB, and llama3.1 405B is 243 GB, far beyond a laptop.

It also can't fix hallucination. A model on your desk invents facts as readily as one in the cloud. Why AI gets things wrong explains the mechanism. For anything legal, medical or financial, a qualified person still has to check the answer.

Workflow checklist
  • Read the download size on the model page, not just the parameter count
  • Keep the file well under your RAM or VRAM so the chat has room
  • Start with 8B or smaller, then size up if it feels fast
  • Turn Wi Fi off once to prove the model runs on your machine
  • Check the license and the source before you download a file
Common questions

Frequently asked questions

Can I run a local LLM without a GPU?

Yes, with a small model. LM Studio's docs recommend 4 GB of dedicated VRAM on Windows but also say 8 GB Macs may work with smaller models. On Apple Silicon the chip's own graphics do the work. Expect slower answers than on a good graphics card.

What is the difference between Ollama and LM Studio?

Ollama starts models from a terminal command, and LM Studio gives you a chat window with a model search tab. Both run open weights on your machine, so the model you load matters more than the app. See the table near the top for each first step.

Can I run an LLM without internet?

Yes, once the model is downloaded. LM Studio's offline page says nothing you enter while chatting leaves your device. You still need a connection to search for models, download them, fetch runtimes and update the app.

What is the best way to run an LLM locally on a Mac?

Use an app that supports Apple Silicon. LM Studio needs an Apple Silicon Mac on macOS 14 or newer and recommends 16 GB of RAM, though an 8 GB Mac may work with smaller models. Ollama and Jan also document desktop installs.

Is a local LLM as good as ChatGPT?

Not at the sizes a normal computer can hold. The largest models are 142 GB or more on Ollama, so a laptop runs smaller ones and trades some quality for privacy. Test your real tasks on a small model first.

Is a local LLM more private than ChatGPT?

For chats with a local model, yes: LM Studio says nothing you enter while chatting leaves your device. Downloads and cloud models still use the internet. To see what ChatGPT does with chats, use the does ChatGPT train on your chats tool.