Can I run it? What a model needs

The big idea

Here is the question almost everyone asks first: can my 16 GB laptop run an 8B model? Take a real one, Qwen3-8B. Its model page says it has 8,190,735,360 parameters stored as BF16, and its Files tab adds up to 16.4 GB. That is more than the whole laptop, before the operating system and your browser take their share. So, as published: no.

But the same model is also published as a GGUFA single-file model format used by llama.cpp, Ollama and other local apps. It stores the weights, usually quantized, together with metadata such as the chat template. file squeezed to about 4.5 bits per weight. That file is 5.0 GB. It loads with room to spare. Same model, same laptop, a different answer, and the difference is just arithmetic you can do before you download anything.

Think of moving into a flat. The furniture (the weights) has to fit inside, and you need some floor left to walk around and cook (working memory while the model runs). Flat-pack furniture (quantization) takes far less room, at a small cost in finish. This lesson teaches you to measure the furniture and the flat before the van arrives.

How it works

Step 1: weights are parameters times bytes

A model is mostly a huge list of numbers called parametersThe learned numbers inside a model. A model's size, such as 8B, is how many of them it has, in billions. or weights. The "8B" in a model's name means about 8 billion of them. How much room each one takes depends on its precisionHow each weight is stored: float32 uses 4 bytes, float16 and bfloat16 use 2, 8-bit formats about 1 and 4-bit formats about half a byte.:

Most models on the Hub are published in 16-bit (BF16 or F16), so "billions of parameters times 2" is the size of the download and the memory you need just to hold it. An 8B model is about 16 GB; a 70B model is about 140 GB.

Step 2: GGUF quantizations and their real sizes

QuantizationStoring weights in fewer bits, for example 4 instead of 16, using a shared scale per small block of weights. Saves memory at a small cost in quality. stores each weight in fewer bits. GGUF repos usually offer several versions of one model, named by their quantization type. Each block of weights also stores a scale (and sometimes a minimum), so the real cost is a little more than the headline bits. The Hugging Face GGUF docs list:

Files you download are usually called something like Q4_K_M or Q5_K_M. The _M versions mix in some tensors at higher precision, so they run a bit bigger than the table. The real files in the Qwen/Qwen3-8B-GGUF repo: Q8_0 is 8.7 GB (8.51 bits per parameter), Q6_K 6.7 GB (6.57), Q5_K_M 5.9 GB (5.71) and Q4_K_M 5.0 GB (4.91). Very small models shrink less: the Q4_K_M file of a 0.5B model works out to 6.44 bits per parameter, partly because the mixed-in higher-precision parts are a bigger share of a small model.

How much quality do you lose? 8-bit is usually very close to the original, and good 4-bit versions are usually close enough for chat. Quantization for inference explains how the scales work and how to check the quality cost yourself. A common first choice for a local chat model is Q4_K_M: Ollama picks it by default when a repo has it.

Step 3: add working memory, and the KV cache for long chats

Holding the weights is not the same as running them. Hugging Face's own memory estimator measures only the loading size and its docs say to add up to about 20 percent for inference, a figure they take from EleutherAI. So the 5.0 GB Q4_K_M file needs about 6 GB to run comfortably.

Long conversations need more. The model keeps a KV cacheStored keys and values for every earlier token in the conversation, so each new token does not recompute them. It grows with the length of the chat. for every token it has seen, and it grows with the length of the chat. For a short chat it is small; for a very long document it can be gigabytes. GPU memory math shows exactly how to calculate it.

Where the memory lives

The number you compare against depends on what runs the model:

• NVIDIA GPU: the model should fit in the card's own memory, called VRAMVideo RAM: the memory on a graphics card. Models run fast only when their weights fit in it., which is often 8 to 24 GB on a gaming card. Your laptop's 32 GB of RAM does not count here. Libraries can spill layers that do not fit to the CPU, but those layers then run much more slowly.
• Apple Silicon Mac: the CPU and GPU share one pool of unified memoryOn Apple Silicon Macs, a single pool of memory shared by the CPU and the GPU, so a model can use most of the machine's RAM for GPU work., and llama.cpp uses the GPU through Metal. A 16 GB Mac can use much of that for a model, but macOS and your apps live in the same 16 GB.
• CPU only: the model runs from ordinary RAM. It works, especially with GGUF files in llama.cpp or Ollama, but generation is slower than on a GPU, so small or quantized models are the realistic choice.

Check before you download

Three ways, from quickest to most precise:

1. The Files tab. Add up the weight files (.safetensors, or the single .gguf you would pick). The download size is the size of the weights. The model page also shows the parameter count and tensor type.
2. accelerate estimate-memory. It reads only the model's config and builds the model on a virtual device, so it downloads almost nothing and works even for huge models. This is the real output for a small chat model:

pip install accelerate transformers
accelerate estimate-memory Qwen/Qwen2.5-0.5B-Instruct

The tool prints a boxed table; it is shown here as a plain table. Its "GB" are 1024-based: 942.29 MB is the model's 494,032,768 parameters times 2 bytes, which is 988 million bytes, the same as the 0.99 GB .safetensors file on the Files tab.

3. The model memory usage Space. Hugging Face hosts the same estimator as a web app (hf-accelerate/model-memory-usage), where you paste a model id. A community Space, Vokturz/can-it-run-llm, goes further and lets you enter your GPU.

Try it

1. Start at the default: an 8B model in F16 on 16 GB. It needs 16.0 GB of weights plus 3.2 GB of overhead, 19.2 GB: does not fit. Switch the format to Q4_K. How much is left, and what is the largest model that fits comfortably in Q4_K on this machine?
2. Choose 70B and 48 GB. Which formats fail? Q4_K only just loads (47.3 GB needed, "tight"). Now raise the memory to 64 GB and then 80 GB. At 80 GB, why can you afford Q5_K but still not F16?
3. Pick 3B in F16 on 8 GB, a typical small laptop. It is tight (7.2 GB). Which format would you pick, and what else would you have to close to make room?

The same arithmetic for real models, using the exact parameter counts from their Hub pages:

# Will it fit? Weights = parameters x bits per weight / 8, then add up to 20 percent
# for inference (the Accelerate docs' rule of thumb). Parameter counts are the exact
# totals the Hub shows for each model (Qwen2.5 Instruct models and Qwen3-8B).
MODELS = {
    "Qwen2.5-0.5B": 494_032_768,
    "Qwen2.5-3B": 3_085_938_688,
    "Qwen3-8B": 8_190_735_360,
    "Qwen2.5-14B": 14_770_033_664,
    "Qwen2.5-32B": 32_763_876_352,
    "Qwen2.5-72B": 72_706_203_648,
}
BITS = {"F32": 32, "F16": 16, "Q8_0": 8.5, "Q6_K": 6.5625, "Q4_K": 4.5}

def gb(params, bits):
    return params * bits / 8 / 1e9

print("GB of weights")
print(f"{'model':13}" + "".join(f"{f:>7}" for f in BITS))
for name, n in MODELS.items():
    print(f"{name:13}" + "".join(f"{gb(n, b):7.1f}" for b in BITS.values()))

budget = 16  # GB: a 16 GB laptop, before the OS and your browser take their share
print(f"\nWith 20 percent extra for inference, what fits in {budget} GB?")
for name, n in MODELS.items():
    ok = [f for f, b in BITS.items() if gb(n, b) * 1.2 <= budget]
    print(f"  {name:13} {', '.join(ok) if ok else 'nothing: rent a GPU or go hosted'}")

# Check the math against the real files in the Qwen/Qwen3-8B-GGUF repo (bytes)
real = {"Q8_0": 8_709_518_112, "Q6_K": 6_725_899_040, "Q4_K_M": 5_027_783_488}
print("\nQwen3-8B: predicted vs real GGUF file")
for f, size in real.items():
    pred = gb(MODELS["Qwen3-8B"], BITS[f[:4]])
    print(f"  {f:7} predicted {pred:4.1f} GB, real file {size / 1e9:4.1f} GB")

GB of weights
model            F32    F16   Q8_0   Q6_K   Q4_K
Qwen2.5-0.5B     2.0    1.0    0.5    0.4    0.3
Qwen2.5-3B      12.3    6.2    3.3    2.5    1.7
Qwen3-8B        32.8   16.4    8.7    6.7    4.6
Qwen2.5-14B     59.1   29.5   15.7   12.1    8.3
Qwen2.5-32B    131.1   65.5   34.8   26.9   18.4
Qwen2.5-72B    290.8  145.4   77.3   59.6   40.9

With 20 percent extra for inference, what fits in 16 GB?
  Qwen2.5-0.5B  F32, F16, Q8_0, Q6_K, Q4_K
  Qwen2.5-3B    F32, F16, Q8_0, Q6_K, Q4_K
  Qwen3-8B      Q8_0, Q6_K, Q4_K
  Qwen2.5-14B   Q6_K, Q4_K
  Qwen2.5-32B   nothing: rent a GPU or go hosted
  Qwen2.5-72B   nothing: rent a GPU or go hosted

Qwen3-8B: predicted vs real GGUF file
  Q8_0    predicted  8.7 GB, real file  8.7 GB
  Q6_K    predicted  6.7 GB, real file  6.7 GB
  Q4_K_M  predicted  4.6 GB, real file  5.0 GB

The first table is the furniture; the second is what fits in the flat. The last lines check the formula against the real files: Q8_0 and Q6_K match to a tenth of a gigabyte, and Q4_K_M is 0.4 GB bigger because of the higher-precision tensors it mixes in.

Trade-offs

Common mistakes

• Comparing a model with your laptop's RAM when it will run on a GPU: what counts is the GPU's VRAM.
• Forgetting the operating system and your other apps, especially on a Mac where everything shares unified memory.
• Budgeting for the weights only, then running out of memory in a long chat because of the KV cache.
• Assuming a 4-bit file is exactly a quarter of the 16-bit size. Q4_K_M files are closer to 30 percent, and small models shrink less.
• Using the "Training using Adam" column when you only want to run the model, or the reverse.

In the wild

Check yourself

Recap