Four ways to run a model

The big idea

You found a model on the Hub and you know roughly how big it is. Now you want it to answer you. There are four common ways to get there, and they differ in one question above all: whose computer does the work?

Think of dinner. You can eat at a restaurant (someone else's kitchen, you pay per dish), heat a ready meal at home (your kitchen, someone else did the hard prep), cook from scratch (your kitchen, full control, more effort), or send each guest a meal kit (their kitchen does the work). In model terms:

1. Hosted: Inference Providers run the model on their GPUs and you call an API.
2. A local app: llama.cpp or Ollama runs a ready-made GGUF file on your machine.
3. Python: the transformers library loads the original weights in your own code.
4. The browser: Transformers.js runs a small model on your visitor's device.

Every snippet below that could be run on a test machine was run, and the output shown is the real output. The two that need an account or an app we did not install are shown exactly as the official docs give them and marked as such.

How it works

1. Hosted Send a request with a Hugging Face token; a partner provider runs the model on its GPUs. Nothing to install but a client; you pay per use.
2. Local app llama.cpp or Ollama downloads one GGUF file and runs it on your machine. No account, no per-use cost, but your memory sets the limit.
3. Python transformers loads the original weights with PyTorch inside your program. Full control, the most setup.
4. Browser Transformers.js downloads a small model into the visitor's browser and runs it there. No server at all.

Way 1: hosted, with Inference Providers

Inference ProvidersHugging Face's serverless API that forwards your request to a partner company running the model on its GPUs, billed per use through your Hugging Face account. is one API in front of many partner companies (Groq, Together, Fireworks, Cerebras and others). You send a request with a Hugging Face token; Hugging Face routes it to a provider that already has the model running. It is the only way here that can run a model far bigger than your own computer.

This is the example from the Inference Providers docs. It was not run for this lesson (it needs a token and credits):

import os
from huggingface_hub import InferenceClient

client = InferenceClient()

completion = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {
            "role": "user",
            "content": "How many 'G's in 'huggingface'?"
        }
    ],
)

print(completion.choices[0].message)

You log in once with hf auth login. By default the router picks the fastest provider for that model; add :cheapest to the model id for the lowest price per output token, or a provider name such as :groq to choose one yourself.

Pick it when the model is too big for your hardware, you want something running in minutes, or you are prototyping before deciding to self-host. Avoid it when your data must not leave your machine, or steady heavy use would make per-token costs add up.

Way 2: a local app with a GGUF file

Local apps run a single GGUF file (see Can I run it? What a model needs ) on your own CPU or GPU. On a model page, "Use this model" lists the apps and gives a ready command. Two popular ones:

llama.cpp downloads the file you name (here a 4-bit file of a 0.5B chat model, 398 MB) and starts a server with an OpenAI-compatible endpoint. Tested with llama.cpp build b11755 on an Intel Mac, CPU only:

llama-server -hf bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M --port 8094

The server printed, among other lines:

srv    load_model: loading model 'bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M'
srv  llama_server: model loaded
srv  llama_server: listening on http://127.0.0.1:8094

Then, from another terminal:

curl -s http://localhost:8094/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "In one sentence, what is a GGUF file?"}],
    "max_tokens": 60,
    "temperature": 0
  }'

The reply (trimmed to the interesting fields):

{
  "choices": [{
    "finish_reason": "stop",
    "message": {
      "role": "assistant",
      "content": "A GGUF file is a type of file format used for storing and transmitting data, particularly in the context of digital signatures and digital signatures."
    }
  }],
  "model": "bartowski/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M",
  "usage": { "completion_tokens": 29, "prompt_tokens": 40, "total_tokens": 69 }
}

It works, and it is confidently wrong: a 0.5B model is tiny, which is why it runs anywhere. The plumbing is the lesson here; for useful answers you would pick a bigger model that still fits, such as an 8B in Q4_K_M on a 16 GB laptop.

Because the endpoint speaks the OpenAI chat format, the same InferenceClient from Way 1 can talk to it. The huggingface_hub docs show this for local servers, and this part was tested against the server above:

from huggingface_hub import InferenceClient

client = InferenceClient(model="http://localhost:8094")
response = client.chat.completions.create(
    messages=[{"role": "user", "content": "Say hello in five words."}],
    max_tokens=20,
)
print(response.choices[0].message.content)

Hello, how are you?

Four words, not five. Small models follow instructions loosely.

Ollama does the same with less to configure. As documented by Hugging Face (not run here; Ollama was not installed on the test machine):

ollama run hf.co/{username}/{repository}
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

Without a tag, Ollama uses Q4_K_M when the repo has it, and otherwise picks a reasonable quantization from the repo.

Pick it when you want a private, offline chat model or a local API for your own tools, with no bills. Avoid it when the model you need does not fit your memory, or it has no GGUF version yet.

Way 3: Python with transformers

The transformers library loads the original weights, as published, into PyTorch. It is the most flexible route: any task (text, speech, images), access to every layer, fine-tuning later. It is also the heaviest to install.

The quickest version is a pipeline, which handles the tokenizer, chat template and decoding for you. Tested with transformers 5.20.2 and torch 2.14.1 on CPU (in Docker):

from transformers import pipeline

chat = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct")
messages = [{"role": "user", "content": "In one sentence, what is Hugging Face?"}]
out = chat(messages, max_new_tokens=40, do_sample=False)
print(out[0]["generated_text"][-1]["content"])

Hugging Face is an open-source platform for natural language processing and machine learning that provides tools and resources to build and deploy models for various applications like text generation, sentiment analysis, and more.

The first run downloads the model (about 1 GB of weights) into the Hugging Face cache; later runs load it from disk.

For more control, load the model and tokenizer yourself. Two arguments matter for memory: dtype="auto" keeps the weights in the precision they were saved in, and device_map="auto" puts them on the fastest device available (a GPU if there is one), spilling to the CPU if they do not fit.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Qwen/Qwen2.5-0.5B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")
print("dtype:", model.dtype, "| device:", model.device)
print(f"parameters: {model.num_parameters():,}")
print(f"memory for weights: {model.get_memory_footprint() / 1e9:.2f} GB")

messages = [{"role": "user", "content": "Name three things you can find on the Hugging Face Hub."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
ids = model.generate(**inputs, max_new_tokens=60, do_sample=False)
print(tokenizer.decode(ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

dtype: torch.bfloat16 | device: cpu
parameters: 494,032,768
memory for weights: 0.99 GB
Sure! Here are three things that you can find on the Hugging Face Hub:

1. **Open Source Libraries**: The Hugging Face Hub is home to a vast collection of open-source libraries and tools for various tasks in natural language processing (NLP). These include models, datasets, frameworks,

The numbers match the arithmetic from Can I run it? What a model needs : 494 million parameters in bfloat16 at 2 bytes each is 0.99 GB, the same as the file on the Files tab. The answer stops mid-sentence because max_new_tokens=60 ran out.

Precision is where memory goes wrong. The transformers docs recommend dtype="auto" because PyTorch's own default is float32. In the version tested here, leaving dtype out also kept bfloat16, but forcing float32 doubled the memory:

no dtype argument -> torch.bfloat16 0.99 GB
dtype='float32'   -> torch.float32 1.98 GB

For an 8B model that is the difference between 16 GB and 33 GB. Writing dtype="auto" makes your code keep the saved precision whatever version it runs on.

For a big model on a CUDA GPU, the transformers docs add 4-bit loading with bitsandbytes: pass quantization_config= set to BitsAndBytesConfig(load_in_4bit=True). That cuts weight memory to roughly a quarter ( Quantization for inference ). It was not tested here, as the test machine has no CUDA GPU.

Pick it when you are building something in Python, need a task other than chat, want to fine-tune, or need the exact published model. Avoid it when you just want to chat with a big model on a laptop: a GGUF app is lighter.

Way 4: in the browser with Transformers.js

Transformers.jsA JavaScript library from Hugging Face with the same pipeline API as transformers, running models with ONNX Runtime in the browser or in Node.js. brings the same pipeline idea to JavaScript. Models run with ONNX Runtime, on the CPU through WebAssembly by default or on the GPU with WebGPU. Nothing runs on your server: each visitor downloads the model once and runs it on their own device.

Tested with @huggingface/transformers 4.3.1 on Node.js 22 (Linux, in Docker; the same code runs in a browser page):

import { pipeline } from "@huggingface/transformers";

const classify = await pipeline(
  "sentiment-analysis",
  "Xenova/distilbert-base-uncased-finetuned-sst-2-english",
);
console.log(await classify("I finally got a model running on my own laptop!"));
console.log(await classify("The download took forever and then crashed."));

[ { label: 'NEGATIVE', score: 0.7719244956970215 } ]
[ { label: 'NEGATIVE', score: 0.9987988471984863 } ]

That first line is a real mistake: a small sentiment model read "finally" as a complaint. The second is right, and confidently so. This is the normal experience with small models, and the reason to try a model on your own examples before you build on it.

In WebAssembly the default is 8-bit weights (q8); you can pass { dtype: "q4" } for smaller downloads or { device: "webgpu" } to use the GPU. WebGPU is still experimental in many browsers, so keep a CPU fallback.

Pick it when you want a feature in a web page with no server bill and private data that never leaves the user's device: classification, embeddings for search, speech to text with a small model. Avoid it when you need a large chat model: the download and the visitor's memory are the limits.

Going deeper

One request format everywhere

Inference Providers, llama-server, and many other servers accept the same OpenAI-style chat completions request: a list of messages with roles. That is why one InferenceClient worked for both a hosted model and a local server above. Write your app against that format and you can move between hosted and local by changing one URL, which makes "start hosted, move local later" (or the reverse) a small change.

Renting your own GPU

Between "their API" and "my laptop" sits a rented GPU: a Space or Job with GPU hardware, or a dedicated Inference Endpoint. You pay by the hour whether or not requests arrive, in exchange for running exactly the model and settings you want.

Trade-offs

A quick way to decide:

1. Is the model bigger than your memory (the lab in Can I run it? What a model needs )? Use Inference Providers, or rent a GPU.
2. Do you just want to chat with it, or call it from your own tools? Use a GGUF in llama.cpp or Ollama.
3. Are you writing Python, need a non-chat task, or want to fine-tune? Use transformers.
4. Should it run inside a web page with no server? Use Transformers.js with a small model.

Common mistakes

• Running a model locally that does not fit and blaming the tool: do the memory sum from Can I run it? What a model needs first.
• Committing a Hugging Face token to Git, or putting it in front-end code that every visitor can read.
• Expecting a 0.5B model to answer like a frontier model: tiny models are for plumbing and simple tasks.
• Loading 16-bit weights as float32 in transformers (doubling the memory), or leaving them on the CPU when a GPU is there: pass dtype="auto" and device_map="auto".
• Shipping a large model to the browser and making every visitor download hundreds of megabytes before anything works.

In the wild

Hugging Face

Inference Providers documentation (opens in a new tab)

The docs describe one API in front of many partner providers, from Groq and Cerebras to Together and Replicate, covering chat, image and video generation, embeddings and speech to text. It works with the Python and JavaScript clients or as a drop-in replacement for the OpenAI chat completions API, uses a single Hugging Face token, and passes provider rates through with no extra markup. That is Way 1 in this lesson, and the reason the same client code also works against a local server.

Check yourself

Recap