AI AI-16.05

Accounts, tokens, downloads and datasets

Get a Hugging Face token without leaking it, download models with the hf CLI, see where files land on disk, work offline and load your first dataset.

Beginner 14 min Last reviewed 2026-10-20

The big idea

The first time you ran a pipeline, 268 MB of model weights arrived from somewhere and landed somewhere, and the second run started faster. This lesson opens that box. Who are you to the Hub, how do files come down, where do they live on your disk, and how do you keep working when the network is gone?

Think of the Hub as a public library and your laptop as a bookshelf at home. You can borrow public books without a card. A card (your account and its access token) lets you borrow restricted books, and some cards also let you put new books on the library's shelves. Once a book is on your shelf at home, you never fetch it again. And on a plane, you tell yourself "no library today" and read what is on the shelf.

Datasets work the same way. They are repositories too, and the datasets library pulls them onto the same shelf, or reads them page by page without carrying the whole thing home.

How it works

The Hub Public models and datasets download with no account. Gated and private repos, and uploads, need a token.
Download hf download in a terminal, or hf_hub_download in Python. Libraries such as transformers call the same code.
The cache Each file is stored once, named by its hash. A snapshot folder per commit points at those files.
Your code pipeline() and load_dataset() find the files in the cache and load them from disk.
Offline HF_HUB_OFFLINE=1 skips the network entirely. Everything you need must already be in the cache.

Step 1: what an account unlocks

You do not need an account to download public models or datasets. Every download in this lesson ran with none. The Hub did print a hint each time:

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

An account becomes necessary for gated models (you must log in and agree to the authors' terms), private repos, uploading anything, creating Spaces, and paying for hosted inference. Signing up is free at huggingface.co/join.

Step 2: access tokens

Scripts cannot click "log in", so they send a token instead. You create tokens under Settings, Access Tokens, and each has a role:

Role	What it can do	Use it for
`read`	Read repos you can read, public and private	Downloading, inference
`write`	Also push to repos you can write to	Uploading a model, pushing a Space
`fine-grained`	Only the specific repos and actions you tick	Production apps, shared machines

Hugging Face recommends one token per app or machine (your laptop, a notebook, a server), so you can kill one without breaking the others, and fine-grained tokens for production: if one leaks, it opens only what you ticked.

Found someone else's leaked token, say in a public repo? The Hub has a revoke endpoint that needs no rights over the owner's account. This is the call from the security docs (not run here, since we had no leaked token to revoke):

# LEAKED_HF_TOKEN should contain the raw token value to revoke
curl -X POST "https://huggingface.co/api/credentials/revoke" \
  -H "Content-Type: application/json" \
  -d "{\"credentials\": [\"${LEAKED_HF_TOKEN}\"]}"

Step 3: logging in

On your own machine, log in once from a terminal:

hf auth login

By default it opens a browser flow: it prints a URL and a short code, you approve, and the token is saved under your Hugging Face home folder (~/.cache/huggingface/token). From then on every library on that machine uses it. You can also paste a token you created. Check who you are with hf auth whoami. Before logging in, that is exactly what it says:

$ hf auth whoami
Error: Not logged in. Run `hf auth login` first.

There are three other ways, in the order you will meet them:

In a notebook or script, from huggingface_hub import login; login() does the same login from Python.
The HF_TOKEN environment variable. It takes priority over the saved token, and it is how a Space secret or a Colab secret named HF_TOKEN passes a token in.
The token parameter on loading functions (from_pretrained, hf_hub_download, load_dataset). The docs discourage it unless you cannot store a token on the machine or must juggle several at once, and say to load the value from a secure place rather than typing it in. If you do use it, read it from the environment:

import os
from transformers import AutoModel

# The token comes from the environment, never from the source code.
model = AutoModel.from_pretrained("private/model", token=os.environ["HF_TOKEN"])

private/model is the docs' placeholder name; this snippet needs a real token and repo, so it is the one Python block here we did not run.

Step 4: gated models from a script

A gated model page (the badge you met in How to read a model page ) shows a form. You can only request access in the browser: clicking Agree shares your username and email with the authors. Some gates approve at once, others wait for a person. Authors can also withdraw access at any time.

After approval, the browser works because you are logged in. A script needs the token: run hf auth login once, or set HF_TOKEN, and the same from_pretrained or hf download call works for the gated repo. Nothing else in the code changes.

Step 5: the hf CLI

The hf command ships with huggingface_hub. Before downloading a model, ask what it would cost. A dry run lists the files without fetching them (progress bars trimmed):

$ hf download distilbert/distilbert-base-uncased-finetuned-sst-2-english --dry-run
[dry-run] Will download 16 files (out of 17) totalling 1.3G.
FILE                         SIZE
---------------------------- ------
.gitattributes               399.0
README.md                    10.5K
config.json                  -
map.jpeg                     81.6K
model.safetensors            267.8M
onnx/model.onnx              268.0M
pytorch_model.bin            267.8M
rust_model.ot                267.8M
tf_model.h5                  267.9M
...

(Eight small files are cut from the table. config.json shows - because an earlier command had already cached it.) Look at the big files: the same model, saved five times in five formats. Python with transformers needs only model.safetensors. Ask for just what you need:

$ hf download distilbert/distilbert-base-uncased-finetuned-sst-2-english \
    --include "*.json" --include "vocab.txt" --include "model.safetensors" \
    --exclude "onnx/*" --dry-run
[dry-run] Will download 3 files (out of 4) totalling 268.1M.

268 MB instead of 1.3 GB. Drop --dry-run and the real download took 13 seconds here. Running the same command again took under 2 seconds: nothing was fetched. For a single file, name it, and -q prints only where it landed:

$ hf download distilbert/distilbert-base-uncased-finetuned-sst-2-english README.md -q
/root/.cache/huggingface/hub/models--distilbert--distilbert-base-uncased-finetuned-sst-2-english/snapshots/714eb0fa89d2f80546fda750413ed43d93601a13/README.md

(Our test ran as root inside a container, so home is /root.) The Python equivalent is hf_hub_download(repo_id=..., filename=...), which returns the same path, and snapshot_download fetches a whole repo with the same allow_patterns and ignore_patterns filters.

Step 6: where the files go

Everything lands in one cache, ~/.cache/huggingface/hub, shared by every Hugging Face library. Set HF_HOME to move the whole Hugging Face folder (token included), or HF_HUB_CACHE to move only the downloads, for example to a bigger disk. Inside, each repo gets a folder. This is what our download left, condensed from find and ls -l:

models--distilbert--distilbert-base-uncased-finetuned-sst-2-english/
  blobs/       4 files named by their hash (the real bytes)
  refs/main    714eb0fa89d2f80546fda750413ed43d93601a13
  snapshots/714eb0fa89d2f80546fda750413ed43d93601a13/
    config.json           -> ../../blobs/b57fe5dfcb8ec3f9bab35ed427c3434e3c7dd1ba
    model.safetensors     -> ../../blobs/7c3919835e442510166d267fe7cbe847e0c51cd26d9ba07b89a57b952b49b8aa
    tokenizer_config.json -> ../../blobs/3ed34255a7cb8e6706a8bb21993836e99e7b959f
    vocab.txt             -> ../../blobs/fb140275c155a9c7c5a3b3e0e77a9e839594a938
  trees/714eb0fa89d2f80546fda750413ed43d93601a13.json

This is the trick that means a file is never downloaded twice. blobs holds the contents, named by hash. snapshots has one folder per commit, full of symlinks to those blobs. refs/main remembers which commit main meant. When the author fixes a typo in the README, the next download fetches only the new README; the 268 MB weights keep the same hash and are shared by both snapshots. (On Windows without symlink support the cache copies files instead, and loses this saving.)

To see and clean it, use hf cache:

$ hf cache ls
ID                                         SIZE LAST_ACCESSED LAST_MODIFIED REFS
---------------------------------------- ------ ------------- ------------- ----
dataset/HuggingFaceFW/fineweb             44.3K 4 minutes ago 4 minutes ago main
dataset/cornell-movie-review-data/rot... 888.5K 4 minutes ago 4 minutes ago main
model/distilbert/distilbert-base-unca... 268.1M 6 minutes ago 8 minutes ago main

Found 3 repo(s) for a total of 3 revision(s) and 269.0M on disk.

hf cache rm model/distilbert/distilbert-base-uncased-finetuned-sst-2-english deletes one repo (add --dry-run to preview: "About to delete 1 repo(s) totalling 268.1M."), and hf cache prune removes old revisions no branch points to and leftovers from interrupted downloads. The cache never deletes anything on its own, so check it when your disk fills up.

Step 7: working offline

Once the files are cached, you can cut the network. We ran a sentiment pipeline in a container with networking switched off and HF_HUB_OFFLINE=1 set:

# HF_HUB_OFFLINE=1 python offline.py
from transformers import pipeline

clf = pipeline("sentiment-analysis",
               model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
print(clf("This works perfectly on a plane with no internet."))

[{'label': 'POSITIVE', 'score': 0.9940329194068909}]

Then we ran the same script, still with no network, without the variable. It got the same answer, but only after the library spent minutes retrying the Hub ("Retrying in 8s [Retry 5/5]", "Temporary failure in name resolution"): 4 minutes 28 seconds against about a minute. Even with everything cached, libraries normally ask the Hub whether a newer version exists. HF_HUB_OFFLINE=1 tells them not to ask.

For a single call, local_files_only=True does the same. Ask for a file that was never downloaded and it fails fast and clearly, which is what you want in a locked-down server:

LocalEntryNotFoundError
Cannot find the requested files in the disk cache and outgoing traffic has been disabled. To enable hf.co look-ups and downloads online, set 'local_files_only' to False.

Step 8: your first dataset

Datasets are repos too, and the datasets library loads them in one line. A classic first one is Rotten Tomatoes movie reviews:

from datasets import load_dataset

ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
print(ds)
print(ds.features["label"])
for row in ds.select([0, 4500]):
    print(row["label"], row["text"][:70])

Dataset({
    features: ['text', 'label'],
    num_rows: 8530
})
ClassLabel(names=['neg', 'pos'])
1 the rock is destined to be the 21st century's new " conan " and that h
0 the most anti-human big studio picture since 3000 miles to graceland .

8,530 labeled reviews, each a row you can index like a list. (If you later publish a dataset built from real users' text, strip personal data first: see Privacy and PII handling .) The raw files sit in the Hub cache (888.5K in the listing above); datasets also keeps a converted Arrow copy in ~/.cache/huggingface/datasets. With HF_HUB_OFFLINE=1 and no network, the same script printed the same rows, after a note that it was "Using the latest cached version of the dataset".

Some datasets are far too big to download. The English part of FineWeb is 45 terabytes. With streaming=True you get an IterableDataset that reads rows as you loop:

ds = load_dataset("HuggingFaceFW/fineweb", split="train", streaming=True)
print(type(ds).__name__)
for row in ds.take(2):
    print(row["token_count"], repr(row["text"][:60]))

IterableDataset
717 'How AP reported in all formats from tornado-stricken regions'
821 'Did you know you have two little yellow, nine-volt-battery-s'

Afterwards, FineWeb took 44.3K in our cache. Streaming is not instant (on our test machine the first rows took a few minutes to arrive), and you cannot jump to row 5 million without reading the rows before it. For exploring a sample or training in one pass over the data, it is exactly right.

Try it

A toy version of the cache: blobs named by hash, one snapshot of pointers per commit. Commit bbbbbb changes only the README.

cache.pyruns with python3 (standard library)

# A toy model of the Hugging Face cache: blobs hold file contents (named by hash),
# snapshots hold one folder of pointers per commit. Standard library only, no network.
import hashlib

blobs = {}       # hash -> contents (the real bytes on disk)
snapshots = {}   # commit -> {filename: hash} (the symlinks)
fetched = 0      # bytes that came over the network

def download(commit, repo_files):
    global fetched
    print(f"download commit {commit}")
    snap = snapshots.setdefault(commit, {})
    for name, data in repo_files.items():
        h = hashlib.sha256(data).hexdigest()[:8]
        if h in blobs:
            print(f"  reuse  {name:<18} blob {h}")
        else:
            blobs[h] = data
            fetched += len(data)
            print(f"  fetch  {name:<18} blob {h}  {len(data):>7,} bytes")
        snap[name] = h

weights = b"w" * 268_000        # stands in for model.safetensors
v1 = {"config.json": b'{"dim": 768}', "README.md": b"# v1", "model.safetensors": weights}
v2 = dict(v1, **{"README.md": b"# v2, typo fixed"})   # the author only edits the README

download("aaaaaa", v1)
download("bbbbbb", v2)
download("aaaaaa", v1)          # asking again for a commit you already have

print()
print("snapshots:", {c: sorted(f) for c, f in snapshots.items()})
print(f"blobs on disk: {len(blobs)}, bytes fetched: {fetched:,}")
print(f"bytes a naive copy-per-commit would fetch: {3 * sum(map(len, v1.values())):,}")

Output

download commit aaaaaa
  fetch  config.json        blob 3458fbad       12 bytes
  fetch  README.md          blob b9f04a79        4 bytes
  fetch  model.safetensors  blob 0d0c7758  268,000 bytes
download commit bbbbbb
  reuse  config.json        blob 3458fbad
  fetch  README.md          blob a5d05122       16 bytes
  reuse  model.safetensors  blob 0d0c7758
download commit aaaaaa
  reuse  config.json        blob 3458fbad
  reuse  README.md          blob b9f04a79
  reuse  model.safetensors  blob 0d0c7758

snapshots: {'aaaaaa': ['README.md', 'config.json', 'model.safetensors'], 'bbbbbb': ['README.md', 'config.json', 'model.safetensors']}
blobs on disk: 4, bytes fetched: 268,032
bytes a naive copy-per-commit would fetch: 804,048

Downloading two commits and one repeat moved 268,032 bytes instead of 804,048. Which file accounts for almost all the saving, and why does its hash not change between commits?
Change the weights in v2 too (for example b"x" * 268_000). How many bytes are fetched now? That is what a new model version really costs.
The real cache adds refs/main. Why does the cache need to remember which commit main pointed to, if you want to load a model offline by the name main?

Trade-offs

Option	Wins when	Loses when
read token	Downloading and inference on your own machine	Opens every repo you can read if it leaks
fine-grained token	Production and shared machines: only the repos you tick	More setup; edit it when the app needs a new repo
write token	Pushing models and Spaces	The most damage if leaked; never use it where read is enough
Shared cache (default)	Every library shares files; nothing downloads twice	Hidden folder that quietly grows; check it with hf cache ls
--local-dir	You want plain files in a project folder	Files live outside the shared cache, so other projects cannot reuse them
Streaming	Huge datasets, quick looks, one pass of training	No random access; slower to start; network needed every time

Common mistakes

Pasting a token into a notebook or script that later gets shared or committed. Use hf auth login, HF_TOKEN or a secret, and delete any token that has been exposed.
Using a write token everywhere because it "just works", when a read or fine-grained token would do.
Downloading a whole repo with five copies of the same weights. Run --dry-run first and filter with --include.
Editing a file inside the cache. The snapshot files are links to shared blobs, so you corrupt the cache; copy the file out, or download with --local-dir.
Going offline without HF_HUB_OFFLINE=1, then wondering why a cached model takes minutes to load while the library retries the network.

In the wild

Hugging Face

huggingface_hub quick start (opens in a new tab)

The official quick start covers the same ground as this lesson: downloading a single file with hf_hub_download, which caches it so the next call loads from disk, pinning a version with the revision parameter, and authenticating with hf auth login, login(), the HF_TOKEN variable or a token parameter. It recommends a read token unless you need to create or edit a repository, warns that hard-coded tokens are a major leak risk, and ends with creating a repository and uploading a file, which needs a write token.

Check yourself

Your production app needs to download one gated model and nothing else. Which token should it use?
Show answer

C. A fine-grained token with read access to that one model A fine-grained token can be limited to exactly that repo, so a leak exposes one model instead of everything your account can read or write. Gated models always need a token from an account that was granted access.
A model author fixes a typo in the README. You call from_pretrained again. What is downloaded?
Show answer

B. Only the new README; the weights keep their hash and are shared The cache stores contents by hash. The new commit gets its own snapshot folder, but unchanged files point at blobs already on disk, so only the changed file is fetched.
You load a cached model on a train with no internet and it hangs for minutes before working. What fixes it?
Show answer

B. Set HF_HUB_OFFLINE=1 so the library does not try to reach the Hub Without the variable, the library checks the Hub for newer versions and retries when the network fails. HF_HUB_OFFLINE=1 skips those calls and reads straight from the cache.
You want to look at 100 rows of a 45 TB dataset. What do you do?
Show answer

B. load_dataset with streaming=True and take(100) Streaming reads rows as you iterate, so only what you touch comes over the network. The trade-off is no random access.
You find a coworker's Hugging Face token in a public GitHub repo. What is the right move?
Show answer

C. Revoke it with the Hub's revoke endpoint and tell them to create a new one The revoke endpoint invalidates a leaked token without any rights over its account, and the owner is emailed. Never use a token that is not yours.

Recap

Public downloads need no account; gated models, private repos, uploads and Spaces need one and a token.
Tokens are read, write or fine-grained. Use one per app, fine-grained in production, and keep them out of code; delete or refresh any that leak.
Log in with hf auth login, login() in a notebook, or the HF_TOKEN variable, which takes priority.
hf download fetches files into ~/.cache/huggingface/hub (moved by HF_HOME or HF_HUB_CACHE), where blobs and snapshots make sure nothing downloads twice; hf cache ls, rm and prune manage it.
HF_HUB_OFFLINE=1 or local_files_only=True runs from the cache with no network. load_dataset loads datasets the same way, and streaming=True reads huge ones row by row.