Parameter-efficient fine-tuning: LoRA and QLoRA

The big idea

Full fine-tuning updates every weight in the model. For a 7-billion-parameter model trained with Adam in mixed precision, the weights, gradients and optimizer state alone need over 100 GiB, before counting activations. And every fine-tuned variant is another full-size copy of the model to store and serve.

LoRALow-Rank Adaptation: fine-tuning that freezes the pretrained weights and learns a small low-rank update (two thin matrices B and A) added to selected weight matrices. takes a different route. It freezes the original weights and learns a small correction next to selected weight matrices, written as the product of two thin matrices. Only those thin matrices are trained: often well under 1 percent of the parameters. QLoRALoRA fine-tuning on top of a base model stored in 4-bit precision, which cuts the memory for the frozen weights by about four times compared with 16-bit. goes further and stores the frozen base model in 4-bit precision, so a large model fits on a single GPU.

Think of renovating a rented flat. You are not allowed to knock down walls (the frozen weights), but you can add furniture, rugs and lamps (the adapters) that change how the place feels. The changes are small, cheap, and easy to take out again, and the next tenant can bring their own set into the same flat.

How it works

The low-rank update

A layer normally computes $h=Wx$. LoRA adds a trainable update $\Delta W=BA$:

$W$ is $d\times d$ and frozen. $A$ is $r\times d$ and $B$ is $d\times r$, where the rankIn LoRA, the inner dimension r of the two adapter matrices. It caps how complex the learned weight update can be; common values range from 4 to 64. $r$ is small, typically 4 to 64. For $d=4,096$ and $r=16$, the full matrix has $4,096^2=16,777,216$ entries, while $A$ and $B$ together have $2\times 4,096\times 16=131,072$: 128 times fewer.

Why should a low-rank update be enough? Fine-tuning adapts a model that already knows a lot; the change needed for a new task or style seems to live in a small number of directions. LoRA bets on that, and in practice it usually holds.

Initialisation and alpha

$B$ starts at zero and $A$ at small random values, so $BA=0$ and training starts from exactly the pretrained model. The scaling factor alphaA LoRA hyperparameter that scales the adapter's output by alpha / r, controlling how strongly the adapter changes the layer. divided by $r$ sets how strongly the adapter speaks. Many people set $\alpha$ to $r$ or $2r$ and tune the learning rate instead.

Which matrices to adapt

The original paper adapted the attention query and value projections. Today it is common to adapt all attention projections and often the MLP matrices too, which helps on harder tasks at a modest cost in parameters.

QLoRA: a 4-bit base

The frozen weights do not need to be updated, so they do not need high precision for training to work. QLoRA stores them in a 4-bit format and dequantises them on the fly during the forward and backward passes, while the adapters train in 16-bit. Memory for the base drops by about 4 times compared with 16-bit, which is what makes fine-tuning very large models on one GPU possible ( Quantization for inference covers quantization).

Merging and serving adapters

After training you can merge the adapter into the weights, $W^{\prime }=W+\frac{\alpha }{r}BA$, and serve a normal model with no extra latency. Or keep adapters separate: an adapter is a small file (megabytes rather than gigabytes), so one base model in GPU memory can serve many fine-tuned variants, swapping or batching adapters per request. This is how platforms host many customer fine-tunes cheaply.

Try it

Try this:

1. At the defaults (a 7-billion-parameter model, hidden size 4,096, rank 16, 4 matrices in each of 32 layers), LoRA trains 16.8M parameters, 0.24 percent of the model. Memory for weights and optimizer state drops from about 104 GiB for full fine-tuning to 13.3 GiB, and to 3.5 GiB with QLoRA.
2. Raise the rank to 64. Trainable parameters quadruple to 67.1M, yet LoRA memory only grows from 13.3 to 14.0 GiB: the frozen base dominates. That is why QLoRA attacks the base.
3. Set hidden size 8,192, 80 layers and a 70-billion-parameter model. Full fine-tuning needs over 1,000 GiB; QLoRA needs about 34 GiB, within reach of a single large GPU once activations are added.

The calculator treats every adapted matrix as $d\times d$ (MLP matrices are larger, so it slightly undercounts) and uses the usual mixed-precision Adam cost of 16 bytes per trained parameter. Activation memory, which grows with batch size and sequence length, is left out.

LoRA on one matrix in NumPy:

# LoRA on one weight matrix: learn a low-rank update B @ A, keep W0 frozen.
import numpy as np

rng = np.random.default_rng(0)
d, n = 64, 500
W0 = rng.normal(size=(d, d)) / np.sqrt(d)            # "pretrained" weights
U, V = rng.normal(size=(d, 4)), rng.normal(size=(4, d))
W_task = W0 + 0.3 * U @ V / np.sqrt(d)               # the task needs a rank-4 change
X = rng.normal(size=(n, d))
Y = X @ W_task.T + 0.01 * rng.normal(size=(n, d))    # input -> desired output

def train_lora(r, alpha=8, steps=1000, lr=0.02):
    A = rng.normal(size=(r, d)) * 0.01               # small random
    B = np.zeros((d, r))                             # zero: starts as W0 exactly
    s = alpha / r                                    # LoRA scaling
    for _ in range(steps):
        err = X @ (W0 + s * B @ A).T - Y             # forward with the adapter
        G = err.T @ X / n                            # dLoss/dW (only used via A, B)
        B, A = B - lr * s * G @ A.T, A - lr * s * B.T @ G
    return B, A, s

base_err = np.mean((X @ W0.T - Y) ** 2)
print(f"frozen W0 only:  error {base_err:.4f}   trainable 0")
print(f"full fine-tune:  trainable {d * d:,}")
for r in (1, 4, 8):
    B, A, s = train_lora(r)
    err = np.mean((X @ (W0 + s * B @ A).T - Y) ** 2)
    print(f"LoRA rank {r}:     error {err:.4f}   trainable {B.size + A.size:,}")

# Merging: fold the adapter into the weights. Same outputs, no extra latency.
W_merged = W0 + s * B @ A
x = X[:3]
same = np.allclose(x @ W_merged.T, x @ W0.T + s * (x @ A.T) @ B.T)
print("merged weights give identical outputs:", same)

frozen W0 only:  error 0.3804   trainable 0
full fine-tune:  trainable 4,096
LoRA rank 1:     error 0.2196   trainable 128
LoRA rank 4:     error 0.0001   trainable 512
LoRA rank 8:     error 0.0001   trainable 1,024
merged weights give identical outputs: True

The task needs a rank-4 change to the frozen matrix. A rank-1 adapter cannot express it and the error only falls from 0.380 to 0.220. Rank 4 drives it to the noise floor (0.0001) with 512 trainable numbers instead of 4,096, and rank 8 adds nothing more. The last line confirms that merging the adapter gives exactly the same outputs.

Going deeper

Choosing the rank

Higher rank gives the adapter more capacity but more parameters and more risk of overfitting small datasets. For style, format and narrow tasks, ranks 8 to 16 are a common start. If quality plateaus below full fine-tuning, try adapting more matrices before cranking the rank.

Learning rates and limits

LoRA usually needs a higher learning rate than full fine-tuning. It also tends to forget less of the base model's abilities, because the base is frozen. The flip side: for tasks that need a lot of new knowledge or a big shift in behaviour, such as continued pretraining on a new language, full fine-tuning can still win.

Other parameter-efficient methods

LoRA is the most widely used PEFTParameter-efficient fine-tuning: methods that adapt a large model by training only a small number of new or selected parameters. method, but not the only one. Prefix and prompt tuning learn a few virtual tokens prepended to the input; adapter layers insert small trainable blocks between layers (and add some latency). Variants of LoRA change how the update is parameterised or initialised.

Trade-offs

Common mistakes

• Forgetting activation memory. The calculator's numbers are a floor; long sequences and big batches add a lot.
• Using the learning rate from full fine-tuning and concluding LoRA does not work.
• Expecting LoRA to inject large amounts of new factual knowledge. Use retrieval for facts ( Prompting vs RAG vs fine-tuning: choosing ).
• Merging an adapter trained on a quantised base into a 16-bit base without checking quality.
• Picking a huge rank by default. Start small, measure, and adapt more matrices first.

In the wild

Check yourself

Recap