Few-shot prompting and examples

The big idea

Describing what you want in words leaves room for interpretation. "Classify the ticket, be consistent" could mean many things. Showing two or three finished examples of input and ideal output removes most of that room. This is few-shot promptingIncluding a few worked examples of inputs and their desired outputs in the prompt, so the model infers the task, format and standard from them. With no examples it is called zero-shot., and the model's ability to pick up a task from examples in its context is called in-context learningA model's ability to perform a task from instructions and examples in its prompt alone, without any change to its weights..

Examples are powerful because the model copies them closely: their format, length, tone, the set of labels they use, even quirks you did not intend. That is the whole point, and the whole risk. Good examples are diverse, consistent with each other and representative of real inputs. Badly chosen ones bias every answer.

Think of training a new barista by letting them watch you make a few drinks. Show one flat white and they learn the steps. Show only flat whites and the first person who orders tea gets a flat white.

How it works

Choose diverse, representative examples

Cover the range of inputs you will really see: every label, short and long inputs, formal and messy ones, at least one tricky case. If every example is a short, polite English sentence, the model is less sure what to do with a long, angry message in Spanish. If one label never appears in the examples, the model is less likely to use it at all.

Three to five good examples usually beat twenty mediocre ones. Each one costs tokens on every call, and near-duplicates add cost without adding information.

Keep examples consistent

Every example should follow exactly the format and standard you want: same field names, same capitalisation, same length range. The model will copy inconsistencies too. If one example says Severity: high and another says sev: 2, expect a mix in the output.

Mark examples clearly, for example wrapped in <example> tags, so the model can tell examples apart from the real input and from your instructions.

Order matters

Models are biased toward answers that appear often in the examples and toward the answer in the last example, the one closest to the new input. Research on early large models found that the choice and even the order of examples could swing accuracy from near chance to near state of the art. Modern models are much more robust, but the effects have not vanished. Practical defences:

• Balance the labels across examples rather than mirroring a skewed real distribution.
• Do not put all examples of one label together, or always last.
• Test a couple of orderings on your test set and keep the one that is reliably good.

Negative examples and near misses

Sometimes the clearest way to teach a boundary is a contrast pair: two inputs that look alike but get different labels. "The export button does nothing" is a bug; "How do I export my data?" is a question; "Please add export to PDF" is a feature request. These near missesExamples chosen because they sit close to a decision boundary, such as two similar inputs with different correct answers, to teach the model where the boundary lies. teach the model what the label really depends on.

Showing a bad output marked as bad ("Do not write it like this: ...") can help with style, but use it sparingly. Models sometimes copy the very phrasing you showed as wrong. Prefer a correct example of the right thing, with a short note on what to avoid.

Try it

The lab uses a deliberately simple stand-in for a model: word overlap with each example, plus a pull toward frequent labels and toward the last example's label. Real models are far smarter, but they show the same two biases.

Try this:

1. Pick "two bug examples only". Only the 3 real bugs come out right: everything else is called a bug too. The examples define the label space the model sees.
2. Pick "one per label", then "reversed". Both get 7 of 9 right, but not the same 7. A ticket that matches no example, such as "the save button does nothing", falls to the label of the last example: question in the first order, bug in the reversed one. Reordering moves the mistakes around.
3. Pick "plus near misses". The contrast examples about export and invoices fix the boundary cases and all 9 are right.

Going deeper

The same toy in Python

# A toy in-context classifier with two biases real models show in few-shot prompts:
# it leans toward labels that appear often in the examples, and toward the last example's label.
STOP = set("i a the my to is it of for do how when in on and be would can me this was "
           "we our you your get please was have by instead today".split())
words = lambda s: {w for w in s.lower().replace(",", "").split() if w not in STOP}

POOL = {"b1": ("app crashes when I upload a photo", "bug"),
        "b3": ("the export button does nothing", "bug"),
        "p1": ("I was charged twice this month", "billing"),
        "p2": ("send me an invoice for last month", "billing"),
        "f1": ("please add dark mode", "feature"),
        "f2": ("it would be great to export to excel", "feature"),
        "q1": ("how do I change my email address", "question"),
        "q2": ("how do I export my data", "question")}
TESTS = [("app crashes when I open settings", "bug"), ("the save button does nothing", "bug"),
         ("the app feels sluggish today", "bug"), ("charged twice for my plan", "billing"),
         ("can I pay by invoice instead", "billing"), ("please add a calendar view", "feature"),
         ("please add export to pdf", "feature"), ("how do I change my password", "question"),
         ("how do I export my invoices", "question")]
LABELS = ["bug", "billing", "feature", "question"]

def predict(text, shots, freq=0.3, recency=0.25):
    s = {l: 0.0 for l in LABELS}
    for ex in shots:                                  # similarity to each example
        t, l = POOL[ex]
        s[l] += len(words(text) & words(t)) + freq     # plus a pull toward frequent labels
    s[POOL[shots[-1]][1]] += recency                  # plus a pull toward the last label
    return max(LABELS, key=lambda l: s[l])

SETS = {"one label only": ["b1", "b3"],
        "one per label": ["b1", "p1", "f1", "q1"],
        "same, reversed": ["q1", "f1", "p1", "b1"],
        "+ near misses": ["b1", "f1", "p1", "f2", "b3", "p2", "q1", "q2"]}
for name, shots in SETS.items():
    wrong = [t for t, y in TESTS if predict(t, shots) != y]
    print(f"{name:15} {len(TESTS) - len(wrong)}/{len(TESTS)} right; wrong: {wrong}")

one label only  3/9 right; wrong: ['charged twice for my plan', 'can I pay by invoice instead', 'please add a calendar view', 'please add export to pdf', 'how do I change my password', 'how do I export my invoices']
one per label   7/9 right; wrong: ['the save button does nothing', 'can I pay by invoice instead']
same, reversed  7/9 right; wrong: ['can I pay by invoice instead', 'how do I export my invoices']
+ near misses   9/9 right; wrong: []

What models actually learn from examples

One study found that, for classification tasks, replacing the labels in the examples with random ones hurt accuracy surprisingly little. What mattered more was that the examples showed the set of possible labels, the kind of input text, and the overall format. The practical lesson: examples are mostly a specification of format and scope, so make those exactly right, and put the decision rules in the instructions as well.

Choosing examples per input

For large example banks, you can pick examples at request time: embed the new input and retrieve the most similar examples from a bank ( Semantic search basics ). This keeps the prompt short and the examples relevant, at the cost of a retrieval step and less predictable prompts.

Trade-offs

Common mistakes

• Examples that all look alike, so the model is lost on anything different.
• A label that never appears in the examples, so the model rarely uses it.
• Inconsistent formats across examples, which the model faithfully reproduces.
• Always putting the same label last, adding a quiet bias toward it.
• Showing only what not to do, so the model copies the bad phrasing.

In the wild

Check yourself

Recap