Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Data: the most important part

Machine Learning ML-02.02

Data collection and labeling

Where training data and labels come from, how to measure label quality with annotator agreement, and how active learning cuts labeling cost.

Beginner 12 min Last reviewed

The big idea

Supervised learning needs examples with answers attached. The examples are often cheap: your logs already hold millions of support tickets, photos or transactions. The answers are not. Someone, or something, has to decide that this ticket is urgent, this photo shows a cracked pipe, this payment was fraud. Labels are usually the slowest and most expensive part of an ML project, and their quality caps how good your model can ever be.

Think of a student learning from a workbook. If the answer key is full of mistakes, the student learns the mistakes, no matter how bright they are. And if writing the answer key costs money per page, you want to spend it on the questions the student actually finds hard, not the ones they already get right. Those two ideas, label quality and spending labeling effort wisely, are this lesson.

How it works

Step 1: Where data comes from

  • Your own product: application logs, databases, clickstreams, sensor feeds. Usually the most relevant data, because it matches what the model will see in production.
  • Public datasets: useful for learning and for pretraining, but check the license and whether the data resembles your users.
  • Bought or partner data: fills gaps, but you inherit its biases and its terms of use.
  • Synthetic data: generated by simulation or by a model. Handy for rare cases, risky if it does not match reality.

Whatever the source, record where each row came from and when. You will need that later to debug the model and to answer privacy questions.

Step 2: Where labels come from

  • In-house experts: doctors label scans, fraud analysts label transactions. Highest quality and highest cost; the only option when the task needs real expertise or the data is sensitive.
  • CrowdsourcingSplitting labeling work across many paid workers on a platform, usually with several workers per item and quality checks.: many paid workers label simple items, such as "is there a cat in this photo". Cheap and fast, noisier, so each item usually gets several labels that are combined by majority vote.
  • Implicit labels from behaviour: clicks, purchases, "mark as spam", a support agent changing a ticket's priority. Free and plentiful, but they measure what people did, not always what is true. A click is not the same as "this result was relevant".
  • Weak supervisionCreating approximate labels automatically from rules, heuristics or other sources, and combining many such noisy labelers instead of labeling by hand.: write labeling rules ("an email containing an unsubscribe link is probably marketing"), apply them to everything, and combine many imperfect rules into one probabilistic label. You trade some accuracy for enormous scale.
  • Model-assisted labeling: a model, increasingly a large language model, proposes labels and humans review or correct them. Reviewing is much faster than labeling from scratch. Always check a sample by hand, because a model's systematic mistakes become your dataset's systematic mistakes.

Step 3: Label quality and annotator agreement

Give the same 200 items to two careful people and they will disagree on some. Part of that is carelessness; much of it is that the task is genuinely ambiguous or the instructions are vague. You measure this with inter-annotator agreementHow often independent labelers give the same label to the same item, often measured with Cohen's kappa so that chance agreement is discounted..

Raw agreement is misleading: if 90 percent of tickets are "not urgent", two people who both say "not urgent" every time agree 90 percent of the time while being useless. Cohen's kappaAn agreement score for two labelers that subtracts the agreement expected by chance: 1 is perfect, 0 is no better than chance. corrects for that:

where is the observed agreement and is the agreement you would expect by chance from each labeler's label frequencies. In the code below two people agree on 10 of 12 tickets (), but because chance alone would give , kappa is only 0.66. Values around 0.8 and above are usually considered strong agreement; low kappa means fix the guidelines before labeling more.

How to raise quality:

  • Write a labeling guide with clear definitions and worked examples of hard cases.
  • Run a pilot of a few hundred items, measure agreement, discuss disagreements, update the guide.
  • Hide known-answer "gold" items in the queue to check each labeler.
  • Get two or three labels for ambiguous items and resolve conflicts.

Step 4: Active learning: label what the model finds hard

Most unlabeled examples are easy. Once a model has seen a few hundred obvious spam emails, labeling a thousand more obvious ones teaches it almost nothing. Active learningA loop where the current model picks which unlabeled examples a human should label next, usually the ones it is least sure about. lets the model choose what gets labeled next.

The active learning loop A large pool of unlabeled examples feeds a model trained on the small labeled set. The model scores the pool and picks the examples it is least sure about, which glow. A human labels just those, they join the labeled set, and the model retrains. The loop repeats until the budget runs out. score pick 20 labels retrain unlabeled pool 2,000 examples current model scores every example most uncertain p close to 0.5 human labeler labeled set grows by 20 per round Figure 1. Active learning. The model scores the unlabeled pool, sends only its most uncertain examples (glowing) to a human, and retrains on the result.

The simplest choice is uncertainty samplingAn active learning strategy that picks the examples whose predicted probability is closest to a coin flip, where the model is least confident.: pick the examples whose predicted probability is closest to 50/50. Those sit near the decision boundary, exactly where a new label moves the boundary most. The simulation below compares it with labeling random examples:

active.pyruns with python3 + numpy + scikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import cohen_kappa_score

# Inter-annotator agreement: two people labeled the same 12 tickets (1 = urgent).
ann_a = [1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0]
ann_b = [1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0]
raw = np.mean(np.array(ann_a) == np.array(ann_b))
print(f"raw agreement {raw:.2f}, Cohen's kappa {cohen_kappa_score(ann_a, ann_b):.2f}")

# Active learning vs random labeling on the same unlabeled pool.
X, y = make_classification(n_samples=3000, n_features=10, n_informative=3,
                           n_redundant=0, n_clusters_per_class=1,
                           class_sep=1.0, flip_y=0.0, random_state=0)
X_pool, y_pool, X_test, y_test = X[:2000], y[:2000], X[2000:], y[2000:]

def run(strategy, budget=200, start=20, batch=20, seed=0):
    rng = np.random.default_rng(seed)
    labeled = list(rng.choice(len(X_pool), start, replace=False))
    while len(labeled) < budget:
        model = LogisticRegression(max_iter=1000).fit(X_pool[labeled], y_pool[labeled])
        rest = np.setdiff1d(np.arange(len(X_pool)), labeled)
        if strategy == "uncertain":   # closest to 50/50 = most unsure
            p = model.predict_proba(X_pool[rest])[:, 1]
            pick = rest[np.argsort(np.abs(p - 0.5))[:batch]]
        else:
            pick = rng.choice(rest, batch, replace=False)
        labeled += list(pick)          # "ask the human" for these labels
    model = LogisticRegression(max_iter=1000).fit(X_pool[labeled], y_pool[labeled])
    return model.score(X_test, y_test)

for budget in (40, 100, 200):
    r = np.mean([run("random", budget, seed=s) for s in range(5)])
    u = np.mean([run("uncertain", budget, seed=s) for s in range(5)])
    print(f"{budget:>3} labels: random {r:.3f}   uncertainty sampling {u:.3f}")
full = LogisticRegression(max_iter=1000).fit(X_pool, y_pool).score(X_test, y_test)
print(f"all 2000 labels: {full:.3f}")

Output

raw agreement 0.83, Cohen's kappa 0.66
 40 labels: random 0.879   uncertainty sampling 0.923
100 labels: random 0.951   uncertainty sampling 0.987
200 labels: random 0.968   uncertainty sampling 0.989
all 2000 labels: 0.986

With 100 labels picked by uncertainty, the model already matches one trained on all 2,000. Random picking needs more labels to get close. Notice too that at 40 labels the gain is modest: the very first model is poor, so its sense of "uncertain" is poor too. Active learning pays off once the model is roughly right.

Trade-offs

OptionWins whenLoses when
Expert labels The task needs domain knowledge, mistakes are costly, data is sensitive You need tens of thousands of labels on a small budget
Crowdsourcing Simple, well-defined tasks at large volume Ambiguous tasks, specialist knowledge, private data
Implicit labels (clicks, actions) Huge volume for free, labels arrive continuously Behaviour is a biased stand-in for the truth you care about
Weak supervision and model-assisted You can write good heuristics or have a capable model, and humans review Nobody audits the output and errors become systematic
Active learning Labeling is the bottleneck and the model is already roughly right Tiny starting sets, or the picked examples are mostly unlabelable noise

Common mistakes

  • Starting to label at scale before a pilot shows the guidelines produce consistent labels.
  • Reporting raw agreement on an imbalanced task instead of kappa.
  • Treating clicks or other behaviour as ground truth without asking what they really measure.
  • Trusting model-generated labels without auditing a random sample by hand.
  • Keeping no record of who labeled what, so you cannot trace or fix a bad batch later.

In the wild

Large image datasets were built with crowdsourcing plus multiple labels per image and quality checks. Search engines and recommender systems learn heavily from implicit feedback such as clicks and dwell time, and spend real effort correcting for its biases. Weak supervision tools let teams write labeling functions instead of labeling by hand, and many teams now use a large language model to draft labels that humans then review. Medical imaging projects typically rely on expert labels and report agreement between specialists, because even experts disagree on hard cases.

Check yourself

  1. Two labelers agree on 95 percent of items, but 95 percent of items are negative. What should you check?

    Show answer

    B. Cohen's kappa, because much of that agreement could be chance If both mostly say negative, they agree often by chance. Kappa subtracts chance agreement and reveals whether they agree on the hard, positive cases.

  2. An active learning loop using uncertainty sampling picks which examples next?

    Show answer

    C. The ones with predicted probability closest to 0.5 Examples near 0.5 sit near the decision boundary, where a new label changes the model the most.

  3. A video app wants to train a 'quality' model using whether users watched to the end as the label. What is the main risk?

    Show answer

    B. Watching to the end measures behaviour, which may reward clickbait or short videos rather than quality Implicit labels are cheap and plentiful but are a stand-in for what you want. Short or sensational videos may be finished more often without being better.

  4. Your labelers' kappa is 0.4 after a pilot. What is the best next step?

    Show answer

    B. Review the disagreements, tighten the guidelines with examples, and re-run the pilot Low agreement usually means the task definition is unclear. More labels under unclear rules just adds more noise.

Recap

  • Labels are often the most expensive part of ML, and their quality caps the model's quality.
  • Labels come from experts, crowds, user behaviour, weak supervision rules, or models reviewed by humans.
  • Measure label quality with agreement between annotators, using kappa to discount chance.
  • Fix vague guidelines with a pilot before labeling at scale.
  • Active learning labels the examples the model is least sure about and can reach the same accuracy with far fewer labels.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up