Data

Data Efficiency

ActivePublished

Key innovation

Shifts the learning objective from maximizing quality given unlimited data to maximizing quality obtained per unit of data or labels.

How it works

Data efficiency is achieved by leveraging knowledge from outside the target labeled set and by selecting data intelligently. Main mechanisms: (1) transfer learning and pretraining — the model learns representations on a large, cheap or unlabeled corpus and is then fine-tuned on a small target set; (2) self-supervised learning — creates a supervision signal from the data itself (e.g. contrastive tasks, masking), reducing the need for labels; (3) few-/one-/zero-shot learning and meta-learning — generalization from a few (or zero) examples per class; (4) active learning — the algorithm itself selects the most informative examples to label (e.g. uncertainty sampling, query-by-committee, expected model change), needing fewer labels than ordinary supervised learning; (5) data augmentation — synthetically expanding the set through label-preserving transformations; (6) knowledge distillation — transferring knowledge from a larger teacher model. Progress is measured with learning curves (quality as a function of the number of examples or labels) and by the sample complexity required to reach a target accuracy.

Problem solved

Classical supervised learning requires large, manually labeled datasets that in many domains are expensive, scarce, or entirely unavailable, and large language models see orders of magnitude more data during training than a human needs to acquire language. Data efficiency addresses this cost and disproportion by reducing the number of examples and labels needed to reach useful quality.

Components

Sample efficiencyDimension of data efficiency concerning the number of examples

A measure of quality attained per training example; how many samples are needed to reach a target accuracy.

Label efficiencyDimension of data efficiency concerning the number of labels

Quality attained per label; critical when data is available but labeling it is expensive.

Transfer learning and pretrainingPrimary technique for achieving data efficiency

Reusing representations learned on a large source corpus for a target task with little data.

Official

Self-supervised learningTechnique that boosts label efficiency

Creating a supervision signal from the data itself (masking, contrastive tasks), reducing the need for labels.

Official

Few-shot / meta-learningTechnique for extremely low example counts

Generalization from a few (few-shot), one (one-shot), or zero (zero-shot) examples per class.

Official

Active learningTechnique for intelligent selection of data to label

The algorithm itself selects the most informative examples to label (uncertainty sampling, query-by-committee), reducing the number of labels required.

Official

Data augmentationTechnique for increasing the effective number of examples

Synthetically expanding the training set through label-preserving transformations.

Official

Implementation

Reference implementations

facebookresearch/deit

Python · Meta AI (FAIR)

Official

modAL (active learning framework)

Python · modAL-python

Implementation pitfalls

Data leakage in augmentation and splittingHigh

Augmentations or duplicates leaking between train and test sets inflate results and mask the true data efficiency.

Fix:Split before augmenting and deduplicate at the level of source examples.

Sampling bias in active learningMedium

Model-driven iterative selection can yield a non-representative set and overestimated quality.

Fix:Combine query strategies with random sampling and monitor distribution coverage.

High evaluation variance on small setsHigh

With few examples, results depend heavily on the random draw, making comparisons unreliable.

Fix:Report mean and deviation over multiple seeds and few-shot episodes.

Negative transfer (distribution mismatch)Medium

Pretraining on a corpus too distant from the target task can hurt rather than help quality.

Fix:Choose a source corpus close to the target domain and validate the transfer.

Evolution

2009

Active Learning Literature Survey (Settles)

A systematic survey of active-learning strategies consolidated the idea of reducing the number of labels needed through intelligent example selection.

2010

A Survey on Transfer Learning (Pan and Yang)

Formalized transfer learning as a way of leveraging source-task knowledge for a data-scarce target task.

2020

SimCLR — contrastive learning for label efficiency

Contrastive self-supervised learning showed that representations from unlabeled data enable high quality with few labels downstream.

2020

GPT-3 — few-shot in-context learning

Inflection point

Demonstrated that a large language model can perform tasks from a few in-prompt examples without weight updates — a strong manifestation of data efficiency at inference time.

2021

DeiT — data-efficient image transformers

Training a vision transformer solely on ImageNet (without hundreds of millions of external images) via attention distillation popularized the term 'data-efficient' in computer vision.

2023

BabyLM Challenge — sample-efficient pretraining

Inflection point

A shared task on pretraining language models under a budget of about 100 million words (with a 10M track), inspired by the cognitively plausible exposure of a child to language, made data efficiency a central research goal in NLP.