Data efficiency is achieved by leveraging knowledge from outside the target labeled set and by selecting data intelligently. Main mechanisms: (1) transfer learning and pretraining โ the model learns representations on a large, cheap or unlabeled corpus and is then fine-tuned on a small target set; (2) self-supervised learning โ creates a supervision signal from the data itself (e.g. contrastive tasks, masking), reducing the need for labels; (3) few-/one-/zero-shot learning and meta-learning โ generalization from a few (or zero) examples per class; (4) active learning โ the algorithm itself selects the most informative examples to label (e.g. uncertainty sampling, query-by-committee, expected model change), needing fewer labels than ordinary supervised learning; (5) data augmentation โ synthetically expanding the set through label-preserving transformations; (6) knowledge distillation โ transferring knowledge from a larger teacher model. Progress is measured with learning curves (quality as a function of the number of examples or labels) and by the sample complexity required to reach a target accuracy.
Classical supervised learning requires large, manually labeled datasets that in many domains are expensive, scarce, or entirely unavailable, and large language models see orders of magnitude more data during training than a human needs to acquire language. Data efficiency addresses this cost and disproportion by reducing the number of examples and labels needed to reach useful quality.
A measure of quality attained per training example; how many samples are needed to reach a target accuracy.
Quality attained per label; critical when data is available but labeling it is expensive.
Reusing representations learned on a large source corpus for a target task with little data.
Official
Creating a supervision signal from the data itself (masking, contrastive tasks), reducing the need for labels.
Official
Generalization from a few (few-shot), one (one-shot), or zero (zero-shot) examples per class.
Official
The algorithm itself selects the most informative examples to label (uncertainty sampling, query-by-committee), reducing the number of labels required.
Official
Synthetically expanding the training set through label-preserving transformations.
Official
Augmentations or duplicates leaking between train and test sets inflate results and mask the true data efficiency.
Model-driven iterative selection can yield a non-representative set and overestimated quality.
With few examples, results depend heavily on the random draw, making comparisons unreliable.
Pretraining on a corpus too distant from the target task can hurt rather than help quality.
A systematic survey of active-learning strategies consolidated the idea of reducing the number of labels needed through intelligent example selection.
Formalized transfer learning as a way of leveraging source-task knowledge for a data-scarce target task.
Contrastive self-supervised learning showed that representations from unlabeled data enable high quality with few labels downstream.
Demonstrated that a large language model can perform tasks from a few in-prompt examples without weight updates โ a strong manifestation of data efficiency at inference time.
Training a vision transformer solely on ImageNet (without hundreds of millions of external images) via attention distillation popularized the term 'data-efficient' in computer vision.
A shared task on pretraining language models under a budget of about 100 million words (with a 10M track), inspired by the cognitively plausible exposure of a child to language, made data efficiency a central research goal in NLP.
Maximum number of examples that can be labeled.
Number of examples per class in a few-shot (k-shot) setting.
Ratio of unlabeled to labeled data in semi-/self-supervised learning.
Size of the source corpus used for pretraining before transfer.
Intensity and diversity of augmentation transformations.
Number of examples selected for labeling in one active-learning iteration.
Data efficiency is an algorithmic/statistical property about data usage, independent of any specific hardware accelerator type.
