Your annotation budget needs a re-labeling line

If your budget pays for 10,000 annotation actions and reserves none for review, it produces 10,000 new labels. It leaves no room to discover that some of the most influential labels are wrong.

An initial experiment in "Deep Active Re-Labeling: Toward Noise-Resilient Annotation Efficiency" tested a different split. The researchers spent 80% of a fixed budget on new labels and 20% on rechecking randomly selected existing labels. The broader study then tested targeted re-labeling under fixed budgets across four image-classification datasets and beat a no-review baseline.

That is a strong argument for putting review in the budget before annotation starts. It is not a universal 80/20 rule.

Why active learning is vulnerable to bad labels#

Uncertainty-sampling active learning tries to stretch an annotation budget by choosing what to label next. Instead of sampling images at random, the model sends humans the examples it is most uncertain about. Those examples sit near the model's decision boundary, so a correct label can teach it more than another easy example.

The same leverage works in reverse. A wrong label on an influential example can distort the boundary more than a wrong label on a random one.

Published at the 2025 IEEE International Conference on Big Data and posted to arXiv in June 2026, the study by Md Abdullah Al Forhad and Weishi Shi tested whether active learning should revisit its own training set. Their method ranks existing labels by two signals: proximity to the decision boundary and inconsistency with nearby examples. It then spends part of the remaining annotation budget on checking the most suspicious labels.

The authors used MNIST, FashionMNIST, CIFAR-10, and PathMNIST. They injected random label errors at a 30% rate for the main experiments, tested 10% and 50% rates separately, and also tested structured errors between similar classes on MNIST. The images were real benchmark data, but the annotation errors were simulated. The result supports the principle that targeted review can beat all-new labeling under a fixed budget. It does not establish the right review percentage for every production dataset.

Bad labels change decisions#

There is separate evidence that label errors survive serious dataset review. A NeurIPS 2021 study examined the test sets of ten widely used machine-learning benchmarks. After algorithmic screening and human validation, the authors estimated an average error rate of at least 3.3%, including at least 6% in the ImageNet validation set.

That study measured test sets, not the training data in a typical annotation project, so it cannot tell you your own error rate. It shows something narrower and still important: a dataset can be heavily used, repeatedly evaluated, and not be clean.

The authors then simulated what would happen if originally mislabeled examples were 6% more prevalent in ImageNet. On corrected labels, ResNet-18 overtook ResNet-50 in that scenario. The experiment shows how enough label noise could reverse a ranking a practitioner might use to choose a model.

Turn review into a budget, not a rescue mission#

Start by budgeting annotation actions rather than dataset size. Every new label consumes one action. So does every second look. Decide how many actions belong to each pool before the deadline makes the decision for you.

Then make the review pool selective. The paper targets labels near a decision boundary or inconsistent with their neighbors. A project without that machinery can still use evidence it already has: annotator disagreements, classes that are often confused, images with unusually complex geometry, and batches labeled before the conventions settled. These are candidates for review, not proof of error.

Track the yield. Record how many reviewed items changed and what caused each change. If review rarely finds errors, move more of the next batch toward new labels. If it finds a cluster of the same mistake, fix the written convention and review the affected class or batch. The budget split should respond to what the project is learning.

Labelme's file-based format keeps this loop simple. Each annotation is a JSON file, stored next to its image by default or in a chosen annotation directory. A review queue can be a list of filenames: reopen the annotations, correct labels and shapes in place, then use a file diff to audit exactly what changed. The format removes an export-and-reimport cycle; the evidence from your project decides what deserves another look.

The default budget asks how many new images you can label. A better budget asks how many annotation decisions you can afford and which deserve a second look, because labels are measurements, not ground truth. Plan for correction.