Data science & ML
LabelWise
Active learning on a labelling budget
An active-learning strategy for a fixed labelling budget: committee disagreement chooses what to label, and validated rebalancing and confidence-gated pseudo-labels help a minority class.
Technical lab Status: Completed Academic
Workflow diagram
The problem
Labels are expensive: with a budget of 5,000 oracle queries, which records should be labelled to classify a 33% minority class well?
My Role & Leadership
Project Owner · Technical Lead · Developer. Owned, designed and implemented the strategy file of a course assignment: the acquisition, calibration and pseudo-labelling strategy, within a provided oracle, data and training framework.
- Led the strategy design: query-by-committee acquisition that uses the forest's own trees as the committee, so disagreement costs no extra models.
- Worked within the framework's limits: rebalancing tuned on internal splits because the decision threshold is fixed, pseudo-labels gated at 85% confidence, and a 42-second time budget.
The algorithm
Each round, the forest's own trees vote on every unlabelled record; the records they disagree on most go to the oracle, and the model is retrained on the growing labelled set.
What I developed
- Query-by-committee acquisition over four rounds, scoring each record by the disagreement of the forest's own trees plus a small tilt toward the minority class.
- A rebalancing ratio chosen on repeated internal train and validation splits, because the framework only calls predict and the decision threshold cannot be moved directly.
- Pseudo-labels only for records the model is at least 85% sure about, added once and never duplicated.
- A self-imposed 42-second time budget that degrades to the best ratio found so far instead of timing out.
Implementation decisions
- Use the forest's own trees as the committee, so disagreement costs no extra models.
- Tune rebalancing on internal splits, because the framework fixes the decision threshold.
- Gate pseudo-labels at 85% confidence, so the model does not teach itself its own mistakes.
What was provided
- The oracle, the dataset, the training framework and the scoring harness.
Key technologies
Known limits
No score is recorded in the supplied files; the only figure is a comment in the code, so none is claimed. The dataset's source and licence are unknown.