Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning

A new data pruning framework called IF-Beta can train compact machine learning models that outperform those trained on full datasets, while using less data and less compute, according to research submitted in June 2026 [1]. Knowledge distillation (KD) is a widely used technique to create smaller, efficient models for resource-constrained environments, but the computational cost of the distillation process itself is often ignored [1]. The researchers behind IF-Beta argue that existing data pruning methods are not built for KD, introducing heavy overhead through retraining or relying on heuristic rules that miss what distillation actually needs [1]. IF-Beta addresses this by combining influence functions with a learnable sampling policy. Influence functions act as an efficient estimator of a sample's impact when only a pre-trained teacher model is available [1]. The sampling policy is parameterized by a Beta distribution, a flexible two-parameter family that adapts to different pruning ratios instead of using fixed heuristics [1]. The framework then optimizes this policy through a bilevel objective: an inner loop runs fast proxy training in the teacher's feature space with a KD-aligned objective, while an outer loop updates the policy to maximize distillation performance [1]. Experiments on CIFAR-10/100 and ImageNet showed IF-Beta consistently outperformed other baselines across a wide range of pruning ratios [1]. The authors report that students trained on pruned data surpassed the performance of students distilled on the full dataset [1]. The work lands as broader efforts to make machine learning more data-efficient continue to expand. Researchers in catalysis informatics, for instance, have explored how datasets like OC20 and OC22 can complement each other through transfer learning or joint training to improve model performance without requiring entirely new data collection [4]. Similarly, the United Nations' Sustainable Development Goals, adopted in 2015, have faced persistent challenges in tracking progress, with a 2025 report finding only 35% of targets on track or making moderate progress and 18% regressing, underscoring the difficulty of achieving ambitious global benchmarks with limited resources [6]. While IF-Beta focuses on computer vision tasks, the underlying principle of doing more with less data resonates across fields. In molecular biology, transcription factors regulate gene expression by binding to specific DNA sequences, a process that depends on precise, context-aware signals rather than brute-force molecular interactions [7]. The IF-Beta framework's reliance on a learnable, adaptive policy echoes this biological efficiency, where a small set of regulatory proteins coordinates complex cellular behavior without wasteful overproduction [7].