Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts

A new distillation framework called DiverseDistill recovers up to 114% of the performance gap when transferring knowledge from large foundation models to compact domain-specific models, according to research accepted at the 1st Workshop on Resource-Efficient Learning and Knowledge Discovery at KDD 2026 [1][3]. The work addresses a persistent challenge in machine learning: distilling knowledge from massive, general-purpose foundation models into smaller, efficient application models. Direct distillation often fails because the two model types have substantial gaps in capacity, architecture, and the data modalities they process [5]. In one experiment, distilling from a 76M-parameter language model to a 2M-parameter recommender closed less than 40% of the performance gap between the undistilled student and the teacher [2]. To bridge this divide, the researchers propose forming a teaching committee that includes both the foundation model and complementary domain-specific experts. These complementary teachers share architectural characteristics with the student model, aiming to smooth the knowledge transfer [4]. However, standard multi-teacher methods struggle with such heterogeneous groups. The paper notes that naively combining diverse teachers can degrade performance below what a single-teacher distillation achieves [2]. DiverseDistill overcomes this through a learnable Question-Answer mechanism. The framework generates teacher-conditioned queries and aligns the heterogeneous outputs from different teachers into the student's representation space [3]. Crucially, the teachers remain frozen throughout the process. The method uses only forward-pass inference through their intermediate layers, requiring no parameter updates, no co-training, and no architectural surgery [2]. A dynamic teacher importance mechanism further reduces training cost by filtering out low-relevance teachers for each sample. For recommendation tasks, this results in approximately 30% fewer forward passes with no loss in quality [2]. After training, the entire Distillation Module is discarded, adding zero inference overhead to the final compact model [3]. Evaluations on recommendation and vision tasks showed the framework recovering 73–114% of the teacher-student performance gap, consistently outperforming all single- and multi-teacher baselines [1][2]. The recommendation task achieved a 38x compression ratio, while the vision task reached 3.6x compression [2]. The paper was authored by researchers including Qingyun Liu and was last revised in June 2026 [1].