Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search

An automated pipeline that searched for optimal 4-Expert Mixture-of-Experts architectures within the LEMUR dataset ecosystem generated 4,463 candidate models over 28 days, but the entire explored space was anchored to a single neural-network family due to an enumeration bias, researchers reported. The pipeline, described in a paper submitted to arXiv, replaced manual design with a deterministic code-assembly generator that systematically combined base architecture families from the LEMUR database into heterogeneous MoE4 ensembles [1]. Each ensemble was governed by a convolutional gating network with temperature scaling, mixup augmentation, and cosine-annealed learning rate scheduling [1]. The campaign ran on an NVIDIA RTX 4090 graphics card across 197 batches, with 1,021 of the 4,463 generated models evaluated successfully [1]. A critical finding concerned the search-space coverage. Because the generator enumerated families alphabetically using itertools.combinations, every model examined was anchored to AirNet, a single family within the LEMUR ecosystem [1]. The explored space represented just 4.8 percent of the 23,751 possible 4-family combinations [1]. The authors characterized this coverage bias precisely, identified the root cause in the generator, and proposed a stratified random sampling fix [1]. Within the AirNet-anchored scope, ShuffleNet and MobileNetV3 consistently co-produced the highest-accuracy ensembles, with mean accuracy reaching 0.632 [1]. FractalNet and MNASNet were identified as low-yield families that the authors recommend excluding from future campaigns [1]. The LEMUR ecosystem belongs to a broader class of machine-learning datasets that have become integral to the field. Major advances in machine learning can result from improvements in learning algorithms, computer hardware, and the availability of high-quality training datasets [4]. High-quality labeled datasets for supervised and semi-supervised algorithms are typically difficult and expensive to produce because of the time required for labeling [4]. The pipeline's search process shares conceptual ground with cluster analysis, an exploratory data-analysis technique that partitions objects into groups such that members of the same group exhibit greater similarity to one another than to those in other groups [3]. Cluster analysis is not an automatic task but an iterative process of knowledge discovery that involves trial and failure, often requiring modification of data preprocessing and model parameters until results achieve the desired properties [3]. The corrected generator and analysis artifacts have been released as part of the open-source NNGPT project [1].