Learning Subset-Shared Invariances for Domain Generalization with Mixture-of-Experts

A new machine-learning framework proposes that models should learn invariances shared only within subsets of training domains, rather than enforcing a single global invariance, to improve generalization to unseen environments [1]. The approach, detailed in a paper submitted on 24 Jun 2026 to arXiv, targets a core tension in domain generalization (DG) [1]. DG trains models on one or more source domains so they perform accurately on a target domain whose data is unavailable during training [2]. A widely used strategy forces a model's internal representations to be invariant across all source domains, on the assumption that the predictive structure is globally shared [3]. The authors argue that this assumption backfires as the number and diversity of source domains grow: enforcing invariance across more domains progressively shrinks the usable representation space and discards transferable factors that are not universally present [4]. To bypass that over-constraint, the researchers introduce subset-shared invariance [1]. Instead of one global invariance, predictive structure is assumed stable only within domain subsets [2]. The idea is implemented with a mixture-of-experts (MoE) architecture [3]. A shared encoder produces a feature representation, a router outputs a soft distribution over experts, and each expert generates an expert-specific representation [4]. The final representation is a weighted sum of expert outputs, allowing different subset-invariant components to activate for different samples [4]. Training objectives promote selective alignment, confident and balanced routing, and diverse expert specialization [2]. The MoE design builds on earlier DG work. A 2022 paper, HMOE, used hypernetwork-based mixture of experts to separate mixed-domain data into clusters without requiring domain labels, achieving state-of-the-art results on DomainBed benchmarks [5]. The new framework differs by explicitly conditioning invariance on latent subset assignments and jointly learning the routing mechanism with the representation [3]. Experiments on DomainBed benchmarks showed improved out-of-domain generalization and greater robustness as domain heterogeneity increased [1]. The results suggest that DG should shift from enforcing a single global invariance toward modeling invariance through partially shared structure across domain subsets [2]. The work was developed within arXivLabs, a framework that lets collaborators build and share new features on arXiv's platform [1].