Ensemble Learning for Large Language Models in Text and Code Generation: A Survey

A new survey categorizes ensemble learning methods for large language models into seven distinct approaches, aiming to address inconsistencies and biases found in single-model text and code generation systems [1]. The preprint, posted on arXiv and last revised in June 2026, examines how combining multiple large language models (LLMs) can improve output reliability. Individual generative pretrained transformers (GPTs)—the foundational architecture behind most modern LLMs—often produce inconsistent results and reflect biases present in their training data, limiting their ability to represent diverse language patterns [1][6]. The closed-source nature of many powerful models further restricts industry adoption due to data privacy concerns [1]. The authors, including Jingzhi Gong and Zheng Wang, organize LLM ensemble techniques into seven main methods: weight merging, knowledge fusion, mixture-of-experts, reward ensemble, output ensemble, routing, and cascading [1][3]. From this taxonomy, they focus on four methods that demonstrate strong performance and potential for broader application, analyzing their modeling steps, training methods, and output features [4]. The survey builds on a growing body of work in LLM ensembling. A separate 2025 systematic review introduced a broader taxonomy that classifies ensemble methods by when they occur relative to inference: before, during, or after [5]. That framework groups routing-based selection as an ensemble-before-inference strategy, token-level aggregation as ensemble-during-inference, and full-response combination as ensemble-after-inference [5]. LLMs are neural networks trained on vast text corpora, typically using transformer architectures, and are capable of generation, summarization, and translation [6]. GPTs, a specific type of LLM, are pre-trained to predict the next word in a sequence and are often fine-tuned to follow instructions [6][7]. OpenAI introduced the first GPT model in 2018, and subsequent releases—including GPT-3.5, GPT-4o, and GPT-5—have expanded capabilities to multimodal processing and reasoning [7]. The survey highlights that ensemble techniques are increasingly being explored for code generation, inspired by their success in text tasks [1][4]. The authors note that ensemble approaches can improve diversity representation, enhance output quality, and offer greater flexibility for real-world applications [1][4]. They also suggest that the methods reviewed could lay groundwork for extending ensemble strategies to multimodal LLMs [1][3].