Ensemble Learning for Large Language Models in Text and Code Generation: A Survey

The article presents a survey on ensemble learning techniques for Large Language Models (LLMs) in text and code generation, categorizing methods into seven types: weight merging, knowledge fusion, mixture-of-experts, reward ensemble, output ensemble, routing, and cascading. It emphasizes the potential of these ensemble approaches to improve output quality, representation diversity, and application flexibility, addressing limitations of individual LLMs such as inconsistency and bias. This work is significant for practitioners as it provides a framework for selecting and implementing ensemble strategies, potentially enhancing performance in real-world applications and paving the way for multimodal LLM advancements.