CausalMoE: A Billion-Scale Multimodal Foundation Model for Granger Causal Discovery with Pattern-Routed Heterogeneous Experts

CausalMoE is a billion-scale multimodal foundation model designed for Granger causal discovery, introducing a Pattern-Routed Mixture of Heterogeneous Experts that dynamically routes data to specialized experts based on latent temporal patterns. It features a Causality-Aware Self-Attention mechanism for interpretable graph recovery, yielding sparse Granger causal graphs, and integrates large language models (LLMs) and vision-language models (VLMs) to align numerical signals with textual and visual information. This model sets a new state-of-the-art in fully supervised benchmarks and demonstrates robust generalization capabilities in few-shot scenarios, addressing limitations of existing neural GCD methods.