FiLM-Coordinated Dual-Branch Transformer for Global-Local Dependency Modeling in Language Modeling

The article introduces a FiLM-coordinated dual-branch Transformer architecture designed for improved modeling of global and local dependencies in language tasks. This model features distinct global and local branches within each layer, utilizing feature-wise linear modulation (FiLM) for dynamic coordination, which enhances channel-wise calibration over traditional methods. Experimental results demonstrate that this architecture outperforms single-branch baselines on benchmarks like TinyShakespeare and a subset of WikiText-2, indicating its potential for more efficient representation learning in language modeling.