PermDoRA -- Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometry

The paper introduces PermDoRA, a framework for analyzing adapter interference in large language models (LLMs) using a hierarchical adapter composition method called DoRA-RBAC. It compares conventional Euclidean merging with a Riemannian-inspired merging strategy on models LLaMA-3.1-8B and Mistral-7B across multiple QA benchmarks, revealing that geometry-aware merging does not consistently outperform standard averaging in multi-domain scenarios. This research indicates that adapter interference may be more influenced by interactions in shared nonlinear representations rather than parameter-space geometry, which has implications for practitioners seeking to optimize multi-domain performance in LLMs.