Dimensionality Controls When Modularity Helps in Continual Learning

The study investigates how modular architectures and representational dimensionality influence compositional continual learning, particularly in a sequential A-B-A task paradigm. It compares a task-partitioned recurrent network against a single-network baseline across high- and low-dimensional regimes, finding that modularity significantly enhances performance in lower-dimensional contexts by creating task-specific subspaces that improve interpretability and organization. These insights highlight the importance of representational dimensionality and initialization scale in optimizing continual learning systems, suggesting a shift in focus towards adaptive allocation of representational subspaces for improved safety and robustness.