Towards Critical Branching Mechanism in Recurrent Neural Networks

The paper presents an analysis of hidden-state dynamics in trained LSTM networks, revealing that small networks near optimal training epochs exhibit near-critical dynamics characterized by scale-free avalanche statistics and branching parameters approaching unity, while larger models show subcritical behavior. The study introduces a mixture branching process framework to explain the coexistence of subcritical branching with robust $1/f^{\beta}$ noise, suggesting that critical-like behavior in LSTMs is an emergent feature dependent on model capacity. This research provides insights into the dynamical regimes of RNNs, which could inform the design and training of more effective neural architectures.