Robustness of Similarity-based Positional Encoding Under Rotations: Theoretical Analysis and Experimental Validation

The paper presents a theoretical and experimental analysis of similarity-based positional encoding (simPE) in Transformer architectures, particularly focusing on its robustness under rotational perturbations. It demonstrates that while simPE is not inherently rotation-invariant, it remains stable under mild Lipschitz conditions, with derived perturbation bounds in Frobenius norm. Experimental results across multiple datasets, including FashionMNIST, show that simPE outperforms standard learned positional encoding in accuracy and other metrics when subjected to rotations, making it a valuable approach for applications in medical imaging and other domains where geometric robustness is critical.