GEMS: Geometric Constraints Enable Multi-Semantic Superposition in LLMs

The paper introduces GEMS, a training-free method for multi-semantic superposition in large language models (LLMs), addressing issues of distributional deviation and directional interference during activation steering. GEMS employs geometric constraints—norm-preserving weighted superposition and real-time orthogonalization—to maintain model performance, achieving a 98% accuracy on GSM8K with multiple concurrent directions, compared to a baseline of 92%. This approach is significant for practitioners as it enables more complex semantic manipulations in LLMs without the need for retraining, enhancing the flexibility and robustness of model behavior during inference.