Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

The paper introduces Orthogonal Representation Editing (ORE), a novel approach for batch knowledge editing in Large Language Models (LLMs) that addresses performance degradation due to semantic representation entanglement. ORE operates in the hidden representation space by creating a general semantic subspace and applying orthogonal constraints on edit vectors, which enhances editing precision. The method includes a gated non-linear representation head for adaptive learning of editing locations, demonstrating superior performance over existing techniques, particularly in cross-lingual scenarios, with the code available at GitHub.