Protein contacts are already in the attention: a single-forward-pass alternative to the Categorical Jacobian

The paper presents a novel approach to reading protein contacts using a single forward pass by leveraging a small subset of attention heads, significantly improving efficiency over the Categorical Jacobian (CJ) method, which requires approximately 19 forward passes. The authors demonstrate that averaging the top-K contact-relevant heads selected from as few as 10 labeled proteins outperforms CJ on leakage-clean data for various bidirectional models, with a notable improvement of +9 percentage points on the ESM-2-650M model. This work is significant for practitioners as it offers a more efficient and potentially more accurate method for extracting protein contact information from language models, which could enhance the development of models in bioinformatics and protein structure prediction.