Structured Hyperedge Adaptation for Parameter-Efficient Fine-Tuning of Vision Transformers

The paper introduces HyperAdapter, a novel hypergraph-based adapter architecture for parameter-efficient fine-tuning of vision transformers (ViTs). By adapting in hyperedge space rather than token space, HyperAdapter leverages structured relationships among tokens through soft token routing, resulting in improved feature refinement and performance across various visual benchmarks. This approach demonstrates that the adaptation space significantly impacts the effectiveness of parameter-efficient transfer methods, particularly for tasks necessitating structured reasoning.