Where Does Texture Evidence Live in SAM? Features, Proposal Masks, and Texture Segmentation

The article investigates the limitations of Segment Anything Models (SAM) in texture segmentation, revealing that while SAM is not inherently designed for this task, it retains valuable texture-related evidence in its frozen state. The study analyzes two evidence spaces—multiscale features and the automatic proposal bank—without fine-tuning the model, demonstrating that failures in texture segmentation stem from issues in representation, proposal support, and readout processes rather than a lack of texture awareness. This insight is crucial for practitioners, as it suggests that leveraging existing features and proposals in SAM may enhance texture segmentation performance without necessitating extensive retraining.