Sparse probes and murky physics: a case study of interpretability challenges in a foundation model for continuum dynamics

The article discusses the evaluation and interpretability challenges of the foundation model Walrus by Polymathic, designed for continuum dynamics. It employs a sparse autoencoder (SAE) to probe internal mechanisms and assess over 20,000 features against enstrophy, revealing piecewise consistency in feature roles but intermittent structure that does not align with traditional physical decompositions. This research underscores the need for improved methods to identify meaningful features and differentiate between genuine internal representations and artifacts, which is critical for practitioners aiming to integrate generative AI models in scientific applications.