The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection

The paper introduces the Zero-shot Procedural Mistake detection (ZeProM) framework, which utilizes a single pre-trained Video-Language Model (VLM) to jointly address procedural mistake detection and temporal action segmentation without the need for task-specific training datasets. Evaluation on benchmarks EgoPER and CaptainCook4D demonstrates that ZeProM can achieve performance comparable to or exceeding fully supervised methods, with a notable 4.4 point improvement in EDA and a 2.0 point increase in F1@.5 across EgoPER tasks. This advancement suggests a shift towards more streamlined and broadly applicable solutions in procedural mistake detection, reducing reliance on complex multi-stage pipelines.