A Projection-Based Surrogate Gradient Interpretation for Neural Codec Wrappers

The paper presents a novel interpretation of surrogate gradients for neural codec wrappers, which enhance traditional video codecs by facilitating end-to-end learning despite the non-differentiability of the encoding process. The authors introduce a first-order local approximation of the video codec via the SCALED method, which improves compression performance and generalizes across various codecs and tasks, achieving BD-Rate (PSNR) reductions of up to -23.59% on x264 and -20.07% on VVenC compared to standard resampling methods. This advancement is significant for practitioners as it enhances the training of neural pre-processors and full neural wrappers, improving overall compression efficiency in video processing applications.