You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences

The paper introduces Temporal Difference in Vision (TDV), a novel self-supervised learning paradigm for visual representation from video that eliminates reliance on strong inductive biases such as augmentations and cropping. Instead, TDV utilizes a causal assumption where past frames inform future representations, training a joint image and motion encoder. This approach matches state-of-the-art performance on dense spatial tasks, indicating a significant shift towards more flexible learning methods as data scales, which is crucial for practitioners aiming to build robust AI systems without the constraints of traditional assumptions.