Beyond Bayer: Task-Optimal Sensor Co-Design for Robust Autonomous-Driving Segmentation

The paper presents a novel approach to sensor co-design for autonomous driving segmentation, emphasizing the importance of optimizing camera measurements rather than solely relying on larger models. It introduces a differentiable RAW-to-task pipeline that learns optimal spectral colour-filter-array (CFA) weights, achieving improvements in mean Intersection over Union (mIoU) by +0.017 on the KITTI-360 dataset and +0.023 on ACDC, while demonstrating that co-designing optics leads to negative outcomes. This work is significant for practitioners as it highlights the potential for sensor-level optimizations to enhance model performance in diverse environmental conditions, independent of the downstream model architecture.