Fixed RAG Compression Collapses Measured Reader Scaling

The paper introduces ragscale, a toolkit for auditing reader scaling in Retrieval-Augmented Generation (RAG) compression, highlighting that fixed compression can distort accuracy measurements and model rankings across various readers and benchmarks. It demonstrates that compression can improve weak readers while detrimental to strong ones, with significant results across 20 readers and multiple QA and summarization benchmarks. This toolkit enables practitioners to evaluate the impact of compression on model performance more effectively, addressing a critical gap in current RAG evaluation methods.