The Token Tax of Epistemic Accuracy: Comparing RAG and Long-Context Architectures for Document-Grounded Generative AI Applications

This study compares retrieval-augmented generation (RAG) and long-context prompting architectures for document-grounded generative AI, focusing on their impact on epistemic accuracy in high-stakes applications. Long-context prompting achieved a correctness rate of 73.1% compared to RAG's 65.4%, but incurred a significant "token tax," costing 26 times more per query due to increased input token consumption. These findings highlight the trade-offs between accuracy and resource efficiency, which are critical considerations for practitioners working with large language models in knowledge-intensive domains.