Token Complexity of Certifying Stochastic-Oracle Reliability

The paper introduces a framework for certifying the reliability of stochastic oracles, defining "certification token complexity" as the minimum expected token cost to distinguish between oracles that meet a specified reliability level and those that do not. It presents a Sequential Probability Ratio Test (SPRT)-based Stochastic-Oracle Turing Machine (SOTM) that effectively queries oracles and computes correctness scores while ensuring two-sided error guarantees. This work is significant for practitioners as it provides theoretical bounds on token complexity, informing the design of efficient certification processes in AI systems that rely on stochastic oracles.