Critically Engaged Pragmatism: Scientific Norm and Social, Pragmatist Epistemology for AI Science Evaluation Tools

The article proposes a framework called Critically-Engaged Pragmatism aimed at improving the transparency and reliability of AI science evaluation tools. It emphasizes the need for developers to disclose comprehensive details about the design, training, and benchmarking of these tools to assess their purpose-specific reliability and potential biases. This approach challenges the notion that such tools can serve as objective measures of scientific credibility, advocating instead for a critical discourse around their use and implications in the scientific community.