Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims

The article introduces FRANZ, an automated framework designed for auditing large language model (LLM) responses based on four dimensions: cultural positioning, generalizing language, anthropomorphic cues, and adherence to conversational maxims. It also presents SQUARE, a corpus of 376,000 subjective questions from various subreddits, which allows for a comprehensive evaluation of LLM responses across different cultural contexts. This framework is significant for practitioners as it provides a multi-dimensional approach to assess how LLMs communicate, highlighting the importance of response framing in subjective queries, which could enhance the design and deployment of LLMs in culturally sensitive applications.