Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact

The paper presents findings that psychological profiles assigned to large language models (LLMs) using human-designed instruments are largely measurement artifacts rather than true representations of model traits. Analysis of 56 instruction-tuned LLMs reveals that 81-90% of the variance in model responses is attributed to directional response bias, with the remaining variance reflecting actual traits. The study emphasizes the need for tailored assessment tools that account for response orthogonality to improve the validity of psychological evaluations for LLMs, which has implications for their usability and safety in research contexts.