Do LLM Embedding Spaces Recover Expert Structure?

The paper investigates the ability of pretrained and fine-tuned Qwen3 embedding spaces (0.6B and 4B parameters) to recover expert-defined structures in mental health-related language. Through representational similarity analysis and various controls, it finds that while pretrained embeddings align with expert structures, fine-tuning enhances this alignment, particularly at finer category levels. This research highlights the importance of evaluating embedding geometry against explicit confounds, suggesting that LLM embeddings can effectively capture expert-relevant category relationships, which is crucial for practitioners leveraging embeddings in specialized domains.