Exposing the Unsaid: Visualizing Hidden LLM Bias through Stochastic Path Aggregation

The paper introduces TreeTracer, a visual analytics tool designed to evaluate biases in Large Language Models (LLMs) by utilizing stochastic path aggregation. It employs a systematic perturbation analysis to replace terms in prompts, aggregate stochastic generations into a hierarchical structure, and visualize biases through a custom Sankey diagram. This approach allows for direct comparison of semantic contexts and aids in the detection of hidden biases, validated through case studies comparing GPT-2 XL and Apertus models, demonstrating its effectiveness in reducing cognitive load for analysts.