What Does Agreement Mean? Confidence-Aware Evaluation of LLM Values

As large language models (LLMs) become widely deployed, evaluating the values reflected in their outputs is increasingly important. Unlike factual correctness, value alignment lacks objective ground truth and depends on the evaluator’s standpoint. Existing methods rely on either predefined references that overlook value pluralism or costly human preferences marked by substantial disagreement. To address this limitation, a recent work instead lets LLMs judge one another, it offers a scalable way to compare language models under explicit value constitutions, but an aggregate score obscures how judges arrive at their preferences. Similar scores can arise from conflicting preferences or widespread ties, while high agreement need not indicate support for a particular response. We introduce ConfStruct, a framework for examining the structure of LLM value judgments across levels of reported confidence. ConfStruct maps each judge’s confidence scores to CDF smoothed within-judge percentiles and sweeps a shared threshold to obtain model preference trajectories. Alongside trajectory-based rankings, ConfStruct reports four complementary panel-level diagnostics: consensus, verdict diversity, polarization, and tie prevalence. Experiments involving 18 models divided into two panels, three constitutions, and three scenario datasets show that Deep Ecology consistently elicits more ties and lower polarization than the other constitutions, revealing that apparent agreement can coexist with limited discrimination between candidate responses. The resulting rankings are robust to constitution rewording, with pairwise correlations of at least 0.90, while a sanity check on GPQA Diamond recovers the ground-truth accuracy ranking with a correlation of 0.90. These results demonstrate the importance of evaluating value alignment through both model rankings and the confidence-dependent structure underlying them.