acceptodds
Under review as a conference paper at ICLR 2027

What Does Agreement Mean? Confidence-Aware Evaluation of LLM Values

Abstract

As large language models (LLMs) become widely deployed, evaluating the values reflected in their outputs is increasingly important. Unlike factual correctness, value alignment lacks objective ground truth and depends on the evaluator’s standpoint. Existing methods rely on either predefined references that overlook value pluralism or costly human preferences marked by substantial disagreement. To address this limitation, a recent work instead lets LLMs judge one another, it offers a scalable way to compare language models under explicit value constitutions, but an aggregate score obscures how judges arrive at their preferences. Similar scores can arise from conflicting preferences or widespread ties, while high agreement need not indicate support for a particular response. We introduce ConfStruct, a framework for examining the structure of LLM value judgments across levels of reported confidence. ConfStruct maps each judge’s confidence scores to CDF smoothed within-judge percentiles and sweeps a shared threshold to obtain model preference trajectories. Alongside trajectory-based rankings, ConfStruct reports four complementary panel-level diagnostics: consensus, verdict diversity, polarization, and tie prevalence. Experiments involving 18 models divided into two panels, three constitutions, and three scenario datasets show that Deep Ecology consistently elicits more ties and lower polarization than the other constitutions, revealing that apparent agreement can coexist with limited discrimination between candidate responses. The resulting rankings are robust to constitution rewording, with pairwise correlations of at least 0.90, while a sanity check on GPQA Diamond recovers the ground-truth accuracy ranking with a correlation of 0.90. These results demonstrate the importance of evaluating value alignment through both model rankings and the confidence-dependent structure underlying them.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-judge, value alignment, confidence calibration, LLM evaluation, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WpE2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a new method to examine LLM judgement structures with taking into account their confidence scores. The method, called ConfStruct, normalize confidence scores using CDF kernel so that normalized scores are uniform. Thanks to that, ConfStruct then can compare judgements from different LLMs, aggregate them in coherent trajectories and single scores. The paper goes further with introducing complementary panel-level diagnostics---consensus, verdict diversity, polarization, and tie prevalence.

In experiments, the paper demonstrates how ConfStruct is able to track judgement structures of 18 LLMs. The paper also shows that the proposed aggregated score ranking highly correlated with ground-truth annotation on a dataset.

Critical strengths

This is an interesting paper tackling a challenge (but very useful if solved), that is how to aggregate agents opinions. The ideas are interesting, and to me, novel. The paper presents pretty strong math framework.

The paper also presents lots of information in the appendices.

The results shown are interesting, demonstrating how the work can be used helpfully in the community.

Critical weaknesses

One of the major problems of the paper is presentation.

  • First of all, the maths presentation should come with clear explanations. For instance, what is the intuition behind equation 4? What are domains/ranges of introduced functions/values? w_i(x, j) is not well-formed when x doesn't contain y_j. The claim that R (equation 11) is well defined isn't correct when all models agree with each other (in this case, spread(lambda, c) == 0 and thus g == 0 always).

  • Second, in the experiments, I believe that it's extremely crucial to demonstrate that the proposed ConfStruct is faithful. Therefore, the key experiment results should be Table 1 and section 4.4. Hence, my suggestion is to move the results in table 1 and section 4.4. to the beginning of section experiments. Other results should be considered as how ConfStruct is used.

Following the second point above, ConfStruct should have tested on more datasets/benchmarks with groudtruth (like table 1) to prove its faithfulness. Besides, although section 4.4 is important, it's too brief, especially, it's unclear why d_{H,M} vs d_{M,M} can tell anything meaningful about ConfStruct (w/o reading the corresponding appendix).

Some minor points really need to be clarified.

  • Line 156, wikjl = 1 − wijkl. Is it true when using LLMs? Very often an LLM can give different output for different input orders e.g. (y_i, y_k) vs (y_k, y_i)
  • Line 255, "The factor 4 normalizes the score to peak at 1...", why?
  • Line 431, "GPQA Diamond (Rein et al., 2023), where a correct answer exists but is never revealed to the judges" --- but is it true that the used LLMs weren't trained on this dataset?
Questions for authors

Please see above.

Reviewer_mDt6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ConfStruct, whcih studies how model-judgment rankings and panel-level agreement patterns change across reported-confidence thresholds. Each judge's confidence is transformed into within-judge percentiles. The method then constructs preference trajectories and reports consensus, verdict diversity, polarisation and tie prevalence, together with an integrated trajectory-based ranking. Experiments involve 18 models organised into two panels, three constitutions and three scenario datasets. Additional checks examine constitution rewording, panel composition, a factual-question ranking and a small human-judgment study.

Critical strengths
  1. The paper makes a useful distinction between different sources of apparent agreement. A panel dominated by ties is not equaivalent to one that consistently prefers the same candidate, and neither is equivalent to two opposing groups whose votes cancel. Reporting these patterns jointly is more informative than using a single ranking or agreemnet statistic.

  2. The empirical examination is broader than a single illustrative panel. The three constitutions, two panels, multiple scenario sets and external checks provide useful observations about when rankings compress or distinguish candidates. The repeated finding of greater tie prevalence under Deep Ecology is a concrete descriptive result.

Critical weaknesses
  1. The contribution of the distinctive confidence-aware pipeline is not sufficiently isolated. The paper appropriately treats transformed confidence as a relative within-judge quantity instead of a probability of correctness. However, it remains unclear what confidence-based filtering and trajectory integration add beyond confidence-independent aggregation of the same judgments. Appendix E usefully compares judgment-level and judge-level weighting, but both variants retain confidence-based thresholding. Moreover, the zero-threshold setting still retains confidence weights in Equation 4, so it is not a fully confidence-independent baseline. The integrated ranking also weights thresholds using score dispersion and tie prevalence, without isolating the value of this choice relative to uniform integration. The GPQA and human checks demonstrate that the outputs contain meaningful signal, but do not identify the contribution of these specific components. This concern is about the justification and validation of the proposed integrated ranking, not a requirement that every descriptive diagnostic outperform a predictive baseline.

  2. The interpretation of ties is underdetermined. The paper explicitly interprets tie prevalence as collective abstention, but a tie can indicate two equally good responses, two equally poor responses, criterion irrelevance, or inability to discriminate. These possibilities have different implications for a constitution's evaluative behaviour. Applying the same constitutions to additional datasets does not by itself resolve relevance differences, particularly for ecological criteria. The data support the descriptive finding of more ties and less polarisation, but do not establish that this pattern specifically represents abstention or a general property of the constitution.

Questions for authors
  1. Using the existing judgments, how do the rankings, external agreement, and main diagnostic conclusions change under confidence-independent aggregation, confidence shuffled within each judge and evaluation setting, and uniform integration over thresholds? Which findings specifically depend on the proposed confidence mechanism?

  2. Can a small, independently reviewed sample distinguish substantive equivalence from criterion irrelevance or inability to judge? In particular, does the Deep Ecology pattern remain on scenarios for which the ecological criteria are clearly relevant?

Reviewer_urKaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

ConfStruct evaluates LLM values through confidence-aware peer judgments under explicit constitutions. It transforms each judge's confidence with a smoothed CDF to express relative confidence, constructs trust trajectories across thresholds, and aggregates them into rankings. Four diagnostics measure consensus, pluralism, polarization, and tie rate. Experiments cover 18 models, two panels, three constitutions, and three scenario datasets. Frequent Deep Ecology ties show that apparent consensus need not indicate a shared response preference. Validation includes cross-dataset comparisons, single-model removal, Conservatism rewording, GPQA ranking recovery, and a human study under Conservatism.

Critical strengths

1. Motivation & Usefulness

  • Aggregate rankings can hide agreement, disagreement, and ties. Distinguishing them helps interpret value evaluations.
  • Confidence trajectories reveal ranking sensitivity as less confident judgments are removed.

2. Related Works & Novelty

  • Confidence-conditioned trajectories and panel diagnostics may usefully extend peer-based evaluation beyond final rankings.
  • Separating ties from directional preferences distinguishes unanimous ties from evenly divided response preferences.

3. Experiments & Robustness

  • Experiments cover multiple model families and three scenario domains. Qwen3-8B removal and Conservatism rewording provide useful sensitivity checks.
  • The GPQA ranking correlation of 0.90 provides supporting evidence. Aggregate human-model ranking agreement is 0.97 for nine candidates under Conservatism. Eight human judges evaluate the same 100 comparisons as the models.

4. Presentation

The paper is generally clear. Its framework illustration and trajectory plots explain the method, and the appendix provides evaluation prompts.

Critical weaknesses

The problem is relevant, but the added value of confidence conditioning over prior work remains unclear. Aggregation and validation gaps leave the rankings and their interpretation insufficiently supported. Resolving them could change my assessment.

1. Related Works & Novelty

  1. Novelty is insufficiently distinguished from prior work. EigenBench [1] already links frequent Deep Ecology ties to scenario relevance (ICLR version, Figure 3 caption). Appendix A (lines 855–857) says previous methods do not report disagreement, but the cited Language Model Council [2] studies inter-judge agreement and disagreement with the majority.

2. Theory & Methodology

  1. Appendix J.1 compares every ordered pair, but Eq. (4) and Algorithm 1 normalize each candidate's credit over all retained comparisons. Treatment of items not involving that candidate is unspecified. If they contribute zero credit, all-tie judgments can yield unequal intermediate scores s_j through retained confidence mass. Balanced initial comparisons do not prevent this after weighting or filtering. For an entirely tied panel, g = 0 makes the final R_j undefined in Eq. (11); no fallback is stated. Retained-exposure statistics would clarify the possible influence on reported rankings. Its empirical impact remains unresolved.
  2. Appendix G's uniformity claim (lines 1288–1294) needs more support when judges use few confidence values. Equal raw scores map to equal values; smoothing preserves their probability masses. The transform can express relative confidence, but the probability integral transform alone does not establish approximate uniformity here. Figure 11 illustrates approximate uniformity for four judges, but omits their identities, distinct score counts, and retained fractions across thresholds. It does not establish the general claim for discrete confidence anchors.
  3. In Section 3.1, Eq. (5) is undefined for an item with no surviving judge. With one judge, its pluralism term is 0/0 and its consensus contribution is 1 (Eqs. (6)–(7)). The paper does not report whether these cases occur or how they are handled. Higher consensus may partly reflect fewer judges. Retained judge counts are needed to interpret the sweep; a null comparison matched for per-item judge counts and confidence weights could help assess this effect.

3. Experiments & Robustness

  1. Appendix E (lines 1154–1165) reports agreement between weighting schemes at ρ ≥ 0.93. Scheme A's constant weight cancels, but both schemes still filter by confidence. Scheme A at λ = 0 gives an unweighted, unfiltered baseline. Final trajectory scores are not compared with it on GPQA or human agreement, leaving the benefit of thresholding and integration unclear.
  2. Section 3.1 (lines 259–264) describes ties as collective abstention and indistinguishability. These differ: responses may be equally aligned, or criteria irrelevant. The human study covers only Conservatism. Interpreting Deep Ecology ties as abstention lacks direct validation through ecology-relevant comparisons and labels explaining ties.
  3. Section 4.3 (lines 417–422) claims preserved cross-domain rankings, with extreme-tier models staying at the top and bottom. Yet Qwen3-8B moves from first on AskReddit to seventh on AIRiskDilemmas under small-panel Universal Kindness (Tables 8 and 5). Figure 7 reports correlations of 0.47–0.82, including 0.48 and 0.50 for Deep Ecology; Appendix C.1 acknowledges instability. Replicated ties do not establish model-ranking robustness. The other checks remove only Qwen3-8B and reword only Conservatism, so they support narrower claims than general panel or wording independence.

4. Presentation & Reproducibility

  1. Tables 6 and 7 differ despite both reporting AskReddit at λ = 0. Large-panel Deep Ecology has tie rates 0.826 versus 0.8550 and consensus 0.846 versus 0.8822 (Appendix F, lines 1181–1185). No specified difference in settings or weighting reconciles these values, limiting reproducibility.

References

[1] Chang et al. EigenBench: A Comparative Behavioral Measure of Value Alignment. ICLR, 2026. https://proceedings.iclr.cc/paper_files/paper/2026/file/762306432afa0ecbbf1d615be551e96b-Paper-Conference.pdf

[2] Zhao et al. Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks. NAACL, 2025. https://aclanthology.org/2025.naacl-long.617/

Questions for authors
  • Using the same judgments, how does the final trajectory score compare with Scheme A at λ = 0 on GPQA and human ranking agreement?

  • How does Eq. (4) treat items not involving a candidate, and how does retained confidence mass vary across candidates? What conventions handle empty or single-judge items and an all-tie panel?

  • Can the authors distinguish Deep Ecology ties caused by equal alignment, irrelevant criteria, and uncertainty? Does the high tie rate persist when ecological criteria clearly apply?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.