acceptodds
Under review as a conference paper at ICLR 2027

A LANGUAGE-MODEL JUDGE’S RESOLUTION IS PREDICTED BEFORE IT IS MEASURED

Abstract

A language-model judge cannot tell every pair of answers apart. Below some difference in quality it is indifferent, and that threshold decides whether it can grade the work in front of it. We register two gates. The first is that the threshold can be predicted before it is measured, and the other is that a second budget can reverse the judge’s preference outright. Judges grade worksheets whose quality is known exactly. A calibration block records the scores a judge actually uses, and from that block alone we predict, and commit with its hash, how often the judge will order fresh pairs correctly at every difference in quality, on three rating scales and six ways of reading its report. Only then is the test block’s seed drawn. Resolution. The prediction held on every graded judge, the largest deviation 0.078 in accuracy against a registered bar of 0.14. The scale a judge is offered is the wrong budget, and no one number drawn from its reports is the right one. The 7B judge asked for a score from 0 to 100 writes nine distinct scores and resolves 5.3 wrong answers in twenty where the scale promises one, and on every scale of every judge the channel it writes fits its accuracy better than the nominal scale does. Reading more of the report lowers the threshold by the predicted amount on five of the six graded judges. Reversal. Give the worse of two worksheets a confident header and a check mark on every line. With no reasoning budget the judge prefers the decorated worse worksheet, only 0.125 of test pairs favouring the better one, and at 1024 tokens of reasoning it favours the better one in 0.95. Measuring what the evidence and the decoration are worth in separate blocks, in which the two never conflict, predicts that reversal, and a second ladder that brackets the crossing locates it, predicted at 72 tokens and observed at 57. On a second vendor’s judge it is predicted at 187 and observed at 166, from that judge’s own calibration. The prediction and the comparison against the nominal scale replicate on a second vendor’s models served through an API, on summaries checked against a passage, which no judge does exactly, and on readers’ reviews scored against the star rating each writer gave. Read to 200 characters, to 400, and whole, the judge’s resolution on those reviews coarsens in that order by the amount each budget’s own calibra- tion predicted. Every bar was fixed before the first of these runs and none was refitted. What predicts a judge’s resolution is its measured channel, of which a scalar budget is one summary.

Primary area:
learning theory
Keywords:
LLM-as-a-judge, evaluation methodology, preregistration, discrimination threshold, semiorder, effective codebook, reasoning budget, held-out prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sUrvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a calibration-based approach to predict LLM judges’ resolution and demonstrates how reasoning budgets can reverse their preferences.

Critical strengths

The research question of resolution is meaningful and the main claim is clear, also the paper is well organized.

Critical weaknesses
  1. The writing of abstract needs to abbreviate, and also make it more intuitive. For example, when introducing the resolution example, authors can name the phenom and then measure its impact. Also, what’s the relationship of the 3 ‘R’s? Are they parallel? Current writing needs big improvement.
  2. Authors should clarify what qwen models they use in table 1, to consist with the notion of gemma 3-4B.
  3. After adapting the now existing measurement problem solution in LLM as a judge area, do authors post any new methods? Seems current method can be directly get from a very simple statistical process. Thus, there will be necessary to add these baselines: (1)Empirical channel (to see what GET actually added) (2)IRT/GRM (3) Simple regression
  4. The theoretical proof on GET provides a framework, but the current version hasn’t proven it’s necessary and can not be substitute.
Questions for authors

See above

Reviewer_6wmHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper shows that we can predict how accurate an LLM judge will be on held out multiplication worksheets by observing how accurate it is on a calibration set. This allows us to understand what resolution of quality differences a judge can see. The prediction is related to the scale used by the judge, how much of the distribution is involved in the quality judgment, and reasoning budget where available. The scale used by the judge is not necessary the scale proposed by the person who set up the judge. For example, when asked to score quality on a 0-9 scale, Qwen 7B only outputs 8 different scores; on a 0-100 scale, it only uses 9 scores. The distributions tested are greedy sampling i.e. a 1 sample estimate, an 8 sample estimate, and expectation over the full distribution. Predicted and observed resolution improves as more of the distribution is used. Experiments looking at the contribution of reasoning show how budget relates to accuracy.

Critical strengths

The experiments use preregistered hypotheses recorded in a commit log in the associated code repository. Preregistration should be more widely adopted in AI research.

The reasoning experiments are nicely designed for exploiting LLM judge preference for decorations

Critical weaknesses

The paper unpublishable because it is extremely unclear. The description of experimental settings is poor, leaving the reader to guess what was done. Many features of the experiments are described with terms that are undefined in the paper and do not relate to my experience of existing literature. The overall structure of the main text is disorienting.

Questions for authors

Suggestions: Start the main paper by summarizing the first 3 appendices to improve clarity. Consider moving some experimental results referenced in the abstract from the appendix to the main paper if they are really helpful to your claim.

Reviewer_82Z5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether calibration data can predict the quality differences that language-model judges can reliably distinguish. It introduces a theoretical framework connecting resolution, indifference, and resource constraints, and evaluates predictions derived from conditional score distributions. Experiments examine models, scoring scales, read-outs, and computational budgets, with extensions to summary verification and review-rating prediction.

Critical strengths
  1. Separation of calibration and testing. Predictions are evaluated on fresh comparisons, providing meaningful evidence of predictive usefulness.

  2. Systematic experimental comparisons. The model, scale, and read-out comparisons show how evaluation choices affect measured resolution.

  3. Transparent reporting of limitations. The paper reports unsuccessful cases, floor censoring, and non-monotonic reasoning-budget effects.

  4. Useful operational perspective. Measuring distinguishable quality gaps complements aggregate agreement metrics and makes judge limitations more interpretable.

Critical weaknesses
  1. The empirical contribution of the theoretical framework remains unclear. The main predictor uses a measured conditional score distribution to predict fresh comparisons. This demonstrates the usefulness of calibration, but does not sufficiently establish what additional predictive power the geometric framework provides. Appendix D compares the full channel against simpler summaries; a calibrated noisy baseline would better distinguish the benefits of modeling score variability from those specifically attributable to the proposed framework.

  2. The ordering claims need a stronger treatment of uncertainty and censoring. Fixed acceptance thresholds prevent retrospective tuning, but their false-alarm probabilities can depend on the judge and experimental design, as Appendix G illustrates. The Gemma 4 31B ordering failure coincides with floor-censored predictions, but censoring is not established as its cause. Likewise, agreement at the minimum tested gap cannot establish the ordering of thresholds below that gap. Design-matched null analyses and uncertainty estimates would strengthen these conclusions.

  3. The reasoning-budget experiment does not isolate improved discrimination from forced-interruption effects. Intermediate token caps suppress evidence discrimination as well as cue influence. Consequently, crossing 50% preference does not necessarily indicate restored task competence. Although the paper acknowledges this non-monotonicity, naturally completed or budget-aware reasoning controls are needed to support a stronger interpretation. Separate calibration at each budget demonstrates prediction under the tested protocol, rather than independently establishing a causal relationship between budget and resolution.

  4. The natural-language extensions support relatively narrow generalization. These experiments have limited model coverage, and the full input-length result is graded on one configuration. Predicting a Goodreads reviewer’s book rating also differs from evaluating answer quality. Additional model families and tasks with independently established answer-quality differences would strengthen the broader claims. The longer-worksheet experiment should also separate worksheet length from the sampled correctness range.

  5. Important operational details need clearer specification. The manuscript should enumerate all six read-outs and explain score extraction and normalization under truncated API log-probability access. The reversal experiment also needs a clearer description of how separate evidence and cue calibration distributions yield predicted win shares, together with the crossing interpolation procedure.

Questions for authors
  1. Which empirical predictions specifically require the geometric framework, and how do they compare with a simple noisy score model fitted using the same calibration budget?

  2. How robust are the ordering conclusions to threshold uncertainty, floor censoring, and design-specific false-alarm rates?

  3. Does the reasoning-budget reversal persist when reasoning completes naturally or is explicitly adapted to the available budget? Can you provide the finer ladder’s evidence and cue curves?

  4. Do the natural-language findings transfer to another model family and to tasks measuring independently established answer quality? Can worksheet length and correctness range be varied separately?

  5. Can you provide complete definitions of the six read-outs, probability-normalization procedures, and the mapping from calibration distributions to predicted win shares?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.