acceptodds
Under review as a conference paper at ICLR 2027

More Reasoning Does Not Always Make a Better Judge: Understanding Inference-Time Compute in LLM Evaluation

Abstract

LLM judges are increasingly used to evaluate outputs requiring complex reasoning, yet it remains unclear whether additional inference-time reasoning improves their decisions or merely produces longer rationales. We investigate this question using a simple forced-continuation intervention as a diagnostic probe across open-domain, safety, mathematics, and code preference benchmarks. The strong open-source judges we test generally fail to benefit: additional reasoning often causes semantic drift and overconfident errors. To isolate the training conditions that enable useful scaling, we train a 7B judge with reflection-augmented supervised fine-tuning followed by reinforcement learning with verifiable outcome rewards, and analyze checkpoints throughout training. Across four benchmarks, this judge improves aggregate accuracy by roughly five percentage points over comparable open-source judges and achieves a 4.1% average relative gain from additional reasoning, whereas the baselines exhibit negative scaling. Checkpoint and ablation studies show that this capability emerges primarily during reinforcement learning rather than from reflective supervision alone. We characterize successful and failed scaling through semantic alignment to oracle rationales, entropy dynamics, and calibration. Successful scaling preserves semantic alignment and uncertainty during deliberation; failed scaling exhibits drift and premature entropy collapse. These differences yield substantially better calibration and support statistically valid risk-controlled selective prediction. Our results show that more reasoning does not inherently produce better judgments: converting extra compute into reliable evaluation is a learned property.

Primary area:
interpretability and explainable AI
Keywords:
Reasoning, LLM-as-a-Judge, Test-time scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wrXUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether LLM judges benefit from test-time scaling, and under which conditions this benefit emerges. The authors apply probe judges with simple test-time scaling (STTS), which replaces the end token of thinking steps </think> with reflective tokens to make model continue generation. The paper also introduces a training recipe for a judge model, DJ-7B including generating reasoning traces with DeepSeek-R1, filtered by rejection sampling for cold-start SFT, followed by RL training. Experiments on four benchmarks show that DJ-7B outperforms several open-source judge models, and that DJ-7B exhibits a positive trend of test time scaling while two baseline models do not. The paper further examines how the scaling benefit across RL training stage, and analyses semantic alignment, entropy dynamics, uncertainty, and risk-controlled selective prediction.

Critical strengths
  1. The problem of whether LLM judges benefit from additional inference-time reasoning is really important, given the wide use of LLM-as-a-judge and test-time scaling.
  2. The training recipe is clearly described, and the main comparison (Table 1) covers diverfised models on four benchmarks.
  3. The paper provides analysis of semantic alignment, uncertainty and risk-controlloed prediction.
Critical weaknesses
  1. Paper structure: The paper is not well-structured and consistent. The introduction part states a lot of background and motivation and "When does additional inference-time reasoningimprove an LLM judge?", while in Section 3 with the title "Simple Test-Time Scaling for LLM-as-a-Judge", mainly describes the training recipe for DJ-7B about how to fine-tune a judge model, with simple test-time scaling only covered in subsection 3.4. In terms of the three stated contributions, the instantiation of DJ-7B leads the main experimental results which is fine. But for "Empirical Characterization of Judge Scaling", the result is presented on Table 2, which includes only two baseline judges. The "Training Dynamics" conclusion that "useful scaling emerges primarily during RL" is supported by a short paragraph in subection 4.5 whose results (Figures 15–18) appear only in the appendix and are barely explained.

  2. The training method is not new: Rejection-sampled data for cold-start SFT followed by RL with verifiable rewards is now a standard recipe for finetuning a specific task, and it has already been applied to judge models such as the citation in this paper JudgeLRM, DeepSeek-GRM. DeepSeek-GRM explicitly studies inference-time scaling for generative reward models, which weakens the contribution and contradicts the motivation in introduction part "test-time scaling techniques have primarily been studied for problem solving rather than evaluation". The training recipe occupying the main method section and the main results is not new.

  3. Baseline selection is a bit confusing: Since the paper chooses to use GPT-4o, o1-mini, then why not just using recent models like GPT-5 or mini? similar to those RL trained judge models.

  4. STTS is not explained sufficiently: Section 3.4 describes the approach of replacing </think> with a reflective tokens and continuing generation from the prefix, but missed decoding settings. In addition, this intervention assumes an explicit thinking format so how about judge models are not trained with CoT format? and those Proprietary models? and breaking their format may also cause artifacts of intervention.

  5. The STTS results are not convincing: 5.1Only two baseline judges are evaluated. Given that STTS only requires prompting with prefix, it would be cheap to evaluate on more models, at least on DeepSeek-R1 which you use to generated the training data.
    5.2 The gains are actually small (even with the confusing "delta relative%" metrics amplifying the absolute number): at Attempt 4, the improvement is roughly +1.3pp, +0.5pp, +0.4pp, and +0.3pp on four benchmarks, which is a bit trivial. And no multiple seeds, or multiple decoding settings for inference makes the gain even weaker.

  6. The conclusion that "useful scaling emerges during RL rather than SFT" is not supported: Table 2 shows that the proposed model RL-trained DJ-7B does have a positive STTS trend while the baselines do not, and the STTS trend evaluation of the SFT-only DJ-7B is missing. Section 4.5 relies on multiple checkpoints results in Figures 15–18 but the r-value could also be affected by accuracy or length bias etc. Moreover, the appendix contradicts the conclusion: models trained with GRPO show degrading STTS performance (Appendix F.2), and the RL data mixture changes the STTS trend (Appendix F.3).

  7. The analysis section is complicated, but not well explained, and uses inappropriate comparisons. The oracle is GPT-4o but in the main result table 1, GPT 4o is not even the best one over these baselines. In the figure 2 right, the length bias is clear. The length controlled ablation is required to attribute the difference to semantic alignment. Figure 2 and 3 are both a bit reluctant since the variances are just too high to just claim anything by looking at mean.

  8. The theory in section 5.3 is confusing (appendix G.1). Theorem 1 is not necessary and only stating the assumption not proving. And the section 5 is still comparing between DJ model and baseline, trying to prove the better performance but the claim in section 5 is about scaling or not.

Questions for authors

solve above questions.

Reviewer_cGETOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether additional inference-time reasoning improves LLM-as-a-Judge decisions. Using Simple Test-Time Scaling (STTS) as a diagnostic probe on four pairwise preference benchmarks (RewardBench, RewardMATH, Anthropic Harmless and CodePrefBench), the authors find that some existing open-source judges show flat or negative scaling. They further present DJ-7B, a judge model built from Qwen2.5-7B with SFT and RLVR, which reaches superior accuracy and relative gain under STTS. Checkpoint analyses suggest this scaling behavior strengthens over RL training. Further analyses characterize successful versus failed scaling and support risk-controlled selective prediction.

Critical strengths
  • The work investigates a timely and practically important question: do longer judge rationales actually lead to better decisions? Its diagnostic framing usefully complements the many method-focused papers on reasoning judges.

  • The empirical evaluation is broad, covering four domains as well as proprietary, open-source and scalar reward model baselines. The resulting DJ-7B is itself a strong 7B judge, and linking test-time scaling to calibration and selective prediction adds a valuable deployment-oriented perspective.

Critical weaknesses
  • The main findings appear to largely mirror what is already established for LRMs in problem solving (e.g., that scaling emerges via RL [1]. It remains unclear what is specific to the evaluation setting. Further discussion of the properties unique to LLM evaluators would help readers understand the paper’s distinct contribution.

  • Currently only sequential scaling (STTS) is studied, despite the general framing around “inference-time compute in LLM evaluation.” There appears to be no empirical comparison with parallel scaling such as majority voting/self-consistency (to the best of my understanding, Appendix A only compares diversity properties and some explanations as to why they focus on sequential scaling). At the least, some forms of compute-matched comparison would be very informative and provide a more comprehensive picture of TTS in LLM evaluations.

  • The negative scaling of Skywork-Critic and RISE-Judge baseline seems somewhat expected – they are not by design LRMs trained with long CoT. Reasoning-native judges would make a more fair comparison, such as JudgeLRM (in Table 1 but not Table 2) or RM-R1 [8].

--

[1] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

[2] Chen, Xiusi, et al. "Rm-r1: Reward modeling as reasoning." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  • Somewhat related to W1: What fundamentally prevents practitioners from using reasoning models that already scale well as judges (since evaluation can simply be framed as problem solving)? Evaluating such models under STTS, for example, a same-size one like R1-Distill-Qwen-7B, might be an informative baseline relevant to the question.
Reviewer_JGNSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper asks whether forcing an LLM judge to keep reasoning improves its pairwise preference decisions.
  • Existing open judges show flat or negative scaling. The authors train DJ-7B and shows positive scaling that emerges mainly during RL.
  • They analyze semantic alignment to an oracle, entropy, and calibration, and apply risk-controlled selective prediction.
Critical strengths
  • Whether extra reasoning helps judges is practically important, and the simple Wait probe makes the comparison easy to reproduce.
  • Checkpoint and SFT-vs-RL comparisons give a plausible picture that useful scaling emerges during RL.
Critical weaknesses

-Scaling evidence is weak. Absolute gains are only 0.3–1.3, with no seeds or confidence intervals. Comparisons omit other RL-trained judges and compute-matched voting, confounding RL with model choice.

  • Mechanistic and calibration claims are weakly supported. Analyses use one benchmark and a general-model baseline; the GPT-4o underperforms DJ-7B. Entropy interpretations are unconvincing, confidence is undefined, and cross-model ECE comparisons do not support the STTS calibration benefits.
  • Theorem 1 largely restates its assumption, Proposition 1 conflates token-level entropy with H(Z)
  • Limited evaluation and novelty: with no contamination checks, and closely related test-time scaling work is absent.
Questions for authors
  • How do RL-trained reasoning judges (JudgeLRM, RM-R1) and majority voting at the same token budget behave under the same probe?
  • How is the confidence score computed? Does DJ-7B's ECE change with and without STTS?
  • Why is GPT-4o a suitable oracle when it underperforms DJ-7B?
  • Evaluate on RewardBench 2 or JudgeBench and check training–test overlap?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.