- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether LLM judges benefit from test-time scaling, and under which conditions this benefit emerges. The authors apply probe judges with simple test-time scaling (STTS), which replaces the end token of thinking steps </think> with reflective tokens to make model continue generation. The paper also introduces a training recipe for a judge model, DJ-7B including generating reasoning traces with DeepSeek-R1, filtered by rejection sampling for cold-start SFT, followed by RL training. Experiments on four benchmarks show that DJ-7B outperforms several open-source judge models, and that DJ-7B exhibits a positive trend of test time scaling while two baseline models do not. The paper further examines how the scaling benefit across RL training stage, and analyses semantic alignment, entropy dynamics, uncertainty, and risk-controlled selective prediction.
Critical strengths
- The problem of whether LLM judges benefit from additional inference-time reasoning is really important, given the wide use of LLM-as-a-judge and test-time scaling.
- The training recipe is clearly described, and the main comparison (Table 1) covers diverfised models on four benchmarks.
- The paper provides analysis of semantic alignment, uncertainty and risk-controlloed prediction.
Critical weaknesses
-
Paper structure: The paper is not well-structured and consistent. The introduction part states a lot of background and motivation and "When does additional inference-time reasoningimprove an LLM judge?", while in Section 3 with the title "Simple Test-Time Scaling for LLM-as-a-Judge", mainly describes the training recipe for DJ-7B about how to fine-tune a judge model, with simple test-time scaling only covered in subsection 3.4. In terms of the three stated contributions, the instantiation of DJ-7B leads the main experimental results which is fine. But for "Empirical Characterization of Judge Scaling", the result is presented on Table 2, which includes only two baseline judges. The "Training Dynamics" conclusion that "useful scaling emerges primarily during RL" is supported by a short paragraph in subection 4.5 whose results (Figures 15–18) appear only in the appendix and are barely explained.
-
The training method is not new: Rejection-sampled data for cold-start SFT followed by RL with verifiable rewards is now a standard recipe for finetuning a specific task, and it has already been applied to judge models such as the citation in this paper JudgeLRM, DeepSeek-GRM. DeepSeek-GRM explicitly studies inference-time scaling for generative reward models, which weakens the contribution and contradicts the motivation in introduction part "test-time scaling techniques have primarily been studied for problem solving rather than evaluation". The training recipe occupying the main method section and the main results is not new.
-
Baseline selection is a bit confusing: Since the paper chooses to use GPT-4o, o1-mini, then why not just using recent models like GPT-5 or mini? similar to those RL trained judge models.
-
STTS is not explained sufficiently: Section 3.4 describes the approach of replacing </think> with a reflective tokens and continuing generation from the prefix, but missed decoding settings. In addition, this intervention assumes an explicit thinking format so how about judge models are not trained with CoT format? and those Proprietary models? and breaking their format may also cause artifacts of intervention.
-
The STTS results are not convincing: 5.1Only two baseline judges are evaluated. Given that STTS only requires prompting with prefix, it would be cheap to evaluate on more models, at least on DeepSeek-R1 which you use to generated the training data.
5.2 The gains are actually small (even with the confusing "delta relative%" metrics amplifying the absolute number): at Attempt 4, the improvement is roughly +1.3pp, +0.5pp, +0.4pp, and +0.3pp on four benchmarks, which is a bit trivial. And no multiple seeds, or multiple decoding settings for inference makes the gain even weaker. -
The conclusion that "useful scaling emerges during RL rather than SFT" is not supported: Table 2 shows that the proposed model RL-trained DJ-7B does have a positive STTS trend while the baselines do not, and the STTS trend evaluation of the SFT-only DJ-7B is missing. Section 4.5 relies on multiple checkpoints results in Figures 15–18 but the r-value could also be affected by accuracy or length bias etc. Moreover, the appendix contradicts the conclusion: models trained with GRPO show degrading STTS performance (Appendix F.2), and the RL data mixture changes the STTS trend (Appendix F.3).
-
The analysis section is complicated, but not well explained, and uses inappropriate comparisons. The oracle is GPT-4o but in the main result table 1, GPT 4o is not even the best one over these baselines. In the figure 2 right, the length bias is clear. The length controlled ablation is required to attribute the difference to semantic alignment. Figure 2 and 3 are both a bit reluctant since the variances are just too high to just claim anything by looking at mean.
-
The theory in section 5.3 is confusing (appendix G.1). Theorem 1 is not necessary and only stating the assumption not proving. And the section 5 is still comparing between DJ model and baseline, trying to prove the better performance but the claim in section 5 is about scaling or not.
Questions for authors
solve above questions.