- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether automated paper reviewers retain agreement with human ratings after domain-specific reinforcement learning using synthetic review scores. Starting from a shared Qwen3.5-9B reviewer fine-tuned on human reviews, the authors perform domain adaptation and evaluate changes in rating error, signed bias, and output concentration. Two matched training runs improve agreement with synthetic domain labels while worsening agreement with held-out human ratings. An offline evaluation of six domain checkpoints finds increased human-referenced rating error in every case, with predictions shifting upward or becoming nearly constant.
Critical strengths
-
The paper examines a relevant concern for adapting automated evaluators: improvements against a new supervision source need not preserve agreement with an earlier human reference.
-
The paired offline comparisons are appropriate for measuring changes on the same evaluation papers. Reporting signed bias and output concentration alongside MAE helps distinguish upward shifts from nearly constant predictions.
-
The authors acknowledge important limitations, including the already concentrated SFT baseline, unmatched training conditions for four checkpoints, and the distinction between uncertainty across evaluation papers and variability across training runs. The manuscript generally avoids claiming that the experiments establish a causal mechanism.
Critical weaknesses
-
The contribution remains limited beyond documenting degradation under shifted supervision. The synthetic training sources have substantially higher mean ratings than the human training source: 6.42–7.38 versus 5.21. Training against these labels and subsequently observing higher predictions on the original reference distribution is therefore not particularly surprising. Score collapse and sensitivity to rating imbalance are also closely related to existing observations. For example, ReviewRL explicitly discusses mid-range rating collapse and uses score balancing to address it. Although the present setting differs, the paper needs stronger evidence of what this setting teaches us beyond those existing concerns.
-
An empirical paper need not propose a new method, but it should establish an informative pattern, boundary condition, or explanation. Here, the conclusions are narrowed so extensively that the remaining contribution is largely descriptive.
-
The experimental design does not distinguish the most plausible explanations. Adaptation changes the document distribution, label source, and score distribution simultaneously. There is no matched continued-SFT baseline on the same synthetic data, nor a control that changes the labels while holding the papers fixed. Consequently, the results do not establish whether the observed behavior is especially associated with RL, synthetic supervision, domain shift, or their combination.
-
I appreciate that the authors explicitly acknowledge this limitation. My concern is that distinguishing at least some of these explanations is central to making the study informative. A particularly useful control would compare human and generated labels on the same training papers, or compare continued SFT and GRPO using identical synthetic examples.
The starting reviewer is already highly concentrated, and the evaluation lacks simple baselines that would clarify what is being retained. The SFT reviewer assigns a rating of 6 to 82.4% of parsed papers. It is therefore unclear how much paper-specific judgment the baseline possesses before adaptation. Likewise, lower active-domain MAE could reflect learning the synthetic score distribution rather than improving discrimination between papers.
Constant predictors based on training-label medians should be included on both the human-reference and active-domain evaluations. Rank correlation or pairwise agreement would also help determine whether the less concentrated checkpoints preserve useful distinctions between papers. These measurements are especially relevant because the paper repeatedly distinguishes a scale shift from a loss of discriminative information, but does not directly evaluate the latter.
The reward-versus-rating discrepancy is insufficiently analyzed. Equation 4 combines errors across four score fields and penalizes incomplete outputs. Aggregate reward can therefore improve while Rating MAE worsens simply because formatting or another score component improves. The Finance and Physics results demonstrate this possibility, but the paper does not decompose the observed reward changes.
Reporting each score component and full-output validity, together with comparisons on a shared set of fully valid outputs, would reveal whether the discrepancy reflects a substantive tradeoff between evaluation dimensions or primarily a formatting effect. Without this decomposition, the aggregate-reward observation provides limited additional insight.
Several reporting omissions make the results difficult to reproduce and reconcile.
Section 4.2 describes the annotation generator only as a vision-capable OpenAI-compatible endpoint. The model identity, version, annotation prompt, and generation settings are important because the generated scores are central to the study.
The human-reference validation set used in Table 2 is not adequately specified. Its initial MAEs are approximately 1.30, whereas the offline SFT MAE is 0.838. The initial values also differ between the two runs despite their shared initialization. These differences may have legitimate explanations, but the evaluation subsets and generation protocols should be stated explicitly.
Table 3 reports each policy’s parse count rather than the actual paired denominator used for its confidence interval. The paired counts should be provided directly.
The training settings and checkpoint-selection details for the four supporting checkpoints are insufficiently documented to assess their evidential value.
The empirical scope is too narrow to establish a robust finding. The study uses one model, one training seed per domain, and the first 200 papers from one test split. Only two domain runs follow the matched protocol. The bootstrap intervals support degradation for the evaluated checkpoints on the sampled papers, but do not establish reproducibility across training runs. A larger, clearly sampled evaluation and repeated runs for the two central experiments would strengthen the contribution considerably.
Questions for authors
Do the active-domain improvements exceed those of a constant predictor using the domain training-label median?
How does continued SFT on the same synthetic data compare with GRPO?
What explains the different initial human-reference MAEs in Table 2 and their difference from the offline SFT result?
For CS and Math, does the upward shift preserve ranking agreement, and how much error can a simple offset correction remove when fitted on a separate validation set?