acceptodds
Under review as a conference paper at ICLR 2027

Rating-Scale Drift in Automated Paper Reviewers after Domain-Specific Reinforcement Learning

Abstract

Domain-specific post-training can improve an automated reviewer's agreement with its training labels, but does it preserve agreement with human ratings? We study this question by adapting a Qwen3.5-9B reviewer, initially fine-tuned on human reviews, with reinforcement learning against generated domain scores. In two source-matched Computer Science and Mathematics runs, active-domain Rating error decreases while human-referenced validation error increases by 43.3% and 29.3%, respectively. A separate offline evaluation compares these checkpoints and four additional, unmatched domain checkpoints with the same SFT baseline on jointly parsed human-rated papers. All six checkpoints have higher paired Rating error than SFT, accompanied by upward shifts or near-constant predictions. Some checkpoints also improve aggregate rule reward despite worse Rating agreement. The SFT baseline is itself concentrated, so these comparisons measure changes from a fixed, imperfect reviewer. We characterize these observations as rating-scale drift and report agreement, signed bias, and output concentration separately. The results document a separation between generated-domain progress and human-rating retention in the evaluated runs, without identifying its causal mechanism.

Primary area:
foundation or frontier models, including LLMs
Keywords:
automated peer review, large language model evaluation, reinforcement learning, domain adaptation, rating-scale drift

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_uETsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether automated paper reviewers retain agreement with human ratings after domain-specific reinforcement learning using synthetic review scores. Starting from a shared Qwen3.5-9B reviewer fine-tuned on human reviews, the authors perform domain adaptation and evaluate changes in rating error, signed bias, and output concentration. Two matched training runs improve agreement with synthetic domain labels while worsening agreement with held-out human ratings. An offline evaluation of six domain checkpoints finds increased human-referenced rating error in every case, with predictions shifting upward or becoming nearly constant.

Critical strengths
  • The paper examines a relevant concern for adapting automated evaluators: improvements against a new supervision source need not preserve agreement with an earlier human reference.

  • The paired offline comparisons are appropriate for measuring changes on the same evaluation papers. Reporting signed bias and output concentration alongside MAE helps distinguish upward shifts from nearly constant predictions.

  • The authors acknowledge important limitations, including the already concentrated SFT baseline, unmatched training conditions for four checkpoints, and the distinction between uncertainty across evaluation papers and variability across training runs. The manuscript generally avoids claiming that the experiments establish a causal mechanism.

Critical weaknesses
  • The contribution remains limited beyond documenting degradation under shifted supervision. The synthetic training sources have substantially higher mean ratings than the human training source: 6.42–7.38 versus 5.21. Training against these labels and subsequently observing higher predictions on the original reference distribution is therefore not particularly surprising. Score collapse and sensitivity to rating imbalance are also closely related to existing observations. For example, ReviewRL explicitly discusses mid-range rating collapse and uses score balancing to address it. Although the present setting differs, the paper needs stronger evidence of what this setting teaches us beyond those existing concerns.

  • An empirical paper need not propose a new method, but it should establish an informative pattern, boundary condition, or explanation. Here, the conclusions are narrowed so extensively that the remaining contribution is largely descriptive.

  • The experimental design does not distinguish the most plausible explanations. Adaptation changes the document distribution, label source, and score distribution simultaneously. There is no matched continued-SFT baseline on the same synthetic data, nor a control that changes the labels while holding the papers fixed. Consequently, the results do not establish whether the observed behavior is especially associated with RL, synthetic supervision, domain shift, or their combination.

  • I appreciate that the authors explicitly acknowledge this limitation. My concern is that distinguishing at least some of these explanations is central to making the study informative. A particularly useful control would compare human and generated labels on the same training papers, or compare continued SFT and GRPO using identical synthetic examples.

The starting reviewer is already highly concentrated, and the evaluation lacks simple baselines that would clarify what is being retained. The SFT reviewer assigns a rating of 6 to 82.4% of parsed papers. It is therefore unclear how much paper-specific judgment the baseline possesses before adaptation. Likewise, lower active-domain MAE could reflect learning the synthetic score distribution rather than improving discrimination between papers.

Constant predictors based on training-label medians should be included on both the human-reference and active-domain evaluations. Rank correlation or pairwise agreement would also help determine whether the less concentrated checkpoints preserve useful distinctions between papers. These measurements are especially relevant because the paper repeatedly distinguishes a scale shift from a loss of discriminative information, but does not directly evaluate the latter.

The reward-versus-rating discrepancy is insufficiently analyzed. Equation 4 combines errors across four score fields and penalizes incomplete outputs. Aggregate reward can therefore improve while Rating MAE worsens simply because formatting or another score component improves. The Finance and Physics results demonstrate this possibility, but the paper does not decompose the observed reward changes.

Reporting each score component and full-output validity, together with comparisons on a shared set of fully valid outputs, would reveal whether the discrepancy reflects a substantive tradeoff between evaluation dimensions or primarily a formatting effect. Without this decomposition, the aggregate-reward observation provides limited additional insight.

Several reporting omissions make the results difficult to reproduce and reconcile.

Section 4.2 describes the annotation generator only as a vision-capable OpenAI-compatible endpoint. The model identity, version, annotation prompt, and generation settings are important because the generated scores are central to the study.

The human-reference validation set used in Table 2 is not adequately specified. Its initial MAEs are approximately 1.30, whereas the offline SFT MAE is 0.838. The initial values also differ between the two runs despite their shared initialization. These differences may have legitimate explanations, but the evaluation subsets and generation protocols should be stated explicitly.

Table 3 reports each policy’s parse count rather than the actual paired denominator used for its confidence interval. The paired counts should be provided directly.

The training settings and checkpoint-selection details for the four supporting checkpoints are insufficiently documented to assess their evidential value.

The empirical scope is too narrow to establish a robust finding. The study uses one model, one training seed per domain, and the first 200 papers from one test split. Only two domain runs follow the matched protocol. The bootstrap intervals support degradation for the evaluated checkpoints on the sampled papers, but do not establish reproducibility across training runs. A larger, clearly sampled evaluation and repeated runs for the two central experiments would strengthen the contribution considerably.

Questions for authors

Do the active-domain improvements exceed those of a constant predictor using the domain training-label median?

How does continued SFT on the same synthetic data compare with GRPO?

What explains the different initial human-reference MAEs in Table 2 and their difference from the offline SFT result?

For CS and Math, does the upward shift preserve ranking agreement, and how much error can a simple offset correction remove when fitted on a separate validation set?

Reviewer_JdWkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether automated paper reviewers can maintain consistency with human ratings after domain-specific reinforcement learning (RL). The authors first performed SFT on Qwen3.5-9B using human review data, and then performed GRPO training using synthetic ratings from six domains. The results showed that CS and Math checkpoints improved in active-domain validation, but the Rating MAE on the human-reference set increased by 43.3% and 29.3%, respectively. All six domain checkpoints exhibited higher paired human-reference errors, accompanied by upward rating shifts or near-constant predictions. The authors termed this phenomenon “rating-scale drift” and suggested reporting Rating MAE, signed bias, and output concentration simultaneously.

Critical strengths

1.The research question is of practical significance.

The paper clearly distinguishes between active-domain progress and human-rating retention, avoiding the direct interpretation of improvements against synthetic labels as improvements in review quality.

2.The diagnostic analysis is quite convincing.

The authors also report Rating MAE, signed bias, and mode share, which can distinguish between upward rating shift and rating collapse, and reveal degradation that may be masked by aggregate rewards.

Critical weaknesses

1.Limited experimental coverage.

The paper studies only one base model, with only one training seed per domain, and only CS and Math belong to source-matched controlled runs. The other four checkpoints cannot be considered strictly matched replications, and key baselines such as continued SFT or human-provided domain labels are lacking.

2.The robustness of human-reference evaluation is insufficient.

Evaluation uses only the first 200 papers in the test set, and Table 3 does not report the exact intersection size for each paired comparison. Parsing failures at different checkpoints may lead to different evaluation subsets. Furthermore, the paper does not assess whether ranking ability is preserved.

Questions for authors

1.Could you provide multi-random seed results and training progress curves for CS and Math, along with the exact number of paired samples for each comparison, and sensitivity analysis on a fixed subset of papers that all models can successfully parse? These results would help confirm whether rating-scale drift is stable and not caused by a specific random seed, final checkpoint, or parsing failure.

2.Could you add matched controls, such as continued SFT, training with human-provided domain labels, or matching a synthetic rating distribution, to differentiate the impact of the RL algorithm, label distribution, and domain content on rating-scale drift?

Reviewer_DEvMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies rating-scale drift in automated paper reviewers after domain-specific reinforcement learning. Starting from a reviewer fine-tuned on human reviews, the authors apply GRPO using synthetic domain-specific reviews. They find that improved agreement with domain labels can come with worse agreement with human ratings, alongside upward score shifts or more concentrated predictions.

Critical strengths

A relevant and practical research question.

The paper asks a useful question: when we adapt an automated reviewer to a new domain, do its scores still mean the same thing? This matters in practice, since users may compare ratings across model updates without realizing that the underlying scale has shifted. I appreciate the paper’s focus on this issue.

Useful diagnostics beyond aggregate reward.

Evaluating the same papers against a fixed human reference is a sensible way to track changes in rating behavior. Signed bias and output concentration also help show what MAE alone can miss. Together, these diagnostics make the mismatch between reward improvement and human-rating agreement easier to see.

Critical weaknesses

The experiments show drift, but do little to explain it.

My main concern is that several things change at once: the supervision source, rating distribution, document domain, and optimization method. In particular, the synthetic domain ratings have much higher means (6.42–7.38) than the human training ratings (5.21), so an upward shift after adaptation is perhaps unsurprising.

I would have liked to see a few straightforward controls, especially continued SFT on the same domain data and GRPO with calibrated rating labels. These would help clarify whether RL contributes anything beyond adaptation to a different supervision distribution. The authors appropriately acknowledge that they do not establish a causal mechanism, but the contribution still feels mostly descriptive.

It is unclear how well the baseline distinguishes paper quality in the first place.

The SFT model assigns a rating of 6 to 82.4% of parsed papers. That is already a highly concentrated output distribution, and it makes me question how much meaningful rating discrimination the baseline has learned.

A comparison with simple constant-rating predictors would help put the MAE results in context. Ranking metrics, such as Spearman correlation or pairwise accuracy, would also show whether the models can distinguish stronger papers from weaker ones. Without these checks, I find it difficult to interpret further concentration after RL as a clear loss of evaluation capability.

Key details needed to reproduce and interpret the results are missing.

The synthetic reviews come from a “vision-capable OpenAI-compatible endpoint,” but the teacher model is not identified, and the prompting and generation setup are not described in enough detail. This is a significant gap because the synthetic ratings—and their distribution—are central to the paper’s findings.

The image-only PDF pipeline also needs a clearer description, including rendering resolution, effective visual-token usage, truncation, and how much of each paper actually reaches the model. The GPU model is unspecified as well. These details would make the experiments easier to reproduce and help readers assess whether the document-processing setup introduces additional confounds.

The robustness of the findings is not yet convincing.

The human-reference evaluation uses only the first 200 papers from a single test split, without random or stratified sampling. Each domain is trained with a single random seed, and four of the six checkpoints are not source-matched reruns. This leaves some uncertainty about whether the same pattern would hold across training runs and evaluation subsets.

The bootstrap confidence intervals are useful for assessing evaluation-sample uncertainty, but they do not tell us how stable the training outcomes are. I also could not find the exact paired sample sizes used for the reported MAE differences. Given these limitations, I would be more convinced by repeated runs and independently sampled reference subsets, along with a clearer explanation of how comparable the additional checkpoints are.

Questions for authors

Q1. How do continued SFT and GRPO compare when trained on the same domain data? Does the drift persist when the synthetic ratings are calibrated to the human rating distribution?

Q2. How does the SFT baseline compare with a constant-rating predictor? Could you also report Spearman correlation or pairwise accuracy to show whether the models meaningfully distinguish paper quality?

Q3. Which teacher model, prompts, and generation settings were used to produce the synthetic reviews? Please also clarify the PDF rendering resolution, image/token truncation strategy, and GPU configuration.

Q4. Do the main findings hold across multiple training seeds and independently sampled human-reference subsets? Please report the exact paired sample sizes and explain how the four additional checkpoints compare with the two controlled runs.

Reviewer_ygDbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether domain-specific reinforcement learning (RL) preserves the alignment between automated paper reviewers' numerical ratings and human review scores. Starting from a Qwen3.5-9B reviewer supervised fine-tuned on human reviews from DeepReview, the authors perform domain-specific RL using synthetically generated review scores across six scientific domains.

The main finding is that improvements in agreement with generated domain-specific labels do not necessarily translate into improved agreement with human ratings. In two controlled training runs (Computer Science and Mathematics), the human-referenced Rating MAE increases by 43.3% and 29.3%, respectively, despite improvements on the corresponding domain validation sets. Additional offline evaluations of six domain-specific checkpoints reveal increased human-referenced Rating MAE, accompanied by either upward shifts in predicted ratings or near-constant predictions.

The authors characterize these behaviors as rating-scale drift and advocate reporting human-referenced error, signed bias, and output concentration separately from aggregate reward.

Critical strengths

1. Relevant and practically important problem. The paper addresses an important reliability concern in automated scientific reviewing: whether numerical ratings remain comparable after domain-specific adaptation. This is particularly relevant when reviewer models are continuously updated using newly collected or synthetically generated training data.

2. Clear separation of adaptation performance and human-rating retention. The evaluation distinguishes agreement with generated domain labels from agreement with a fixed human reference. This distinction is meaningful because optimizing a domain-specific objective does not guarantee preservation of the original rating behavior.

3. Paired evaluation with uncertainty estimates. The authors compare SFT and RL checkpoints on jointly parsed examples and report paired bootstrap confidence intervals for changes in Rating MAE. This design reduces confounding due to differences in evaluated examples.

4. Useful characterization of different failure patterns. The distinction between upward rating shifts and output collapse is informative. The results demonstrate that models can exhibit different rating distributions while showing similar aggregate errors. The authors also appropriately acknowledge that the SFT baseline itself produces highly concentrated ratings.

5. Clear presentation and appropriately scoped conclusions. The paper is generally well organized, and its limitations are explicitly discussed. In particular, the authors avoid attributing the observed drift to a specific causal mechanism without sufficient evidence.

Critical weaknesses
  1. Limited novelty relative to existing literature. Prior work has already investigated rating collapse in RL-based paper reviewing (ReviewRL, Zeng et al., EMNLP 2025), numerical scoring bias after alignment (Sato et al., 2026), and the compatibility of fine-tuned LLM judges under distribution changes (Singh et al., ICLR 2026). While the specific setting of domain-specific RL with generated review scores is interesting, the paper mainly applies existing diagnostic metrics to a related failure phenomenon. It is unclear whether the findings provide sufficiently new insights beyond these studies.

  2. Insufficient controlled experiments to explain the observed drift. The generated domain ratings have substantially higher means than the human training ratings, suggesting that the observed drift may be driven by differences in label distributions. However, the paper does not include controlled label-distribution experiments or a continued-SFT baseline using the same domain data. Moreover, only two domains have matched training runs, with one training seed per domain. These limitations make it difficult to distinguish the effects of rating-prior shifts, domain adaptation, and RL optimization.

  3. Incomplete evaluation of rating discrimination. The analysis mainly relies on MAE, signed bias, and output concentration. However, the SFT baseline already assigns a rating of 6 to 82.4% of evaluated papers, and some RL checkpoints produce almost constant predictions. A model can achieve moderate MAE while having little ability to distinguish papers of different quality. Ranking-based metrics, such as Spearman correlation or pairwise ranking accuracy, would provide a more complete assessment of whether domain RL changes only the numerical scale or also degrades the reviewer's discriminative ability.

Questions for authors

1. Have the authors considered matching the synthetic rating distributions to the human rating distribution, or comparing domain RL against continued SFT on the same generated data? These controls would help clarify the source of the observed drift.

2. How does domain-specific RL affect the correlation or pairwise ranking agreement between predicted ratings and human ratings? Do the CS and Math checkpoints retain meaningful discrimination despite their upward rating shifts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.