acceptodds
Under review as a conference paper at ICLR 2027

ReCrit: Transition-Aware Reinforcement Learning for Scientific Critic Reasoning

Abstract

Large language models can fail in critic interaction not only by answering incorrectly, but also by abandoning an initially correct scientific solution after user criticism. We frame critic interaction as an inter-turn correctness-transition problem rather than a final-answer accuracy problem, and identify three challenges: transition awareness, decoupling useful correction from harmful sycophancy, and scalable rollout. We propose ReCrit, a transition-aware reinforcement learning framework that decomposes Initial-to-Critic behavior into four quadrants: Correction, Sycophancy, Robustness, and Boundary. ReCrit rewards correction and robustness, penalizes sycophancy, and treats persistent errors as weak boundary signals. To make interaction training practical, ReCrit further uses dynamic asynchronous rollout with tail-adaptive completion to reduce rollout waiting. On three scientific reasoning benchmarks, ChemBench, TRQA, and EarthSE, ReCrit improves average Critic accuracy from 38.15 to 51.49 on Qwen3.5-4B and from 45.40 to 55.59 on Qwen3.5-9B. Ablations show that final-answer rewards provide little interaction-level gain, while transition-aware rewards and quadrant weighting produce more distinguishable training signals and larger net Critic-stage improvement.

Primary area:
reinforcement learning
Keywords:
llm, critic, reasoning, science

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_716cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Models often switch their answers (even when they are correct) when they are subject to criticism. The paper describes a strategy to train models to teach models to flip when their answers are incorrect, and not flip when their answer is correct.

Critical strengths
  • The paper is well written and easy to follow.
  • The proposed approach improves over compared baselines.
Critical weaknesses
  • I am not clear about the practical significance of the experimental setup where critic feedback is randomly chosen during training. (Please refer Q2 and Q3 for more clarity)
Questions for authors
  • For the results reported in Table 1. could the authors clarify if during training and evaluation the critic feedback is randomly chosen among the three possibilities (opposing , neutral, and supportive?)
  • How sensitive are the results to the chosen values of the transition reward scores?
  • If the critic feedback is chosen randomly from three options, it provides zero information to the policy about whether its answer is correct, right? I'm a bit confused about why studying such a setup is useful? In practice, wouldn't critic feedback contain more context that the model could use to update its initial answer?
  • Could the authors clarify how their rollout strategy differs from Jiang et al. (2026)?
14 comments
AuthorsOpenReview ↗

Response (1/3): Random feedback and the purpose of the task

Dear Reviewer 716c,

Thank you for your careful review. We appreciate your recognition of the paper's clear presentation and improvements over the compared baselines. Your questions help clarify the role of feedback, the reward calibration and the rollout mechanism.

Q1: Feedback selection during training and evaluation.

Response. Both training and the main Table 1 evaluation randomly choose opposing, neutral or supportive feedback. Main evaluation is not fixed-neutral. We will make this explicit in Section 3.2 and the evaluation protocol. This specifies the selection rule, without assuming uniform probabilities or paired per-question feedback and generation seeds that have not been reported.

Two additional EarthSE evaluations give net Gain two questions above and one question below the original result. With 250 questions, these correspond to +0.80 and -0.40 percentage points (pp), a 1.20-pp range across the three observations. They provide limited evaluation-repeat evidence, not repeated-training robustness or an error bar for final accuracy. The checkpoint and random factors of these repeats are not fully specified, so we do not use them as the main evidence of reliability.

Q2: Practical meaning of feedback without correctness information.

Response. The attitudes provide pressure to reconsider, not evidence of correctness. The task asks whether a model can re-examine its answer without treating objection as a signal to change or praise as a signal to retain it. A correct solution should survive unsupported objection, while an incorrect solution should remain open to correction despite approval. This is an interaction-stability problem, complementary to using evidence-bearing expert corrections.

The supplementary 9B comparison uses 30 questions from each benchmark, or 90 per condition. C and W denote correct and wrong:

FeedbackC to CC to WW to CW to W
Neutral37.78%6.67%7.78%47.77%
Supportive40.00%4.44%4.44%51.12%
Opposing31.11%12.22%15.56%41.11%
No feedback38.89%7.78%7.78%45.55%

Correctness-changing fractions are 14.44%, 8.89%, 27.78% and 15.56%, respectively. Opposition has more observed correction and harmful switching. These are descriptive comparisons: Initial correct counts differ, so they are not fixed-Initial causal estimates. They motivate covering multiple pressures, without proving mixed-attitude training superior or necessary. We will replace "forces content-based reasoning" with this design intention and retain the scope limitation.

The existing reward comparison supports training within this limited task. Table 2's final-solution and transition-aware reward rows use the same initialization, data, two-turn feedback structure, training budget, loss coverage and evaluation interface, with token weights 1/1 and no finalizer; only the reward differs. Critic accuracy is 45.03% versus 50.12%. This isolates the reward replacement in the reported setting, not the benefit of mixed versus single-attitude training.

AuthorsOpenReview ↗

Response (2/3): Sensitivity of transition rewards

Q3: Sensitivity to the transition-reward scores.

Response. The scores affect the correction/preservation trade-off, and the existing Boundary sweep shows material sensitivity. Figure 4 reports:

Boundary penaltyCorrection (%)Harmful switching (%)Net Gain (pp)
03.484.12-0.64
0.15.094.85+0.24
0.22.176.81-4.64
0.41.887.86-5.98

Net Gain is computed from the reported correction minus harmful-switch fractions. The selected 0.1 gives the best net Gain among the tested values; higher penalties substantially degrade the outcome. It does not lower every risk relative to zero penalty. We will describe this as calibration of a trade-off, rather than claim all reward directions are insensitive.

The Boundary term distinguishes persistent failure from abandoning a correct answer. A weak penalty limits pressure on trajectories that remain beyond the model's capability; a stronger penalty can encourage more harmful changes. The sweep supports the observed trade-off, while that mechanism is an interpretation rather than an independently isolated causal finding.

Other available weight checks are auxiliary. Historical Robustness-only comparisons change its weight from 0.6 to 0.3 in a two-turn setting and from 0.6 to 0.4 in a three-turn setting. Over the final 20 common logged steps, the maximal four-transition-fraction shifts are approximately 0.70 and 0.60 pp. Those runs use an older 1,980-example collection and are training diagnostics, not current held-out sensitivity results.

The additional Initial-reward check retains a large final benefit in one current setting. On 4B EarthSE, adding Initial-accuracy reward 0.1 gives 57.20% final accuracy versus default ReCrit 58.00% and Base 37.60%; Initial becomes 40.40% versus Base 39.20%. This is a single objective perturbation, not a sweep of the four original weights, and the Initial-accuracy reward differs from turn-loss weighting.

The formal sensitivity answer rests on Figure 4. The other checks support only their stated local observations. We will report the tested values and limits explicitly in the analysis and appendix; unswept reward directions remain a limitation.

AuthorsOpenReview ↗

Response (3/3): Relation to existing rollout strategies

Q4: Rollout design relative to Jiang et al. (2026).

Response. We adopt dependency-ready dispatch rather than claim that principle as new. FlexMARL [1] already starts downstream requests when upstream dependencies are ready. Immediate Initial-to-Critic dispatch follows that principle; we will expand the mechanism comparison in Section 3.6 and related work.

The implementation differs in where asynchrony occurs. FlexMARL overlaps micro-batch gradient computation with unfinished rollout and commits updates after the full batch. ReCrit alternates rollout and training; its asynchrony mainly advances turns within rollout. Its task-specific tail rule uses a completion threshold and fills missing Critic outputs with Initial answers, creating synthetic keep transitions labeled Robustness or Boundary.

The threshold is not a rule to forcibly abort exactly 25% of requests. All Initial responses must complete, and in-flight requests finish their current turn. We will align Appendix Algorithm 2 with this behavior. Synthetic keep is an imputation, not an observed successful reflection or a guaranteed unbiased replacement for completed interaction.

Table 3 reports the internal throughput/quality trade-off:

ReCrit completion settingTime per iteration (s)Critic accuracy (%)
No completion threshold20456.42
75% completion19155.59
60% completion18253.59

These results support a configurable trade-off within ReCrit. They do not show superiority to FlexMARL, and the cross-algorithm GRPO-to-ReCrit accuracy difference cannot be attributed to scheduling alone. Rollout is a supporting engineering component; the primary reward evidence is the matched pre-finalizer comparison described in Part 1.

We will also distinguish this tail rule from AsyncFlow's streaming RL pipeline [2] and RollPacker's rescheduling of long prompts [3]. This clarification preserves the practical engineering contribution while crediting the established scheduling principles and reporting the cost of synthetic keep transitions.

We hope these responses clarify the task's practical meaning and the bounds of the evidence. If they address your concerns, we would be grateful if you could consider raising your score. We welcome further discussion and thank you again for your time.

References

  • [1] Rollout-Training Co-Design for Efficient LLM-Based Multi-Agent Reinforcement Learning (FlexMARL).
  • [2] AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training.
  • [3] RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training.
Reviewer_SF7kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReCrit rewards correctness transitions between an initial answer and a revision after critic feedback. It distinguishes correction, preservation, harmful switching, and persistent error, with greater emphasis on critic-stage updates. A separately supervised finalization model consolidates answers at inference. Experiments cover three scientific QA benchmarks and two model sizes.

Critical strengths
  1. The transition decomposition makes revision behavior easier to inspect than final accuracy alone. Table 2 shows a meaningful gain from transition rewards before adding finalization.
  2. Final accuracy improves across all three domains and both scales, largely through substantially more successful corrections.
Critical weaknesses
  1. Correct-to-wrong switching is labeled sycophancy without isolating the effect of feedback attitude. It could include ordinary reconsideration errors. These switches also increase against most baselines: on 4B ChemBench, 1.5% becomes 6.0%. The evidence supports a correction/degradation trade-off more directly than reduced sycophancy.
  2. The same-scale finalizer adds supervision and inference computation. Whether baselines receive it is unclear. Table 2 shows it raises harmful switching from 1.68% to 4.58%, so its effect cannot be treated as formatting alone.
  3. Appendix A/C omits training-set sizes and the exact critic-stage weights or token masks. These details are central to the reported ablation and reproducibility.
Questions for authors
  1. How do harmful-switch rates vary under opposing, neutral, and supportive feedback, relative to a feedback-free second attempt?
  2. Apply the same finalizer to all baselines and report inference cost; please also specify the critic-stage loss weights and masks.
5 comments
AuthorsOpenReview ↗

Response (1/3): Harmful switching and feedback attitude

Dear Reviewer SF7k,

Thank you for your detailed review. We appreciate your recognition that the transition decomposition makes revision behavior easier to inspect, that Table 2 shows a meaningful gain before finalization, and that final accuracy improves across the evaluated domains and scales. We address the behavioral interpretation, finalizer comparison and reproducibility details below.

Q1: Interpretation of correct-to-wrong transitions and the correction/degradation trade-off.

Response. Correct-to-wrong is a measured harmful transition, not proof of sycophantic motivation. We will use "harmful switching" as the measurement label and qualify the anti-sycophancy interpretation throughout the paper.

We agree with your 4B ChemBench observation. Harmful switching rises from 1.50% to 6.00%, while correction rises from 1.00% to 16.50%. ReCrit therefore has a net Initial-to-Critic Gain of +10.50 percentage points (pp), with final accuracy 61.00% versus Base 53.00%. This is improved net performance with a real degradation cost; a larger correction benefit does not make the harmful modifications disappear. The full-system result supports a trade-off rather than universal reduction of harmful switching.

Q2: Feedback-attitude results and the feedback-free second attempt.

Response. The supplementary 9B condition comparison shows different observed revision profiles across feedback attitudes. Both training and main evaluation randomly choose opposing, neutral or supportive feedback. For the condition comparison, each row contains 30 questions from each of the three benchmarks, totaling 90. C and W denote correct and wrong answers:

FeedbackC to CC to WW to CW to W
Neutral37.78%6.67%7.78%47.77%
Supportive40.00%4.44%4.44%51.12%
Opposing31.11%12.22%15.56%41.11%
No feedback38.89%7.78%7.78%45.55%

Opposing feedback has more observed harmful switching and correction. Harmful-switch fractions over all questions are 6.67%, 4.44%, 12.22% and 7.78%, respectively. Among initially correct answers, the corresponding fractions are 6/40 = 15.00%, 4/40 = 10.00%, 11/39 = 28.21% and 7/42 = 16.67%.

This table is descriptive rather than a fixed-Initial paired causal test. Initial correct counts differ, and the table does not compare ReCrit with Base under each attitude. It therefore does not establish a causal attitude effect or a training-induced reduction in sycophancy. It does motivate studying varied interaction pressure. We will describe mixed attitudes as intended to discourage a tone-only revision rule, replacing the stronger statement that they "force" content-based reasoning.

AuthorsOpenReview ↗

Response (2/3): Finalizer fairness, policy evidence and inference cost

Q3: Separating the reward-policy contribution from additional finalizer supervision.

Response. The primary reward comparison uses the same two-turn training structure and no finalizer. We clarify that the two Table 2 reward configurations share initialization, data, feedback structure, budget, loss coverage, Initial/Critic token weights 1/1, and evaluation interface. Only the reward changes:

Reward, same 9B two-turn trainingInitial (%)Critic (%)Correction (%)Harmful switching (%)
Final-solution reward44.8645.033.893.72
Transition-aware reward45.9150.127.543.33

Critic improves by 5.09 pp, Initial by 1.05 pp, correction by 3.65 pp and harmful switching by -0.39 pp. This comparison addresses the reward objective without extra finalizer training or inference. It is one reported matched configuration, not a multi-seed significance result; its matching conditions do not automatically extend to every later progressive row.

The later pre-finalizer 9B policy reaches 53.00% Critic accuracy, versus Base 45.40%, with harmful switching 1.68% versus 2.68%. For context, the strongest reported 9B policy-baseline mean is Critique-GRPO at 48.93%. These existing results support policy improvement; the 4.07-pp arithmetic margin is not an independently compute-matched estimate.

We have not completed the requested common-finalizer comparison for every baseline. The supplementary GRPO/EarthSE pilot changes transition counts by -1/+2/+1/-2 in C-to-C/C-to-W/W-to-C/W-to-W order. Final correct count is unchanged, but Initial correct count changes by one, so it is not a fixed-trace control and cannot establish the module's effect for all baselines. We retain that limitation.

We will make the matched pre-finalizer reward comparison the main evidence for the training objective. Table 1 will explicitly identify finalizer-assisted ReCrit as a system result, rather than an equal-supervision or equal-compute comparison. Its entire margin will not be attributed to the reward.

Q4: Finalizer-induced changes and inference cost.

Response. The same-scale supervised finalizer can reason and change substantive answers. It aggregates the two candidate answers and produces a structured final option; it is not pure format normalization. Table 2 reports:

ConfigurationCritic (%)Correction (%)Harmful switching (%)
Before finalization53.008.111.68
With finalization55.5913.544.58

We will report both the higher accuracy and increased harmful switching. Initial also differs slightly between these rows, so the differences are configuration comparisons, not a fixed-trace module effect.

The measured inference-time shares are Initial 33%, Critic 56% and finalizer 11%. Relative to the first two stages, finalization adds approximately 11/89 = 12.36% time. This is a within-pipeline breakdown; the proportions do not establish absolute latency, training cost or matched-budget superiority over baselines. Section 3.5 and the result captions will make these boundaries explicit.

AuthorsOpenReview ↗

Response (3/3): Training data, loss coefficients and reproducibility

Q5: Training-data size and stage-specific reporting.

Response. The training collection contains 3,354 examples. Appendix A provides the training-stage settings, and Appendix B/Table 12 describes synthesis and filtering. We will add the count explicitly and consolidate these details rather than imply the submitted appendix already states that number.

The total does not by itself determine each stage's allocation, data reuse or the DPO pair count. Those quantities remain incompletely documented and will not be inferred from 3,354. Format warm-up, transition RL and finalizer SFT will be distinguished in the configuration description.

Q6: Critic-stage coefficients, token masks and normalization.

Response. Initial and Critic model-output tokens are trained; user and template tokens are masked. The coefficients map to Table 2 as follows:

Policy configurationInitial token weightCritic token weight
Final-solution reward1.01.0
Transition-aware reward1.01.0
+ Quadrant weights1.01.0
+ Critic-stage weights0.31.0

For x -> y0 -> c -> y1, question x, user feedback c, template bridges and padding have zero weight. In the weighted configuration, all output tokens in y0 receive 0.3 and those in y1 receive 1.0. Each trajectory's weighted token loss is divided by 0.3 * len(y0) + len(y1) before batch averaging. This is token weighting, not a sum of independently averaged turn losses. The 0.3/1.0 override must be explicit because the code fallback is 1/1.

These coefficients are separate from quadrant rewards +1/-1/+0.6/-0.1. The additional Initial-accuracy reward 0.1 belongs only to the supplementary 4B EarthSE check, not to Table 2's reward replacement. The finalizer uses a separate supervised objective. We will add this mapping and normalization to Section 3.4 and Appendix A/C.

Q7: Consolidated optimization and sampling settings.

Response. We will consolidate the submitted settings and additional saved baseline parameters in the reproducibility section. Full-parameter training uses bfloat16, maximum training length 8192 and warmup ratio 0.05. The baseline settings are:

ParameterSFTDPOGRPO
Learning rate1e-65e-75e-7
Device batch / accumulation1 / 161 / 82 / 2
OptimizerFused AdamWFused AdamWFused AdamW
Adam beta1 / beta2 / epsilon0.9 / 0.95 / 1e-80.9 / 0.95 / 1e-80.9 / 0.95 / 1e-8
Weight decay / grad clip0.1 / 1.00.1 / 1.00.1 / 1.0
Seed / data seed42 / 4242 / 4242 / 42

DPO uses beta 0.1 and sigmoid loss. GRPO uses KL beta 0.04, clip epsilon 0.2, group size 4, temperature 1.0, top-p 0.9, top-k 50 and maximum completion 4096. These saved baseline settings are not substituted for ReCrit's separate trainer. ReCrit's reported critic temperature is 1.0, per-stage generation limit 4096 and completion ratio 0.75. Appendix A, Tables 9-11 separate the warm-up, RL and finalizer stages.

We hope these responses clarify the empirical contribution and its limits. If they address your concerns, we would be grateful if you could consider raising your score. We welcome further discussion and thank you for your constructive feedback.

Reviewer_6vBLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how large language models can respond more appropriately to user criticism. The authors propose ReCrit, a reinforcement learning method that assigns different rewards to four types of correctness transitions between the initial and post-feedback answers. The system also incorporates asynchronous rollout, tail-adaptive completion, and a final-answer finalization module. Across three scientific question-answering benchmarks and two model scales, the authors report improvements in average post-feedback (Critic) accuracy from 38.15% to 51.49% for Qwen3.5-4B and from 45.40% to 55.59% for Qwen3.5-9B.

Critical strengths
  • The paper addresses a valuable problem. Preserving correct answers when challenged while revising incorrect ones is an important capability.
  • The method improves performance over the reported baselines, and the paper includes comprehensive ablation studies.
  • The proposed asynchronous rollout is well suited to the training setting considered in the paper.
Critical weaknesses
  • The methodological novelty is limited, as similar reward designs have appeared in many reinforcement learning methods. Moreover, the paper lacks direct comparisons with closely related baselines, e.g., SCoRe.
  • The finalization module is a separately trained language model of the same scale as the base model. It takes the question and candidate answers from the interaction, performs an additional reasoning pass, and outputs an answer option. Because this process can change the answer itself, the final scores may not solely reflect whether the Critic policy has learned to revise answers appropriately.
  • The experiments do not clearly separate the benefit of training on the evaluation's two-turn interaction structure from the benefit of the proposed reward design. ReCrit is explicitly trained to generate an initial answer, receive criticism, and answer again, whereas baselines are not. Some gains may therefore reflect adaptation to this interaction structure.
Questions for authors
  • The introduction emphasizes the importance of correction for complex scientific problems, but the benchmarks used in the evaluation are relatively simple.
  • The method may, to some extent, incentivize the model to produce an incorrect initial answer. For Qwen3.5-4B, initial-answer accuracy decreases across all three benchmarks relative to the base model.
7 comments
AuthorsOpenReview ↗

Response (1/3): Reward attribution and finalization

Dear Reviewer 6vBL,

Thank you for your careful review. We appreciate your recognition of the value of preserving correct answers while correcting errors, the reported improvements and ablations, and the suitability of our rollout design. We first clarify the control that separates the reward contribution from two-turn training and additional finalization.

Q1: Contribution of the reward under the same two-turn training structure.

Response. Table 2 contains a reward-only comparison within the same two-turn training setting. The final-solution-reward and transition-aware-reward configurations use the same initialization, training data, feedback structure, training budget, loss coverage and evaluation interface. Both use Initial/Critic output-token weights 1/1, and neither uses the finalizer. Only the reward changes.

The reported 9B results are unweighted means over the three benchmarks:

Reward in the same two-turn settingInitial (%)Critic (%)Correction (%)Harmful switching (%)
Final-solution reward44.8645.033.893.72
Transition-aware reward45.9150.127.543.33

Replacing the reward improves Critic accuracy by 5.09 percentage points (pp), Initial by 1.05 pp and net Gain by 4.04 pp. The comparison therefore provides evidence beyond adapting to the two-turn structure: both configurations already train that structure. These are observed differences in the reported configuration, rather than a multi-seed significance claim. We will state the matching conditions explicitly in Section 4.3 and the Table 2 caption.

An additional EarthSE check using two-turn GRPO with reward only on the second answer answers four fewer questions correctly than the single-turn GRPO baseline: 4/250 = 1.60 pp. This illustrates that an extra trained turn need not help in that setting. The matched Table 2 comparison, rather than this auxiliary check, is our main evidence for the reward contribution.

Q2: Critic-policy performance and the finalization module.

Response. The critic policy improves before finalization. Table 2 reports the following 9B configurations:

ConfigurationInitial (%)Critic (%)Correction (%)Harmful switching (%)
Base45.2945.402.782.68
ReCrit policy, before finalization46.5753.008.111.68
ReCrit with finalization46.6355.5913.544.58

The pre-finalizer policy improves Critic accuracy by 7.60 pp and reduces harmful switching by 1.00 pp relative to Base. Thus the additional model cannot account for all policy improvement. The finalizer is nevertheless a same-scale, separately supervised answer aggregator: it combines candidate answers, reasons and can change the substantive answer. We agree that it is more than format normalization.

The full system has both a higher final score and more harmful switching. Since Initial also differs slightly between the two configurations, their differences are not a fixed-trace causal estimate of the module. We will distinguish policy outputs from finalizer-assisted system outputs in Section 3.5 and Tables 1-2. The matched reward rows establish the reward comparison; later progressive rows and complete-system margins have their separate evidence scope.

AuthorsOpenReview ↗

Response (2/3): Related work and scientific-task scope

Q3: Novelty relative to reward shaping and SCoRe.

Response. ReCrit builds on established reward shaping and self-correction RL. We agree that the reward-allocation idea alone is not new. SCoRe [1] is a close predecessor: it already uses progress rewards and a two-stage procedure with first-response constraints. We will cite it directly and expand the related-work comparison, rather than describe prior self-correction RL as rewarding only final answers.

The specific design difference is independent control over the four correctness transitions. Expanding SCoRe's second-response correctness-plus-progress term gives:

TransitionSCoRe term: S1 + alpha(S1 - S0)ReCrit default
Wrong to correct1 + alpha+1.0
Correct to wrong-alpha-1.0
Correct to correct1+0.6
Wrong to wrong0-0.1

This table compares that reward term, not SCoRe's entire objective. In ReCrit, the raw correct-versus-incorrect Critic reward gap is 1.1 for an initially wrong answer and 1.6 for an initially correct answer; both gaps are 1 + alpha in the displayed SCoRe term. ReCrit can therefore adjust recovery and preservation pressures separately. These are raw reward gaps, not gradient multipliers: group normalization and the remaining optimization also matter.

Our contribution is a concrete training recipe and empirical study of these trade-offs under supportive, neutral and opposing user feedback in scientific QA. The matched reward ablation in Part 1 supports usefulness in this setting. It does not establish superiority over SCoRe. We have not completed a matched SCoRe experiment and will acknowledge that limitation. We will also distinguish ReCrit's stage weighting from SCoRe's two-stage procedure, without claiming either a new RL principle or the first transition-dependent reward.

Q4: Scope and difficulty of the scientific benchmarks.

Response. The evaluation concerns closed-form scientific QA, rather than open-ended scientific discovery. The benchmark papers provide the following context:

BenchmarkDescription in the benchmark workScope of our evaluation
ChemBench [2]Chemical knowledge and reasoning, including reasoning and calculationClosed-form chemistry QA
TRQA [3]Target-research QA in a biomedical research settingMultiple-choice biomedical QA
EarthSE [4]Earth-science QA includes analysis and calculation, alongside separate exploration tasksMultiple-choice earth-science QA

These tasks involve domain knowledge and reasoning while allowing reliable correctness judgments before and after criticism. We agree that they do not cover the complexity of real scientific research. We will make that boundary explicit in the introduction and limitations, without attributing capabilities of other benchmark subsets to the subsets evaluated here.

References

  • [1] Training Language Models to Self-Correct via Reinforcement Learning.
  • [2] A Framework for Evaluating the Chemical Knowledge and Reasoning Abilities of Large Language Models against the Expertise of Chemists.
  • [3] OriGene: A Self-Evolving Virtual Disease Biologist Automating Therapeutic Target Discovery.
  • [4] EarthSE: A Benchmark Evaluating Earth Scientific Exploration Capability for Large Language Models.
AuthorsOpenReview ↗

Response (3/3): Initial-answer accuracy and reward incentives

Q5: Initial-answer degradation and the incentive to start incorrectly.

Response. We acknowledge the 4B Initial decrease and the potential incentive to obtain a correction reward by starting incorrectly. Higher post-feedback accuracy alone does not rule out that behavior. Our focus is improved self-critique, but both initial and post-feedback performance should be reported.

An additional 4B EarthSE check adds an Initial-accuracy reward of 0.1 to the ReCrit objective. The dataset contains 250 questions:

SettingInitial correctInitial (%)Final correctFinal (%)
Base9839.209437.60
Default ReCrit9738.8014558.00
ReCrit + Initial reward 0.110140.4014357.20

The adjusted setting answers three more Initial questions correctly than Base (+1.20 pp), while answering two fewer final questions correctly than default ReCrit (-0.80 pp). Its final accuracy remains 19.60 pp above Base. This demonstrates an adjustable trade-off in this setting, rather than showing that gaming has been eliminated or that the same cost holds across tasks.

The 9B evidence also shows that improvement need not require lowering Initial accuracy. In the matched reward comparison, Initial rises from 44.86% to 45.91%; in the later pre-finalizer configuration, it reaches 46.57% versus Base 45.29%. These observations do not cancel the 4B limitation, which we will retain explicitly.

The Initial reward and turn-loss coefficients are separate quantities. Table 2 uses output-token coefficients 1/1 before Critic-stage weighting and 0.3/1 after adding that component. User questions, feedback, template bridges and padding are masked. The additional Initial-accuracy reward of 0.1 belongs only to the 4B EarthSE check above; it is not an extra change between Table 2's two reward rows. The finalizer has its own supervised training.

We will clarify the reward controls in Section 4.3, expand the comparison with SCoRe, distinguish policy and system results, and report the first-answer trade-off in the limitations. These clarifications preserve the demonstrated post-feedback improvement while specifying what the results establish.

We hope these responses address your concerns. If so, we would be grateful if you could consider raising your score. We welcome further questions and thank you again for your careful review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary: Reviewer Discussion, Additional Evidence, and Manuscript Updates

Dear Area Chair and Reviewers,

Thank you for the reviews and discussion. Below are the main responses, PDF updates and latest exchanges.

  • Reviewer 6vBL recognized the problem's value, baseline improvements, ablations and rollout suitability. We addressed novelty relative to SCoRe, reward attribution versus two-turn training/finalization, task scope and Initial-answer trade-offs through direct comparisons, the matched reward control and qualified claims.
  • Reviewer SF7k recognized interpretable transitions, gains before finalization, and improvements across domains and scales. We clarified harmful switching as an observed transition rather than proof of sycophantic motivation, separated auxiliary inference from policy results, and specified available data, loss weights, masks and normalization.
  • Reviewer 716c recognized clear writing and baseline improvements. We explained random feedback as unsupported pressure, clarified reward sensitivity and prior rollout principles, and addressed shared initialization, SCoRe, finalization and quadrant variability.

Changes implemented in the revised PDF:

  1. Policy-only main results. All main-table methods are scored directly on the second answer; the finalizer is excluded from the main text and method. ReCrit's three-benchmark means are 50.80%/54.91% on 4B/9B, versus Base Model means of 38.15%/45.40%. The abstract, introduction, tables and conclusion use these results.
  2. Complete SCoRe comparison and reward attribution. Table 1 includes both scales and all three benchmarks, with shared SFT initialization, data, total training budget and feedback-evaluation protocol. SCoRe's means are 47.94%/52.98%; ReCrit is higher in all six settings. This compares full training recipes. Separately, Table 2's matched 9B two-turn reward-only pair improves accuracy from 45.03% to 50.12%; only that pair isolates reward replacement. Related work and the method acknowledge SCoRe's progress rewards and two-stage procedure.
  3. Training and rollout clarification. Warm-up is the common scientific-QA SFT stage. We document the 3,354-example collection, token weights/masks, normalization and synthetic keep loss. We credit prior dispatch principles and separate runtime observations from accuracy claims. Configuration limits remain explicit.
  4. Direct-policy evidence in Appendix G. We include the 9B Base attitude pilot, full 4B ReCrit mixed/opposing quadrant profiles, Initial-reward check, two-turn GRPO and scoped historical diagnostics. On the full 200/172/250-question sets, the largest mixed/opposing quadrant changes are 1.00/0.58/0.00 percentage points. These condition comparisons do not estimate across-run variance or statistical significance.
  5. Auxiliary interface and results in Appendix H. We corrected the original description of the finalizer as separately trained. It reuses the trained ReCrit checkpoint, with no extra SFT or independent dataset. Its prompt, possible answer changes, auxiliary results/repeats, GRPO pilot and inference cost are reported separately. These repeats do not estimate direct-policy quadrant variance.

Latest discussion and our responses:

  • 6vBL's November 18 follow-up. The reviewer wrote, "Thank you for the clarifications and additional experiments." They also stated that novelty and gain attribution were only partly addressed and maintained the original score. We value this acknowledgment and take the remaining concerns seriously. In our submitted follow-up, we explain that constant terminal-correctness groups yield zero relative advantage from that reward component; distinct transition scores can retain information when Initial states differ. Other rewards or KL can still produce updates. We present this as a plausible mechanism, without claiming measured group frequencies, gradient attribution or guaranteed learning efficiency. The observed 5.09-point gain comprises 1.05 points in Initial accuracy and 4.04 points in net revision gain.
  • 716c's November 19 follow-up. The reviewer previously appreciated direct second-answer reporting and now maintains their score. Their remaining concern is opposing-versus-supportive quadrant changes in trained ReCrit. Table 14 compares mixed/opposing feedback for 4B ReCrit; Table 13 concerns 9B Base. Neither provides the requested ReCrit comparison, which remains open. Appendix H.1 explicitly corrects the original separately trained finalizer description: it reuses ReCrit's checkpoint without extra training. Table 11 gives its inference settings; main results score second answers directly.

We have responded with explanations, evidence and PDF revisions while retaining unresolved limitations. Thank you for your continued time and scrutiny. We welcome further questions and ask that the revised evidence inform the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.