- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Models often switch their answers (even when they are correct) when they are subject to criticism. The paper describes a strategy to train models to teach models to flip when their answers are incorrect, and not flip when their answer is correct.
Critical strengths
- The paper is well written and easy to follow.
- The proposed approach improves over compared baselines.
Critical weaknesses
- I am not clear about the practical significance of the experimental setup where critic feedback is randomly chosen during training. (Please refer Q2 and Q3 for more clarity)
Questions for authors
- For the results reported in Table 1. could the authors clarify if during training and evaluation the critic feedback is randomly chosen among the three possibilities (opposing , neutral, and supportive?)
- How sensitive are the results to the chosen values of the transition reward scores?
- If the critic feedback is chosen randomly from three options, it provides zero information to the policy about whether its answer is correct, right? I'm a bit confused about why studying such a setup is useful? In practice, wouldn't critic feedback contain more context that the model could use to update its initial answer?
- Could the authors clarify how their rollout strategy differs from Jiang et al. (2026)?
14 comments
Response (2/3): Sensitivity of transition rewards
Q3: Sensitivity to the transition-reward scores.
Response. The scores affect the correction/preservation trade-off, and the existing Boundary sweep shows material sensitivity. Figure 4 reports:
| Boundary penalty | Correction (%) | Harmful switching (%) | Net Gain (pp) |
|---|---|---|---|
| 0 | 3.48 | 4.12 | -0.64 |
| 0.1 | 5.09 | 4.85 | +0.24 |
| 0.2 | 2.17 | 6.81 | -4.64 |
| 0.4 | 1.88 | 7.86 | -5.98 |
Net Gain is computed from the reported correction minus harmful-switch fractions. The selected 0.1 gives the best net Gain among the tested values; higher penalties substantially degrade the outcome. It does not lower every risk relative to zero penalty. We will describe this as calibration of a trade-off, rather than claim all reward directions are insensitive.
The Boundary term distinguishes persistent failure from abandoning a correct answer. A weak penalty limits pressure on trajectories that remain beyond the model's capability; a stronger penalty can encourage more harmful changes. The sweep supports the observed trade-off, while that mechanism is an interpretation rather than an independently isolated causal finding.
Other available weight checks are auxiliary. Historical Robustness-only comparisons change its weight from 0.6 to 0.3 in a two-turn setting and from 0.6 to 0.4 in a three-turn setting. Over the final 20 common logged steps, the maximal four-transition-fraction shifts are approximately 0.70 and 0.60 pp. Those runs use an older 1,980-example collection and are training diagnostics, not current held-out sensitivity results.
The additional Initial-reward check retains a large final benefit in one current setting. On 4B EarthSE, adding Initial-accuracy reward 0.1 gives 57.20% final accuracy versus default ReCrit 58.00% and Base 37.60%; Initial becomes 40.40% versus Base 39.20%. This is a single objective perturbation, not a sweep of the four original weights, and the Initial-accuracy reward differs from turn-loss weighting.
The formal sensitivity answer rests on Figure 4. The other checks support only their stated local observations. We will report the tested values and limits explicitly in the analysis and appendix; unswept reward directions remain a limitation.
Response (3/3): Relation to existing rollout strategies
Q4: Rollout design relative to Jiang et al. (2026).
Response. We adopt dependency-ready dispatch rather than claim that principle as new. FlexMARL [1] already starts downstream requests when upstream dependencies are ready. Immediate Initial-to-Critic dispatch follows that principle; we will expand the mechanism comparison in Section 3.6 and related work.
The implementation differs in where asynchrony occurs. FlexMARL overlaps micro-batch gradient computation with unfinished rollout and commits updates after the full batch. ReCrit alternates rollout and training; its asynchrony mainly advances turns within rollout. Its task-specific tail rule uses a completion threshold and fills missing Critic outputs with Initial answers, creating synthetic keep transitions labeled Robustness or Boundary.
The threshold is not a rule to forcibly abort exactly 25% of requests. All Initial responses must complete, and in-flight requests finish their current turn. We will align Appendix Algorithm 2 with this behavior. Synthetic keep is an imputation, not an observed successful reflection or a guaranteed unbiased replacement for completed interaction.
Table 3 reports the internal throughput/quality trade-off:
| ReCrit completion setting | Time per iteration (s) | Critic accuracy (%) |
|---|---|---|
| No completion threshold | 204 | 56.42 |
| 75% completion | 191 | 55.59 |
| 60% completion | 182 | 53.59 |
These results support a configurable trade-off within ReCrit. They do not show superiority to FlexMARL, and the cross-algorithm GRPO-to-ReCrit accuracy difference cannot be attributed to scheduling alone. Rollout is a supporting engineering component; the primary reward evidence is the matched pre-finalizer comparison described in Part 1.
We will also distinguish this tail rule from AsyncFlow's streaming RL pipeline [2] and RollPacker's rescheduling of long prompts [3]. This clarification preserves the practical engineering contribution while crediting the established scheduling principles and reporting the cost of synthetic keep transitions.
We hope these responses clarify the task's practical meaning and the bounds of the evidence. If they address your concerns, we would be grateful if you could consider raising your score. We welcome further discussion and thank you again for your time.
References
- [1] Rollout-Training Co-Design for Efficient LLM-Based Multi-Agent Reinforcement Learning (FlexMARL).
- [2] AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training.
- [3] RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training.
Response (1/3): Random feedback and the purpose of the task
Dear Reviewer 716c,
Thank you for your careful review. We appreciate your recognition of the paper's clear presentation and improvements over the compared baselines. Your questions help clarify the role of feedback, the reward calibration and the rollout mechanism.
Response. Both training and the main Table 1 evaluation randomly choose opposing, neutral or supportive feedback. Main evaluation is not fixed-neutral. We will make this explicit in Section 3.2 and the evaluation protocol. This specifies the selection rule, without assuming uniform probabilities or paired per-question feedback and generation seeds that have not been reported.
Two additional EarthSE evaluations give net Gain two questions above and one question below the original result. With 250 questions, these correspond to +0.80 and -0.40 percentage points (pp), a 1.20-pp range across the three observations. They provide limited evaluation-repeat evidence, not repeated-training robustness or an error bar for final accuracy. The checkpoint and random factors of these repeats are not fully specified, so we do not use them as the main evidence of reliability.
Response. The attitudes provide pressure to reconsider, not evidence of correctness. The task asks whether a model can re-examine its answer without treating objection as a signal to change or praise as a signal to retain it. A correct solution should survive unsupported objection, while an incorrect solution should remain open to correction despite approval. This is an interaction-stability problem, complementary to using evidence-bearing expert corrections.
The supplementary 9B comparison uses 30 questions from each benchmark, or 90 per condition. C and W denote correct and wrong:
Correctness-changing fractions are 14.44%, 8.89%, 27.78% and 15.56%, respectively. Opposition has more observed correction and harmful switching. These are descriptive comparisons: Initial correct counts differ, so they are not fixed-Initial causal estimates. They motivate covering multiple pressures, without proving mixed-attitude training superior or necessary. We will replace "forces content-based reasoning" with this design intention and retain the scope limitation.
The existing reward comparison supports training within this limited task. Table 2's final-solution and transition-aware reward rows use the same initialization, data, two-turn feedback structure, training budget, loss coverage and evaluation interface, with token weights 1/1 and no finalizer; only the reward differs. Critic accuracy is 45.03% versus 50.12%. This isolates the reward replacement in the reported setting, not the benefit of mixed versus single-attitude training.