- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the data-value bottleneck in LLM self-improvement: the training value of verified data depends on the current checkpoint, training method, and configuration. Through experiments with SFT, RFT, and GRPO on several Qwen models, the authors show that the same data can improve performance, have little effect, or degrade other capabilities, while the diagnostics tested before training cannot reliably predict these outcomes. Motivated by the observation that short pilot runs are more useful for flagging harmful updates than for ranking candidates, the authors propose a veto-and-commit policy: reject potentially harmful candidates, proceed only with credible evidence of positive utility, and otherwise stop.
Critical strengths
- The paper studies the data-value bottleneck and asks whether the value of a candidate update can be assessed before committing to a full training run. This is a meaningful and timely question for LLM self-improvement.
- The paper provides detailed empirical evidence showing how the training value of data varies across checkpoints, training methods, and configurations.
Critical weaknesses
- My main concern is the effectiveness of the proposed veto-and-commit policy. The claim that it never selected an update with negative final utility provides limited evidence of effective selection, since Table 5 shows that the primary policy chooses STOP in three of the four prospective tests. Its advantage over the 25-step point argmax baseline is also unclear. At 2B, it stops while achieves a utility of +5.0; at 4B, both select the same update, and at 27B, both avoid the destructive update. This weakens the methodological contribution.
- The definition of in Section 2, is central to the proposed policy. However, its parameters and are fixed at 1 and 2 pp in the experiments. The robustness of the policy’s decisions and reported effectiveness to these parameter choices is insufficiently evaluated.
Questions for authors
- Could you address the concerns raised in the Weaknesses section?
2 comments
Thank you for the responses and additional experiments. I appreciate the authors' efforts to address my concerns.
Weakness 1: I appreciate the additional comparisons with baseline methods. However, the results do not demonstrate a clear advantage of the proposed veto-and-commit policy. In particular, it appears to underperform a simpler early-stopping baseline. Therefore, my original concern regarding the methodological contribution remains.
Weakness 2: The sensitivity analyses adequately address my concern. Although some failures occur in the 4B setting under certain parameter choices, I consider this weakness largely resolved.
Overall, while I appreciate the additional experiments and clarifications, my concern regarding Weakness 1 remains. I therefore maintain my original rating.
Thank you for the clear and fair review. We ran both analyses you asked for.
1. Effectiveness, and the advantage over P. Four prospective tests are few, so we replayed both policies on all 579 subsets of the candidate sets, using the same records:
P trusts the 25-step ranking, which §5 shows is unreliable on fresh evaluation sets. Its +5.0 at 2B and its −1.4 at 1.5B come from the same behavior. We do not claim that V is the best selector: other simple rules that can stop are also safe (new Table 4). What the evidence supports is that an early negative read can be trusted and a ranking cannot.
2. Robustness to λ and δ. App. I re-runs everything over λ ∈ {0.5, 1, 2, 5}, δ ∈ {0, 1, 2, 3}, and a hard constraint. V's decisions at 2B, 1.5B, 27B and in the 9B replay never become negative. The exception is the 4B commit: its loss is on MMLU-Pro, which no decision set measured. Adding an MMLU-Pro decision set removes 8 of these 9 negative cases (details in our reply to gxtC, Q1).