acceptodds
Under review as a conference paper at ICLR 2027

Learning What Is Worth Training: The Data-Value Bottleneck in LLM Self-Improvement

Abstract

As high-quality human data runs short, reasoning models are increasingly improved by training on their own verified solutions. These self-improvement loops usually stall after a few rounds, and the usual remedy is to generate more, or harder, verified data. We argue that the harder problem is deciding what to train on. The same verified data can help one model, do nothing for another, and damage abilities that should be kept, depending on the checkpoint it is applied to and on how it is used. We call this the data-value bottleneck and ask whether the value of a candidate update can be known before paying for a full training run. In a pre-registered study of mathematical reasoning models from 1.5B to 27B parameters, we find that training value depends on both the current checkpoint and the training method. At a 9B checkpoint, a set of 322 problems had little effect when their solutions were used for supervised fine-tuning, but moved reinforcement learning one round ahead when the problems were used as prompts. The same teacher-generated solutions improved 4B and 9B checkpoints but reduced performance on protected tasks at 1.5B. None of the diagnostics we computed before training reliably distinguished useful updates from harmful or ineffective ones. Short pilot runs were more informative about harm. Every candidate whose estimated value was negative after full training already had a negative estimated value after 25 optimizer steps, on the evaluation sets used to make the decision. The converse did not hold: some useful candidates also looked negative early, and pilots did not reliably rank candidates with similar final performance. We therefore propose a conservative veto-and-commit policy: reject candidates with evidence of harm, select a candidate only when its estimated value is credibly positive, and otherwise stop. In four prospective tests, the policy never selected an update with negative value on a sealed evaluation, one that is read only once, after the decision is committed. This caution had a measurable cost: at a 2B checkpoint, the policy stopped even though the best candidate later produced a 5-point gain. It also prevented a much larger failure: at a near-ceiling 27B checkpoint, it rejected a teacher update that reduced sealed target accuracy by as much as 39 points after full training. We release the policy, evaluation protocol, code, pre-registration records and per-example results.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM post-training, reasoning models, training data selection, learning progress, GRPO, supervised fine-tuning, catastrophic forgetting, model state

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HN15OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the data-value bottleneck in LLM self-improvement: the training value of verified data depends on the current checkpoint, training method, and configuration. Through experiments with SFT, RFT, and GRPO on several Qwen models, the authors show that the same data can improve performance, have little effect, or degrade other capabilities, while the diagnostics tested before training cannot reliably predict these outcomes. Motivated by the observation that short pilot runs are more useful for flagging harmful updates than for ranking candidates, the authors propose a veto-and-commit policy: reject potentially harmful candidates, proceed only with credible evidence of positive utility, and otherwise stop.

Critical strengths
  • The paper studies the data-value bottleneck and asks whether the value of a candidate update can be assessed before committing to a full training run. This is a meaningful and timely question for LLM self-improvement.
  • The paper provides detailed empirical evidence showing how the training value of data varies across checkpoints, training methods, and configurations.
Critical weaknesses
  • My main concern is the effectiveness of the proposed veto-and-commit policy. The claim that it never selected an update with negative final utility provides limited evidence of effective selection, since Table 5 shows that the primary policy chooses STOP in three of the four prospective tests. Its advantage over the 25-step point argmax baseline PP is also unclear. At 2B, it stops while PP achieves a utility of +5.0; at 4B, both select the same update, and at 27B, both avoid the destructive update. This weakens the methodological contribution.
  • The definition of UU in Section 2, is central to the proposed policy. However, its parameters λ\lambda and δ\delta are fixed at 1 and 2 pp in the experiments. The robustness of the policy’s decisions and reported effectiveness to these parameter choices is insufficiently evaluated.
Questions for authors
  • Could you address the concerns raised in the Weaknesses section?
2 comments

Thank you for the clear and fair review. We ran both analyses you asked for.

1. Effectiveness, and the advantage over P. Four prospective tests are few, so we replayed both policies on all 579 subsets of the candidate sets, using the same records:

negative commitsworst sealed U
V0 / 5790.0
P13 / 579−78.8

P trusts the 25-step ranking, which §5 shows is unreliable on fresh evaluation sets. Its +5.0 at 2B and its −1.4 at 1.5B come from the same behavior. We do not claim that V is the best selector: other simple rules that can stop are also safe (new Table 4). What the evidence supports is that an early negative read can be trusted and a ranking cannot.

2. Robustness to λ and δ. App. I re-runs everything over λ ∈ {0.5, 1, 2, 5}, δ ∈ {0, 1, 2, 3}, and a hard constraint. V's decisions at 2B, 1.5B, 27B and in the 9B replay never become negative. The exception is the 4B commit: its loss is on MMLU-Pro, which no decision set measured. Adding an MMLU-Pro decision set removes 8 of these 9 negative cases (details in our reply to gxtC, Q1).

Thank you for the responses and additional experiments. I appreciate the authors' efforts to address my concerns.

Weakness 1: I appreciate the additional comparisons with baseline methods. However, the results do not demonstrate a clear advantage of the proposed veto-and-commit policy. In particular, it appears to underperform a simpler early-stopping baseline. Therefore, my original concern regarding the methodological contribution remains.

Weakness 2: The sensitivity analyses adequately address my concern. Although some failures occur in the 4B setting under certain parameter choices, I consider this weakness largely resolved.

Overall, while I appreciate the additional experiments and clarifications, my concern regarding Weakness 1 remains. I therefore maintain my original rating.

Reviewer_5GgrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes how the value of training data depends on the model checkpoint and training method, and proposes using short pilot runs to decide whether to proceed with an update.

Critical strengths

None

Critical weaknesses
  1. The proposed method is trivial: it runs a short training pilot to decide whether to proceed with the full run.

  2. The paper is poorly written and difficult to follow.

  3. The main finding that data value depends on the model checkpoint and training method is not surprising. It is unclear what new insight the analysis provides beyond this observation.

  4. The paper describes when updates help or hurt but provides limited explanation of why. It remains unclear how the findings can guide the design of better training data or methods beyond testing each candidate.

Questions for authors

See weaknesses.

Please explain why "Every prospective claim was frozen with a SHA-256 hash before the runs it predicts, and every miss is reported." Is this sentence fully produced by AI?

1 comment

Thank you for your review. We are sorry the paper was hard to follow, and we have rewritten the parts that caused the most trouble. Below we first state in plain terms what problem the paper addresses and why it matters, and then answer each point.

The problem. A common way to improve an LLM is self-improvement: the model generates solutions, a checker keeps the correct ones, and the model is trained on them. This is repeated over many rounds. In each round, someone must decide what to train on next and with which method. Today this choice is usually made by intuition or a fixed recipe. A full training run is expensive, and a bad choice can do more than waste compute: it can quietly damage skills the model already had.

The paper asks a practical question: before paying for a full training run, can we tell whether an update is worth training?

What we found.

  • The value of the same data is not fixed. Data that helps one model can be useless or harmful for a slightly different one. The checks we computed before training (for example, how many problems sit at the edge of the model's ability, or how similar the new data is to what the model must keep) did not tell these cases apart.
  • A short trial run is reliable in one direction only. Every update that ended up harmful already looked harmful after a short trial, so dropping such candidates early is safe. But a short trial could not reliably tell which of two promising updates is better.
  • This gives a simple, cheap rule: try each candidate briefly, drop any that look harmful, train fully only when there is clear evidence of a gain, and otherwise make no update.

Where it is useful. Any team or automated pipeline that updates a model round after round with its own generated data, for example to improve a math or coding model, needs a check that stops the loop from making the model worse. The rule is such a check. It needs no new model or extra data, and it costs about half of training every candidate. Its one requirement, which the revision states clearly, is that the skills to be protected must be measured during the check.

Your points.

  1. The method is simple. We agree, and we see this as a strength: it is easy to adopt. The contribution is the evidence behind it, namely which signal from a short trial can be trusted (an early harmful result, as a reason to drop a candidate) and which cannot (an early ranking). The revision adds a direct comparison with other simple rules: the rules that act on early harmful results stayed safe, the rules that trust rankings did not.
  2. The finding is not surprising. That data value depends on the model is expected in general. What is new is how strong and how unpredictable the effect is: it can flip from helping to hurting between nearby models, and the usual checks before training miss it. In practice, this means a data or method choice cannot be made once and reused; it must be tested cheaply at each round, and the paper shows how.
  3. Why, and how this guides design. The paper explains part of the why. Short trials cannot rank candidates because the differences between candidates are smaller than the measurement noise. When our rule accepted an update that hurt another skill, that skill had not been measured during the check. The practical guidance: test each candidate on the current model, measure every skill you want to protect, and treat an early harmful result as a veto.
  4. Writing. §2 now defines the key terms in plain words, Fig. 4 is larger, and the statements in §5 are corrected.

The SHA-256 sentence. The sentence describes a procedure we followed for every experiment in the paper. Its purpose is to rule out a common problem in empirical work: adjusting the hypothesis after seeing the results.

  • Before each experiment, we wrote the protocol and our predictions into a text file.
  • We computed the file's SHA-256 hash, a fingerprint that changes if a single character of the file changes, and recorded it with a timestamp in a ledger. This happened before any training for that experiment started.
  • After the experiment, we scored every prediction against the frozen file and reported every outcome. About 40% of our predictions failed, and the paper lists each failure with its diagnosis (Appendix J).

The ledger and all protocol files are in the supplement (protocols/FREEZE_LEDGER.json). Anyone can recompute the hashes and check that the predictions were fixed before the results existed. The analysis we added in this revision followed the same procedure.

On AI use: as stated in the paper's "Use of generative AI" section, AI tools helped with planning, code and writing, including drafting text. The procedure in this sentence was actually carried out, and every number in the paper comes from executed code and the released per-item records. The authors checked every statement. We have rewritten the sentence in plain words.

Reviewer_gxtCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the benefit of a self-improvement update can be predicted before completing training. It varies the model checkpoint, training data, and update procedure across Qwen models. The same verified data can help one model and hurt another. Several inexpensive data statistics fail to predict these outcomes. Short training runs identify harmful updates more reliably than they rank useful ones. The paper uses this observation to build a veto-and-commit policy, which either continues a candidate update or stops. It also documents evaluation and implementation choices that can reverse apparent training gains.

Critical strengths

The paper asks a useful question that is often obscured by reporting only the final training result. The experiments distinguish the quality of a solution from its usefulness for a particular checkpoint and update procedure. The examples where verified teacher data helps larger models but damages the smaller model make this distinction concrete. I also appreciate the negative results for difficulty, entropy, and gradient-based predictors. The authors generally keep those conclusions within the settings they actually tested.

The prospective evaluation is a strong part of the work. Decisions are made before reading the sealed evaluation, and the paper reports missed opportunities as well as successful vetoes. The evaluation audit is also informative. Changes in prompting, adapter handling, and batching can be comparable to the claimed training gains. These observations should be useful to researchers running similar self-improvement experiments.

Critical weaknesses

The evidence for the decision policy is much narrower than the evidence for the diagnosis. The primary policy stops in three of the four prospective tests. It misses a +5.0 update at 2B and accepts only the 4B update. That update has positive utility but loses 7.2 points on MMLU-Pro. The utility explicitly permits this tradeoff, so the result does not establish preservation of the protected tasks. The paper acknowledges its small action set and limited prospective evidence. Still, I would like to see how the decisions change when preservation is enforced as a constraint or given a larger penalty. This would clarify what practical objective the policy can reliably serve.

The computational argument also needs clarification. Figure 4 says that pilots cost only additional readouts because they are prefixes of full runs. That accounting assumes the candidate full runs would be performed anyway. A deployed selector must also pay for the prefixes of candidates it eventually discards. Please report the total selection cost, including those candidates and validation decoding. A comparison with a simple early-stopping or successive-halving rule under the same budget would help establish the value of the proposed decision rule. The related-work section should discuss Non-stochastic Best Arm Identification and Hyperparameter Optimization. It directly addresses selecting training runs from partial learning curves, although the preservation objective here is different.

The presentation could be more direct. Terms such as “surface,” “wall,” and “contract” accumulate quickly, and Figure 4 uses very small text for several important details. There is also a concrete inconsistency in Section 5: “No policy committed an update with negative sealed utility” conflicts with the reported −1.4 outcome for the point-argmax policy at 1.5B. The statement should distinguish the primary policy from the comparison policies.

Questions for authors
  1. How do the prospective decisions change under a hard preservation constraint or a stronger penalty on protected-task losses? Does the 4B update still qualify for commitment?
  2. What is the total cost of choosing and executing an update, including discarded pilots? Under that budget, how does the policy compare with validation-based early stopping or successive halving?
  3. Can you correct the statement about all policies avoiding negative sealed utility and clarify whether “destructive” has a separate numerical definition?
1 comment

Thank you for the careful and constructive review. Your three questions shaped most of the revision.

Q1. Under a hard preservation constraint, does the 4B update still qualify? No, and thank you for raising it. The update lost 7.2 points on the sealed MMLU-Pro set. None of our decision sets measured MMLU-Pro, so V could not see the loss. Its sealed utility is negative for λ ≥ 2, and it violates a hard constraint.

To check that the cause is the missing domain and not the rule, we added a 500-item MMLU-Pro decision set (disjoint from the sealed set) and re-ran the 4B decision. Sealed utility of V's commit, δ = 2:

beforewith MMLU-Pro decision set
λ = 14B update, +2.9same, +2.9
λ = 24B update, −2.3teacher code, +2.1
λ = 54B update, −17.9teacher code, +2.1
hard4B update, violatesteacher code, +2.1

Over the full λ/δ grid, 8 of the 9 negative cases disappear. One small case remains (λ = 2, δ = 3, −0.3), because the MMLU-Pro loss was still small at 25 steps (−1.4) and only reached −7 by the end. The sealed results were already known, so App. I reports this as a replay, not a prospective test, together with the predictions we wrote down before it, including one that failed.

The revision states this limitation directly: a decision set must include every domain one wants to protect. At the other four states, V never commits a negative update under any setting (App. I, Table 9).

Q2. Total cost, and comparison with early stopping / successive halving. You are right that the old caption ignored discarded pilots. Counting them, V uses 45–52% of the training steps of running every candidate to the end. Replaying simple rules on all 579 subsets of our candidate sets:

rulenegative commitsworstcost
V (ours)00.00.52
successive halving6−78.80.50
successive halving + final check00.00.50
early stopping at a negative read00.00.36
  • Rules that can stop and that drop candidates on early negative reads are safe. Rules that cannot stop commit the −78.8 update at 27B when it is the best candidate left.
  • V is not the cheapest safe rule. We now present the result as evidence about which pilot signal to trust (an early negative read), not as V being the best selector.

Related work now covers best-arm identification, Hyperband and learning-curve extrapolation.

Q3. "No policy committed …" and the definition of "destructive". Corrected: V committed no negative update, and P did once (−1.4 at 1.5B). "Destructive" now means that the 95% upper bound of the sealed utility is below zero; by that definition, no policy made a destructive commit.

Comments on the paper

1 comment

We thank the reviewers for their careful reading and for questions that pointed at the weakest parts of the paper. Below we restate each main question, then say what we did and what we found. All changes are in blue in the revised PDF.

1. What does V really cost? The old Fig. 4 caption counted only the steps of the chosen run and ignored the short pilot runs of the candidates that were dropped. We now count every pilot. In total, V uses 45–52% of the training steps needed to train every candidate to the end.

2. Does V still avoid harmful updates when the utility parameters change? The utility has two parameters: λ (how strongly a loss on a protected domain is penalized) and δ (how large a loss is tolerated). We re-ran every decision for λ ∈ {0.5, 1, 2, 5} and δ ∈ {0, 1, 2, 3}, and also under a hard constraint (any loss beyond δ is unacceptable). At four of the five states, V never commits a harmful update. The exception is 4B: V commits an update that loses 7.2 points on the sealed MMLU-Pro set, and this update is harmful in 9 of the 20 settings, all with λ ≥ 2 or the hard constraint.

3. Why did V miss the 4B loss, and can it be fixed? V only sees the decision sets it is given, and none of them included MMLU-Pro. To test whether this is the cause, we built a new 500-question MMLU-Pro decision set, with no overlap with the sealed set, and re-ran the 4B decision. This needed only evaluations of saved checkpoints, no new training. With MMLU-Pro included, V switches to a safe update in 8 of the 9 settings. The paper now states the lesson directly: the decision sets must cover every domain that should be protected.

4. Is V better than simple budgeted rules such as early stopping or successive halving? We replayed several simple rules on all 579 subsets of our candidate sets (new Table 4). Rules that can stop, and that drop a candidate as soon as its early result is negative, never chose a harmful update. Rules that always commit their best-ranked candidate sometimes did (worst: −78.8 points). V is safe, but it is not the cheapest safe rule, so we no longer present it as the best selector. What the evidence supports is a simpler point: an early negative result can be trusted, while an early ranking cannot.

How this strengthens the paper.

  • The main finding now rests on more than four tests. Across all 579 candidate subsets, the same pattern holds: acting on an early negative result is safe, trusting an early ranking is not.
  • The results do not depend on our choice of λ and δ. At four of the five states, V stays safe in every setting.
  • The one failure has a known cause and a tested fix. The 4B loss came from a domain that was not measured; measuring it removes 8 of the 9 harmful cases. This turns a weakness into concrete guidance: measure every domain you want to protect.
  • The cost is now fully counted. V costs about half of training every candidate, so the trade-off is clear.

What we corrected.

  • §5 said "no policy committed a negative update". It now says that V did not, and that the point rule P did once (−1.4 at 1.5B).
  • "Destructive" is now defined: the 95% upper bound of the sealed utility is below zero.
  • Key terms are defined in §2, Fig. 4 is larger, and the sentence about pre-registration hashes is rewritten in plain words.
  • Related work now covers best-arm identification, Hyperband and learning-curve extrapolation.

Points 1, 2 and 4 reuse the per-item records already in the supplement, so they need no new runs. The revised supplement adds the scripts and the new MMLU-Pro records. Full tables are in Appendix I.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.