- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigates the problem of selecting which dataset to use for post-training, arguing that it depends on the dataset, but also on the state of training (e.g., optimizer state). To this end, it proposes CTV, which trains the model from a checkpoint on the old reference data, as well as the new proposed dataset. If performance goes up with the new dataset (on some held-out dataset), this one is accepted. Experiments show that CTV reports some small improvements, result-wise, for Qwen3-4B and Qwen3-8B trained with LoRA.
Critical strengths
- Interesting question that indeed impacts model training processes: given that training state impacts the selection of the next training dataset, how to best select that dataset? This is an important empirical question faced everyday by many researchers/professionals.
Critical weaknesses
- The paper is really hard to read. E.g., "so the price of a shared verdict is a quantity" [Abstract, line 23]. It is not clear what this means; "The measurement confirms itself before it reports" [Abstract, line 19]; "The cost is the continuations themselves, which the scores CTV replaces never train." [Section 1, lines 78–79]; "measurement has utility −0.50 points relative to the incumbent: a smaller observed loss than the compared scores, with positive one-sided lower bounds against the judge and LESS-style proxy but bounds including zero against execution correctness, Data-DPO, and random selection." [Section 6.1, lines 347–350]; "CTV measures a candidate dataset as an effect at its receiving state." [Section 7, line 452]; ..
- The proposal solution states that CTV "continues training once on the candidate dataset and once on the incumbent" [lines 73–74] and keeps the candidate if held-out accuracy goes up [line 252]; this is standard ML (empirical) approach to decide if a given selection is correct. This could still be a contribution if validated by results, but results seem unreliable given that the main findings are not reproduced when the test dataset is resampled: "Database-cluster resampling recomputes the policies on resampled questions and gives +0.05 points in [−0.15, +0.21]" [Section 6.1, lines 351–353]; this is also stated in Appendix C.2: “The positive gain is thus unresolved at the coarsest available level”.
- The paper stresses that one important variable to consider to select the data is optimizer state, yet the main experiments do not vary it; indeed, the experiments focus on varying the models, not the optimizers: “eight producer families at Qwen3-4B and Qwen3-8B” [Section 6, lines 313–314].
Questions for authors
- (Question) What is the difference between CTV and training a model on different dataset candidates, then checking the results on a held-out dataset?
- (Suggestion) Would it be possible to rewrite all the difficult to read passages in the text, providing an easier reading experience?
- (Suggestion) Can you add experiments where you test the same model with two different optimizer states?
4 comments
Reply to Reviewer tzTc (2/2): resampling of the evaluation databases, and two optimizer states at the same weights
W2.2.
The gain "is not reproduced when the test dataset is resampled".
Answer: Direct CTV's utility stays positive over resamples of the decision views and of the training configurations, and over the 11 evaluation databases it exceeds admit-all and seven of the eight admitting scores and proxies on the same draws. Each database resample draws the 11 databases of the 192 question pairs with replacement and recomputes every policy's admission decisions from scratch, so every policy is compared on identical draws. Against keeping the incumbent, the database-level interval is [−0.15, +0.21], positive in 73.6% of the draws.
| Resampling unit (20,000 draws) | Direct CTV utility, 95% interval (pp) | Admit-all, scores and proxies with Direct CTV minus policy above zero |
|---|---|---|
| 144 decision views | [+0.09, +0.27] | 9 of 9 |
| 12 training configurations | [+0.07, +0.28] | 9 of 9 |
| 11 evaluation databases | [−0.15, +0.21] | 8 of 9 |
Relative to admitting every candidate, the paired database-level difference is +1.83 points with 95% interval [+1.34, +2.27]. The LESS validation gradient is the one admitting policy whose paired interval includes zero over databases, and the revised Appendix C reports every paired interval.
W3 & Q3.
The paper stresses the optimizer state but does not test the same model with two optimizer states.
Answer: The paper evaluates identical weights under two optimizer states in a controlled intervention where resetting the AdamW second moment reverses the data-order effect, and the revision adds an admission-level test on a Qwen3-4B state carrying optimizer history. In the controlled intervention, Qwen3-1.7B continues from identical step-8 weights across two data blocks in both orders under carried or reset AdamW moments, evaluated on 192 validation questions.
| Same weights at step 8 | Order effect (pp) |
|---|---|
| Both AdamW moments carried | −1.30 |
| First moment reset | +0.91 |
| Second moment reset | +3.91 |
| Both moments reset | +2.21 |
| Second-moment component, carried minus reset, averaged over the first-moment treatment | −3.26, simultaneous 95% interval [−5.09, −1.42] |
In the admission-level test, a Qwen3-4B receiving state with 64 prior AdamW steps runs paired continuations on five candidate families and the incumbent, comparing restored AdamW moments and step counter against a fresh optimizer. Across updater seeds 1 to 7 (84 continuations in all), resetting the moments and the step counter shifts the validation effects of the five families by −1.88 to −0.26 points, every simultaneous 95% max- interval covers zero, and the stateless repair loop family changes sign. When each arm admits on the selection questions of six seeds and evaluates on the validation questions of the seventh, both arms admit all 35 cross-fitted candidates (15 harmful) for +0.74 points per decision. Both arms make the same decision in all 35 cases, so the registered equivalence criterion for utility holds. At this warm state the optimizer history moves candidate effects within updater noise and leaves admission unchanged, and the revision states this as the scope of the warm-state evidence. Revised Section 4 reports the step-8 intervention, and Section 5 reports the admission-level test.
If these replies resolve your concerns about readability, novelty and the optimizer state, we would be grateful if you would consider raising your score.
Thank you again for your review and for the concrete list of passages. We're happy to address any further questions you may have.
I thank the authors for their replies. I still think the paper has several significant flaws, well described in the 3 reviews. I agree with reviewer cY8M that fixing them would require changes beyond what is possible in a rebuttal phase. I keep my current score.
Thank you for the follow-up
Thank you for taking the time to reply. We rewrote the paper during the rebuttal phase, passage by passage from the list you gave us, since that's what we understood the rebuttal phase to be for, and we're honestly a little sad that the rewrite reads as work left undone. Every result stands as measured, and none of the three reviews points to an error in a proof, a measurement or a comparison, so we'd hoped the clearer text would let the findings speak for themselves. We're sorry the work didn't land with you, and we're grateful for your time and for your careful list of passages.
Reply to Reviewer tzTc (1/2): readability, and what the measurement adds to trial training
Thank you for your review and for the concrete list of passages, which guided our rewrite of the paper. Here are our replies.
W1 & Q2.
Answer: We rewrote the paper from the abstract to the conclusion for a reader outside the area, replacing every passage the review quotes with a plain statement.
Section 3 now defines the input, output, and objective through a running example on BIRD before introducing any method. Each formal result opens with a plain statement of what it establishes and closes with its practical implication. Section 6 opens with the setting, the metrics, and the compared policies and states each finding in bold before its numbers.
W2.1 & Q1.
Answer: CTV is that trial measurement made at the saved receiving state, and the paper establishes which conditions make it reliable and proves that no cheaper dataset score can replace it in general. CTV restores the saved weights, AdamW moments, and step counter, runs paired continuations on the candidate and the incumbent at equal tokens and steps under a shared updater seed, decides admission on the selection split, and evaluates realized utility on a disjoint validation split that no rule reads. Each of these conditions changes the outcome on the frozen benchmark. The same trial run on a Qwen3-1.7B proxy student realizes −0.52 points per decision, a one-step trial on a micro model realizes −0.38, and the trial at the receiving model with a single updater seed realizes +0.04, against +0.17 with four paired seeds. Dataset-selection methods such as LESS, Data-DPO, trace judges, and influence scores exist to avoid this training cost. Theorem 1 proves that a single dataset score can replace measurement at the receiving state if and only if receiving state and dataset act additively on validation accuracy. Corollary 1 lower-bounds the regret of every state-blind score when a dataset helps one state and harms another.
The frozen benchmark holds 144 decision views (12 training configurations, two split directions, and six evaluation criteria), and admitting the candidate is harmful in 101 of them. Only measurement at the receiving state reaches positive realized utility there.
In both admission studies, the receiving state starts from base weights with fresh AdamW moments, so CTV and seed-paired trial training run the same continuations there, as the revised Section 5 states. The two differ when the receiving state carries optimizer history, which the experiments in our reply to W3 measure.