acceptodds
Under review as a conference paper at ICLR 2027

Data Value Depends on the Full Training State

Abstract

A post-training pipeline has to choose its next dataset before it can read any outcome, so it decides from a score computed ahead of the run. Every such score returns one verdict per dataset, read from the data alone or from the data together with the current parameters. We show that the usefulness of a dataset is a property of the data together with the state that reads it, and that this state carries more than the weights: it carries the optimizer memory and the step clock as well. Two states can therefore share a verdict on a dataset whose effects at those states have opposite signs, and the shared verdict is wrong at one of them. We call the effect at the receiving state causal trace value (CTV) and measure it directly, by restoring the saved state and continuing once on the candidate and once on the incumbent at equal tokens and steps, so that the paired difference in held-out accuracy credits the choice of data alone. The measurement confirms itself before it reports, replaying one continuation and returning no verdict when the replay disagrees. Any rule whose verdict two states share pays a summed regret of at least the smaller of the two effects on a dataset whose effects at those states have opposite signs, so the price of a shared verdict is a quantity. On a frozen benchmark of text-to-SQL decisions, measurement achieves positive mean validation utility over twelve training configurations at a matched admission count, admits a fraction of the candidates that every rival policy must match, and trains continuations whose price we report beside what they buy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data valuation, post-training, synthetic reasoning traces, causal effects, optimizer state

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_tzTcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the problem of selecting which dataset to use for post-training, arguing that it depends on the dataset, but also on the state of training (e.g., optimizer state). To this end, it proposes CTV, which trains the model from a checkpoint on the old reference data, as well as the new proposed dataset. If performance goes up with the new dataset (on some held-out dataset), this one is accepted. Experiments show that CTV reports some small improvements, result-wise, for Qwen3-4B and Qwen3-8B trained with LoRA.

Critical strengths
  1. Interesting question that indeed impacts model training processes: given that training state impacts the selection of the next training dataset, how to best select that dataset? This is an important empirical question faced everyday by many researchers/professionals.
Critical weaknesses
  1. The paper is really hard to read. E.g., "so the price of a shared verdict is a quantity" [Abstract, line 23]. It is not clear what this means; "The measurement confirms itself before it reports" [Abstract, line 19]; "The cost is the continuations themselves, which the scores CTV replaces never train." [Section 1, lines 78–79]; "measurement has utility −0.50 points relative to the incumbent: a smaller observed loss than the compared scores, with positive one-sided lower bounds against the judge and LESS-style proxy but bounds including zero against execution correctness, Data-DPO, and random selection." [Section 6.1, lines 347–350]; "CTV measures a candidate dataset as an effect at its receiving state." [Section 7, line 452]; ..
  2. The proposal solution states that CTV "continues training once on the candidate dataset and once on the incumbent" [lines 73–74] and keeps the candidate if held-out accuracy goes up [line 252]; this is standard ML (empirical) approach to decide if a given selection is correct. This could still be a contribution if validated by results, but results seem unreliable given that the main findings are not reproduced when the test dataset is resampled: "Database-cluster resampling recomputes the policies on resampled questions and gives +0.05 points in [−0.15, +0.21]" [Section 6.1, lines 351–353]; this is also stated in Appendix C.2: “The positive gain is thus unresolved at the coarsest available level”.
  3. The paper stresses that one important variable to consider to select the data is optimizer state, yet the main experiments do not vary it; indeed, the experiments focus on varying the models, not the optimizers: “eight producer families at Qwen3-4B and Qwen3-8B” [Section 6, lines 313–314].
Questions for authors
  1. (Question) What is the difference between CTV and training a model on different dataset candidates, then checking the results on a held-out dataset?
  2. (Suggestion) Would it be possible to rewrite all the difficult to read passages in the text, providing an easier reading experience?
  3. (Suggestion) Can you add experiments where you test the same model with two different optimizer states?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer tzTc (1/2): readability, and what the measurement adds to trial training

Thank you for your review and for the concrete list of passages, which guided our rewrite of the paper. Here are our replies.

W1 & Q2.

The paper is hard to read, e.g. "so the price of a shared verdict is a quantity".

Answer: We rewrote the paper from the abstract to the conclusion for a reader outside the area, replacing every passage the review quotes with a plain statement.

Submitted passageRevised paper
"so the price of a shared verdict is a quantity" (abstract)"any rule unable to distinguish two states where a dataset changes sign incurs regret at least equal to the smaller effect" (abstract)
"The measurement confirms itself before it reports" (abstract)A same-action replay repeats one continuation under the same seed to verify that the saved state is restored deterministically (Section 3)
"The cost is the continuations themselves, which the scores CTV replaces never train"Reading all 781 questions takes 96% of the runtime, against 10 minutes per continuation for the 179 selection questions (Section 6.3)
"measurement has utility −0.50 points relative to the incumbent: a smaller observed loss than the compared scores, with positive one-sided lower bounds …""every policy loses on VV, Direct CTV least (−0.50 points)", and in a separate sentence on the matched comparison, "that bound is positive against the trace judge and the LESS-style proxy and includes zero against execution correctness, Data-DPO and random selection" (Section 6.1)
"CTV measures a candidate dataset as an effect at its receiving state""Choosing the next post-training dataset depends on the full receiving state" (conclusion)

Section 3 now defines the input, output, and objective through a running example on BIRD before introducing any method. Each formal result opens with a plain statement of what it establishes and closes with its practical implication. Section 6 opens with the setting, the metrics, and the compared policies and states each finding in bold before its numbers.

W2.1 & Q1.

CTV "continues training once on the candidate dataset and once on the incumbent", which is the standard way to check a selection on held-out data.

Answer: CTV is that trial measurement made at the saved receiving state, and the paper establishes which conditions make it reliable and proves that no cheaper dataset score can replace it in general. CTV restores the saved weights, AdamW moments, and step counter, runs paired continuations on the candidate and the incumbent at equal tokens and steps under a shared updater seed, decides admission on the selection split, and evaluates realized utility on a disjoint validation split that no rule reads. Each of these conditions changes the outcome on the frozen benchmark. The same trial run on a Qwen3-1.7B proxy student realizes −0.52 points per decision, a one-step trial on a micro model realizes −0.38, and the trial at the receiving model with a single updater seed realizes +0.04, against +0.17 with four paired seeds. Dataset-selection methods such as LESS, Data-DPO, trace judges, and influence scores exist to avoid this training cost. Theorem 1 proves that a single dataset score can replace measurement at the receiving state if and only if receiving state and dataset act additively on validation accuracy. Corollary 1 lower-bounds the regret of every state-blind score when a dataset helps one state and harms another.

The frozen benchmark holds 144 decision views (12 training configurations, two split directions, and six evaluation criteria), and admitting the candidate is harmful in 101 of them. Only measurement at the receiving state reaches positive realized utility there.

Frozen benchmark, 144 decision viewsHarmful / admittedUtility per decision (pp)
Admit every candidate101/144−1.66
Teacher accuracy65/96−1.22
Trace judge39/48−1.24
LESS validation gradient41/72−0.05
Random selection at the same count29/42−0.48
Qwen3-1.7B proxy student37/54−0.52
Direct CTV11/42+0.17

In both admission studies, the receiving state starts from base weights with fresh AdamW moments, so CTV and seed-paired trial training run the same continuations there, as the revised Section 5 states. The two differ when the receiving state carries optimizer history, which the experiments in our reply to W3 measure.

AuthorsOpenReview ↗

Reply to Reviewer tzTc (2/2): resampling of the evaluation databases, and two optimizer states at the same weights

W2.2.

The gain "is not reproduced when the test dataset is resampled".

Answer: Direct CTV's utility stays positive over resamples of the decision views and of the training configurations, and over the 11 evaluation databases it exceeds admit-all and seven of the eight admitting scores and proxies on the same draws. Each database resample draws the 11 databases of the 192 question pairs with replacement and recomputes every policy's admission decisions from scratch, so every policy is compared on identical draws. Against keeping the incumbent, the database-level interval is [−0.15, +0.21], positive in 73.6% of the draws.

Resampling unit (20,000 draws)Direct CTV utility, 95% interval (pp)Admit-all, scores and proxies with Direct CTV minus policy above zero
144 decision views[+0.09, +0.27]9 of 9
12 training configurations[+0.07, +0.28]9 of 9
11 evaluation databases[−0.15, +0.21]8 of 9

Relative to admitting every candidate, the paired database-level difference is +1.83 points with 95% interval [+1.34, +2.27]. The LESS validation gradient is the one admitting policy whose paired interval includes zero over databases, and the revised Appendix C reports every paired interval.

W3 & Q3.

The paper stresses the optimizer state but does not test the same model with two optimizer states.

Answer: The paper evaluates identical weights under two optimizer states in a controlled intervention where resetting the AdamW second moment reverses the data-order effect, and the revision adds an admission-level test on a Qwen3-4B state carrying optimizer history. In the controlled intervention, Qwen3-1.7B continues from identical step-8 weights across two data blocks in both orders under carried or reset AdamW moments, evaluated on 192 validation questions.

Same weights at step 8Order effect F(US)−F(SU)F(US)-F(SU) (pp)
Both AdamW moments carried−1.30
First moment reset+0.91
Second moment reset+3.91
Both moments reset+2.21
Second-moment component, carried minus reset, averaged over the first-moment treatment−3.26, simultaneous 95% interval [−5.09, −1.42]

In the admission-level test, a Qwen3-4B receiving state with 64 prior AdamW steps runs paired continuations on five candidate families and the incumbent, comparing restored AdamW moments and step counter against a fresh optimizer. Across updater seeds 1 to 7 (84 continuations in all), resetting the moments and the step counter shifts the validation effects of the five families by −1.88 to −0.26 points, every simultaneous 95% max-tt interval covers zero, and the stateless repair loop family changes sign. When each arm admits on the selection questions of six seeds and evaluates on the validation questions of the seventh, both arms admit all 35 cross-fitted candidates (15 harmful) for +0.74 points per decision. Both arms make the same decision in all 35 cases, so the registered equivalence criterion for utility holds. At this warm state the optimizer history moves candidate effects within updater noise and leaves admission unchanged, and the revision states this as the scope of the warm-state evidence. Revised Section 4 reports the step-8 intervention, and Section 5 reports the admission-level test.

If these replies resolve your concerns about readability, novelty and the optimizer state, we would be grateful if you would consider raising your score.

Thank you again for your review and for the concrete list of passages. We're happy to address any further questions you may have.

I thank the authors for their replies. I still think the paper has several significant flaws, well described in the 3 reviews. I agree with reviewer cY8M that fixing them would require changes beyond what is possible in a rebuttal phase. I keep my current score.

AuthorsOpenReview ↗

Thank you for the follow-up

Thank you for taking the time to reply. We rewrote the paper during the rebuttal phase, passage by passage from the list you gave us, since that's what we understood the rebuttal phase to be for, and we're honestly a little sad that the rewrite reads as work left undone. Every result stands as measured, and none of the three reviews points to an error in a proof, a measurement or a comparison, so we'd hoped the clearer text would let the findings speak for themselves. We're sorry the work didn't land with you, and we're grateful for your time and for your careful list of passages.

Reviewer_cY8MOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that a dataset’s value depends on the full training state receiving it (weights plus optimizer memory), so one fixed score per dataset can’t be right for every state. It proposes “causal trace value,” which restores the saved state, trains once on the candidate and once on the incumbent, and admits the candidate if accuracy improves. On a text-to-SQL benchmark this beat random selection and scoring baselines.

Critical strengths

None

Critical weaknesses
  • The paper is very hard to read. It looks heavily AI-drafted or polished and needs significant rewriting, with clearer language and examples to illustrate the idea and methodogy.
  • The setting is never stated up front. The input, output and objective should be defined before any methodology is introduced.
  • The first paragraph in Section 4.1 mentions a student and a teacher model without saying where they come from. The setting must be introduced first, or the reader can’t follow.
  • Section 4 and Section 5 are heavy on technical detail. They give no intuition for why the theorems are needed or what result they state.
  • The experiments use only one small dataset, and it covers only NL2SQL tasks.
  • The experiment section is poorly organized. It doesn’t clearly state the setting, metrics and baselines.
Questions for authors
  • What is F^\hat{F} in Equation (2)? What does “the same accuracy on a disjoint selection split” mean?
  • Where do the student and teacher models in the first paragraph come from?
  • What are the input, output and objective of the problem?
  • Can you explain in plain text what result Section 4 is trying to state, and why it is needed? The same question applies to Section 5.
5 comments
AuthorsOpenReview ↗

Reply to Reviewer cY8M (1/2): the setting, the estimator and the models

Thank you for your precise questions on the setting, notation, and experimental organization, which the revised paper now answers directly in the text. Here are our replies.

W1.

The paper is hard to read and needs a rewrite with clearer language and examples.

Answer: We rewrote the paper from the abstract to the conclusion in plain language built around one running example. Section 3 now defines the input, output, and objective before introducing any method and carries the BIRD text-to-SQL running example into Section 5. Each formal result opens with a plain statement and closes with its practical implication. Section 6 opens with the experimental setting, evaluation metrics, and compared policies. The replies below quote the revised text.

W2 & Q3.

What are the input, output and objective of the problem?

Answer: The input is a saved receiving state and candidate datasets, the output is a binary decision to admit each candidate or keep the incumbent, and the objective is post-update execution accuracy on disjoint validation questions that no admission rule reads. Revised Section 3 defines these components directly.

  • Input: a saved receiving state of the student model being trained (its trainable weights, first and second AdamW moments, and step counter), a fixed token budget, an incumbent dataset scheduled for training, candidate datasets of synthetic reasoning traces, and a held-out selection split that the admission rule reads.
  • Output: for each candidate, admit the candidate or keep the incumbent.
  • Objective: higher post-update student accuracy on a disjoint validation split that no rule ever reads. A decision's realized utility equals the candidate's validation effect when admitted and zero when the incumbent is kept.
  • Example: a Qwen3-4B student with a budget of 65,536 tokens over 32 optimizer steps, paired with a candidate dataset from a producer that receives pass-or-fail SQLite execution feedback and revises rejected queries.

Q1.

What is F^\hat F in Equation (2), and what is "a disjoint selection split"?

Answer: F^(y)\hat F(y) is the SQL execution accuracy of a trained state yy on the selection split read by the admission rule, and F(y)F(y) is execution accuracy on the disjoint validation split reserved to evaluate the decision. Separating the selection split from the validation split ensures that no decision is evaluated on the questions used to make it. In the cross-state study, the two splits draw from disjoint databases (179 selection questions and 441 validation questions across 20 databases). The frozen benchmark uses 192 matched question pairs across 11 databases and swaps the selection and validation roles in a second pass. Equation (2) of the submission, Equation (3) in the revision, defines the CTV estimate as the paired continuation accuracy difference on the selection split averaged across updater seeds, and the true validation effect τ\tau substitutes FF for F^\hat F.

W3 & Q2.

Where do the student and teacher models of Section 4.1 come from?

Answer: The student is the receiving model being updated, and the teacher is the offline producer model that generated the synthetic reasoning traces prior to student training. In the motivating experiment of Section 4.1, a Qwen3-1.7B student is fine-tuned from base weights on four trace sets generated offline by a Qwen3-1.7B producer under two demonstration-retrieval rules. In the admission studies, Qwen3-4B and Qwen3-8B serve as students. In the cross-state study, six producer families pair Qwen3-1.7B with a SQL verifier, a solution-bearing verifier, a persistent or stateless runtime, or a repair step, and the incumbent and two retrieval candidates come from Qwen3.5-2B. The frozen benchmark uses three of these Qwen3-1.7B families, the binary verifier, the runtime-state repair loop, and the solution verifier. Every producer generates traces on training databases disjoint from all evaluation databases, and the revised Section 3 introduces both roles up front.

AuthorsOpenReview ↗

Reply to Reviewer cY8M (2/2): what Sections 4 and 5 state, the scope of the evidence, and the experiment section

W4 & Q4.

Explain in plain text what Sections 4 and 5 state and why they are needed.

Answer: Section 4 characterizes when a static dataset score can replace measurement at the receiving state, Section 5 establishes how many updater seeds that paired continuation requires, and Proposition 2 in Section 6.2 specifies when measurement stops paying off.

  • Theorem 1. A single state-independent score per dataset accurately predicts each candidate's effect across receiving states if and only if state and dataset act additively on validation accuracy. Every static dataset score relies on this additivity assumption, and Theorem 1 converts it into a testable state-by-dataset interaction.
  • Corollary 1. Whenever a candidate dataset helps one receiving state and harms another, any rule that assigns the same admission probability to both states incurs expected regret, summed over the two states, at least equal to the smaller effect magnitude, pricing the error of omitting the state.
  • Proposition 1. Under paired continuations, the probability that the measurement gets the sign of a candidate's selection-split effect wrong decays exponentially in the number of updater seeds, so averaging three seeds bounds expected selection-split regret from updater noise at roughly one third of the per-seed noise scale.
  • Proposition 2. As the token budget grows and the mean candidate effect outpaces its cross-candidate spread, the realized utility gain of any selector over admitting every candidate vanishes exponentially, specifying when measurement stops paying off.

W5.

The experiments use one small dataset and only text-to-SQL.

Answer: We chose one task with an exact execution verifier so that every realized utility is a measured accuracy change, and within it the experiments vary the producer family, the receiving model, the LoRA rank and the token budget. Text-to-SQL checks every answer by executing it against SQLite, so program execution measures the utility of an admission decision without LLM scoring. The frozen benchmark spans 12 training configurations with three producer families, two Qwen3 receiving models, LoRA ranks 8 and 32, and budgets from 4,096 to 65,536 tokens. The cross-state study evaluates eight candidate families at two token budgets on 20 databases. CTV applies to any task with a saved receiving state, token-matched candidate packs, and held-out questions with a deterministic verifier.

W6.

The experiment section does not state the setting, metrics and baselines.

Answer: Revised Section 6 opens with dedicated paragraphs for the experimental setting, evaluation metrics, and compared policies, then leads each empirical subsection with its finding in bold.

  • Setting: Qwen3-4B and Qwen3-8B LoRA students trained from base weights on synthetic traces from databases disjoint from evaluation, evaluated on a frozen benchmark of 144 decision views (12 configurations, two split directions, six evaluation criteria) and a registered cross-state study of 16 decisions per token budget.
  • Metrics: realized utility per decision on the validation split relative to keeping the incumbent, harmful admissions, realized utility at a matched admission count, Spearman rank correlation with validation effects, and GPU runtime, reported with 95% bootstrap intervals.
  • Policies: keep the incumbent, admit every candidate, matched-count random selection, teacher accuracy, execution correctness, trace judge, student NLL, RSR, forward learnability, Data-DPO, LESS validation and sign gradients, a LESS-style pool proxy, a Qwen3-1.7B transport student, one-step micro-model CTV, debiased measurement with pooled or score priors, and Direct CTV, grouped by access to the receiving state with published methods cited and constructed baselines marked.

If the revised text answers your questions on the setting and the theory, we would be grateful if you would consider raising your score.

Thank you again for your precise questions. We're happy to address any further questions you may have.

AuthorsOpenReview ↗

Follow-up to Reviewer cY8M: the cross-state study at a current-generation receiving model

W5.

The experiments use one small dataset and only text-to-SQL.

Answer: The cross-state study now includes a third receiving model from the current generation, Qwen3.5-4B. It is the newest general Qwen model under 10B parameters, and it changes the architecture (24 of its 32 layers use Gated DeltaNet linear attention), the tokenizer and the pretraining generation. It receives the same eight candidate families, incumbent, updater seeds and 781 questions at 65,536 tokens, under a decision rule registered before any of its runs:

24 decisions over three receiving models, 17 admissions eachDirect CTV minus the policy (points)One-sided 5% lower bound
Execution correctness+0.32−0.57
Trace judge+0.47−0.30
LESS-style proxy+0.44+0.03
Data-DPO+0.57−0.19

Direct CTV stays ahead of every score in point estimate. Its lower bound is positive against the LESS-style proxy and includes zero against the trace judge, so the registered rule classifies the result as weakened. Over the eight Qwen3.5-4B decisions alone, execution correctness, the trace judge and the LESS-style proxy exceed Direct CTV by 0.07 to 0.18 points. At this budget every policy again loses relative to keeping the incumbent, Direct CTV least (−0.76 points against −1.43 for admitting every candidate), and the family-by-model interaction stays within updater noise. Appendix C.3 of the revised paper reports the study in Table 5, and Section 6 states its result.

We're happy to address any further questions you may have.

Reviewer_z6t5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that training data value depends on the receiving model's full training state, including model parameters, optimizer states, and training progress. The authors propose Causal Trace Value (CTV), which estimates dataset utility by comparing matched training continuations from an identical saved state. They measure the CTV by restoring the saved state and continuing one time on the candidate and one time on the the incumbent at equal tokens and steps for controlled comparison. They also provide theoretical bounds for state-independent data selection. Experiments on BIRD text-to-SQL with Qwen3-4B and 8B show that CTV outperforms existing data selection baselines, but shows higher computational cost.

Critical strengths
  • I believe that the weakness of the paper significantly outweighs the positives, but the paper presents a well-defined, intuitive problem and a good starting point for measuring dataset utility, while significant adjustment to their framework and writing are needed.
Critical weaknesses
  • The writing is somewhat overly complicated unnecessarily, with the theoretical results or additional analyses having insufficient connection to one another. It's hard to follow how these experiments or results relate to each other and support the claims made in the paper. A much clearer presentation is needed for better readership. In short, the paper is hard to follow.

  • Although CTV shows better performance than the presented baselines, the improvement compared to retaining the incumbent is relatively low (0.17pp), in which the confidence interval includes 0 (no gain) and CTV exceeds the random baseline by <1pp. The gains are too incremental considering the additional training that incurs than the compared baselines. Moreover, if the quantitative increments are small, the paper should have some qualitative analyses to back up the claims, such as how can CTV help in some situation to provide insights where this method can be beneficial.

Questions for authors
  • Any insights where CTV can be especially beneficial?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer z6t5 (1/2): how the results connect, and the size of the gain

Thank you for your constructive review. Your question on where receiving-state measurement pays off directly motivated a new finding in the revised Section 6. Here are our replies.

W1.

The writing is complicated, and the theoretical results and analyses are not connected to one another or to the claims.

Answer: We rewrote the paper so that every formal result opens with a plain statement of what it establishes and connects directly to the experiment that tests it.

Formal resultWhat it statesExperiment that tests it
Theorem 1One score per dataset is accurate across states exactly when state and dataset act additivelySection 4: with the weights fixed, resetting the AdamW second moment reverses the effect of a data order
Corollary 1A dataset that helps one state and harms another costs every state-blind rule at least the smaller effectSection 4: the binary-verifier family measures −2.04 points on Qwen3-4B and +0.91 on Qwen3-8B, so a shared verdict would cost at least 0.91 points if these were the true effects
Proposition 1The sign error of the measurement falls exponentially in the number of updater seedsSection 6: utility per decision rises from +0.04 at one seed to +0.17 at four
Proposition 2The gain of any selector over admitting every candidate vanishes once the mean effect outgrows its spreadSection 6: at 524,288 tokens 15 of 16 candidates help and admitting all ties the measurement

Section 3 now defines the input, output, and objective up front through one running example, and Section 6 opens with dedicated paragraphs for the experimental setting, evaluation metrics, and compared policies.

W2.1.

The gain over keeping the incumbent is small, 0.17 points, and its interval includes zero.

Answer: Direct CTV gains +0.60 points per admitted candidate, and its per-decision realized utility of +0.17 points carries a 95% bootstrap interval of [+0.09, +0.27] across the 144 decisions and [+0.07, +0.28] across the 12 training configurations. Filtering out harmful candidates produces that positive utility, because admitting the candidate degrades validation accuracy in 101 of the 144 decision views (12 training configurations, two split directions, and six evaluation criteria).

Frozen benchmark, 144 decision viewsHarmful / admittedUtility per decision (pp)
Admit every candidate101/144−1.66
Teacher accuracy65/96−1.22
LESS validation gradient41/72−0.05
Direct CTV11/42+0.17

Resampling the 11 evaluation databases and recomputing all decisions yields [−0.15, +0.21] relative to keeping the incumbent. On those same draws Direct CTV exceeds admit-all and seven of the eight admitting scores and proxies, with each paired 95% interval above zero.

W2.2.

The measurement exceeds random selection by less than one point and costs extra training.

Answer: At a matched admission count of 42 candidates, random selection incurs −0.48 points of realized utility per decision and makes 29 harmful admissions, and Direct CTV gains +0.17 points and makes 11. Because random admission in this pool degrades the model, Direct CTV's advantage over matched-count random selection is +0.66 points per decision with a one-sided 5% lower bound of +0.31 over resampled configurations, exceeding its gain over keeping the incumbent. For compute cost, each continuation trains in about 12 seconds on one H100 and reading the 179 selection questions takes about 10 minutes. Measuring eight candidates and the incumbent at three seeds at one receiving state takes about 4.4 GPU-hours, against 0.81 GPU-hours for the single scoring pass of the dataset scores at Qwen3-4B (Appendix C). Reading the held-out questions takes 96% of a continuation's runtime when all 781 are read, so the number of selection questions sets the price.

AuthorsOpenReview ↗

Reply to Reviewer z6t5 (2/2): where the measurement pays

W2.3 & Q1.

Where can CTV be especially beneficial, with qualitative analysis to back the claim?

Answer: Receiving-state measurement pays off when harmful candidates are common and pass static dataset filters and when the selection questions come from the evaluation domains the update serves, and it stops paying off once a larger token budget makes nearly every candidate helpful. The revised Section 6 states this as a standalone finding.

SettingHarmful candidatesAdmit every candidate (pp)Direct CTV (pp)
Frozen benchmark, 144 decisions101 of 144−1.66+0.17
Fresh draw of eight families, Qwen3-4B, 65,536 tokens, original databases2 of 8+0.22+0.81
Cross-state study, 65,536 tokens, database-disjoint splits11 of 16−1.18−0.50
Cross-state study, 524,288 tokens1 of 16+2.67+2.67

On the fresh Qwen3-4B draw, selection and validation questions share the original databases, and Direct CTV admits the two highest-gain families, the persistent repair loop (+3.85 points) and the solution-bearing verifier (+2.65), and rejects both harmful retrieval families (−8.31 and −2.19 points, Appendix C). In the cross-state study at 65,536 tokens, harm is as common and the selection and validation questions come from disjoint databases. There the selection-split effect exceeds the validation effect in 13 of 16 decisions, so measurement loses less than admitting every candidate without turning positive. Three mechanisms govern when measurement pays off.

  • Harmful candidates pass dataset-level filters. Teacher accuracy and the trace judge score traces without updating the student and make 65 and 39 harmful admissions on the frozen benchmark, because traces solved accurately by a producer can still degrade the receiving student. In Section 4's motivating study, the trace set with the largest producer gain (+13.5 points) lowers student accuracy by 5.2 points.
  • The receiving state changes a dataset's effect. With the weights fixed, resetting the saved AdamW second moment moves the effect of a data order by 3.26 points. At 65,536 tokens, the binary-verifier family measures −2.04 points on Qwen3-4B and +0.91 points on Qwen3-8B. If these were the true effects, Corollary 1 implies that any state-blind rule assigning a shared verdict would incur at least 0.91 points of regret summed over the two receiving states.
  • The token budget sets selection value. By Proposition 2, the realized utility advantage of any selector over admitting every candidate decays faster than e−z2/2e^{-z^2/2} once the ratio zz of mean candidate effect to spread exceeds 1. Because zz grows with the token budget on BIRD, the value of paired continuations concentrates in small-budget regimes and harm-heavy candidate pools.

If these replies resolve your concerns about the size of the gain and where the method helps, we would be grateful if you would consider raising your score.

Thank you again for your constructive review. We're happy to address any further questions you may have.

Thanks for the reply. Align with all the other reviews, I believe (1) the revised draft is still hard to follow, takes way too much time to digest and (2) still needs substantial rework in terms of formulation, writing, and experiments to be accepted as a top-tier ML conference paper. I tend to keep my score as clear reject.

AuthorsOpenReview ↗

Thank you for the follow-up

Thank you for the follow-up and for staying with the paper through the discussion. We take the point on readability seriously, because a paper that takes this much effort to digest hasn't yet done its job, and that responsibility is ours. We'll keep working on the formulation, the writing and the experiments so the findings come through more directly, and the three reviews will guide that work. Thank you again for your time and care.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.