acceptodds
Under review as a conference paper at ICLR 2027

Not All Rollouts Are Worth Learning: On Trajectory Valuation for Post-Training Reinforcement Learning

Abstract

We consider the problem of trajectory valuation in reinforcement learning: how to identify and mitigate detrimental trajectories during online training. Unlike classification, where data valuation relies on fixed training and validation sets, reinforcement learning involves dynamically generated trajectories without explicit validation signals, making conventional influence-based methods inapplicable. We propose Dynamic Trajectory Valuation (DTV), a simple and efficient framework that estimates trajectory utility at the mini-batch level and filters detrimental trajectories based solely on gradient information. By operating at the optimization level, DTV integrates seamlessly with existing reinforcement learning pipelines with minimal overhead. Extensive experiments across diverse settings, including GRPO, DPO, and PPO, demonstrate that DTV consistently improves performance, enhances data efficiency, and stabilizes optimization.

Primary area:
reinforcement learning
Keywords:
Data-Centric Learning, Data Valuation, Reinforcement Learning, Trajectory Valuation, Gradient-Based Optimization, Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vmLXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies a method that identifies and filters detrimental trajectories during the training of a reinforcement learning algorithm. Inspired by techniques from supervised learning, their score, called dynamic trajectory valuation (DTV), essentially identifies samples in a batch whose induced gradient differs from the overall gradient of the mini-batch.

Critical strengths
  • Simple and attractive idea to filter detrimental trajectories during training which is compatible with any RL algorithm
  • Well explained and motivated method (although I find that the experiments lacking both)
Critical weaknesses
  • Design of experiments: In its current form, the experiments do not convince me enough that the proposed method is beneficial for general reinforcement learning as claimed in the introduction. At best, the RL results for PPO (Appendix D) are early signals that the method could be promising, but their scope is too narrow to derive general statements here. It seems to me that some of the experiments consider supervised fine-tunings, which I would not regard as reinforcement learning and where I do not find it surprising that DTV provides improvements. However, I could also be a misunderstanding at my side as the paper seems very vague at these places (see below).
  • Mixed evaluation schemes and protocols: It seems to me that different evaluation schemes and application regimes are mixed. Moreover, it is not made mathematically precise enough what exactly the "post-training RL" setting is. Section 4.1 treats a mismatched LLM post-training setting without explaining what *mismatched" means and why its important for benchmarking DTV. On the other hand, the PPO experiments train policies from scratch in rather simple tasks (only two MiniGrid tasks). That makes it hard to assess which effects come from the training methodology, the RL algorithm, or the filtering method.
  • Presentation: The presentation of the experiment section must be improved substantially in my opinion. I had a hard time finding essential details. For instance, I could not found which LLM was used in the GSM8K experiments in Section 4.1 and whether DTV was used in GRPO or in a LoRA fine-tuning as said at the end of Section 4.1. Moreover, Section 4 is cluttered with seemingly unnecessary details that essentially restating the numbers shown in the tables instead of providing intuition and interpretation of the results. I miss structured research questions the experiments aim to answer.
Questions for authors
  • What does mismatched mean and why is this setting important for benchmarking DTV? Why are the ACCs reported in Section 4.1 lower than other reported baselines for GSM8K and where does the pretrained model come from?
  • Which of the experiments are RL trainings and which are supervised fine-tunings? What is the precise setting of post-training RL the paper addresses?
  • Why is PPO not tested in a post-training fashion? Why is GRPO only tested in a post-training fashion?
5 comments
AuthorsOpenReview ↗

Response to Reviewer vmLX (Part 1/3)

Thank you, Reviewer vmLX, for asking how our training and evaluation settings differ and how broadly the PPO results apply.

Clarification of RL Training Settings and Generality. We thank the reviewer for raising this concern. We agree that the distinction between supervised fine-tuning, reinforcement learning, and preference optimization should be made more explicit in the manuscript. However, we would like to clarify that our experiments are not primarily based on supervised fine-tuning.

Specifically, our GSM8K and AIME experiments employ online GRPO reinforcement learning, where the policy generates responses, receives reward-based feedback, and is updated using group-relative advantages. On GSM8K, we use LoRA for parameter-efficient policy updates, but LoRA only specifies which parameters are trained; the underlying optimization algorithm remains GRPO rather than supervised fine-tuning. On AIME, we perform full-parameter GRPO training. Furthermore, our PPO experiments involve online interaction with the environment and policy-gradient optimization, constituting standard reinforcement learning.

For UltraFeedback, supervised fine-tuning is used only to obtain a shared initialization checkpoint. DTV is applied during the subsequent DPO stage, not during SFT. We include DPO as an additional offline preference-optimization setting to investigate the applicability of gradient-based trajectory valuation beyond online RL, rather than presenting it as equivalent to online reinforcement learning.

Regarding generality, we agree that no finite collection of benchmarks can establish universal effectiveness across all RL settings. Nevertheless, our experiments cover two distinct online RL algorithms, GRPO and PPO, across language-model reasoning and control tasks, with additional evidence from DPO. We believe these results provide evidence that DTV is not restricted to supervised fine-tuning or a single RL algorithm, while acknowledging that broader evaluations would further strengthen this conclusion.

We will revise the experimental section to explicitly distinguish the training objectives, initialization procedures, and optimization algorithms, and will appropriately qualify our claims about general applicability.

GSM8K Checkpoint and Evaluation. Thank you for pointing out this important detail; we are happy to clarify it. We initialize all methods from the official google/gemma-3-1b-it checkpoint. Before GRPO, this model achieves 47.23% accuracy in our evaluation setup. The lower absolute ACC compared with some published GSM8K results mainly reflects differences in backbone choice, parameterization, generation budget, and evaluation protocol. In particular, our evaluation uses a 768-token generation cap, while reported results for the same model may use substantially longer generation budgets. All methods in our comparison share the same initialization and evaluation procedure, so our analysis focuses on relative improvements under a controlled setting rather than direct comparison across different experimental configurations.

Mixed Evaluation Schemes and Protocols. We appreciate the reviewer's observation that our experiments involve different training regimes and evaluation protocols. We agree that their respective purposes were not sufficiently distinguished in the original submission.

Our primary focus is post-training optimization of pretrained language models. Specifically, GSM8K and AIME employ online GRPO, while UltraFeedback employs offline DPO following a shared SFT initialization. In both cases, DTV is applied during the subsequent optimization stage, rather than during supervised fine-tuning.

The purpose of including different optimization algorithms is to investigate whether the same gradient-based trajectory-valuation principle can be applied across distinct post-training objectives, not to suggest that these algorithms or evaluation protocols are equivalent.

Importantly, we assess the contribution of DTV through within-setting comparisons: the underlying algorithm, initialization, training budget, and evaluation protocol are held fixed across the compared methods, with trajectory filtering as the intervention. We do not use cross-algorithm comparisons to attribute performance differences to DTV. We will clarify this experimental structure and the scope of each comparison in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer vmLX (Part 2/3)

The Role of PPO Experiments. We agree with the reviewer that the original PPO experiments, which train policies from scratch on relatively simple control tasks, do not constitute LLM post-training. This distinction is precisely why we placed these experiments in the appendix: they are intended as supplementary evidence exploring whether DTV's valuation principle can also be applied in conventional RL, rather than as primary evidence for our post-training claims.

To broaden this supplementary evaluation, we have additionally included PPO experiments on HalfCheetah-v5 (Table 7), trained from random initialization, and windy LunarLander-v3 (Table 8), where policies adapt from a shared checkpoint pretrained in the no-wind environment. The latter examines adaptation from an existing policy, although it remains distinct from LLM post-training. Within each task, all compared methods share the same PPO algorithm, initialization protocol, and interaction budget.

We will make the supplementary role of these experiments more explicit.

Table 7: HalfCheetah-v5 PPO from scratch. Evaluation return as mean ± std across five training seeds, with 1,000 evaluation episodes per checkpoint.

RoundVanilla PPOIIFDTVDTV-Loo
50550.28 ± 101.98960.09 ± 430.42571.66 ± 144.37238.85 ± 224.31
1001320.85 ± 270.071829.94 ± 798.381291.91 ± 278.221525.23 ± 516.75
1501755.40 ± 775.942486.67 ± 1123.711616.67 ± 500.492464.70 ± 471.34
2002036.72 ± 1063.513096.76 ± 1054.581909.01 ± 805.322952.97 ± 667.38
2502184.36 ± 1152.163556.64 ± 1157.382225.46 ± 1017.673521.53 ± 587.61
3002336.50 ± 1309.273696.94 ± 1166.142597.53 ± 1131.273824.20 ± 582.17
3502379.19 ± 1251.783842.51 ± 1204.262803.53 ± 1190.183978.12 ± 488.29
4002474.12 ± 1420.843964.54 ± 1249.752941.41 ± 1180.504180.55 ± 454.66
4502219.15 ± 1816.283971.38 ± 1254.963135.55 ± 1307.394363.13 ± 467.57
5002445.22 ± 1646.134082.45 ± 1304.563319.91 ± 1334.564514.31 ± 399.89

Table 8: Windy LunarLander-v3 PPO post-training. Evaluation return as mean ± std across five training seeds, with 1,000 evaluation episodes per checkpoint.

RoundVanilla PPOIIFDTVDTV-Loo
0153.73 ± 0.00153.73 ± 0.00153.73 ± 0.00153.73 ± 0.00
20181.10 ± 19.15150.26 ± 18.89189.05 ± 10.17180.78 ± 21.11
40187.94 ± 22.07174.91 ± 10.16194.65 ± 26.78176.70 ± 29.62
60187.29 ± 33.66178.86 ± 64.42198.41 ± 18.20187.80 ± 16.55
80193.32 ± 14.78187.48 ± 22.57216.40 ± 25.14191.32 ± 35.86
100203.73 ± 9.83187.03 ± 31.10222.70 ± 41.63198.04 ± 42.15
120211.49 ± 13.52220.76 ± 27.50242.96 ± 17.39211.11 ± 18.73
140224.07 ± 41.45211.21 ± 38.49254.61 ± 12.20229.99 ± 13.42
160226.59 ± 27.94199.83 ± 17.62245.61 ± 14.80231.05 ± 17.46
180213.79 ± 16.23197.51 ± 23.30245.55 ± 15.40235.52 ± 17.91
200226.20 ± 31.56228.38 ± 12.29245.28 ± 12.07232.78 ± 22.40
AuthorsOpenReview ↗

Response to Reviewer vmLX (Part 3/3)

Mismatch Setting and Its Motivation. We thank the reviewer for pointing out that the mismatch protocol should have been defined and motivated in the main text.

For GSM8K, Mismatch-20% means that approximately 20% of training prompt groups are selected, and the observed completion rewards within each selected group are reassigned in reverse rank order. This preserves the group's reward values while disrupting their correspondence with the generated completions, thereby introducing unreliable advantage estimates into GRPO training.

For UltraFeedback, the mismatch settings instead perturb selected preference pairs by replacing responses with responses from other prompts and reversing their preference roles. These are distinct corruption protocols, reflecting the different forms of training feedback used by GRPO and DPO.

Our motivation is to examine whether DTV remains useful when the learning signal derived from training feedback is imperfect. These settings provide controlled stress tests of feedback reliability; we do not claim that they reproduce every form of noise encountered in practice. Importantly, evaluation is performed on clean, uncorrupted data.

We will move the definitions and motivation of these mismatch protocols into the main experimental section and clarify their differences.

Presentation and Readability. We sincerely apologize for the difficulty the reviewer experienced in reading and understanding our manuscript. We recognize that it is our responsibility as authors to communicate our methodology and experimental design clearly, and that readers should not have to reconstruct essential details from different sections and appendices. We appreciate the reviewer for bringing these presentation issues to our attention.

To address them, we will reorganize the manuscript around a clearer division between the main text and the appendix. The main text will explain the overall methodological and experimental pipeline, including the role of DTV in each optimization algorithm, the training and evaluation stages, and the purpose of each experimental setting. This will allow readers to understand what is being tested, why it is being tested, and how the reported results should be interpreted.

The appendix will provide the detailed experimental specifications needed for reproducibility, including implementation details, hyperparameters, model configurations, data processing, and evaluation protocols.

Our goal is to make the main paper self-contained and easy to follow, while retaining sufficient technical detail in the appendix for readers who wish to reproduce or examine the experiments more closely. We thank the reviewer again for these constructive suggestions, which will help us substantially improve the clarity of the manuscript.

Reviewer_G6iLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DTV, which filters training units using their gradient alignment with a batch reference. DTV-Loo excludes each unit's own gradient from that reference. The approach is evaluated with GRPO, DPO, and PPO, including controlled feedback corruption. The main results show improved GSM8K accuracy and in-domain preference learning, with additional analysis of the self-term's effect on filtering.

Critical strengths
  1. The filtering rule is simple, requires no separate validation gradient, and has a clear implementation across three training objectives. The self-term decomposition helps explain why DTV and DTV-Loo can behave differently.

  2. The DPO experiments provide useful controlled evidence: matched seeds, paired tests, and larger fixed-rate filtering baselines in Appendix F.3.2. The GSM8K improvement from 0.4828 to 0.5483 is also substantial and merits further investigation.

Critical weaknesses
  1. My main concern is whether batch agreement measures useful training signal. In GRPO, the reference uses completions from the same prompt, while corruption reverses reward ranks within whole groups (E.1–E.2). That reference is not an independent clean signal, so the minority-corruption explanation from DPO does not transfer directly. The influence-function motivation also does not establish that replacing the original reference with the batch mean preserves utility rankings.

  2. The evidence for a general post-training benefit is less convincing than the in-domain results. Table 8 reports lower RB2 Precise IF for DTV-Loo than vanilla DPO, including 0.4238 versus 0.4938 on clean data, with no consistent generation-benchmark gains. There is also an apparent discrepancy between the clean endpoint in Figure 2 and Table 3. These issues keep me at weak rejection.

Questions for authors
  1. Clarification: In GRPO, how do filtering rates differ between corrupted and clean groups, and between positive- and negative-advantage completions?

  2. Clarification: Table 3 gives a clean final gain of 0.0292 over vanilla DPO, whereas Figure 2 appears to end near zero. Do the plotted endpoint, evaluation set, or averaging procedure differ?

  3. Targeted check: Can filtering decisions predict improvement on a separate clean objective at fixed checkpoints? This would help distinguish useful valuation from batch agreement alone and clarify when DTV-Loo should be preferred.

7 comments
AuthorsOpenReview ↗

Response to Reviewer G6iL (Part 1/4)

Thank you, Reviewer G6iL, for revieweing our paper and asking about batch agreement and the scope of our evaluation.

Batch Agreement and Training Utility. We thank the reviewer for raising this important concern. We would like to clarify that DTV is not intended to recover utility rankings defined by an independent clean objective. Rather, it aims to improve policy optimization by identifying training gradients that conflict with the current aggregate learning direction.

The key assumption is that the aggregate gradient remains locally informative, even when some training feedback is noisy. Under this condition, filtering negatively aligned gradients can reduce cancellation against the aggregate direction and potentially improve the resulting policy update. This does not require the reference gradient to be perfectly clean, but it does require the reference direction to retain useful learning information. If corruption makes the aggregate direction misleading, DTV may instead reinforce an undesirable update.

Importantly, we do not treat batch agreement itself as evidence of improved training utility. We assess the effectiveness of DTV through the performance of the resulting policies on separate, uncorrupted evaluation data. In our GSM8K experiments, DTV improves final clean test accuracy over vanilla GRPO under both clean and mismatched training conditions. These results provide end-to-end empirical evidence for the usefulness of the filtering procedure, although they do not establish that individual filtering decisions preserve rankings under an independent clean objective.

We also agree that the minority-corruption intuition from DPO does not directly transfer to GRPO, where reward rankings are reversed within entire prompt groups. Similarly, the influence-function perspective motivates gradient-based valuation but does not establish equivalence between the batch reference and an independent utility gradient. We will clarify these distinctions and the conditional nature of Assumption 1 in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer G6iL (Part 2/4)

Clarification of Evaluation Sets and the Scope of Generalization. We sincerely appreciate the reviewer's thoughtful observations. In particular, the distinction between improving the learning signal of the current training objective and achieving broader downstream generalization is fundamental to how we conceptualize DTV. We are glad that the reviewer has brought attention to this important point, which we did not articulate sufficiently clearly in the original submission. We welcome the opportunity to clarify and discuss it further.

Figure 2 versus Table 3. We apologize for the confusing presentation. Figure 2 and the original Table 3 evaluate two different held-out sets within the same DPO task. Figure 2 tracks preference accuracy on a held-out evaluation subset during training, while Table 3 reports final-checkpoint accuracy on a separate test split. We included both evaluation sets to examine whether improvements from refining the training signal persist across different held-out samples of the same task. However, we did not clearly distinguish these evaluation sets in the original manuscript. We have now revised Table 3 of the manuscript to report results on both sets, making the comparison explicit and transparent. Table 5 presents the revised results.

Table 5: DPO preference-accuracy results on UltraFeedback under clean and mismatch settings. Accuracy is reported on two clean sets: a held-out set for periodic evaluation and a final test set for final evaluation. Results are reported as mean ± std over five seeds, with the best result in each column shown in bold. †The final row reports two-sided paired t-test p-values comparing DTV-Loo with Reward 10% across matched seeds; significant differences (p < 0.05) are shown in bold.

MethodHeld-out Clean ↑Held-out Mismatch-20% ↑Held-out Mismatch-40% ↑Final test Clean ↑Final test Mismatch-20% ↑Final test Mismatch-40% ↑
Vanilla DPO0.6165±0.00390.5959±0.00320.5481±0.01250.5388±0.00730.5014±0.00440.4301±0.0160
Random 5%0.6160±0.00390.5926±0.00300.5461±0.01380.5449±0.00230.5013±0.01520.4234±0.0143
Random 10%0.6150±0.00260.5945±0.00660.5455±0.01040.5421±0.00630.5015±0.01000.4332±0.0122
Reward 5%0.6156±0.00200.6045±0.00290.5699±0.00760.5559±0.00620.5350±0.00920.4683±0.0191
Reward 10%0.6175±0.00270.6098±0.00540.5799±0.00800.5647±0.00630.5443±0.00480.4857±0.0147
DTV (Ours)0.6196±0.00200.6011±0.00800.5572±0.00960.5618±0.00200.5236±0.00680.4413±0.0103
DTV-Loo (Ours)0.6134±0.00200.6176±0.00340.5945±0.00870.5680±0.00400.5537±0.00680.5196±0.0208
p-value†0.07270.10450.00040.36800.14210.0176

Interpretation of Manuscript Table 8 and the Applicability of DTV. We particularly appreciate the reviewer's attention to the downstream results, as they highlight an important question about the scope of trajectory valuation.

DTV is designed to refine the learning signal provided by the current training objective by filtering gradients that conflict with the aggregate optimization direction. Consequently, its most direct expected benefit is improved performance on held-out data aligned with that objective. Such improvements do not necessarily transfer to downstream tasks that evaluate different capabilities.

Manuscript Table 8 is therefore important not only for assessing downstream transfer, but also for understanding the empirical limitations of DTV. In particular, DTV-Loo underperforms vanilla DPO on RB2 Precise IF under clean training (0.4238 versus 0.4938), while the other downstream results are mixed. We consider these negative and mixed findings valuable: a complete evaluation of a trajectory-valuation method should characterize both where it provides benefits and where those benefits do not reliably extend. This is why we report these results rather than restricting the evaluation to favorable benchmarks.

We see the reviewer's observation as an opportunity for a constructive discussion about the objectives and boundaries of trajectory valuation. In our view, improving optimization of the current training objective and improving broader downstream capabilities are related but distinct goals. We will make this distinction clearer in the revised manuscript and ensure that our claims reflect both the positive in-domain results and the limitations revealed by manuscript Table 8.

AuthorsOpenReview ↗

Response to Reviewer G6iL (Part 3/4)

Filtering Rates in GRPO. Thank you for raising this question. We will clarify the filtering rule and report the subgroup rates in the revised manuscript.

Table 2: Overall GRPO filtering rates and accuracy improvements over the shared pretrained model.

MethodClean filter rateClean Δ Acc.Mismatch-20% filter rateMismatch-20% Δ Acc.
DTV1.49%+0.07601.66%+0.0690
DTV-Loo20.54%+0.069322.13%+0.0511

As reported in Table 2, over the full training runs on Clean and Mismatch-20% GSM8K, DTV filters 1.49% and 1.66% of completions, respectively, with Δ\Delta Acc. of +0.0760+0.0760 and +0.0690+0.0690 relative to the shared pretrained model. DTV-Loo filters 20.54% and 22.13%, respectively, with Δ\Delta Acc. of +0.0693+0.0693 and +0.0511+0.0511.

Following Assumption 1, filtering is determined by whether a sample gradient is negatively aligned with the mean gradient of the current batch. Corruption status and advantage sign are therefore not used as explicit filtering criteria, although they can affect the underlying gradient through the GRPO objective and reward.

Table 3: GSM8K filtering breakdown for the first 32 optimizer updates. Each percentage uses all 512 candidate completions in its run as the denominator.

SettingMethodTotal filteredPositive advantageNegative advantageCorrectIncorrect
CleanDTV3.12%1.95%1.17%1.76%1.37%
CleanDTV-Loo25.59%13.09%12.50%11.91%13.67%
Mismatch-20%DTV6.45%4.49%1.95%3.71%2.73%
Mismatch-20%DTV-Loo22.85%10.55%12.30%11.13%11.72%

To further clarify the filtering behavior, we conducted an additional subgroup analysis on Clean and Mismatch-20% GSM8K. Table 3 summarizes the results by advantage sign. Using all 512 candidate completions per run as the denominator, filtered completions with positive and negative advantage accounted for 1.95% and 1.17% for DTV under Clean, and 13.09% and 12.50% for DTV-Loo. Under Mismatch-20%, the corresponding shares were 4.49% and 1.95% for DTV and 10.55% and 12.30% for DTV-Loo. Overall, the filtering rates do not follow a simple rule based on corruption status or advantage sign, consistent with our method using gradient alignment rather than these attributes as explicit filtering criteria.

Reviewer_LpRiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Dynamic Trajectory Valuation (DTV) for RL post-training. In every mini-batch, every training unit is scored by the inner product of its gradient with the mini-batch mean gradient, units with negative scores are removed. A leave-one-out variant, DTV-Loo, removes the unit’s own contribution (the self term). The method rests on a majority-gradient-alignment assumption (Assumption 1), which informs the main part of the method.

Critical strengths
  • The method is simple: filtering-rate hyperparameter , no validation set, no extra rollouts. Clear advantage over GradAlign and LearnAlign.
  • A single rule is applied for GRPO, DPO and PPO.
  • The self/cross-term decomposition is a useful analysis tool.
Critical weaknesses
  • missing prior work on rollout selection: PODS (Xu, Savani, Fang & Kolter, “Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning”, arXiv, April 2025) Compare also with DAPO-style dynamic sampling, and PCGrad, Yu et al. 2020; CAGrad, Liu et al. 2021
  • everything rests on Assumption 1, which is neither tested nor justified. Filtering by agreement with the mean can ermove miority signals including rare correct solutions or negative samples (see Zhu et al. 2025 for importance of negative samples)
  • It is unclear when the two methods DTV and DTV-Loo work and when not (before seeing the test results)
  • empirical evidence is weak, see e.g. GSM8K results, weak / artificial baselines (except for LearnAlign and GradAlign)
  • evaluation setup (e.g. corruption schemes) only in the appendix
Questions for authors
  • How does DTV compare with PODS and DABO dynamic sampling, how with a filter keeping units with gradient norm above median?
  • What fraction of units is filtered, how does that related to advantage, correctness, prompt difficulty in GRPO
  • Can you state under which conditions Assumptions 1 holds, and whether DTV or DTV-Loo should be used (without refering to the test results)?
  • How do the results change with stronger vanilla GRPO, longer training, more AIME seeds, larger models?
  • What are the wall-clock and memory costs compared to vanilla training?
7 comments
AuthorsOpenReview ↗

Response to Reviewer LpRi (Part 1/5)

Thank you, Reviwer LpRi, for reviewing our paper and suggesting additional baselines and asking us to clarify when the batch reference is informative.

Additional Baselines. Thank you for suggesting these valuable comparisons. We agree that comparisons with stronger and more competitive baselines are important for more thoroughly evaluating the effectiveness of our method. Accordingly, we added the five baselines you suggested to our evaluation: PODS, GRPO with DAPO-style dynamic sampling (GRPO+DS), PCGrad, CAGrad, and Grad-norm 50% (retains the top-50% gradient-norm).

Table 1: Final test results on GSM8K under Clean and Mismatch-20%. Acc. and Partial are reported as mean ± std over five training seeds. Δ Acc. is relative to the shared pre-trained model. Bold indicates the best value in each column.

MethodClean Acc. ↑Clean Δ Acc. ↑Clean Partial ↑Mismatch-20% Acc. ↑Mismatch-20% Δ Acc. ↑Mismatch-20% Partial ↑
Pre-trained0.4723±0.00000.00000.4996±0.00000.4723±0.00000.00000.4996±0.0000
Vanilla GRPO0.4828±0.00770.01050.5114±0.00890.4731±0.00940.00080.5001±0.0089
PODS0.4837±0.00630.01140.5119±0.00870.4708±0.0105-0.00150.4996±0.0095
GRPO+DS0.5042±0.01170.03180.5319±0.01050.4907±0.00780.01830.5208±0.0095
PCGrad0.4717±0.0038-0.00060.5010±0.00410.4693±0.0215-0.00300.4963±0.0244
CAGrad0.4984±0.00780.02610.5284±0.00850.4690±0.0200-0.00330.4948±0.0178
Grad-norm 50%0.5592±0.01450.08690.5821±0.01590.5219±0.03940.04960.5491±0.0371
DTV (Ours)0.5483±0.00750.07600.5742±0.00990.5413±0.01390.06900.5660±0.0123
DTV-Loo (Ours)0.5416±0.01410.06930.5647±0.01460.5234±0.01030.05110.5460±0.0076

In Table 1, DTV and DTV-Loo trail only Grad-norm 50% in mean exact-answer accuracy under Clean and rank first and second under Mismatch-20%, respectively. Under Clean, DTV remains competitive with Grad-norm 50% (0.54830.5483 vs. 0.55920.5592). The difference is statistically nonsignificant in a two-sided paired tt-test across five matched seeds (p=0.275p=0.275). Grad-norm's strong performance is plausible when rewards are reliable, since prioritizing large policy gradients can retain completions that provide a strong learning signal. This follows an intuition similar to that of loss-based zero-order valuation, which uses high loss as a proxy for learning potential.

Under Mismatch-20%, DTV retains most of its Clean accuracy and surpasses Grad-norm 50% (0.54130.5413 vs. 0.52190.5219). Reward-rank reversal can yield large gradients that oppose the within-prompt reference, and opposing gradients can have the same norm. The reversal in ranking is consistent with our analysis that signed alignment provides information beyond gradient magnitude, helping identify conflicting completions and preserve the dominant optimization signal using gradients from the current prompt group alone.

For PCGrad and CAGrad, we followed their original procedures where applicable when adapting them from multi-task optimization to GRPO. In our GRPO setting, both adaptations achieve lower mean exact-answer accuracy than DTV and DTV-Loo under both training conditions. PCGrad also trails Vanilla GRPO under Clean.

AuthorsOpenReview ↗

Response to Reviewer LpRi (Part 2/5)

Justification of Assumption 1. The rationale for Assumption 1 is grounded in the fundamental mechanism of gradient-based optimization. Standard optimizers, such as SGD and Adam, rely on aggregating individual sample gradients to obtain a representative optimization direction. This aggregation reflects the principle that the collective gradient can provide a more informative estimate of the underlying descent direction than individual, potentially noisy gradients.

Under this premise, individual gradients negatively aligned with the mini-batch mean counteract the dominant optimization direction, partially canceling the contributions of other gradients during aggregation. By identifying and filtering these conflicting gradients, DTV aims to reduce such cancellation and reinforce the dominant direction captured by the mini-batch gradient. This intuition can also be understood from the perspective of the mini-batch loss. Let gig_i denote the gradient of training unit ii, and let gˉ=1b∑i=1bgi\bar g=\frac{1}{b}\sum_{i=1}^{b}g_i denote the mini-batch mean gradient. For a sufficiently small step size η\eta, the first-order change in the mini-batch loss from an individual update along −gi-g_i is

\approx-\eta\langle g_i,\bar g\rangle.$$ Thus, when $\langle g_i,\bar g\rangle < 0$, the corresponding individual gradient step would locally increase the mini-batch loss. Filtering such gradients therefore removes contributions that oppose descent on the current mini-batch objective, providing a loss-based interpretation of reducing gradient cancellation. Importantly, Assumption 1 concerns whether the aggregated gradient provides a locally informative direction for learning. While gradient aggregation is fundamental to mini-batch optimization, we do not assume that every mini-batch mean perfectly reflects the underlying learning objective. In particular, useful minority gradients may conflict with the batch mean. We will clarify this limitation and emphasize that DTV measures alignment with the current optimization direction rather than guaranteeing the intrinsic utility of every training unit. **Negative Samples.** We thank the reviewer for pointing out this relevant work. We agree with Zhu et al. (2025) that negative samples can provide valuable learning signals. However, the term *negative* refers to different notions in the two approaches. In Zhu et al., negative samples are incorrect responses identified by an external reward signal, and their negative reinforcement can contribute positively to learning. In contrast, DTV defines negative contributions through gradient alignment: a training unit is filtered when its gradient is negatively aligned with the current mini-batch reference direction, regardless of whether its reward or advantage is positive or negative. Moreover, the utility of a training sample is not necessarily static. As model parameters evolve during training, the contribution of a given sample to the current optimization objective may also change, even when its externally assigned reward remains unchanged. This dynamic nature of sample utility is central to the perspective of dynamic data valuation. DTV captures this model-dependent behavior by reassessing gradient alignment throughout training, rather than relying solely on fixed reward-based categories. Therefore, our findings do not contradict Zhu et al. (2025): their work highlights the value of negative reinforcement, whereas ours examines whether an individual training gradient supports the current aggregate optimization direction. We will cite and discuss Zhu et al. in the revised manuscript to clarify this distinction and the connection between the two approaches.
AuthorsOpenReview ↗

Response to Reviewer LpRi (Part 3/5)

When Work, When Fail? The effectiveness of DTV depends on Assumption 1: the mini-batch aggregate gradient should provide a locally informative reference direction. The plausibility of this assumption is supported by the empirical effectiveness of mini-batch gradient-based optimization, which often achieves meaningful learning progress even in the presence of noisy training signals. This suggests that the aggregate gradient can remain informative despite individual noisy or conflicting contributions, providing a reasonable basis for Assumption 1.

When the aggregate gradient captures a meaningful descent direction, DTV can improve optimization by filtering gradients that oppose this direction, thereby reducing gradient cancellation and reinforcing the dominant learning signal.

Conversely, DTV may become ineffective or even detrimental when Assumption 1 is violated. For example, if a mini-batch contains a large proportion of noisy or corrupted samples whose gradients collectively dominate the batch average, the resulting reference direction may deviate substantially from, or even point opposite to, the underlying desirable optimization direction. In this case, DTV may mistakenly filter useful gradients while retaining misleading ones, potentially reinforcing the wrong optimization direction. Similarly, when a mini-batch contains informative minority samples whose gradients disagree with the dominant direction, DTV may discard valuable learning signals. This issue may be particularly pronounced in highly heterogeneous mini-batches, where gradients from different samples or tasks can conflict despite being individually useful.

These cases highlight that DTV does not guarantee the identification of intrinsically harmful samples. Rather, its effectiveness relies on the batch reference being sufficiently informative, as specified in Assumption 1. We will clarify these conditions and limitations in the revised manuscript.

Clarification of the Evaluation Setup. We thank the reviewer for this suggestion. We agree that the key aspects of the evaluation setup, particularly the corruption schemes, should be clearly presented in the main text.

In the revised manuscript, we will provide a high-level overview of the evaluation pipeline in the main text, including the construction of corruption schemes and the corresponding experimental settings, so that readers can understand the overall experimental design without referring to the appendix. We will retain the detailed implementation procedures and technical specifications in the appendix for completeness and reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.