- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies a method that identifies and filters detrimental trajectories during the training of a reinforcement learning algorithm. Inspired by techniques from supervised learning, their score, called dynamic trajectory valuation (DTV), essentially identifies samples in a batch whose induced gradient differs from the overall gradient of the mini-batch.
Critical strengths
- Simple and attractive idea to filter detrimental trajectories during training which is compatible with any RL algorithm
- Well explained and motivated method (although I find that the experiments lacking both)
Critical weaknesses
- Design of experiments: In its current form, the experiments do not convince me enough that the proposed method is beneficial for general reinforcement learning as claimed in the introduction. At best, the RL results for PPO (Appendix D) are early signals that the method could be promising, but their scope is too narrow to derive general statements here. It seems to me that some of the experiments consider supervised fine-tunings, which I would not regard as reinforcement learning and where I do not find it surprising that DTV provides improvements. However, I could also be a misunderstanding at my side as the paper seems very vague at these places (see below).
- Mixed evaluation schemes and protocols: It seems to me that different evaluation schemes and application regimes are mixed. Moreover, it is not made mathematically precise enough what exactly the "post-training RL" setting is. Section 4.1 treats a mismatched LLM post-training setting without explaining what *mismatched" means and why its important for benchmarking DTV. On the other hand, the PPO experiments train policies from scratch in rather simple tasks (only two MiniGrid tasks). That makes it hard to assess which effects come from the training methodology, the RL algorithm, or the filtering method.
- Presentation: The presentation of the experiment section must be improved substantially in my opinion. I had a hard time finding essential details. For instance, I could not found which LLM was used in the GSM8K experiments in Section 4.1 and whether DTV was used in GRPO or in a LoRA fine-tuning as said at the end of Section 4.1. Moreover, Section 4 is cluttered with seemingly unnecessary details that essentially restating the numbers shown in the tables instead of providing intuition and interpretation of the results. I miss structured research questions the experiments aim to answer.
Questions for authors
- What does mismatched mean and why is this setting important for benchmarking DTV? Why are the ACCs reported in Section 4.1 lower than other reported baselines for GSM8K and where does the pretrained model come from?
- Which of the experiments are RL trainings and which are supervised fine-tunings? What is the precise setting of post-training RL the paper addresses?
- Why is PPO not tested in a post-training fashion? Why is GRPO only tested in a post-training fashion?
5 comments
Response to Reviewer vmLX (Part 2/3)
The Role of PPO Experiments. We agree with the reviewer that the original PPO experiments, which train policies from scratch on relatively simple control tasks, do not constitute LLM post-training. This distinction is precisely why we placed these experiments in the appendix: they are intended as supplementary evidence exploring whether DTV's valuation principle can also be applied in conventional RL, rather than as primary evidence for our post-training claims.
To broaden this supplementary evaluation, we have additionally included PPO experiments on HalfCheetah-v5 (Table 7), trained from random initialization, and windy LunarLander-v3 (Table 8), where policies adapt from a shared checkpoint pretrained in the no-wind environment. The latter examines adaptation from an existing policy, although it remains distinct from LLM post-training. Within each task, all compared methods share the same PPO algorithm, initialization protocol, and interaction budget.
We will make the supplementary role of these experiments more explicit.
Table 7: HalfCheetah-v5 PPO from scratch. Evaluation return as mean ± std across five training seeds, with 1,000 evaluation episodes per checkpoint.
| Round | Vanilla PPO | IIF | DTV | DTV-Loo |
|---|---|---|---|---|
| 50 | 550.28 ± 101.98 | 960.09 ± 430.42 | 571.66 ± 144.37 | 238.85 ± 224.31 |
| 100 | 1320.85 ± 270.07 | 1829.94 ± 798.38 | 1291.91 ± 278.22 | 1525.23 ± 516.75 |
| 150 | 1755.40 ± 775.94 | 2486.67 ± 1123.71 | 1616.67 ± 500.49 | 2464.70 ± 471.34 |
| 200 | 2036.72 ± 1063.51 | 3096.76 ± 1054.58 | 1909.01 ± 805.32 | 2952.97 ± 667.38 |
| 250 | 2184.36 ± 1152.16 | 3556.64 ± 1157.38 | 2225.46 ± 1017.67 | 3521.53 ± 587.61 |
| 300 | 2336.50 ± 1309.27 | 3696.94 ± 1166.14 | 2597.53 ± 1131.27 | 3824.20 ± 582.17 |
| 350 | 2379.19 ± 1251.78 | 3842.51 ± 1204.26 | 2803.53 ± 1190.18 | 3978.12 ± 488.29 |
| 400 | 2474.12 ± 1420.84 | 3964.54 ± 1249.75 | 2941.41 ± 1180.50 | 4180.55 ± 454.66 |
| 450 | 2219.15 ± 1816.28 | 3971.38 ± 1254.96 | 3135.55 ± 1307.39 | 4363.13 ± 467.57 |
| 500 | 2445.22 ± 1646.13 | 4082.45 ± 1304.56 | 3319.91 ± 1334.56 | 4514.31 ± 399.89 |
Table 8: Windy LunarLander-v3 PPO post-training. Evaluation return as mean ± std across five training seeds, with 1,000 evaluation episodes per checkpoint.
| Round | Vanilla PPO | IIF | DTV | DTV-Loo |
|---|---|---|---|---|
| 0 | 153.73 ± 0.00 | 153.73 ± 0.00 | 153.73 ± 0.00 | 153.73 ± 0.00 |
| 20 | 181.10 ± 19.15 | 150.26 ± 18.89 | 189.05 ± 10.17 | 180.78 ± 21.11 |
| 40 | 187.94 ± 22.07 | 174.91 ± 10.16 | 194.65 ± 26.78 | 176.70 ± 29.62 |
| 60 | 187.29 ± 33.66 | 178.86 ± 64.42 | 198.41 ± 18.20 | 187.80 ± 16.55 |
| 80 | 193.32 ± 14.78 | 187.48 ± 22.57 | 216.40 ± 25.14 | 191.32 ± 35.86 |
| 100 | 203.73 ± 9.83 | 187.03 ± 31.10 | 222.70 ± 41.63 | 198.04 ± 42.15 |
| 120 | 211.49 ± 13.52 | 220.76 ± 27.50 | 242.96 ± 17.39 | 211.11 ± 18.73 |
| 140 | 224.07 ± 41.45 | 211.21 ± 38.49 | 254.61 ± 12.20 | 229.99 ± 13.42 |
| 160 | 226.59 ± 27.94 | 199.83 ± 17.62 | 245.61 ± 14.80 | 231.05 ± 17.46 |
| 180 | 213.79 ± 16.23 | 197.51 ± 23.30 | 245.55 ± 15.40 | 235.52 ± 17.91 |
| 200 | 226.20 ± 31.56 | 228.38 ± 12.29 | 245.28 ± 12.07 | 232.78 ± 22.40 |
Response to Reviewer vmLX (Part 3/3)
Mismatch Setting and Its Motivation. We thank the reviewer for pointing out that the mismatch protocol should have been defined and motivated in the main text.
For GSM8K, Mismatch-20% means that approximately 20% of training prompt groups are selected, and the observed completion rewards within each selected group are reassigned in reverse rank order. This preserves the group's reward values while disrupting their correspondence with the generated completions, thereby introducing unreliable advantage estimates into GRPO training.
For UltraFeedback, the mismatch settings instead perturb selected preference pairs by replacing responses with responses from other prompts and reversing their preference roles. These are distinct corruption protocols, reflecting the different forms of training feedback used by GRPO and DPO.
Our motivation is to examine whether DTV remains useful when the learning signal derived from training feedback is imperfect. These settings provide controlled stress tests of feedback reliability; we do not claim that they reproduce every form of noise encountered in practice. Importantly, evaluation is performed on clean, uncorrupted data.
We will move the definitions and motivation of these mismatch protocols into the main experimental section and clarify their differences.
Presentation and Readability. We sincerely apologize for the difficulty the reviewer experienced in reading and understanding our manuscript. We recognize that it is our responsibility as authors to communicate our methodology and experimental design clearly, and that readers should not have to reconstruct essential details from different sections and appendices. We appreciate the reviewer for bringing these presentation issues to our attention.
To address them, we will reorganize the manuscript around a clearer division between the main text and the appendix. The main text will explain the overall methodological and experimental pipeline, including the role of DTV in each optimization algorithm, the training and evaluation stages, and the purpose of each experimental setting. This will allow readers to understand what is being tested, why it is being tested, and how the reported results should be interpreted.
The appendix will provide the detailed experimental specifications needed for reproducibility, including implementation details, hyperparameters, model configurations, data processing, and evaluation protocols.
Our goal is to make the main paper self-contained and easy to follow, while retaining sufficient technical detail in the appendix for readers who wish to reproduce or examine the experiments more closely. We thank the reviewer again for these constructive suggestions, which will help us substantially improve the clarity of the manuscript.
Response to Reviewer vmLX (Part 1/3)
Thank you, Reviewer vmLX, for asking how our training and evaluation settings differ and how broadly the PPO results apply.
Clarification of RL Training Settings and Generality. We thank the reviewer for raising this concern. We agree that the distinction between supervised fine-tuning, reinforcement learning, and preference optimization should be made more explicit in the manuscript. However, we would like to clarify that our experiments are not primarily based on supervised fine-tuning.
Specifically, our GSM8K and AIME experiments employ online GRPO reinforcement learning, where the policy generates responses, receives reward-based feedback, and is updated using group-relative advantages. On GSM8K, we use LoRA for parameter-efficient policy updates, but LoRA only specifies which parameters are trained; the underlying optimization algorithm remains GRPO rather than supervised fine-tuning. On AIME, we perform full-parameter GRPO training. Furthermore, our PPO experiments involve online interaction with the environment and policy-gradient optimization, constituting standard reinforcement learning.
For UltraFeedback, supervised fine-tuning is used only to obtain a shared initialization checkpoint. DTV is applied during the subsequent DPO stage, not during SFT. We include DPO as an additional offline preference-optimization setting to investigate the applicability of gradient-based trajectory valuation beyond online RL, rather than presenting it as equivalent to online reinforcement learning.
Regarding generality, we agree that no finite collection of benchmarks can establish universal effectiveness across all RL settings. Nevertheless, our experiments cover two distinct online RL algorithms, GRPO and PPO, across language-model reasoning and control tasks, with additional evidence from DPO. We believe these results provide evidence that DTV is not restricted to supervised fine-tuning or a single RL algorithm, while acknowledging that broader evaluations would further strengthen this conclusion.
We will revise the experimental section to explicitly distinguish the training objectives, initialization procedures, and optimization algorithms, and will appropriately qualify our claims about general applicability.
GSM8K Checkpoint and Evaluation. Thank you for pointing out this important detail; we are happy to clarify it. We initialize all methods from the official
google/gemma-3-1b-itcheckpoint. Before GRPO, this model achieves 47.23% accuracy in our evaluation setup. The lower absolute ACC compared with some published GSM8K results mainly reflects differences in backbone choice, parameterization, generation budget, and evaluation protocol. In particular, our evaluation uses a 768-token generation cap, while reported results for the same model may use substantially longer generation budgets. All methods in our comparison share the same initialization and evaluation procedure, so our analysis focuses on relative improvements under a controlled setting rather than direct comparison across different experimental configurations.Mixed Evaluation Schemes and Protocols. We appreciate the reviewer's observation that our experiments involve different training regimes and evaluation protocols. We agree that their respective purposes were not sufficiently distinguished in the original submission.
Our primary focus is post-training optimization of pretrained language models. Specifically, GSM8K and AIME employ online GRPO, while UltraFeedback employs offline DPO following a shared SFT initialization. In both cases, DTV is applied during the subsequent optimization stage, rather than during supervised fine-tuning.
The purpose of including different optimization algorithms is to investigate whether the same gradient-based trajectory-valuation principle can be applied across distinct post-training objectives, not to suggest that these algorithms or evaluation protocols are equivalent.
Importantly, we assess the contribution of DTV through within-setting comparisons: the underlying algorithm, initialization, training budget, and evaluation protocol are held fixed across the compared methods, with trajectory filtering as the intervention. We do not use cross-algorithm comparisons to attribute performance differences to DTV. We will clarify this experimental structure and the scope of each comparison in the revised manuscript.