acceptodds
Under review as a conference paper at ICLR 2027

When Fine-Grained Credit Assignment Helps and Hurts: Multimodal Agentic Reinforcement Learning through Video Editing

Abstract

Reinforcement learning (RL) has shown promise for training multimodal agents to reason and use tools. However, it remains unclear when fine-grained credit assignment improves learning and when it fails to improve task performance. Video editing provides a useful testbed for this question, combining multimodal perception, long-horizon planning, and tool use with delayed outcome feedback. We construct four movie-trailer editing task types—Content Selection, Narrative Sequencing, Rough Cut, and Fine Cut—that reflect real-world editing workflows and span different levels of task complexity and interaction horizon. Complex, long-horizon tasks benefit more from fine-grained credit assignment, with a 45.4% relative gain over outcome-only GRPO on Fine Cut tasks averaging more than 60 turns under outcome-only GRPO. On simpler tasks averaging 30–40 turns under the same baseline, outcome-only GRPO remains competitive, with fine-grained credit assignment yielding at most a 10.4% relative gain. Process rewards from an LLM judge initially improve tool use and encourage visual inspection. As training progresses, however, GRPO+PRM with a fixed process-reward weight increasingly exhibits excessive reasoning and redundant tool calls: process rewards continue to rise while task completion rates and outcome scores decline. Decaying the process-reward weight to zero mitigates this reward-hacking behavior while preserving the early gains. Despite dense token-level supervision, on-policy self-distillation (OPSD) yields limited and unstable gains and is highly sensitive to the choice of privileged information. Our analysis points to unreliable teacher guidance at student-visited states and dilution of informative signals over long trajectories.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Agents, Agentic Reinforcement Learning, Credit Assignment, Process Rewards, Reward Hacking, On-Policy Self-Distillation, Video Editing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vg2qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors construct a trailer-editing benchmark from 49 films and their released trailers. Shots are aligned to source footage and grouped into chunks within story beats. Each chunk gives rise to four task types: Content Selection, evaluated by shot F1; Narrative Sequencing, evaluated by a permutation metric; Rough Cut, evaluated using timing and both reference-conditioned and reference-free judge scores; and Fine Cut, covering dialogue, production sound, music and transitions on a fixed picture, evaluated entirely by judges.

After Gemini-based auditing, the dataset contains 716 training tasks from 43 films and 81 validation tasks from six held-out films, for 797 tasks in total.

A Qwen3.5-9B agent is trained for 50 updates. The GRPO runs use 16 tasks with eight rollouts each per update. The paper compares outcome-only GRPO, GRPO with a fixed turn-level process reward from Gemini 3.8 Flash at lambda=0.5, and the same reward with lambda decayed to zero over updates 20-30. An on-policy self-distillation (OPSD) arm instead uses one rollout per task and a frozen copy of the initial policy as teacher, conditioned on either reference answers or Gemini-generated turn suggestions.

Outcome-only GRPO raises the overall score from 0.284 to 0.413. Decayed PRM reaches 0.447, with the largest gain on Fine Cut, 0.293 versus 0.201. Fixed PRM reaches only 0.334. OPSD scores 0.277 with references and 0.044 with suggestions. The paper also reports turn-quality analyses, tool-use and reasoning statistics, and diagnostics of the OPSD gradients and KL signal.

Critical strengths

(1) The testbed gives multimodal agents a substantial editing task with an inspectable final product. Film-disjoint splits and hidden references reduce obvious shortcuts. Validating and rendering deliverables, and instructing judges to score the result rather than the agent's declarations, are sensible choices. Episodes of 60-90 turns with delayed and partly subjective outcomes are a useful setting for studying credit assignment. The benchmark and planned release of its construction workflow are the strongest contribution in my view.

(2) The fixed-PRM failure is documented in enough detail to be useful. Figure 1(c) and Fig. 3 show process reward increasing while outcome reward falls after about update 30. Table 5 reports 165.8 tool calls versus 52.2 for GRPO, more frames viewed, and 44.1% truncation. Decaying the process weight reduces truncation to 6.2% and produces the best overall score. Even if this behavior is specific to the reward design, the measurements give practitioners a concrete failure mode to look for.

Critical weaknesses

(1) The evidence does not isolate a benefit from longer horizons, and its reliability across training runs is unknown.

All methods use a single training seed, 42. The overall difference between 0.413 and 0.447, and the Fine Cut difference between 0.201 and 0.293 on 26 validation tasks, need uncertainty estimates and repeated runs before they can support the central conclusion.

There is also no controlled variation of horizon. Fine Cut is the only task type scored entirely by an LLM judge, with channel weights 0/0.4/0.6. It depends on audio the policy cannot hear and on which the process judge must abstain. It has the lowest base score, 0.152, and requires rendering. Any of these differences could affect the value of the process reward.

Both the process judge and the outcome grader, Gemini 3.1 Pro, are Gemini models. Their agreement could contribute to the gain on the task scored entirely by a judge, independently of horizon. Rough Cut, which is 60% judge-scored, shows no PRM benefit, so judge alignment alone is not a complete explanation either. The current experiments cannot distinguish these possibilities.

The mixing analysis in Table 4 uses the same process judge on the untrained policy and is correlational. To support the horizon claim, I would like at least three seeds for the key arms and a comparison within one task type, such as Fine Cut tasks with different numbers of joins or Rough Cut tasks with different numbers of clips.

(2) The three training schemes change more than the granularity of credit assignment.

Adding a turn-level process reward also adds information from a strong external Gemini judge. Moving to OPSD then removes outcome reward, introduces distillation from a frozen 9B teacher with privileged information, and reduces rollouts per task from eight to one. These comparisons are informative about the chosen recipes but do not place trajectory-, turn- and token-level credit on a common footing.

The OPSD diagnostics reinforce this concern. With reference answers, per-token KL is only 0.0019, suggesting that the answer card barely changes the teacher distribution in a long multimodal context. The median trajectory contains 23 turns and 13.7k generated tokens, with a 131k context limit. There may simply be too little distillation signal in this setup. With suggestions, training collapses and 22 of 50 updates exceed the clipping threshold. Neither result establishes that token-level credit itself is ineffective. A comparison retaining outcome reward, for example OPSD combined with GRPO or a token-level value-based advantage, would be more informative.

The fixed-PRM conclusion also depends on one choice, lambda=0.5. Turns that produce new information earn +1, process advantages are normalized across rollouts at each turn index, and there is no step cost. This can reward continued inspection over finishing the edit. Because the shaping is not potential-based in the sense of Ng et al. (1999), it does not preserve the optimal policy by construction. Smaller fixed weights, such as 0.1 and 0.25, would help establish whether decay is necessary or whether the reported collapse follows mainly from an overly strong process reward.

Questions for authors
  1. How much of the fixed-PRM score loss comes from failure to deliver, and how much from poorer completed edits? Scores conditional on delivery would help distinguish these effects.
  2. Can you report uncertainty over validation tasks and, if feasible, repeat the outcome-only and decayed-PRM training runs? Since tasks share source films, please account for that dependence when interpreting intervals.
  3. Were smaller fixed process weights tested, and was the 20–30 decay window selected solely from the fixed run's training curves?
  4. Is there any human agreement check for the Fine Cut outcome grader or the process judge? Even a modest, clearly described sample would help interpret the judge-based improvements.
3 comments
AuthorsOpenReview ↗

Response to Reviewer vg2q

Dear Reviewer vg2q,

We greatly appreciate your thoughtful feedback and constructive suggestions. Below, we provide detailed responses and clarifications. If our responses address your concerns, we would be grateful if you would consider updating your score.

W1. The evidence does not isolate a benefit from longer horizons, and reliability across training runs is unknown

  • Horizon: we agree and will remove the claim. See the response to Reviewer yMuD, W1.4.
  • Uncertainty over validation tasks: see the response to Reviewer yMuD, Q2, and Q2 below.
  • Training seeds: see the general response (part 1/3). A second seed of GRPO+PRM (decayed λ) scores 0.281 on Fine Cut, against 0.293 for the first.
  • Both judges are Gemini models: the blind human evaluation agrees with the Fine Cut ranking. See general response 1/3.

W2. The three training schemes change more than the granularity of credit assignment

  • Process reward versus its granularity: see the response to Reviewer yMuD, W1.1. With the same PRM and no turn-level credit, the result equals outcome-only GRPO.
  • OPSD: we agree that it also changes the objective and the supervision source. We will present it as a comparison of training recipes (see the response to Reviewer yMuD, W1.1 and W1.2).
  • Reference answers carry little signal: we agree, and confirm it directly. See general response 3/3.
  • Turn-level suggestions: the collapse comes from where the suggestion is inserted. See general response 2/3.
  • OPSD combined with outcome reward, and smaller fixed weights (0.1, 0.25): we did not run these. See the response to Reviewer yMuD, Q3.

Q1. How much of the fixed-PRM loss comes from failure to deliver?

Almost all of it comes from failure to deliver. Over the 324 validation episodes (81 tasks, four repetitions):

Outcome-only GRPOGRPO+PRM (fixed λ)
Overall score0.4130.334
Episodes with a score above zero90%72%
Mean score of those episodes0.4600.462

When the fixed-λ model delivers, its edits score the same as those of outcome-only GRPO. It delivers less often.

Q2. Uncertainty over validation tasks, accounting for shared films

GRPO+PRM (decayed λ) against outcome-only GRPO, paired per task, with intervals from resampling whole films:

TasksPaired difference95% interval
Overall81+0.033[−0.002, +0.067]
Fine Cut26+0.092[+0.031, +0.146]

The Fine Cut difference is clear. The overall difference is positive in five of six films, but its interval touches zero. By-film numbers for Fine Cut are in the response to Reviewer yMuD, Q2. Repeated training runs: see the general response (part 1/3).

Q3. Smaller fixed weights, and how the decay window was selected

  • Smaller fixed weights: not tested. See the response to Reviewer yMuD, Q3.
  • Decay window: selected from the training curves of the fixed-λ run. The decay starts at update 20, where the training outcome reward stops improving, and ends at update 30, where it starts to fall. See the response to Reviewer PFUE, Q4, for the numbers.

Q4. Human agreement check for the Fine Cut grader

See general response 1/3.

As the discussion period ends soon, we hope that our rebuttal has addressed your concerns. If any questions remain, we would be happy to provide additional information or clarification. Thank you for your time.

Dear Reviewer,

Thank you for taking the time to review our paper. As the discussion period draws to a close, we wanted to briefly follow up on our rebuttal. If you have a chance to read our response before the discussion closes, we’d appreciate hearing whether any of your concerns remain. We’d be happy to clarify any points you’d like to discuss further.

Thank you again for your time. The authors

Reviewer_PFUEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies outcome-only GRPO, turn-level LLM process rewards with fixed or decaying AI-generated technical reference | Verify before use 4 weights, and two on-policy self-distillation recipes in four stagewise trailer-editing tasks. The dataset contains 716 training tasks from 43 films and 81 validation tasks from six disjoint films. The main empirical finding is that fixed process rewards increase budget truncation, whereas annealing the process weight improves delivery and achieves the best reported overall score. The largest additional gain over GRPO is on Fine Cut. The OPSD experiments illustrate strong sensitivity to the teacher's privileged information.

Critical strengths

-- The paper reports informative negative results rather than only favorable comparisons.

-- the fixed-PRM truncation rate of 44.1%, compared with 0.3% for GRPO and 6.2% for decayed PRM, makes the completion failure concrete (Table 5).

-- The appendices are unusually explicit about loss reduction, reward rubrics, the top-64-plus-tail KL objective, modality limitations and the distinction between evaluation repetitions and training seeds. Several mechanistic interpretations are appropriately presented as hypotheses.

-- The effective process supervision deserves additional characterization.

Critical weaknesses

The experiments do not isolate horizon or credit granularity. Fine Cut differs from the other tasks in sound-design requirements, rendering, available observations and fully model-based scoring (p.5; Appendix B.2), not just interaction length. Interaction length is itself policy-dependent (Table 12).

The principal Fine Cut improvement needs independent quality validation. All of its outcome score comes from the two Gemini 3.1 Pro judge channels, while Gemini 3.8 Flash participates in process scoring and task filtering. Different judge models are preferable to using the identical model for both purposes, but do not establish agreement with human editorial judgment.

The interpretation of fixed-PRM degradation and the choice of annealing require stronger controls. The rising process reward, falling outcome reward and increased truncation are consistent with proxy overoptimization. However, the current results do not show whether a smaller fixed weight, early stopping, another decay interval or a completion/cost-aware AI-generated technical reference | Verify before use 5 reward would produce comparable benefits.

Questions for authors
  • How consistent are the main gains across training seeds and individual held-out films?
  • What happens to OPSD under matched rollout/token budgets and modest learning-rate controls?
  • Do independent human or model judges reproduce the Fine Cut ranking, including without the zero-item penalty?
  • How do smaller fixed process weights, early stopping and alternative decay windows compare, and which data were used to choose the schedule?
3 comments
AuthorsOpenReview ↗

Response to Reviewer PFUE

Dear Reviewer PFUE,

We greatly appreciate your thoughtful feedback and constructive suggestions. Below, we provide detailed responses and clarifications. If our responses address your concerns, we would be grateful if you would consider updating your score.

W1. The experiments do not isolate horizon or credit granularity

  • Credit granularity: we ran a new control with the same PRM and no turn-level credit. See the response to Reviewer yMuD, W1.1.
  • Horizon: we agree and will remove the claim. See the response to Reviewer yMuD, W1.4.

W2. The Fine Cut improvement needs independent quality validation

We ran a blind human evaluation on all 26 Fine Cut validation tasks. See general response 1/3.

W3. Fixed-PRM degradation and the choice of annealing need stronger controls

See the response to Reviewer yMuD, Q3. Gradual decay is standard practice, and due to limited compute we did not run a smaller fixed weight, early stopping, another decay interval, or a cost-aware reward. Our conclusion is limited to the tested comparison.

Q1. Consistency across training seeds and held-out films

  • Films: see the response to Reviewer yMuD, Q2.
  • Training seeds: see the general response (part 1/3). A second seed of GRPO+PRM (decayed λ) scores 0.281 on Fine Cut, against 0.293 for the first.

Q2. OPSD under matched rollout budgets and learning-rate controls

  • Matched rollouts: OPSD with eight trajectories per task scores 0.271, against 0.277 with one. See the response to Reviewer yMuD, W1.2.
  • Learning rate: we did not run this control.

Q3. Do independent human or model judges reproduce the Fine Cut ranking?

  • Human judges: yes. See general response 1/3.
  • Without the zero-item penalty: the ranking is unchanged. Removing the factor (0.75)^z, the Fine Cut scores are 0.172 (untrained), 0.235 (outcome-only GRPO) and 0.321 (GRPO+PRM, decayed λ), against 0.152, 0.202 and 0.293 with it.

Q4. Smaller fixed weights, early stopping, alternative decay windows, and how the schedule was chosen

  • Other schedules: see the response to Reviewer yMuD, Q3.
  • Choice of schedule: we chose the window from the training curves of the fixed-λ run. Its training outcome reward rises up to update 20 (0.25 over updates 1–10, 0.33 over 11–20), stays flat over updates 21–30 (0.35), and then drops (0.20 over 31–40), while the process reward keeps rising (0.25, 0.42, 0.61 over the same three windows). We therefore start the decay at update 20, where the outcome reward stops improving, and reach zero at update 30, where it starts to fall. The decayed run branches from the fixed-λ checkpoint at update 20.

As the discussion period ends soon, we hope that our rebuttal has addressed your concerns. If any questions remain, we would be happy to provide additional information or clarification. Thank you for your time.

Dear Reviewer,

Thank you for taking the time to review our paper. As the discussion period draws to a close, we wanted to briefly follow up on our rebuttal. If you have a chance to read our response before the discussion closes, we’d appreciate hearing whether any of your concerns remain. We’d be happy to clarify any points you’d like to discuss further.

Thank you again for your time. The authors

Reviewer_yMuDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates learning signals at different granularities for multimodal agent training through movie-trailer editing. The authors construct 797 tasks from 49 films, covering content selection, narrative sequencing, rough cutting, and fine cutting. They compare outcome-only GRPO, GRPO augmented with fixed or decaying process-reward weights, and on-policy self-distillation (OPSD) with two forms of privileged information.

The main findings are that outcome-only GRPO remains competitive on the three shorter-horizon task types, while decaying process rewards provides the largest additional improvement on Fine Cut. Fixed-weight process rewards are associated with excessive tool use, increased budget truncation, and declining outcome scores. OPSD exhibits limited gains with reference answers and substantial degradation with turn-level suggestions.

The paper presents a valuable application setting and informative failure cases. In particular, the divergence between increasing process scores and declining task performance, together with the improvement after removing process guidance, has practical relevance. However, the current experiments do not sufficiently isolate credit-assignment granularity or establish the broader explanations for when finer-grained supervision helps or hurts.

Critical strengths

Strength 1: The research question is important, and the environment captures meaningful practical complexity. Video editing combines visual understanding, tool use, execution constraints, and delayed feedback. The stagewise task design reduces contamination from upstream errors, and splitting training and validation data by film provides a stronger generalization test than randomly splitting clips.

Strength 2: The analysis of process-reward failure goes beyond aggregate scores. The paper reports tool calls, visual inspection, and budget truncation alongside outcome performance. Fixed-weight PRM increases the truncation rate from 0.3% under GRPO to 44.1%, while decaying the weight reduces it to 6.2%. The decayed run branches from the fixed-weight run’s update-20 checkpoint, providing useful evidence that reducing subsequent process guidance mitigates degradation in this configuration.

Strength 3: The appendix provides substantial implementation detail and acknowledges important limitations. The authors document reward normalization, restrictions on evidence available to the process judge, the top-K-plus-tail distillation objective, sampling-budget differences, and behavioral metric definitions. They also explicitly distinguish repeated evaluations from independent training seeds and caution against interpreting reflection markers as measures of reasoning quality.

Critical weaknesses

Weaknesses 1: The comparisons do not sufficiently isolate credit-assignment granularity. GRPO, GRPO+PRM, and OPSD differ in their optimization objectives and supervision sources, in addition to their supervision granularity. They respectively optimize terminal reward, terminal reward combined with local process judgments, and teacher-distribution matching. Appendix C.1 further states that the reward-based methods sample eight trajectories per task, whereas OPSD samples one. The one-rollout design is reasonable for OPSD, but matching update counts does not establish a matched sampling or compute budget.

I appreciate the appendix’s more precise characterization of these experiments as comparisons between specified training recipes. However, this qualification conflicts with the introduction’s statement that only credit assignment changes while all other training settings remain fixed. The results establish differences between these recipes, but do not isolate granularity as the explanation.

The horizon-based interpretation has a related confounding problem. Fine Cut differs from the other tasks in action space, audio requirements, rendering requirements, and evaluation methodology. Interaction length also depends on policy efficiency. Consequently, larger gains on Fine Cut do not establish that longer horizons are the primary reason finer-grained supervision helps.

Weaknesses 2: The OPSD degradation is clearly documented, but its mechanism and scope remain insufficiently established. The substantial performance difference between the two forms of privileged information is worth reporting. However, larger teacher–student KL, larger gradient norms, and concentration of KL in a subset of turns describe disagreement and optimization behavior. They do not establish that teacher guidance is unreliable or that signal dilution causes the observed failure. The qualifications in Section 6.3 and the appendix are appropriate; the abstract should maintain the same level of caution.

Targeted sensitivity analysis would also strengthen this negative result. The paper uses a top-64-plus-tail approximation without evaluating its impact. Related OPSD work specifically examines pointwise clipping of token-level, vocabulary-wise divergence contributions to improve stability; the global gradient clipping reported here addresses a different issue. This motivates a focused stabilization ablation, although it does not establish that clipping would resolve the failures in this environment.

An exhaustive evaluation of every OPSD variant is unnecessary. Nevertheless, at least one diagnostic intervention distinguishing ineffective guidance from an unsuitable optimization configuration would make this part of the paper substantially more informative. Otherwise, the conclusions should remain explicitly limited to the tested configurations.

Questions for authors

Question 1: Could the same PRM judgments be used in a controlled comparison between a trajectory-level auxiliary signal and turn-specific signals? This would help distinguish the benefit of additional supervision from the benefit of assigning it at finer granularity. Alternatively, would the authors narrow the central claims to comparisons between the tested training recipes?

Question 2: Is the Fine Cut improvement consistent across films and independent training runs? Could the authors report paired differences with uncertainty estimates, together with valid-submission rates and quality conditional on valid submission?

Question 3: Starting from the same update-20 checkpoint, how does immediately disabling PRM compare with gradually decaying its weight? Does a smaller fixed weight also avoid the observed degradation?

Question 4: Have the authors directly evaluated the execution quality of teacher suggestions at student-visited states, or tested an OPSD-specific stabilization intervention? What evidence currently distinguishes teacher quality, optimization instability, and signal dilution as explanations?

I find this an interesting study and would consider raising my score if the rebuttal shows the Fine Cut improvement is robust across training runs or films (Question 2) and either adds the controlled comparison in Question 1 or narrows the central claims to the tested training recipes.

6 comments
AuthorsOpenReview ↗

Response to Reviewer yMuD (1/3): Weakness 1

Dear Reviewer yMuD,

We greatly appreciate your thoughtful feedback and constructive suggestions. Below, we provide detailed responses and clarifications. If our responses address your concerns, we would be grateful if you would consider updating your score.

W1.1 The three methods differ in more than granularity

We ran a new control that changes only how the process reward is assigned. It uses the same training configuration and the same PRM as GRPO+PRM (decayed λ). The only change is that the per-turn process signal of a trajectory is averaged and added to the outcome signal of that trajectory, so every turn receives the same value and no turn-level credit is assigned.

MethodProcess rewardCredit for the process rewardOverall (avg@4)Relative to untrained
Untrained Qwen3.5-9Bno–0.284–
Outcome-only GRPOno–0.413+45%
GRPO+PRM (decayed λ), trajectory averageyesnone0.418+47%
GRPO+PRM (fixed λ)yesturn-level0.334+18%
GRPO+PRM (decayed λ)yesturn-level0.447+57%
  • With the same PRM but no turn-level credit, the result is the same as outcome-only GRPO (+0.005, 95% interval [−0.023, +0.033]).
  • Assigning the same process reward to turns adds +0.029 (95% interval [+0.009, +0.052]).
  • So within these runs the gain comes from turn-level credit assignment, not from adding the PRM signal.

We agree that OPSD also differs from the reward-based methods in its objective and its source of supervision. In the revision we will present OPSD as a comparison of training recipes, and restrict the credit-assignment claim to the GRPO and GRPO+PRM runs above.

W1.2 Sampling and compute budgets are not matched

We trained OPSD (reference answers) again with eight trajectories per task, the same as the reward-based methods. Everything else is unchanged.

OPSD (reference answers)Overall (avg@4)
One trajectory per task (reported)0.277
Eight trajectories per task0.271
Untrained0.284

The number of sampled trajectories does not change the OPSD result.

W1.3 The introduction conflicts with the appendix

We agree. See W1.1: we will revise the introduction to say that OPSD is compared as a training recipe, and that only the GRPO and GRPO+PRM runs isolate credit assignment.

W1.4 The horizon interpretation is confounded

We agree. Fine Cut differs from the other task types in several ways besides horizon: its action space, its audio and rendering requirements, and its evaluation. Interaction length also depends on the policy. Our results cannot separate these factors, so we will remove the claim that longer horizons are the reason finer-grained supervision helps.

What the results still show about when fine-grained credit helps. The conditions concern how the process signal is used, not the task type:

  • It has to be assigned per turn. The same PRM signal averaged over the trajectory gives no gain over outcome-only GRPO (0.418 vs. 0.413); assigned per turn it gives 0.447 (see W1.1).
  • Kept throughout training, it can hurt. With a fixed weight the process reward keeps rising while the outcome reward falls, and the final score is 0.334, below outcome-only GRPO (0.413). Decaying the weight to zero avoids this and gives 0.447 (see Q3, and Q1 in our response to Reviewer vg2q for the training curves).

We will state these two conditions in the revision in place of the horizon claim.

AuthorsOpenReview ↗

Response to Reviewer yMuD (2/3): Weakness 2

W2.1 and W2.4 The mechanism is not established; a diagnostic intervention is needed

Please see the general response (parts 2/3 and 3/3). It reports two diagnostic interventions that separate ineffective guidance from an unsuitable optimization configuration:

  • Turn-level suggestions: moving the suggestion to a separate message, with the same optimization configuration, removes the collapse.
  • Reference answers: giving the frozen model the reference answer does not improve the answers it writes itself, so the teacher has nothing to transfer.

Both results concern the two teacher constructions we tested. They show that these constructions gave the student little usable learning signal, not that OPSD is unsuitable for this task. We will state this in the revision.

W2.2 The abstract is stronger than Section 6.3

We agree. We will revise the abstract to match the qualifications in Section 6.3 and the appendix.

W2.3 Sensitivity to the top-64-plus-tail approximation and to pointwise clipping

Top-64 plus tail. Truncating the teacher distribution to its top-k entries is common practice in on-policy distillation, for example the forward_kl_topk loss in verl (default k = 32), MOPD (arXiv 2606.30406), and Rethinking OPD(arXiv 2604.13016). To check that it does not matter here, we computed the distillation loss and its gradient twice on the same token positions, once with top-64 plus tail and once with the full vocabulary (248,320 entries). The positions come from 64 training trajectories of the two OPSD runs.

Top-64 plus tail compared with the full vocabularyReference answersTurn-level suggestions
Token positions compared4,0853,082
Loss, as a fraction of the full-vocabulary loss97.7%95.8%
Cosine similarity of the two gradients (1 = same direction)0.999990.99996
Relative difference between the two gradients0.5%1.0%

The approximation changes the gradient by at most 1%, so it does not explain either failure.

Pointwise clipping. We agree that pointwise clipping of per-entry divergence contributions (Zhao et al., arXiv 2601.18734) addresses a different issue from our global gradient clipping, and we did not train with it. On the same positions, with their threshold of 0.05:

Reference answersTurn-level suggestions
Positions with at least one clipped entry1.7%18.1%
  • Reference answers: almost nothing would be clipped, and the gradient never reaches the global clipping threshold. This run is not unstable, so clipping has nothing to stabilize.
  • Turn-level suggestions: clipping would reduce the size of the large contributions, but not the teacher distribution that produces them. The placement ablation removes the collapse without any clipping (see general response 2/3, "Ablation").

We will add these analyses to the appendix and state that our conclusions are limited to the tested configurations, which do not include pointwise clipping.

AuthorsOpenReview ↗

Response to Reviewer yMuD (3/3): Questions

Q1. Trajectory-level versus turn-specific use of the same PRM judgments

We ran this control (see W1.1). Both runs use the same PRM judgments and the same training configuration. One averages the process signal over each trajectory, and the other assigns it per turn.

MethodOverall (avg@4)
Outcome-only GRPO0.413
GRPO+PRM (decayed λ), trajectory average0.418
GRPO+PRM (decayed λ), turn-level0.447

The trajectory-level signal gives no gain over outcome-only GRPO (+0.005, 95% interval [−0.023, +0.033]). Assigning the same signal per turn adds +0.029 (95% interval [+0.009, +0.052]). We will also narrow the OPSD claims to comparisons between the tested training recipes.

Q2. Consistency of the Fine Cut improvement

Across films: the improvement holds in five of the six validation films. Across independent training runs: a second training seed of GRPO+PRM (decayed λ) scores 0.281 on Fine Cut, against 0.293 for the first (see the general response, part 1/3).

The requested numbers, for GRPO+PRM (decayed λ) against outcome-only GRPO on the 26 Fine Cut validation tasks (four evaluation repetitions each):

Outcome-only GRPOGRPO+PRM (decayed λ)
Mean score0.2020.293
Valid-submission rate78%90%
Mean score given a valid submission0.2590.324
Paired difference per task (95% interval, resampling whole films)+0.092 [+0.031, +0.146]
Tasks that improve / get worse17 / 9

Paired difference by film:

FilmFine Cut tasksPaired difference
Deadpool & Wolverine5+0.188
Heretic4+0.138
Sinners4+0.121
They Will Kill You4+0.072
The Drama5+0.056
Whiplash4−0.040

The interval covers the evaluation tasks, not the training seed. For the second training seed, see the general response (part 1/3).

Q3. Disabling PRM immediately, or using a smaller fixed weight

Gradually reducing the weight of an auxiliary signal is standard practice: Kickstarting (Schmitt et al., arXiv 1803.03835) lowers the weight of its teacher loss over training, Scheduled Sampling (Bengio et al., arXiv 1506.03099) decays the use of ground-truth inputs, and SGDR (Loshchilov & Hutter, arXiv 1608.03983) anneals the learning rate. We followed this practice. Due to limited compute, we did not run the two variants (disabling PRM immediately, and a smaller fixed weight). Our conclusion is limited to the tested comparison: decaying the weight to zero (0.447) is better than keeping it fixed (0.334).

Q4. Execution quality of teacher suggestions, stabilization, and competing explanations

  • Did we evaluate the execution quality of the suggestions at student-visited states? Yes, for the first action (general response 2/3). Executing the suggested action scores 0.412, against 0.421 without it. The suggestions are reasonable in themselves, but following them does not raise the score.
  • Did we test an OPSD-specific stabilization intervention? Not in training. See W2.3 ("Pointwise clipping") for an offline analysis.
  • What distinguishes the three explanations?
ExplanationEvidence
Teacher qualityWith the reference answer, the model's own answers do not improve (0.36 vs. 0.36; see general response 3/3). Executing the suggested action does not raise the score (0.412 vs. 0.421; see general response 2/3).
Optimization instabilityMoving the suggestion to a separate message removes the collapse under the same optimization configuration (see general response 2/3, "Ablation").
Signal dilutionWe withdraw this explanation (see W2.2 on revising the abstract and Sec. 6.3).

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (3/3): Why OPSD with reference answers failed

OPSD (reference answers), 0.277 vs. 0.284 untrained: with the reference answer, the teacher is still not a better policy at the states the student visits.

  • We selected 300 trajectories that the student generated during training. In each one we inserted the reference answer before the student's longest reasoning turn and had the frozen initial model generate the answer from there. Mean outcome score: 0.36 with the reference answer vs. 0.36 without it.
  • The next-token distribution of the model with the reference answer (the teacher) is almost the same as that of the model without it (the untrained student), on the same trajectories (3.4M token positions):
Change in the teacher's next-token distribution from the reference answer0.0012 nats per token
Same, from a wrong answer of equal length0.0012 nats per token
Positions where the teacher's top-1 token is unchanged99.0%
Teacher's probability of the student's wrong ID, at the 1,288 IDs the student got wrong93–95%, with or without the reference answer
  • Summary of the mechanism: giving the model the reference answer barely changes its distribution and does not raise its score. The model cannot necessarily make use of an answer that is simply handed to it, so this teacher has nothing to teach the student. This is a property of the reference-answer teacher we tested, not evidence that OPSD cannot work on this task.

We will rewrite Sec. 6.3 and the abstract accordingly, and state that the two variants also differ in where the privileged information is inserted.

AuthorsOpenReview ↗

General Response (2/3): Why OPSD with turn-level suggestions failed

2. Why OPSD failed

Both failures come from the teacher signal at student-visited states, not from optimization instability or signal dilution. The evidence is probes of the frozen teacher on the student's own training trajectories (forward passes, and answers the teacher writes itself), plus one single-factor training run. This comment covers turn-level suggestions, and the next one covers reference answers.

OPSD (turn-level suggestions), 0.044: the inserted suggestion makes the teacher's distribution end reasoning early.

  • The suggestion is inserted right after the assistant message header, which already opens the model's reasoning block. On 32 trajectories sampled from the initial policy (the untrained Qwen3.5-9B), the teacher with the suggestion inserted this way begins 87% of the turns with </think>, the token that closes the reasoning block (mean first-token probability). With the suggestion as a separate message or absent, this is below 0.01% in every turn.
  • The loss is dominated by this one token. The first token of each turn is 0.17% of the tokens but 46% of the distillation loss at update 1, so the student is trained mainly to copy it.
  • The student learns this distribution. Turns that begin by closing the reasoning block go from 0% at update 5 to 100% at update 8. At evaluation every task type ends after about 12 turns (untrained: 15 to 87), and 85% of episodes have no valid deliverable.
  • Ablation: we moved the suggestion out of the assistant turn into a separate user message placed right before it, and ran OPSD again with the same configuration. With this placement no turn starts by closing reasoning or ending. The result still does not beat the untrained model (0.208 vs. 0.284, one validation pass at update 50), because the trained model now does not stop:
UntrainedAblation run
Mean turns per episode4271
Episodes reaching the 100-turn limit12%37%
Trajectories that call the validation tool0%57%
Turns after the deliverable is first written (Rough Cut, median)466

The cause is in the suggestions: at the turn where the student finishes, the suggestion is to continue in 77% of cases (first six batches of the reported run), mostly by calling the validation tool. The student never sees the suggestion, so it learns to keep validating and rewriting instead of stopping.

Do the suggestions help when they are followed? The suggestions are reasonable in themselves. For example, they often ask the model to validate its output once more. The results above indicate that the student either over-applies them (it keeps validating instead of stopping) or cannot make good use of them. We therefore ran one more test. On 16 tasks we replaced the first action of the episode with the suggested action and let the same frozen GRPO-trained policy finish the episode, four times per condition.

First actionScore
Chosen by the untrained model, without a suggestion0.421
The suggested action, executed as given0.412

Executing the suggested action does not raise the score (difference −0.010, 95% interval −0.032 to +0.012). This test covers the first action only.

These results concern the two ways of inserting suggestions that we tested. They do not show that OPSD cannot work on this task.

AuthorsOpenReview ↗

General Response (1/3): Human evaluation and second training seed

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their thoughtful feedback. We are encouraged that reviewers found the video-editing testbed a meaningful setting for studying credit assignment (yMuD, vg2q), the analysis of the fixed-weight process-reward failure concrete and useful (yMuD, PFUE, vg2q), and the negative results and implementation details informative (yMuD, PFUE). Below, we summarize the concerns shared by multiple reviewers and the new experiments we conducted. Individual responses follow. We will include these results in the revised manuscript, and we are glad that the main findings on process rewards hold under the new controls.

1. Human evaluation

Task. Fine Cut, all 26 validation tasks from the six held-out films. For each task, raters compared the edit rendered by the untrained Qwen3.5-9B with the edit from GRPO+PRM (decayed λ) at update 50. The two videos were shown blind as A and B in random order, and the rater chose "A better", "B better" or "tie". Six human raters took part, and each task was judged by three of them, 78 judgments in total.

Result.

Trained betterUntrained betterTie
Judgments (78)431223
Tasks by majority vote (26)1862

The trained model is preferred in each of the six films.

Why raters preferred the trained model. Every judgment came with a written reason. Of the 43 judgments for the trained model, 42 give a usable reason:

ReasonJudgments
Untrained edit has no audio or loses the dialogue12
Untrained edit has stray black frames, freezes or flicker inside the cut10
Untrained render is broken (one frozen frame, or ends after about a second)9
Trained edit has better sound design (continuous music bed, effects in sync, normal level)8
Trained edit follows the scene brief more closely (title card, date, shot order)3

Second training seed. Due to limited compute, we repeated only our best-performing method, GRPO+PRM (decayed λ), with a second training seed. The configuration is unchanged. Scores are avg@4 on the 81 validation tasks:

Content SelectionNarrative SequencingRough CutFine CutOverall
Outcome-only GRPO, seed 42 (reported)0.8210.5000.3560.2020.413
GRPO+PRM (decayed λ), seed 42 (reported)0.8040.5520.3480.2930.447
GRPO+PRM (decayed λ), seed 43 (new)0.7610.5210.3380.2810.427

The Fine Cut score is close across the two seeds (0.293 and 0.281), so the Fine Cut improvement reproduces. The remaining difference in the overall score (0.447 vs. 0.427) is variation between training seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.