acceptodds
Under review as a conference paper at ICLR 2027

When Rewards Disagree: Reward Weighting for Token-Efficient Video Reasoning

Abstract

Human perception interprets video dynamics by constructing structured mental representations of entities, actions, and temporal boundaries before engaging in high-level reasoning. In contrast, current Video-LLMs typically rely on unstructured Chain-of-Thought (CoT), where crucial visual cues are obscured by verbose narration. While recent efforts ground reasoning in intermediate visual evidence, they rely on task-specific schemas that fail to generalize across diverse video understanding tasks. To address this, we introduce Structured Event Evidence, a unified, temporally ordered schema designed for general video reasoning. However, post-training a model to extract structured evidence while reasoning introduces a severe multi-objective optimization bottleneck, as candidate rollouts must simultaneously satisfy schema compliance, task accuracy, and reasoning budget. Standard reinforcement learning (RL) algorithms attempt to balance these objectives via static scalarization, enforcing a rigid accuracy–length trade-off across training and obscuring fine-grained reward signals. We resolve this by proposing Group-Relative Advantage Balancing (GRAB), which dynamically computes Pareto-optimal objective weights within each rollout group via a minimum-norm criterion over centered rewards. Operating strictly in reward space, GRAB incurs virtually zero computational overhead (<0.002% relative runtime to GRPO) and eliminates brittle weight tuning. Supported by our new PARSE-60K dataset (60K annotations across 35K videos) and a three-stage progressive curriculum, we train PARSE-4B. Across temporal grounding and video question-answering benchmarks, GRAB generates 13% to 39% fewer tokens while maintaining competitive accuracy with exhaustively hand-tuned static weights—without requiring hyperparameter weight sweeps.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, Video Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6p78OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Addressing the issues of redundancy in free-text Chain-of-Thought (CoT) and the lack of grounding for temporal evidence in video-based reasoning, this paper proposes using structured event evidence as a task-agnostic intermediate representation. It introduces GRAB, a method that dynamically balances multiple objectives, such as evidence formatting, task accuracy, and reasoning efficiency, within a unified reward framework. Combined with a progressive training curriculum and a custom-constructed dataset, the approach achieves superior performance over baselines on temporal localization and video question-answering benchmarks, while also enhancing interpretability. Overall, the motivation is clear, the method is lightweight, and the experimental evaluation is systematic; however, there remain concerns regarding the scale of data and models, generalization across patterns, the robustness of the reward design, and the consistency of evaluation metrics.

Critical strengths

The motivation is well-articulated, accurately identifying the drawbacks of unstructured video CoT, such as verbosity and a tendency to drift away from critical temporal evidence.

Structured event evidence organizes entities, actions, scenes, and event boundaries into a unified intermediate representation, offering both interpretability and verifiability.

Critical weaknesses

The scale of both the training data and the primary model is limited, leaving the generalization of the evidence-based approach across larger models, diverse domains, and open-world scenarios under-explored.

The use of a fixed-duration assumption for event segmentation limits the precision of boundary localization; the challenge of adaptive or multi-scale event decomposition remains unaddressed.

The evidence schema is centered on individuals and their actions, which leads to degraded information representation in videos that lack human subjects or are not focused on human activity.

Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Author Rebuttal

Thank you for your careful review and constructive feedback. Following your suggestions, we have updated paper, and we provide further analyses to address each concern below.

R1. Scale and generalization. Thank you for raising this point. Our main 4B experiments use full-parameter RL training. Although our computational budget does not support the same setting at 8B, we evaluate four RL methods on the 8B model using LoRA and half the RL data. All methods share the same settings. CTL denotes Charades-TimeLens:

8BVideoMMEVideoMME tokensCTL mIoUCTL tokensTempCompassEventBench
w/o Evidence61.583,02041.6158564.6264.31
w/o Thinking58.378,61038.9271063.8863.19
GRPO64.814,45243.7387565.7964.87
Fixed Weight (Accuracy)67.214,37646.2595066.0865.62
Lagrangian (Length)64.953,86043.6059065.7064.92
GRAB66.983,80546.8161266.2366.10

We believe our results on public benchmarks, including ETBench, VideoMME, MLVU, and NExT-GQA, have supported generalization across diverse video domains and tasks. Moreover, we further examine performance across the six VideoMME domains below:

VideoMME domainQwen3-VL-4B-InstructPARSE-4B
Knowledge65.869.9
Film & Television64.368.7
Sports Competition58.663.3
Artistic Performance65.068.8
Life Record62.266.8
Multilingual63.367.5
All63.467.7

R2. Event segmentation. The 10–30 s range is used only to construct training annotations. Shot detection identifies boundaries from the video content, and half of them are reviewed by human annotators. At inference, segment durations are not restricted to the 10–30 s range. A substantial proportion of generated segments fall outside the 10–30 s range on both benchmarks, showing that this range does not constrain segment durations at inference.

Charades-TimeLensActivityNet-Captions
Number of videos1,3124,885
Mean video duration29.3 s118 s
Number of valid generated segments8,43478,473
Segments outside 10–30 s4,512 (53.5%)34,600 (44.1%)
Median ratio of the longest to shortest segment within a video2.13.4

We next assess boundary precision using [email protected], which applies the strictest of the three IoU thresholds. On these two benchmarks, this metric shows the largest relative gains over Qwen3-VL-4B-Instruct in Table 1. The comparison with GRPO uses the same segmentation pipeline.

MethodCharades-TimeLens [email protected]ActivityNet-Captions [email protected]
Qwen3-VL-4B-Instruct18.421.6
GRPO21.824.7
PARSE-4B (GRAB)24.628.1

To examine whether this improvement extends to short moments, we divide Charades-TimeLens into three groups by annotated moment duration.

Annotated moment durationSamplesShare of samplesQwen3-VL-4B-InstructGRPOPARSE-4B (GRAB)
Shorter than 10 s2,05261%15.218.320.8
10–20 s97529%22.126.129.4
Longer than 20 s33610%27.531.033.8
Overall3,363100%18.421.824.6

The gains also extend to annotated moments shorter than the 10 s lower bound used to construct training segments.

R3. Videos without people. The scene, object, camera, and event_caption fields in the schema do not depend on people. When no person appears, person and human_action are set to None, while the remaining fields describe the video as usual.

To evaluate this case, we group VideoMME videos using PARSE-4B's evidence. A video is assigned to the no-person group when every person field is None. Both models are evaluated on the same groups.

VideoMME subsetQwen3-VL-4B-InstructPARSE-4B
With people63.167.5
Without people65.068.6
All63.467.7

We will continue to refine the schema to cover a broader range of cases.

Dear Reviewer 6p78,

Thank you again for your thoughtful feedback. We hope that our response has helped address your concerns. Please do not hesitate to let us know if there are any remaining questions or issues that we could further clarify.

If you feel that your concerns have been sufficiently addressed, we would sincerely appreciate it if you might kindly consider revisiting your overall assessment.

Dear Reviewer 6p78,

As the discussion period is ending soon, please do not hesitate to let us know if there are any remaining concerns that we could further clarify. Thank you very much for you reviews.

Reviewer_ZEVEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Group-Relative Advantage Balancing (GRAB), which dynamically adjusts reward weights for each rollout group instead of using fixed weights in GRPO. The goal is to balance accuracy, formatting, and reasoning length in video understanding tasks. The authors also introduce a structured video reasoning framework and the PARSE-60K dataset. Experiments show that GRAB reduces generated tokens by 13–39% while maintaining competitive accuracy.

Critical strengths
  • Practical approach to multi-objective RL. GRAB addresses the limitations of fixed reward weighting with a simple, computationally efficient method that avoids extensive weight tuning.

  • Solid empirical evaluation. Experiments across multiple video benchmarks demonstrate favorable accuracy–efficiency trade-offs compared with GRPO, DAPO, GDPO, and fixed-weight baselines.

Critical weaknesses
  • Limited theoretical justification. The minimum-norm optimization operates in reward space rather than gradient space. It is unclear whether this produces Pareto-optimal policy updates. Additionally, the claimed scale invariance is questionable because the final advantages use raw rewards.

  • Insufficient evidence of superiority over tuned fixed weights. GRAB achieves similar accuracy to the strongest fixed-weight baseline while generating fewer tokens. However, the static-weight sweep is limited, making it unclear whether GRAB consistently achieves a better accuracy–length Pareto frontier.

  • Limited effective adaptivity of reward weighting. When only two reward objectives are active, GRAB assigns equal weights (0.5, 0.5), regardless of their correlation. Therefore, meaningful adaptive weighting primarily occurs when three or more objectives are active. The paper does not report how frequently this happens during training, raising concerns about how much of the improvement actually comes from dynamic reward balancing.

Questions for authors
  • What theoretical guarantees connect the proposed reward-space optimization to Pareto-optimal policy updates?

  • How does GRAB maintain scale invariance when final advantages are computed using raw rewards?

  • Can the authors provide a more comprehensive accuracy–length Pareto frontier against carefully tuned fixed-weight baselines?

  • What proportion of rollout groups have three or more active reward objectives during training? How much of GRAB's performance improvement comes from these groups?

3 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

Thank you so much for your deep review and feedback! We have updated paper, and we address each of your concerns in detail below.

Q1. Theoretical justification. GRAB computes weights from rewards, and these weights also determine how the policy gradients of the objectives are combined. Under the conditions below, reward correlations also describe how closely the empirical objective gradients agree, so GRAB's minimum-norm subproblem corresponds to a minimum-norm problem over standardized objective gradients in an empirical inverse-Fisher metric.

For a group of GG responses to input qq, let δi,m=ri,m−G−1∑j=1Grj,m\delta_{i,m}=r_{i,m}-G^{-1}\sum_{j=1}^{G}r_{j,m} be the centered reward of response ii for objective mm, and let si=∇θlog⁡πθ(oi∣q)∣θ=θolds_i=\left.\nabla_\theta\log\pi_\theta(o_i\mid q)\right\rvert_{\theta=\theta_{\mathrm{old}}} be its log-probability gradient at the rollout policy. The gradient estimate for objective mm is gq,m=G−1∑i=1Gsiδi,mg_{q,m}=G^{-1}\sum_{i=1}^{G}s_i\delta_{i,m}. Since the rewards are group-centered, Eq. (4) gives Ai=cq∑mαq,m∗δi,mA_i=c_q\sum_m\alpha_{q,m}^*\delta_{i,m}, where cq=(σA,q+ϵ)−1>0c_q=(\sigma_{A,q}+\epsilon)^{-1}>0 is the group normalization factor, σA,q\sigma_{A,q} is the group standard deviation of the raw weighted centered rewards, and ϵ>0\epsilon>0. Evaluating the gradient at the policy parameters that generated these responses and holding the computed advantages fixed during backpropagation gives the group-averaged reward-ascent gradient, excluding KL regularization:

=c_q\sum_m\alpha_{q,m}^* \left(\frac1G\sum_i s_i\delta_{i,m}\right) =c_q\sum_m\alpha_{q,m}^*g_{q,m}. \tag{W1.1}$$ At the rollout policy, the importance ratios equal one, so clipping does not alter this first-order expression. To connect reward correlations with objective-gradient alignment, we introduce an empirical inverse-Fisher metric motivated by natural policy gradient [2]. Let $S=[s_1,\ldots,s_N]\in\mathbb R^{P\times N}$ contain the score vectors of $N$ responses sampled from the rollout policy, including the $G$ responses in the current group, where $P$ is the number of parameters. Define $\hat F=N^{-1}SS^\top$, and let $\hat F^+$ denote its Moore–Penrose pseudoinverse. Assuming that $S$ has full column rank, $S^\top\hat F^+S=NI_N$, where $I_N$ is the $N$-dimensional identity matrix. Thus, $s_i^\top\hat F^+s_j$ equals $N$ when $i=j$ and zero when $i\ne j$. For active objectives $m$ and $k$, substituting this identity into the objective-gradient inner product eliminates cross terms between different responses, leaving a sum of within-response products of centered rewards: $$\begin{aligned} g_{q,m}^\top\hat F^+g_{q,k} &=\frac1{G^2}\sum_{i,j=1}^{G} \delta_{i,m}\delta_{j,k}\\,s_i^\top\hat F^+s_j\\\\ &=\frac N{G^2}\sum_{i=1}^{G}\delta_{i,m}\delta_{i,k} =\frac NG\sigma_{q,m}\sigma_{q,k}C_{q,mk}. \end{aligned} \tag{W1.2}$$ Here, $\sigma_{q,m}$ is the group reward standard deviation for objective $m$, and $C_{q,mk}$ is the reward correlation coefficient between objectives $m$ and $k$. We use $\sigma_{q,m}^2=G^{-1}\sum_i\delta_{i,m}^2$ and $C_{q,mk}=\sum_i\delta_{i,m}\delta_{i,k}/(G\sigma_{q,m}\sigma_{q,k})$, with positive standard deviations for active objectives. This agrees with the normalization $\hat D_q^\top\hat D_q=GC_q$ in the paper. Under this metric, a higher positive reward correlation corresponds to a smaller angle between the objective gradients, while a negative reward correlation gives a negative gradient inner product. MGDA [1] chooses weights that minimize the squared norm of the combined gradient. Expanding the two sums and substituting Eq. (W1.2) gives $$\begin{aligned} &\left\lVert\sum_m\alpha_m\frac{g_{q,m}}{\sigma_{q,m}}\right\rVert_{\hat F^+}^2\\\\ &=\left(\sum_m\alpha_m\frac{g_{q,m}}{\sigma_{q,m}}\right)^\top \hat F^+ \left(\sum_k\alpha_k\frac{g_{q,k}}{\sigma_{q,k}}\right)\\\\ &=\sum_{m,k}\alpha_m\alpha_k \frac{g_{q,m}^\top\hat F^+g_{q,k}}{\sigma_{q,m}\sigma_{q,k}}\\\\ &=\frac NG\sum_{m,k}\alpha_m\alpha_k C_{q,mk}\\\\ &=\frac NG\alpha^\top C_q\alpha. \end{aligned} \tag{W1.3}$$ The left-hand side measures the squared norm of the standardized gradient combination, and the right-hand side is the GRAB objective multiplied by the fixed positive constant $N/G$. Because this positive constant is independent of the weights, GRAB's minimum-norm subproblem and the standardized-gradient problem in the empirical inverse-Fisher metric have the same minimizers over nonnegative weights that sum to one, under the stated conditions. This correspondence applies before the near-zero fallback in Algorithm 1. This equivalence concerns the weight-selection criterion. Full column rank is an assumption that we have not empirically verified. The actual update applies the weights to raw centered rewards rather than standardized rewards and includes clipping and KL regularization. We therefore do not claim a Pareto-improvement guarantee for the full policy update. We have clarified its scope in the revised manuscript.
AuthorsOpenReview ↗

Author Rebuttal (2/2)

Q2. Scale invariance.

Scale invariance in the paper refers to the weight-selection criterion, rather than the final advantages. When the active objective set remains unchanged, multiplying any reward by a positive factor leaves CqC_q and the set of optimal weights (αq∗\alpha_q^*) unchanged, so no objective dominates the weights because of larger numerical units.

We also accounted for this when designing the rewards: all four rewards are defined on [0, 1] (Appendix G.1), so even when the weights are applied to raw rewards, no component will dominate the advantage because of larger units. Eq. 4 applies the weights directly to the raw centered rewards based on this design. Standardizing before weighting would amplify rare differences. For example, if only 1 of 16 rollouts receives a format reward of 0.5, its standardized format value is about −3.9. Applying the weights to raw rewards keeps the advantage proportional to the actual reward differences.

Prior work [3] also computes weights from normalized gradients and then applies them to the original losses. We will further clarify this property in the camera-ready version.

Q3. Static-weight sweep. We add more static configurations to test whether tuned weights can match GRAB at similar output lengths. These configurations vary both the length and format/task ratios, and we extend the evaluation to VideoMME. Including the configurations in Table 9, all 7 use the same Stage-2 checkpoint, RL data, learning rate and training steps.

The table lists weights as (format, task, length). Static configurations are sorted by output length on Charades-TimeLens (CTL).

MethodCTL [email protected]CTL tokensVideoMMEVideoMME tokens
Fixed weights (1, 2, 0)42.11,18067.54,650
Fixed weights (1, 2, 1)42.993168.04,310
Fixed weights (1, 1, 1) (GRPO)40.684665.64,388
Fixed weights (2, 2, 3)39.366864.83,990
Fixed weights (1, 2, 3)39.860565.33,790
Fixed weights (0.5, 2, 3)40.058865.53,720
Fixed weights (1, 2, 4)37.846863.53,350
Lagrangian (Length)40.255665.73,806
GRAB43.457967.73,736

Each static configuration requires a separate training run to search for a suitable trade-off. GRAB computes its weights within each rollout group, avoiding this search.

Q4. Proportion and contribution of three-objective groups. To measure how often GRAB adapts its weights, we count active objectives during PARSE-4B training with G=16G=16. At most 3 objectives are active (Appendix G.3). We divide training into three phases with equal numbers of rollout groups. Each column reports percentages over all groups in that phase or the full run.

Active objectivesFirst 1/3Middle 1/3Last 1/3Full run
≤ 2 (advantage identical to GRPO)38%58%78%58%
3 (GRAB solves weights per group)62%42%22%42%

With two active objectives, standardization gives the reward columns equal norms, so equal weights solve the minimum-norm problem (Appendix G.3). GRAB therefore matches uniformly weighted GRPO with at most two active objectives and adapts its weights only in three-objective groups.

Ablation: To isolate the effect of adapting weights across groups, we fix the weights only in three-objective groups. We use GRAB's mean weights from Table 11, (0.15, 0.39, 0.46), and keep all other settings unchanged. Parentheses show gains over GRPO.

MethodGroups with ≤ 2 active objectivesThree-objective groupsCTL [email protected]CTL tokens
GRPOUniform weightsUniform weights40.6846
Fixed mean weightsSame as GRPOFixed to the means41.8 (+1.2)636
GRABSame as GRPOSolved per group43.4 (+2.8)579

GRAB improves [email protected] from 41.8 with fixed mean weights to 43.4 through per-group adaptation. This ablation fixes mean weights only in three-objective groups and retains GRPO behavior in the remaining groups, whereas the static-weight sweep applies a fixed weight vector to every group.

References:

[1] Jean-Antoine Désidéri. Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathématique, 350(5–6):313–318, 2012.

[2] Sham M. Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems 14 (NIPS), 2001.

[3] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems 31 (NeurIPS), pp. 525–536, 2018. Official code: https://github.com/isl-org/MultiObjectiveOptimization

Dear Reviewer ZEVE,

Thank you again for your thoughtful feedback. We hope that our response has helped address your concerns. Please do not hesitate to let us know if there are any remaining questions or issues that we could further clarify.

Reviewer_WyXXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address the challenge of balancing task accuracy, structural compliance, and reasoning efficiency in VLMs. Motivated by the limitations of unstructured CoT reasoning, which often produces redundant descriptions and obscures important temporal evidence, the paper proposes explicitly organizing video information into structured event evidence before performing higher-level reasoning.

The paper introduces Group-Relative Advantage Balancing (GRAB), a multi-objective RL algorithm that dynamically adjusts reward weights within each rollout group based on the relationships among different reward objectives. The intent is to balance format compliance, task accuracy, and reasoning length without manually tuning reward weights. The author also introduce PARSE-60K, a dataset containing 60K annotations across 35K videos, and they adopt a three-stage training curriculum consisting of evidence training, thinking warm-start, and RL post-training.

The resulting model, PARSE-4B, is built on Qwen3-VL-4B-Instruct. Evaluations are conducted on Charades-TimeLens, ActivityNet-Captions, VideoMME, MLVU, ETBench, and NExT-GQA, among other benchmarks. Experimental results show that GRAB reduces generated tokens by 13–39%.

Critical strengths
  • Overall, I appreciate the motivation of this paper and its understanding of the challenges involved in balancing multiple potentially conflicting reward objectives during RL post-training.
  • The proposed GRAB algorithm is a simple and computationally efficient approach to dynamically balancing different reward objectives. Its ability to avoid manually tuning reward weights sounds practically appealing.
Critical weaknesses

My main concerns lie in the following aspects:

  • The theoretical motivation behind GRAB could be further justified. Although the method is inspired by Pareto-optimal multi-objective optimization, it performs minimum-norm optimization over centered reward vectors rather than policy gradients. It remains unclear whether the resulting weights guarantee Pareto-optimal improvement in the actual policy optimization process.
  • While the paper provides an additional 8B experiment, the main evaluation focuses on the 4B model. Furthermore, the static-weight sweep is limited to one reward-weight ratio on a single benchmark. A more comprehensive comparison across model scales and reward configurations, together with end-to-end inference latency measurements, would provide stronger evidence for the general effectiveness and practical efficiency of GRAB.
Questions for authors

Please see the weaknesses. A more rigorous theoretical analysis would strengthen the motivation.

3 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

We sincerely thank you for your valuable feedback! We have updated paper, and we address each of your concerns in detail below.

Q1. Theoretical justification. GRAB computes weights from rewards, and these weights also determine how the policy gradients of the objectives are combined. Under the conditions below, reward correlations also describe how closely the empirical objective gradients agree, so GRAB's minimum-norm subproblem corresponds to a minimum-norm problem over standardized objective gradients in an empirical inverse-Fisher metric.

For a group of GG responses to input qq, let δi,m=ri,m−G−1∑j=1Grj,m\delta_{i,m}=r_{i,m}-G^{-1}\sum_{j=1}^{G}r_{j,m} be the centered reward of response ii for objective mm, and let si=∇θlog⁡πθ(oi∣q)∣θ=θolds_i=\left.\nabla_\theta\log\pi_\theta(o_i\mid q)\right\rvert_{\theta=\theta_{\mathrm{old}}} be its log-probability gradient at the rollout policy. The gradient estimate for objective mm is gq,m=G−1∑i=1Gsiδi,mg_{q,m}=G^{-1}\sum_{i=1}^{G}s_i\delta_{i,m}. Since the rewards are group-centered, Eq. (4) gives Ai=cq∑mαq,m∗δi,mA_i=c_q\sum_m\alpha_{q,m}^*\delta_{i,m}, where cq=(σA,q+ϵ)−1>0c_q=(\sigma_{A,q}+\epsilon)^{-1}>0 is the group normalization factor, σA,q\sigma_{A,q} is the group standard deviation of the raw weighted centered rewards, and ϵ>0\epsilon>0. Evaluating the gradient at the policy parameters that generated these responses and holding the computed advantages fixed during backpropagation gives the group-averaged reward-ascent gradient, excluding KL regularization:

=c_q\sum_m\alpha_{q,m}^* \left(\frac1G\sum_i s_i\delta_{i,m}\right) =c_q\sum_m\alpha_{q,m}^*g_{q,m}. \tag{W1.1}$$ At the rollout policy, the importance ratios equal one, so clipping does not alter this first-order expression. To connect reward correlations with objective-gradient alignment, we introduce an empirical inverse-Fisher metric motivated by natural policy gradient [2]. Let $S=[s_1,\ldots,s_N]\in\mathbb R^{P\times N}$ contain the score vectors of $N$ responses sampled from the rollout policy, including the $G$ responses in the current group, where $P$ is the number of parameters. Define $\hat F=N^{-1}SS^\top$, and let $\hat F^+$ denote its Moore–Penrose pseudoinverse. Assuming that $S$ has full column rank, $S^\top\hat F^+S=NI_N$, where $I_N$ is the $N$-dimensional identity matrix. Thus, $s_i^\top\hat F^+s_j$ equals $N$ when $i=j$ and zero when $i\ne j$. For active objectives $m$ and $k$, substituting this identity into the objective-gradient inner product eliminates cross terms between different responses, leaving a sum of within-response products of centered rewards: $$\begin{aligned} g_{q,m}^\top\hat F^+g_{q,k} &=\frac1{G^2}\sum_{i,j=1}^{G} \delta_{i,m}\delta_{j,k}\\,s_i^\top\hat F^+s_j\\\\ &=\frac N{G^2}\sum_{i=1}^{G}\delta_{i,m}\delta_{i,k} =\frac NG\sigma_{q,m}\sigma_{q,k}C_{q,mk}. \end{aligned} \tag{W1.2}$$ Here, $\sigma_{q,m}$ is the group reward standard deviation for objective $m$, and $C_{q,mk}$ is the reward correlation coefficient between objectives $m$ and $k$. We use $\sigma_{q,m}^2=G^{-1}\sum_i\delta_{i,m}^2$ and $C_{q,mk}=\sum_i\delta_{i,m}\delta_{i,k}/(G\sigma_{q,m}\sigma_{q,k})$, with positive standard deviations for active objectives. This agrees with the normalization $\hat D_q^\top\hat D_q=GC_q$ in the paper. Under this metric, a higher positive reward correlation corresponds to a smaller angle between the objective gradients, while a negative reward correlation gives a negative gradient inner product. MGDA [1] chooses weights that minimize the squared norm of the combined gradient. Expanding the two sums and substituting Eq. (W1.2) gives $$\begin{aligned} &\left\lVert\sum_m\alpha_m\frac{g_{q,m}}{\sigma_{q,m}}\right\rVert_{\hat F^+}^2\\\\ &=\left(\sum_m\alpha_m\frac{g_{q,m}}{\sigma_{q,m}}\right)^\top \hat F^+ \left(\sum_k\alpha_k\frac{g_{q,k}}{\sigma_{q,k}}\right)\\\\ &=\sum_{m,k}\alpha_m\alpha_k \frac{g_{q,m}^\top\hat F^+g_{q,k}}{\sigma_{q,m}\sigma_{q,k}}\\\\ &=\frac NG\sum_{m,k}\alpha_m\alpha_k C_{q,mk}\\\\ &=\frac NG\alpha^\top C_q\alpha. \end{aligned} \tag{W1.3}$$ The left-hand side measures the squared norm of the standardized gradient combination, and the right-hand side is the GRAB objective multiplied by the fixed positive constant $N/G$. Because this positive constant is independent of the weights, GRAB's minimum-norm subproblem and the standardized-gradient problem in the empirical inverse-Fisher metric have the same minimizers over nonnegative weights that sum to one, under the stated conditions. This correspondence applies before the near-zero fallback in Algorithm 1. This equivalence concerns the weight-selection criterion. Full column rank is an assumption that we have not empirically verified. The actual update applies the weights to raw centered rewards rather than standardized rewards and includes clipping and KL regularization. We therefore do not claim a Pareto-improvement guarantee for the full policy update. We will add this derivation and clarify its scope in the revised manuscript.
AuthorsOpenReview ↗

Author Rebuttal (2/2)

Q2 (a). Model scale. Thank you for raising this point. The RL stage in our paper uses full-parameter training. Our computational resources only allow us to train the 4B model, but are insufficient to train the 8B model under the same settings.

Nevertheless, we still try our best to provide 8B experiments within these resource limits. For these 8B experiments, we use LoRA in the RL stage with a lower rank and half of the RL data. All methods share the same settings. CTL denotes Charades-TimeLens:

8BVideoMMEVideoMME tokensCTL mIoUCTL tokensTempCompassEventBench
GRPO64.814,45243.7387565.7964.87
Fixed Weight (Accuracy)67.214,37646.2595066.0865.62
Lagrangian (Length)64.953,86043.6059065.7064.92
GRAB66.983,80546.8161266.2366.10

Q2 (b). Additional static-weight comparisons. We add more static configurations to test whether tuned weights can match GRAB at similar output lengths. These configurations vary both the length and format/task ratios, and we extend the evaluation to VideoMME. Including the configurations in Table 9, all 7 use the same Stage-2 checkpoint, RL data, learning rate and training steps.

The table lists weights as (format, task, length). Static configurations are sorted by output length on Charades-TimeLens (CTL).

MethodCTL [email protected]CTL tokensVideoMMEVideoMME tokens
Fixed weights (1, 2, 0)42.11,18067.54,650
Fixed weights (1, 2, 1)42.993168.04,310
Fixed weights (1, 1, 1) (GRPO)40.684665.64,388
Fixed weights (2, 2, 3)39.366864.83,990
Fixed weights (1, 2, 3)39.860565.33,790
Fixed weights (0.5, 2, 3)40.058865.53,720
Fixed weights (1, 2, 4)37.846863.53,350
Lagrangian (Length)40.255665.73,806
GRAB43.457967.73,736

Each static configuration requires a separate training run to search for a suitable trade-off. GRAB computes its weights within each rollout group, avoiding this search.

Q2 (c). End-to-end latency. We add end-to-end latency measurements for both methods using vLLM on the same GPU with batch size 1. Timing includes video preprocessing, visual encoding, prefill and generation, with identical decoding settings. Parentheses show reductions relative to Fixed Weight (Accuracy).

MethodCTL latencyCTL tokensVideoMME latencyVideoMME tokens
GRPO5.2 s84631.8 s4,388
Fixed Weight (Accuracy)5.7 s93131.4 s4,310
GRAB3.9 s57928.0 s3,736

The reduction in latency follows the reduction in output tokens. The relative latency reduction is slightly smaller than the token reduction because the video preprocessing, visual encoding and prefill are largely independent of output length.

References:

[1] Jean-Antoine Désidéri. Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathématique, 350(5–6):313–318, 2012.

[2] Sham M. Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems 14 (NIPS), 2001.

Dear Reviewer WyXX,

Thank you again for your thoughtful feedback. We hope that our response has helped address your concerns. Please do not hesitate to let us know if there are any remaining questions or issues that we could further clarify.

Comments on the paper

1 comment

Thank you very much for further clarifying your concern. We respectfully disagree with dismissing these results because the training setup differs from the 4B experiments. Full-parameter RL training at 8B under the same conditions is beyond our capability, that was why we used LoRA and half of the RL dataset.

We would like to kindly emphasize the controlled comparisons within each model scale, rather than comparisons across backbones trained under different settings. All four methods share the same 8B model, LoRA configuration, and training data. Under these exactly same configurations, GRAB improves over GRPO on all four benchmarks and reduces output length on both benchmarks with reported token counts.

We acknowledge that further experiments under additional training configurations would be valuable, but believe the 8B evaluation meaningfully addresses the your original concern regarding model scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.