acceptodds
Under review as a conference paper at ICLR 2027

When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning

Abstract

A failed vision-language-action (VLA) rollout does not imply that every action contributed to failure. Yet in group-relative policy optimization (GRPO), a single trajectory-level advantage is assigned to all actions in a rollout. Potentially useful recovery attempts can thus be penalized alongside the errors they aim to correct. Ideal stopping conditions for failure feedback are characterized using success-probability and policy-score martingales. At these stops, the removed gradient contribution has zero conditional expectation given the retained information. Truncation therefore preserves the expectation of a reference GRPO group-gradient estimator and can reduce its covariance. One such stop occurs when the conditional probability of success is zero, motivating error and recovery cues for selecting candidate feedback boundaries. Martingale GRPO (MART-GRPO) is introduced with a vision-language model (VLM) for event-guided truncation. The VLM is fine-tuned on our annotated training set using low-rank adaptation (LoRA). The last visible error without observed recovery is localized by the frozen VLM in complete rollout videos. Failure feedback is retained through the selected error interval and removed afterward, while successful-rollout loss terms and original loss normalization are preserved. For retrospective boundaries, changes in expectation and variance are analyzed to assess departures from ideal stopping. Experiments with OpenVLA-OFT on RoboTwin 2.0 show that MART-GRPO improves success rates over full-trajectory GRPO across multiple tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Reinforcement Learning, Temporal Credit Assignment, Martingales, Stopping Times, Variance Reduction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UeBwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MART-GRPO addresses the uniform penalization of actions in failed GRPO rollouts. A fine-tuned, frozen VLM reviews complete failed trajectories and identifies the last unrecovered error. Negative feedback is retained through that error and removed afterward. A martingale analysis establishes conditions under which truncation preserves the reference gradient expectation and reduces variance.

Critical strengths

The motivation is clear: a failed trajectory may contain useful actions, particularly recovery attempts, that should not receive the same penalty as the errors. The method integrates easily into GRPO and adds no deployment-time VLM cost. Across seven RoboTwin 2.0 tasks, average success increases from 65.01% to 68.64%. Preserving the original loss normalization and including uniform-scaling and random-prefix controls also strengthen the evaluation.

Critical weaknesses
  1. The connection between the theory and the practical truncation rule remains unclear. The theoretical guarantee requires the removed gradient contribution to have zero conditional expectation. However, an absence of observed recovery does not establish that subsequent actions have no learning value. The VLM also uses future observations, violating the prefix-based condition of the ideal stopping rule. Although the paper analyzes potential bias, it does not measure gradient mean shifts, variance changes, or recovery probabilities after the selected boundaries in the robotics experiments. Direct evidence is needed to connect the theoretical analysis to the observed gains.

  2. The stability and computational cost of the gains are insufficiently evaluated. Comparisons mainly involve Full GRPO, without directly related methods such as PCM. The two ablations cover only two tasks, and no results across independent training seeds are reported. Evaluation over different scene seeds does not replace repeated training runs. Moreover, adaptive rollout refilling means that 40 updates may require different numbers of environment interactions, while MART-GRPO additionally invokes a 27B VLM. Total interactions, training time, and results across multiple training seeds are needed to assess the benefit relative to its cost.

  3. Annotation quality and localization evaluation need stronger support. The paper does not clearly explain who produced the reference annotations, how they were verified, or their inter-annotator agreement. Localization is evaluated only on 220 localizable, unrecovered failures, leaving incorrect truncation on ambiguous or recovered errors insufficiently assessed. Appendix E.3 also states that test diagnostics informed training duration, so the test set was not fully isolated from development. An independent test set and sensitivity analysis of earlier or later boundaries would strengthen the evidence.

  4. How would the method perform on long-horizon, contact-sensitive tasks such as insertion or screw tightening, where recovery may require repeated retreats and adjustments with little visible progress? If the initial policy rarely recovers, could the frozen VLM truncate feedback too early and leave ineffective or harmful recovery actions unpenalized? Repeated continuations from the same failure states, evaluated by recovery success, would help establish whether the method improves recovery learning.

Questions for authors

see weakness.

3 comments

Thank you for identifying the key questions about theoretical validity and empirical evidence. We conducted additional continuation, gradient, boundary-sensitivity, and multi-seed experiments.

W1. Connecting theory to practical VLM boundaries

Theorem 2 characterizes ideal prefix-based stopping; Proposition 4 and Eq. (11) describe the bias–variance tradeoff for MART's retrospective boundaries.

Recovery feasibility. We restored chunk-boundary states from 67 failed trajectories across four tasks and sampled 32 continuations per state under the initial policy and remaining budget. Fixed cuts match Table 10's retained feedback fractions.

TaskRetained chunksMean success0/32 successes
Hammer4/82.9%12/16
Handover5/163.1%18/23
Container on Plate3/65.1%12/14
Empty Cup4/87.1%12/14

Overall, 75–86% of trajectories have zero successes in 32 attempts. Combining continuation profiles with empirical VLM boundary distributions gives estimated success probabilities at the first masked chunk of 8.7–9.0% on Hammer and 5.1–5.3% on Handover.

Credit alignment. Mean signed alignment with a continuation-estimated reference is 0.505 for prefixes, 0.313 for Full feedback, and 0.218 for equal-size random subsets (64 trajectories). Prefix alignment exceeds Full on 52/64 trajectories and Random on 56/64.

Actual-mask bias and variance. Replaying VLM masks on a 512-trajectory Hammer bank at the frozen initial policy reduces recorded parameter-subspace variance by 33.7%, from 10.81×10−410.81\times10^{-4} to 7.16×10−47.16\times10^{-4}. The squared mean shift is (4.09±0.68)×10−4(4.09\pm0.68)\times10^{-4}. Under Eq. (11), estimated MSE change is (0.45±0.60)×10−4(0.45\pm0.60)\times10^{-4} (descriptive jackknife SE across four 128-trajectory minibatches).

Boundary sensitivity. On the same bank, shifting the boundary one chunk earlier or later yields 35.8% or 26.0% variance reduction, versus 33.7% at the original boundary.

These measurements connect recovery feasibility, temporal credit alignment, and the gradient behavior of retrospective masks.

W2. Training stability and interaction budget

Each original table entry used one run. We added two seeds, giving three runs per method on Hammer and Handover, with 256 evaluation episodes at update 40. Results are mean ± SD.

TaskFull GRPOUniformRandom PrefixMART
Hammer74.48 ± 0.81%75.91 ± 1.76%79.69 ± 7.04%82.81 ± 2.38%
Handover38.54 ± 4.39%38.54 ± 1.80%40.23 ± 1.04%40.89 ± 2.74%

MART achieves the highest mean on both tasks. Gains over Full, Uniform, and Random are 8.33, 6.90, and 3.12 points on Hammer and 2.34, 2.34, and 0.65 points on Handover. Hammer and Handover were selected after encouraging seven-task results; we will clarify this exploratory selection.

Each run uses 40 updates and 20,480 accepted trajectories. Adaptive refill changes generated trajectory counts; comparisons use fixed update and accepted-trajectory budgets, not interaction-matched budgets.

W3. Annotation quality and evaluation protocol

Among 220 localizable, unrecovered failures, predicted boundaries match 150 human-annotated references (68%) exactly and fall within one chunk in 202 cases (92%). In 34 cases (15%), predictions are earlier. The other 36 cases comprise 32 Move Can Pot failures, three non-localizable Stack Bowls cases, and one Container on Plate case with uncertain recovery.

The policy experiments use the validation-selected update-304 localizer. Test diagnostics later informed extending training from update 361 to 437, while validation continued to select update 304. Videos come from Full-GRPO checkpoints at updates 5–40; the frozen localizer also generates masks on online MART rollouts (Table 10).

These metrics evaluate interval localization, not recovery classification. When recovery is uncertain or localization invalid, MART retains full feedback.

W4. Recovery in long-horizon tasks

MART retains full feedback when recovery is reported or uncertain, or localization is invalid. This is important for contact-sensitive tasks such as insertion and screw tightening. Evaluating trained MART and Full-GRPO policies from identical failure states with equal remaining budgets is a natural next step. We will illustrate recovery and fallback cases in Figure 3.

Compute and implementation

Measured end-to-end time per update is:

TaskFull GRPOMART
Hammer13.1 min32.6 min
Handover17.3 min36.3 min

The additional cost primarily comes from VLM inference through a shared eight-GPU service. The localizer is frozen during training, deployment requires no VLM inference, and MART preserves original loss normalization.

Thank you for the response, which has addressed most of my concerns. I will maintain my original score.

Thank you for your follow-up and careful review. Your constructive suggestions have helped us improve the manuscript, and we sincerely appreciate your time and engagement throughout the discussion.

Reviewer_oPftOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks which actions in a failed rollout should get GRPO's negative advantage. The theory part shows that if the failure penalty is cut at a stopping time after which the remaining actions carry no credit for the outcome - certified failure is one such case - the masked estimator keeps the same expectation and has no larger variance. The method MART-GRPO uses a fine-tuned VLM (Qwen3.8 27B) to locate the last unrecovered error in each failed video and masks the negative advantage after it. On seven RoboTwin 2.0 tasks with OpenVLA-OFT, mean success moves from 65.0% (Full GRPO) to 68.6%. On two tasks, MART also beats ablations that take away the same amount of negative feedback by uniform scaling or at random points.

Critical strengths
  1. Clean theory with an honest scope. The condition for masking failure feedback to be "free" is precise, and both the proofs and the appendix counterexamples check out. The paper also says openly that its own VLM rule falls outside the guarantee.
  2. A simple change, tested against the right controls. The method only changes a mask on the advantage, so the loss change itself is easy to add to an existing GRPO pipeline, although the method also needs annotated videos to train the localizer and a 27B VLM pass over every failed rollout. The mass-matched ablations (uniform scaling and random prefix) are exactly the comparisons needed to separate where the penalty is cut from how much is cut and the paper reports raw counts for the ablations and openly shows the task where MART loses (Container on Plate).
Critical weaknesses
  1. The gains may be within noise. Each method and task appears to rest on a single RL run with no seeds and no error bars. With 256 evaluation episodes most per-task gaps fall inside evaluation noise and the pooled gap (z ~ 2.3) still ignores run-to-run variance. Uniform scaling, which uses no VLM during training (its ratios come from a finished MART run), already gets half or more of MART's gain on Hammer and Handover and what is left is about one standard error. So the claim I find most important - that the VLM-chosen boundary matters beyond simply reducing negative feedback, is not yet supported. This is the main reason for my score.
  2. The theory does not cover the method that is actually used. The guarantee needs a cut chosen from the observed prefix, after which the remaining actions carry no credit (certified failure is the clearest case). MART picks the boundary in hindsight from the full video which is the case the paper itself shows can bias the estimator. Nothing that links the two is measured - no estimate of the bias term or the gradient variance, and no check that success was really unreachable after the boundary. In Figure 1 the robot keeps trying to recover for several chunks after the error. If recovery was still possible, masking those steps could hurt learning to recover.
Questions for authors

Clarifications

  1. How many RL runs are behind Tables 1 and 2, and how were Hammer and Handover chosen for the ablations? Results over at least three seeds for Full GRPO, uniform scaling, random prefix and MART on these tasks would address W1.
  2. How accurate is the localizer on the recovery decision (Table 8 does not cover 36 of the test failures)? Have you checked its masks on rollouts from the RL runs rather than only on the annotation corpus?
  3. Per update how much compute does the VLM add?

Suggestions (future work, not decisive)

  1. In simulation you can test the theory's condition directly - restore the state right after the boundary, resample continuations, and estimate the success probability
  2. A useful baseline would be a rule that uses only the observed prefix or a simulator signal of certain failure. If it matches MART then the 27B localizer is not needed
6 comments

Thank you for the thoughtful review. We conducted additional multi-seed experiments and gradient-level analyses of temporally selective negative feedback.

W1 / Q1. Training variability and feedback ablations

Each cell in the original Tables 1–2 used one RL run. We added two independent seeds, giving three runs per method on Hammer and Handover. Each run uses 256 evaluation episodes at update 40. Results are mean ± SD across seeds.

TaskFull GRPOUniformRandom PrefixMART
Hammer74.48 ± 0.81%75.91 ± 1.76%79.69 ± 7.04%82.81 ± 2.38%
Handover38.54 ± 4.39%38.54 ± 1.80%40.23 ± 1.04%40.89 ± 2.74%

On Hammer, MART improves over Full, Uniform, and Random Prefix by 8.33, 6.90, and 3.12 points. On Handover, gains are 2.34, 2.34, and 0.65 points. MART achieves the highest mean success on both tasks. Uniform scaling matches MART's retained negative-feedback mass; Random Prefix samples from MART's boundary-length distribution independently of trajectory content. Both preserve successful-rollout weights and original token normalization.

Hammer and Handover were selected for the original ablations after encouraging seven-task results. We will clarify that this selection was exploratory.

W2. Connecting practical boundaries to theory

Recovery feasibility. We restored chunk-boundary states from 67 failed trajectories across four tasks and sampled 32 continuations per state under the initial policy and remaining budget. Fixed cuts match Table 10's retained feedback fractions.

TaskRetained chunksMean success0/32 successes
Hammer4/82.9%12/16
Handover5/163.1%18/23
Container on Plate3/65.1%12/14
Empty Cup4/87.1%12/14

Across tasks, 75–86% of trajectories have zero successes in 32 attempts. Combining continuation profiles with the VLM boundary distribution during training (8.7k–13.0k failed rollouts/run), estimated success probability at the first masked chunk is 8.7–9.0% on Hammer and 5.1–5.3% on Handover.

Temporal credit alignment. Using a continuation-estimated credit reference, mean signed alignment is 0.505 for prefixes, 0.313 for Full feedback, and 0.218 for equal-size random subsets (64 trajectories). Prefix alignment exceeds Full on 52/64 trajectories and Random on 56/64.

Actual-mask bias and variance. Replaying VLM masks on a 512-trajectory Hammer bank at the frozen initial policy reduces recorded parameter-subspace variance by 33.7%, from 10.81×10−410.81\times10^{-4} to 7.16×10−47.16\times10^{-4}. The squared mean shift is (4.09±0.68)×10−4(4.09\pm0.68)\times10^{-4}. Under Eq. (11), estimated MSE change is (0.45±0.60)×10−4(0.45\pm0.60)\times10^{-4} (descriptive jackknife estimates across four 128-trajectory minibatches).

Theorem 2 characterizes ideal prefix-based stopping; Proposition 4 and Eq. (11) characterize the bias–variance tradeoff for retrospective boundaries. These experiments connect recovery feasibility, credit alignment, and actual-mask gradient behavior.

Q2. Localizer accuracy and robustness on RL rollouts

Of the 36 failures outside Table 8, 32 belong to Move Can Pot, an eighth annotation task whose MART run ended at update 36; three are non-localizable Stack Bowls cases, and one is a Container on Plate case with uncertain recovery.

Among 220 localizable, unrecovered failures, predicted boundaries exactly match 150 human-annotated references (68%) and fall within one chunk in 202 cases (92%). In 34 cases (15%), predictions are earlier than references.

Despite imperfect exact localization, MART achieves the highest mean success across three seeds on both Hammer (82.81%) and Handover (40.89%). This suggests that favorable policy-level performance does not require exact boundary localization in every rollout and that MART can tolerate moderate chunk-level localization errors.

All annotation videos come from Full-GRPO checkpoints at updates 5–40 and are RL-generated rollouts. The frozen localizer also generates masks on online MART rollouts (Table 10). When recovery is uncertain, MART retains full negative feedback.

Q3. Additional compute

TaskFull GRPOMART
Hammer13.1 min32.6 min
Handover17.3 min36.3 min

MART adds 19.5 and 19.0 minutes per update, respectively (2.5× and 2.1× total time). The additional cost primarily comes from VLM inference through a shared eight-GPU service. Deployment requires no VLM inference.

Summary. These experiments provide evidence for temporally selective negative feedback through improved multi-seed performance, limited recovery after evaluated boundaries, better credit alignment, and reduced gradient-subspace variance with actual VLM masks. We will incorporate these results into the revision.

Thanks for the extra experiments and for explaining the 36 test failures. I corrected a few points in my review, including Question 2. Three seeds on hammer show a clear gain over full GRPO and uniform scaling. However Handover stays within the seed spread. Random prefix on Hammer is too noisy to separate from MART and the other five tasks are still single runs. The recovery check and the mask replay are what I asked for. Success after the cut is not zero though and there is no clear MSE gain, so I read theory as motivation not a guarantee. The 2x training cost should be stated next to the gains. For now I am keeping my score. I would raise it to 3 if the revision includes these results and states the claim that the VLM boundary matters beyond reducing negative feedback only as far as these results support it, with the theory as motivation.

AuthorsOpenReview ↗

Revised manuscript incorporating the additional experiments

Thank you for your kind comments. The revised manuscript has been uploaded, which includes the additional experiments below, with changes marked in blue.

  1. Three-run results and claim scope (Section 4.3, Table 2). We report mean ± SD across three training runs for all four methods on Hammer and Handover. On Hammer, MART's mean success exceeds Full GRPO and Uniform by 8.33 and 6.90 points. Its mean is 3.12 points above Random Prefix, which shows greater observed variability across runs (SD 7.04 versus 2.38 for MART). On Handover, MART also achieves the highest observed mean success, exceeding Full and Uniform by 2.34 points and Random Prefix by 0.65 points; the run-to-run SDs range from 1.03 to 4.39 points across methods. The other five tasks are identified as single-run results. Appendix F.2 provides exact per-run success counts in Table 11 and individual outcomes in Figure 8.

  2. Recovery and gradient diagnostics (Section 4.3; Appendix F.3, Tables 12–13). We include the fixed-cut continuation tests, boundary-weighted recovery estimates, credit alignment, and replay of the actual VLM masks. Mean continuation success at the fixed cuts ranges from 2.9% to 7.1%. On Hammer, replay at the frozen initial policy reduces variance in the recorded parameter subspace by 33.7%, with a mean shift. For n = 1 in Equation 11, the estimated MSE change relative to Full GRPO is (+0.45 ± 0.60) × 10⁻⁴ (descriptive jackknife SE).

  3. Theoretical scope (Section 3.3). We explicitly present the theory as motivation for the practical boundary rule and state that the ideal projection condition is not established by visible-event localization.

  4. Training cost alongside the gains (Section 4.3; Table 14). MART's per-update wall-clock time is 2.5× and 2.1× that of Full GRPO on Hammer and Handover, respectively. These costs appear immediately after the ablation results.

The explanation of the 36 remaining test failures is also included in Appendix E.3 and the Table 8 caption.

Thank you again for your careful reading and constructive suggestions. Your feedback has helped us clarify the empirical evidence and the scope of our claims, and we have revised the manuscript accordingly. We appreciate the opportunity to discuss these points with the research community and welcome any further questions or comments.

Reviewer_erQBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MART-GRPO, which removes negative training feedback from the suffix of failed robot trajectories. A fine-tuned VLM identifies the last visible error without subsequent recovery, and the policy update retains negative feedback only through the selected boundary. A martingale analysis motivates this approach under conditions on the remaining policy-gradient contribution.

Critical strengths
  • The paper addresses a meaningful credit-assignment problem: assigning the same negative advantage to an entire failed trajectory can penalize reasonable recovery attempts.
  • The theoretical analysis distinguishes ideal stopping rules from the retrospective boundaries used in practice and discusses the resulting bias–variance tradeoff.
Critical weaknesses
  • The practical boundary rule identifies errors without observed recovery, whereas the theoretical guarantees require conditions on the remaining gradient contribution. The paper does not empirically establish that its selected boundaries satisfy or approximate these conditions; accurate error localization alone is insufficient.
  • The method is evaluated only with GRPO, whose shared trajectory-level advantage directly motivates the proposed masking. It remains unclear whether the benefits extend to actor–critic methods such as PPO, where timestep-specific advantages may already mitigate this issue.
  • Evaluation is restricted to seven tasks in one simulation benchmark, using one VLA backbone and one VLM backbone, with no real-world experiments. It remains unclear whether the gains generalize across benchmarks and model choices or remain reliable under real-world visual and execution noise.
  • Some figures hinder readability. Figure 3, in particular, contains small rollout images and dense annotations that make the relationship between the predicted error interval and retained feedback difficult to inspect.
Questions for authors
  1. Can you evaluate whether the selected boundaries remove uninformative gradient contributions, through repeated continuations from boundary states or direct gradient-variance measurements?
  2. How sensitive are the gains to the choice of VLA and event-localization VLM? Do they extend to another benchmark or real-world tasks?
  3. Can the proposed feedback masking also benefit PPO with a learned value function and timestep-specific advantages? Such a comparison would help determine whether the method addresses a broader credit-assignment problem or primarily compensates for GRPO’s trajectory-level advantage assignment.
5 comments

Thank you for the thoughtful review. We conducted additional continuation, gradient, and multi-seed experiments to further examine MART-GRPO.

W1 / Q1. Theoretical conditions and empirical gradient measurements

Theorem 2 characterizes ideal prefix-based stopping with zero remaining conditional credit. For MART's retrospective boundaries, Proposition 4 and Eq. (11) characterize the bias–variance tradeoff.

Recovery feasibility. We restored chunk-boundary states from 67 failed trajectories across four tasks and sampled 32 continuations per state under the initial policy and remaining execution budget.

TaskRetained chunksMean success0/32 successes
Hammer4/82.9%12/16
Handover5/163.1%18/23
Container on Plate3/65.1%12/14
Empty Cup4/87.1%12/14

Across tasks, 75–86% of trajectories have zero observed successes in 32 attempts. Combining continuation profiles with empirical VLM boundary distributions gives estimated success probabilities at the first masked chunk of 8.7–9.0% on Hammer and 5.1–5.3% on Handover.

Credit alignment. Using a continuation-estimated reference, mean signed alignment is 0.505 for prefixes, 0.313 for Full feedback, and 0.218 for equal-size random subsets (64 trajectories). Prefix alignment exceeds Full on 52/64 trajectories and Random on 56/64.

Actual-mask gradient effects. Replaying VLM masks on a 512-trajectory Hammer bank at the frozen initial policy reduces recorded parameter-subspace variance by 33.7%, from 10.81×10−410.81\times10^{-4} to 7.16×10−47.16\times10^{-4}. The squared mean shift is (4.09±0.68)×10−4(4.09\pm0.68)\times10^{-4}. Under Eq. (11), estimated MSE change is (0.45±0.60)×10−4(0.45\pm0.60)\times10^{-4} (descriptive jackknife SE across four 128-trajectory minibatches). Shifting the boundary one chunk earlier or later yields 35.8% or 26.0% variance reduction, respectively, compared with 33.7% at the original boundary.

These measurements connect recovery feasibility, temporal credit alignment, and the gradient behavior of practical VLM masks.

W2 / Q2. Training reliability and generalization

We conducted three independent training runs for Full GRPO, uniform scaling, Random Prefix, and MART on Hammer and Handover, with 256 evaluation episodes per run at update 40. Results are mean ± SD.

TaskFull GRPOUniformRandom PrefixMART
Hammer74.48 ± 0.81%75.91 ± 1.76%79.69 ± 7.04%82.81 ± 2.38%
Handover38.54 ± 4.39%38.54 ± 1.80%40.23 ± 1.04%40.89 ± 2.74%

MART achieves the highest mean on both tasks. Gains over Full, Uniform, and Random are 8.33, 6.90, and 3.12 points on Hammer and 2.34, 2.34, and 0.65 points on Handover.

Our evaluation covers seven RoboTwin 2.0 tasks using OpenVLA-OFT and Qwen3.8-27B. The additional seeds further assess training reliability. Extending MART to other VLA backbones and event-localization VLMs, additional benchmarks, and real-world robots is a natural direction for broader validation.

W3 / Q3. Extension to PPO with a learned value function

Our current work focuses on critic-free GRPO, where a shared trajectory-level advantage assigns the same negative coefficient to all action chunks in a failed rollout. For the undiscounted terminal-success objective, an exact history-conditioned value function, Vπ(hk)=PkV^\pi(h_k)=P_k, yields zero action advantages after certified failure.

Extending MART to actor–critic methods such as PPO is a natural next step. In particular, event-guided masking could be investigated in conjunction with learned value functions and timestep-specific advantages to further improve credit assignment under imperfect critic estimates.

W4. Figure readability

We will enlarge the rollout frames in Figure 3 and align the predicted error interval, boundary B, and retained feedback mask on a common chunk timeline. Recovery and semantic-fallback examples will also be included.

Overall, the additional experiments provide evidence through continuation-based recovery measurements, temporal credit alignment, actual-mask gradient diagnostics, and independent policy training runs. We will incorporate these findings into the revision.

AuthorsOpenReview ↗

Follow-up on the revised manuscript and additional experiments

Thank you for your review and suggestions. We are following up on our last response ahead of the author-reviewer discussion deadline on 18 November (AoE). The revised manuscript incorporates the additional experiments and updated Figure 3, with changes marked in blue.

W1 / Q1. Recovery and gradient measurements. Theorem 2 characterizes ideal stopping conditions, while Proposition 4 and Equation 11 analyze expectation and variance changes for retrospective boundaries. Section 3.3 states that visible-event localization does not establish the ideal projection condition and that the theory motivates boundary selection. We report the following measurements using the frozen initial policy (Section 4.3; Appendix F.3). Continuation tests and credit alignment use fixed task-specific cuts that approximate MART's retained feedback fractions (Table 10). Boundary-weighted recovery combines continuation profiles with the VLM boundary distributions, while gradient replay uses the recorded VLM masks.

  • Continuation tests: We restored states at fixed task-specific cuts in 67 failed trajectories across four tasks, with 32 continuations per state. Mean success ranges from 2.9% to 7.1% across tasks, and 75–86% of trajectories have 0/32 successes (Table 12).
  • Boundary-weighted recovery: Combining the continuation profiles with the observed VLM boundary distributions gives estimated success probabilities at the first masked chunk of 8.7–9.0% on Hammer and 5.1–5.3% on Handover (Appendix F.3).
  • Actual-mask replay: On a 512-trajectory Hammer bank, replaying the VLM masks reduces variance in the recorded parameter subspace by (3.65 ± 0.69) × 10⁻⁴, or 33.7%. The estimated squared mean shift is (4.09 ± 0.68) × 10⁻⁴. For n = 1 in Equation 11, the estimated MSE change is (+0.45 ± 0.60) × 10⁻⁴, defined as MSE(MART) minus MSE(Full GRPO), with both measured relative to the same reference mean. Errors are descriptive jackknife SEs (Table 13).
  • Credit alignment: At the same fixed cuts, across 64 trajectories, mean signed alignment with a continuation-estimated credit reference is 0.505 for the retained prefix, 0.313 for full feedback, and 0.218 for equal-size random subsets. The prefix has higher alignment on 52/64 and 56/64 trajectories, respectively (Appendix F.3).

W2 / Q3. Relation to actor–critic methods. Our empirical evaluation concerns critic-free GRPO, where one trajectory-level coefficient is shared across action chunks. Under the undiscounted terminal-success objective, an exact history-conditioned value function yields zero action advantages after certified failure under the current policy; this follows from the local action advantage in Appendix B.2 (Equation 24). Under these conditions, the advantage term supplies no negative feedback for those actions. MART targets the critic-free setting. Whether masking benefits PPO with a learned, imperfect critic has not been evaluated here; the conclusion identifies this comparison as future work.

W3 / Q2. Evaluation scope and training variability. The current evaluation uses OpenVLA-OFT and the Qwen3.8-27B localizer on seven RoboTwin 2.0 simulation tasks. We have not yet evaluated sensitivity to alternative VLA or VLM backbones, transfer to another benchmark, or performance on real robots. These evaluations cannot be completed within the discussion period and will be a priority for our future work.

The additional training runs assess variability within this setting. Table 2 reports mean ± SD over three training runs per method on Hammer and Handover. On Hammer, MART's mean success exceeds Full GRPO and Uniform by 8.33 and 6.90 points. Its mean is 3.12 points above Random Prefix, whose observed SD is 7.04 versus 2.38 for MART. On Handover, MART also achieves the highest observed mean, exceeding Full and Uniform by 2.34 points and Random Prefix by 0.65 points; the run-to-run SDs range from 1.03 to 4.39 points across methods. The other five tasks remain single-run results. Appendix F.2 provides exact per-run counts in Table 11 and individual outcomes in Figure 8.

W4. Figure readability. Figure 3 now has larger chunk-level frames and aligns the predicted onset interval [1,2], boundary B=2, and feedback mask (1,1,1,0) on a common chunk timeline. Its caption points to an example of recovery followed by a later error (Figure 5) and a schematic fallback (Appendix E.1).

Thank you again for your careful reading and constructive suggestions. We appreciate the opportunity to discuss these points with the research community and welcome any further questions or comments.

Thank you for the additional experiments and for clarifying how the method might extend to PPO. The multi-seed results strengthen the evidence for MART-GRPO on the two tasks tested. My main concern about generalization remains, since the evaluation still covers a single simulation benchmark and a single VLA backbone, with no experiments on real robots. So far, the empirical evidence shows a benefit only for GRPO. Whether the same masking strategy also helps actor–critic methods such as PPO remains open.

Comments on the paper

3 comments

Dear Reviewers,

Please read the authors’ response and let us know whether it addresses your concerns. If any concerns remain, please briefly indicate which ones.

Thank you for your time and for your thoughtful reviews.

Best,

AC

AuthorsOpenReview ↗

Summary of manuscript revisions and additional experiments

Dear Reviewers and Area Chair,

Thank you for your careful reviews and for the discussion. We have revised the manuscript, with changes highlighted in blue. Below we summarize the main revisions and indicate the corresponding reviewer comments for reference.

  • A martingale stopping criterion for failure feedback [erQB W1/Q1, oPft W2, UeBw W1]. Reaching again for a dropped object can improve a robot’s chance of success even if the rollout ultimately fails. GRPO can assign this recovery attempt the same negative advantage as the initial error. We pair two martingales, the conditional success probability and the cumulative policy score, to characterize remaining conditional credit: the first-order sensitivity of continuation success to the policy. Under the assumptions of Theorem 2, stopping when this credit vanishes makes the truncated GRPO group-gradient estimator the conditional expectation of the full-feedback estimator given the retained information. Its expectation is therefore preserved and its covariance cannot increase. This stopping criterion can hold even when recovery remains possible. Section 3.3 clarifies how it motivates the practical VLM boundary rule.
  • Recovery and gradient measurements [erQB W1/Q1, oPft W2, UeBw W1/W4]. We restored states from 67 failed trajectories across four tasks and sampled 32 continuations per state under the initial policy. At the fixed cuts tested, mean continuation success ranged from 2.9% to 7.1% across tasks. For boundaries selected retrospectively by the VLM, Proposition 4 and Equation (11) characterize how changes in the gradient mean and variance affect estimation error. On Hammer, replaying the actual masks reduced variance in the recorded gradient subspace by 33.7% and shifted the mean. Section 4.3 and Appendix F.3 also report credit alignment and the effect of shifting the boundary by one chunk.
  • Seven-task results and three-run ablations [oPft W1/Q1, UeBw W2]. Table 1 reports one training run per method and task. MART achieved higher success than Full GRPO on six of seven tasks, with a 3.63-percentage-point gain in the seven-task average. We then compared all four methods over three training runs each on Hammer and Handover (Table 2). On Hammer, MART’s mean success rate exceeded Full GRPO by 8.33 points and mass-matched uniform scaling by 6.90 points. The greater variability of Random Prefix limits what the three runs establish about the additional benefit of trajectory-specific localization. On Handover, MART had the highest observed mean (40.89%), with differences within the observed run-to-run variation. Appendix F.2 gives the exact per-run counts.
  • Training cost and interaction budget [oPft Q3, UeBw W2]. Section 4.3 reports wall-clock time alongside the policy comparisons. Each MART update took 2.5 times as long as Full GRPO on Hammer and 2.1 times as long on Handover. Each run used 40 updates and 20,480 accepted trajectories. Adaptive refill allowed the number of generated interactions to vary. Appendix F.4 gives the timings.
  • Localization, recovery handling, and Figure 3 [erQB W4, oPft Q2, UeBw W3/W4]. Among 220 localizable test failures without observed recovery, 150 predicted endpoints matched the reference exactly and 202 fell within one chunk. Appendix E.3 accounts for the other 36 test failures. Figure 3 now aligns the predicted error interval, selected boundary, and feedback mask on a common timeline. Its caption points to a recovery example and a schematic fallback. When recovery is reported or uncertain, MART retains full feedback.
  • Scope of evaluation [erQB W2/Q3, erQB W3/Q2]. Our policy experiments use OpenVLA-OFT with critic-free GRPO on RoboTwin 2.0. In our response, we explain how timestep-specific advantage estimates change the question for PPO with a learned critic. Testing the masking rule in that setting, across other backbones and benchmarks, and on real robots are priorities for future work.

We would be grateful if you would consider the revised evidence and the theoretical contribution in your final assessments. If a specific concern remains, we would welcome the opportunity to address it.

Thank you again for your time and consideration.

The Authors

Dear Area Chair and Reviewers,

Thank you to the reviewers for reading the manuscript so carefully and to the Area Chair for guiding the discussion. Reviewer oPft confirmed that the revision addressed the points they had raised. UeBw said our response addressed most of their concerns, and erQB noted that the additional training runs strengthened the evidence for the proposed MART-GRPO.

Summary of Our Work

A robot that drops an object may still make a sensible attempt to pick it up. In outcome-supervised GRPO, all steps in a rollout share one group-normalized advantage, so the initial error and the recovery attempt receive the same penalty if the rollout fails. We ask when negative feedback can stop while preserving the expectation of the full-feedback GRPO gradient estimator.

Our stopping criterion is based on remaining conditional credit. Two martingales track the conditional probability of success and the cumulative policy score. Together they characterize the remaining actions’ first-order contribution to success. Under the ideal stopping conditions, once this credit vanishes, the truncated estimator equals the conditional expectation of the full-feedback estimator given the retained information (Theorem 2). It has the same expectation and no greater covariance. Certified failure is one such stop (Corollary 3). Remaining credit can also vanish while recovery is still possible (Appendix C).

This stopping criterion inspired the design of MART-GRPO. A vision-language model (VLM) reviews the full video of a failed rollout to locate the last visible error with no subsequent sign of recovery. When it identifies such an error, MART-GRPO retains failure feedback through the end of that error interval. We analyze how these retrospective boundaries affect the gradient estimator (Proposition 4 and Equation 11) and measure the resulting changes in its mean and variance.

Strengths Identified by the Reviewers

  • Credit assignment. Reviewers erQB and UeBw singled out recovery attempts as a reason to reconsider how GRPO assigns failure feedback. A rollout can contain both the action that caused an error and a sensible response to it, yet its shared advantage penalizes both.
  • Theoretical analysis. Reviewer oPft highlighted the proof that truncation preserves the expected full-feedback gradient under the stated stopping condition, along with the counterexamples that clarify its scope. Reviewer erQB highlighted the treatment of boundaries chosen in hindsight, including the tradeoff between a shift in the gradient mean and a reduction in variance.
  • Method and controls. Reviewers oPft and UeBw noted that the mask change fits into an existing GRPO update. UeBw emphasized that it preserves the original loss normalization and adds no VLM inference at deployment. Both reviewers valued the uniform-scaling and random-prefix controls for separating how much failure feedback is retained from where it is applied. Reviewer oPft also appreciated the raw success counts reported for the ablations.

Additional Experiments and Revisions

Policy performance and feedback selection. In single-run comparisons, MART exceeded Full GRPO on six of seven RoboTwin 2.0 tasks, with a mean gain of 3.63 percentage points. On Hammer and Handover, we trained each of the four methods three times per task. To test whether it matters where failure feedback ends, we compared MART with a control that scaled feedback uniformly along each failed rollout. We set the scale to the fraction of total negative feedback mass retained in the paired MART run. On Hammer, mean success with MART was 6.90 percentage points higher than with uniform scaling and 8.33 points higher than with Full GRPO. The revision reports individual run outcomes and training time alongside these comparisons.

Recovery and mask replay. To measure how often the policy recovered after a cut, we restored states at fixed cuts in 67 failed trajectories across four tasks and sampled 32 continuations per state. At these cuts, we also compared two feedback directions, one from the retained prefix and one from the full trajectory, with a credit estimate derived from continuations. Prefix feedback was closer to that reference on 52 of the 64 trajectories with usable estimates. On Hammer, we replayed the recorded MART masks and a mass-matched uniform control at the frozen initial policy to measure how each changed the gradient mean and variance.

Localization and presentation. Figure 3 now aligns the predicted error interval, selected boundary, and feedback mask on one timeline. Among 220 localizable failures without observed recovery, 150 predicted boundaries matched the reference exactly and 202 fell within one chunk.

We sincerely appreciate the reviewers' feedback and the discussions. Your feedback helped us strengthen the paper. Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.