- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MART-GRPO addresses the uniform penalization of actions in failed GRPO rollouts. A fine-tuned, frozen VLM reviews complete failed trajectories and identifies the last unrecovered error. Negative feedback is retained through that error and removed afterward. A martingale analysis establishes conditions under which truncation preserves the reference gradient expectation and reduces variance.
Critical strengths
The motivation is clear: a failed trajectory may contain useful actions, particularly recovery attempts, that should not receive the same penalty as the errors. The method integrates easily into GRPO and adds no deployment-time VLM cost. Across seven RoboTwin 2.0 tasks, average success increases from 65.01% to 68.64%. Preserving the original loss normalization and including uniform-scaling and random-prefix controls also strengthen the evaluation.
Critical weaknesses
-
The connection between the theory and the practical truncation rule remains unclear. The theoretical guarantee requires the removed gradient contribution to have zero conditional expectation. However, an absence of observed recovery does not establish that subsequent actions have no learning value. The VLM also uses future observations, violating the prefix-based condition of the ideal stopping rule. Although the paper analyzes potential bias, it does not measure gradient mean shifts, variance changes, or recovery probabilities after the selected boundaries in the robotics experiments. Direct evidence is needed to connect the theoretical analysis to the observed gains.
-
The stability and computational cost of the gains are insufficiently evaluated. Comparisons mainly involve Full GRPO, without directly related methods such as PCM. The two ablations cover only two tasks, and no results across independent training seeds are reported. Evaluation over different scene seeds does not replace repeated training runs. Moreover, adaptive rollout refilling means that 40 updates may require different numbers of environment interactions, while MART-GRPO additionally invokes a 27B VLM. Total interactions, training time, and results across multiple training seeds are needed to assess the benefit relative to its cost.
-
Annotation quality and localization evaluation need stronger support. The paper does not clearly explain who produced the reference annotations, how they were verified, or their inter-annotator agreement. Localization is evaluated only on 220 localizable, unrecovered failures, leaving incorrect truncation on ambiguous or recovered errors insufficiently assessed. Appendix E.3 also states that test diagnostics informed training duration, so the test set was not fully isolated from development. An independent test set and sensitivity analysis of earlier or later boundaries would strengthen the evidence.
-
How would the method perform on long-horizon, contact-sensitive tasks such as insertion or screw tightening, where recovery may require repeated retreats and adjustments with little visible progress? If the initial policy rarely recovers, could the frozen VLM truncate feedback too early and leave ineffective or harmful recovery actions unpenalized? Repeated continuations from the same failure states, evaluated by recovery success, would help establish whether the method improves recovery learning.
Questions for authors
see weakness.
3 comments
Thank you for the response, which has addressed most of my concerns. I will maintain my original score.
Thank you for your follow-up and careful review. Your constructive suggestions have helped us improve the manuscript, and we sincerely appreciate your time and engagement throughout the discussion.
Thank you for identifying the key questions about theoretical validity and empirical evidence. We conducted additional continuation, gradient, boundary-sensitivity, and multi-seed experiments.
W1. Connecting theory to practical VLM boundaries
Theorem 2 characterizes ideal prefix-based stopping; Proposition 4 and Eq. (11) describe the bias–variance tradeoff for MART's retrospective boundaries.
Recovery feasibility. We restored chunk-boundary states from 67 failed trajectories across four tasks and sampled 32 continuations per state under the initial policy and remaining budget. Fixed cuts match Table 10's retained feedback fractions.
Overall, 75–86% of trajectories have zero successes in 32 attempts. Combining continuation profiles with empirical VLM boundary distributions gives estimated success probabilities at the first masked chunk of 8.7–9.0% on Hammer and 5.1–5.3% on Handover.
Credit alignment. Mean signed alignment with a continuation-estimated reference is 0.505 for prefixes, 0.313 for Full feedback, and 0.218 for equal-size random subsets (64 trajectories). Prefix alignment exceeds Full on 52/64 trajectories and Random on 56/64.
Actual-mask bias and variance. Replaying VLM masks on a 512-trajectory Hammer bank at the frozen initial policy reduces recorded parameter-subspace variance by 33.7%, from 10.81×10−4 to 7.16×10−4. The squared mean shift is (4.09±0.68)×10−4. Under Eq. (11), estimated MSE change is (0.45±0.60)×10−4 (descriptive jackknife SE across four 128-trajectory minibatches).
Boundary sensitivity. On the same bank, shifting the boundary one chunk earlier or later yields 35.8% or 26.0% variance reduction, versus 33.7% at the original boundary.
These measurements connect recovery feasibility, temporal credit alignment, and the gradient behavior of retrospective masks.
W2. Training stability and interaction budget
Each original table entry used one run. We added two seeds, giving three runs per method on Hammer and Handover, with 256 evaluation episodes at update 40. Results are mean ± SD.
MART achieves the highest mean on both tasks. Gains over Full, Uniform, and Random are 8.33, 6.90, and 3.12 points on Hammer and 2.34, 2.34, and 0.65 points on Handover. Hammer and Handover were selected after encouraging seven-task results; we will clarify this exploratory selection.
Each run uses 40 updates and 20,480 accepted trajectories. Adaptive refill changes generated trajectory counts; comparisons use fixed update and accepted-trajectory budgets, not interaction-matched budgets.
W3. Annotation quality and evaluation protocol
Among 220 localizable, unrecovered failures, predicted boundaries match 150 human-annotated references (68%) exactly and fall within one chunk in 202 cases (92%). In 34 cases (15%), predictions are earlier. The other 36 cases comprise 32 Move Can Pot failures, three non-localizable Stack Bowls cases, and one Container on Plate case with uncertain recovery.
The policy experiments use the validation-selected update-304 localizer. Test diagnostics later informed extending training from update 361 to 437, while validation continued to select update 304. Videos come from Full-GRPO checkpoints at updates 5–40; the frozen localizer also generates masks on online MART rollouts (Table 10).
These metrics evaluate interval localization, not recovery classification. When recovery is uncertain or localization invalid, MART retains full feedback.
W4. Recovery in long-horizon tasks
MART retains full feedback when recovery is reported or uncertain, or localization is invalid. This is important for contact-sensitive tasks such as insertion and screw tightening. Evaluating trained MART and Full-GRPO policies from identical failure states with equal remaining budgets is a natural next step. We will illustrate recovery and fallback cases in Figure 3.
Compute and implementation
Measured end-to-end time per update is:
The additional cost primarily comes from VLM inference through a shared eight-GPU service. The localizer is frozen during training, deployment requires no VLM inference, and MART preserves original loss normalization.