acceptodds
Withdrawn from ICLR 2027

PIVOT: Pivot-aware On Policy Self Distillation for Multi-Turn VLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards (RLVR) via Group-Relative Policy Optimization (GRPO) is widely used for multi-turn VLM agent training, yet it suffers from zero-gradient silence on uniform failures and coarse episode-level credit assignment. While On-Policy Distillation (OPD) and On-Policy Self-Distillation (OPSD) mitigates sparse rewards using hindsight information, its underlying mechanisms remain poorly understood. Through controlled counterfactual rollback probes across five multi-turn VLM agent benchmarks, we reveal that performance gains in OPSD/OPD are largely driven by physical state rollback at the pivot step, defined as the first unrecoverable action without remaining step budget. However, physical state rollbacks are computationally prohibitive and infeasible in real-world environments. To bridge this gap, we present Pivot-Aware Internalized Visual On-Policy Training (PIVOT), an RL framework that internalizes pivot localization and state restoration directly into token-level parameter updates, eliminating environment rollbacks during RL training and additional skill hints at test time. PIVOT unifies three functional roles within a single architecture: a failure Analyzer non-invasively localizes the pivot step and diagnoses failure modes from visual trajectory collages and action logs; a detached Teacher re-scores failed tokens under this privileged diagnostic context; and a Student optimizes joint GRPO and confidence-gated OPD objectives. At test time, both Teacher and Analyzer branches are stripped. Evaluated on five multi-turn VLM agent tasks across cognitive grid puzzles, 3D embodied control and navigation, and generative reasoning, PIVOT achieves overall accuracy on Qwen2.5-VL-3B ( over SFT+GRPO baseline and over previous SOTA) and scales to on Qwen3-VL-2B ( over SFT+GRPO baseline).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM Agents, On-Policy Distillation, GRPO, Credit Assignment, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jgkhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PIVOT trains multi-turn VLM agents through a shared Analyzer–Teacher–Student framework. It localizes critical failure steps and distills privileged visual hindsight into the acting policy using GRPO and gated self-distillation.

Critical strengths

This paper addresses an important limitation of rl agent training: extracting useful supervision from failed trajectories when episode-level rewards provide weak or zero advantages. Experimental results report promising improvements across five tasks and two Qwen backbones, with ablations supporting the value of localized failure diagnostics.

Critical weaknesses
  • End-to-end cost-effectiveness is not established. The 95 s versus 102 s RL-update comparison covers only Sokoban and excludes complete data preparation and SFT costs. Removing the Analyzer and Teacher at inference avoids additional branches, but deployment latency, token consumption, and cost per successful episode remain unreported.
  • The proprietary VLM comparison (such GPT-5.6-Sol or 6-Astra) is insufficient to establish frontier competitiveness (especially given that the authors proposed a domain-specific model). Table 1 largely inherits published baseline results and lacks an updated, controlled comparison against the strongest contemporary proprietary models. Comparing task-trained PIVOT with frozen API models also limits the interpretation of its claimed superiority.
Questions for authors

What training-level evidence distinguishes “internalizing rollback” from learning with additional privileged hindsight supervision? I understand the "internalization" means that the student will eventually match or even exceed its teacher after imitation.

Reviewer_WA4GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why on-policy distillation (OPD) and on-policy self-distillation (OPSD) help multi-turn VLM agents trained with GRPO, and proposes a method based on the answer. The motivating problem is that GRPO gives no gradient to groups whose rollouts all receive the same return and assigns credit only at the episode level. The authors first run a counterfactual rollback probe on failed rollouts of the untrained model. They define a “pivot step” t* as the first action after which, according to a task-specific feasibility check, the goal is unreachable within the remaining step budget. They roll the environment back to the state at t*, resample one suffix under four conditions (no hint, no hint rolled back to t*+1, self-generated skill hints from the 3B model, and skill hints from a Qwen3-VL-8B teacher), and measure how often the new suffix return exceeds the old one. They report that rollback alone yields most of the lift, that rolling back one step later reduces it, and that adding hints changes it only slightly. From this they conclude that the gains of OPD/OPSD in this setting come mainly from state rollback at the pivot step rather than from the hints.

Experiments use five VAGEN tasks, namely Sokoban, FrozenLake, Navigation, PrimitiveSkill and SVG reconstruction, with Qwen2.5-VL-3B and Qwen3-VL-2B as backbones. The full variant, M+P+R, reaches an average of 0.90 on the 3B model against 0.83 for SFT-initialized GRPO, and 0.92 on the 2B model against 0.82. Baselines include open-source and proprietary VLMs, two earlier results on the same benchmark, and a reproduced OPD baseline. The earlier results are VAGEN-Full at 0.81 and GLANCE-Full at 0.86, and the reproduced baseline, SDAR, reaches 0.79.

Ablations vary the teacher’s privileged context, which consists of the failure mode, the pivot panel and the guidance text, and replace the predicted pivot with a random step. Further analyses report the Analyzer’s pivot-localization accuracy on FrozenLake over training, and per-step training time: 95 s for PIVOT, 102 s for vanilla GRPO and 373 s with a frozen external 8B teacher.

Critical strengths

The main strengths are:

  1. The problem is well motivated and the proposed loss targets it directly. The paper does not just assert that GRPO stalls on uniform failures. It measures how often this happens. On the untrained Qwen2.5-VL-3B, 48.3% of Sokoban groups and 16.5% of PrimitiveSkill groups have zero return variance, and the appendix separates this from the all-fail rate and states how groups are formed and discarded. The design then follows from this diagnosis. The distillation term is applied only to failed trajectories, so when a whole group fails and the GRPO term vanishes, a token-level training signal remains. The mechanism is simple and easy to implement.
  2. The paper asks a diagnostic question worth asking. Much of the OPD and OPSD literature reports that hindsight or teacher signals help without asking where the gain comes from. Here the authors try to separate the effect of resetting the environment from the effect of the textual signal. They do this with a probe that has no-hint, shifted-reset, self-hint and larger-teacher arms. Asking this question for multi-turn VLM agents is useful for the community, and the idea of connecting a diagnostic to a method that avoids the costly part is a sound way to structure a paper.
  3. Reporting is transparent and the work is easy to audit. The appendix gives the full prompts for all three roles in every environment, the SFT data sizes, the pivot and failure-mode rules for each task, the hyperparameters, and the details of the SDAR reproduction. The main text also reports results that do not favor the method. On the 3B model, adding guidance text lowers the DreamSim score on SVG. On the 2B model, the variant without guidance text beats the full variant on Sokoban. The limitations section openly states that the labels depend on environment solvers, that only small models were tested, and that all experiments are in simulation. This makes it much easier for a reviewer to check the claims.
Critical weaknesses

The main weaknesses are:

  1. The claim that PIVOT “internalizes rollback” is never tested against whether rollback would have helped on the cases where PIVOT improves. Section 2 shows that resetting the untrained model to st∗s_{t*} and resampling raises suffix return on failed rollouts. Section 4 shows that the trained PIVOT policy scores higher on validation. Nothing links the two. The paper never splits instances by whether recovery from t∗t^* is actually possible, and never reports whether PIVOT’s gains fall in the recoverable group. Gains concentrated where rollback succeeds would be consistent with the explanation, though still not proof. Gains concentrated where it does not would leave them unexplained.

  2. The empirical gains are modest, partly saturated, and not statistically supported. The components the paper emphasizes contribute little. On the 3B model, M already reaches 0.88 overall and M+P also 0.88, so the pivot panel adds nothing on average, and FrozenLake falls from 0.83 to 0.77. M+P+R adds 0.02, mostly on Sokoban and FrozenLake. SVG DreamSim drops from 0.74 to 0.71, and on the 2B model Sokoban drops from 0.97 (M+P) to 0.92. PrimitiveSkill is 1.00 for every variant from M onward on both backbones, so it cannot separate them. Most of the gain over SFT-GRPO is already present with the failure-mode text alone. The random-pivot control uses Sokoban and Navigation only, and compares against Vanilla-GRPO even though all PIVOT variants start from the SFT weights, where the reference is 0.82/0.85. No seed count or variance is reported. Validation uses 128 episodes per environment, where the binomial standard error near 0.9 is about 0.03, similar to most ablation differences. Reported numbers are running-best validation points on the same set (App. J), which biases results upward, and the curves in Figs. 9 and 10 fluctuate considerably.

Questions for authors

Regarding Weakness 1, please answer the following:

To test the claim that PIVOT internalizes rollback, I would ask the authors to do four things.

  • stratify validation instances by the slack at the pivot state, which the existing solvers can compute, and report both the SFT-GRPO versus PIVOT difference and the rollback outcome in each stratum.
  • report the true success rate of rollback from the pivot state, not only the fraction of resampled suffixes with a strictly higher return, so that it is clear in how many cases rollback actually recovers the task.
  • add a rollback-only training baseline, where failed rollouts are restored to the pivot state and the resampled suffixes are used for GRPO updates without any distillation, at a matched environment-step budget.
  • measure the actual cost of rollback in these simulators. The paper compares training time only against an external 8B teacher and never against a rollback baseline, so the claim that rollback is prohibitive is currently unmeasured.

Regarding Weakness 2, please answer the following:

  • the ablation should be run against the right reference and with wider coverage. Table 2 covers only Sokoban and Navigation and compares against Vanilla-GRPO, but every PIVOT variant is initialized from the Analyzer SFT weights, so the relevant reference is SFT-GRPO (0.82 and 0.85 on those two tasks). Measured against it, the random-pivot arm is 0.07 and 0.05 below, and the single-channel variants are much closer. I would ask for the ablation on all five tasks and both backbones, with SFT-GRPO as the reference. The random-pivot control should also be complemented. Replacing the predicted step with a random one makes the panel, the pivot index and the guidance text inconsistent with each other, so it tests mismatch and not the value of localization. Controls such as no panel with the same text, a fixed offset from the predicted step, or always using the last step would separate these effects.
  • the efficiency claim needs more support. Figure 5 reports per-step time on Sokoban only, and the explanation that PIVOT converges to shorter trajectories that reduce rollout time is not measured. Sokoban also has a horizon of only 9 steps. The efficiency claim rests on a single measurement, 9595 s vs. 102102 s per step on Sokoban, where Tmax⁡=9T_{\max}=9. The stated cause, shorter trajectories, is not measured, and Stage I (failed rollouts, solver labels, guidance text from the 8B model) appears not to be counted. Supporting it requires per-step time on all five environments split by component, mean trajectory length over training, and total wall-clock time to reach a fixed score, including Stage I.
Reviewer_6LdqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reward assignment for multi-turn VLM agents trained with GRPO. The authors use probes to analyse failed trajectories on these VAGEN tasks to identify a "pivot" step. Their proposed PIVOT methodology includes:

  • An Analyzer: predicts the pivot step, a failure mode and a corrective hint from a frame collage and action log
  • A Teacher: re-scores the original failed tokens with a privileged context from Analyser
  • A Student: trained with GRPO plus a sigmoid-gated OPD term on failed trajectories.
Critical strengths
  • Motivation is well-founded: zero-variance groups and episode-level reward assignment in multi-turn/step agentic RL
  • The counterfactual-restoration probe is a good experimental idea, and the raw numbers in Tables 7/8 look comprehensive
  • The proposed methodology shows consistent gains over SFT-GRPO baseline across the five tasks (with no additional overhead/cost)
Critical weaknesses
  • The probe does not actually test OPD/OPSD since the environment state is restored and resampled from in their setup (instead of just re-scoring on-policy samples under a teacher or privileged context)
  • The OPD term just seems to be an unsigned self-imitation of failed trajectories
    • Minimizing it will always raises the probability of the sampled tokens on failed trajectories
    • It never pushes probability away from the pivot action
  • Missing or weak baselines: the “prohibitive cost” claim can't be justified without running VinePPO or other relevant comparisons
  • The method does not do what the probe says matters (never restores or resamples from s_t*)
Questions for authors
  • Is the OPD coefficient in your implementation exactly the stop-gradient sigmoid gate in Eq. 13, or is there a signed component?
  • When t < t_^, the teacher sees future frames, does the OPD effect differ for tokens before, at and after t_^?
  • Can you add s_0-resample, (t* - k) and hint-without-restoration to the probe, with CIs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.