- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies two concrete limitations in the standard two-stage pipeline for visual latent reasoning and addresses each one. In stage 1, prior methods encode a training-only helper image with a frozen general-purpose vision encoder to obtain the latent target. The authors instead train a scaffolding encoder whose target is optimized directly by the downstream answer cross-entropy, then train the VLM to regress that target from the original input alone. In stage 2, they argue that VLPO only penalizes latent drift with a fixed-variance Gaussian and does not sample latent actions. Scaffolding RL therefore learns an input-adaptive mean and variance over residual actions around the generated latent, and inserts the sampled action into GRPO together with the text tokens. At inference, both the scaffolding encoder and the helper image are discarded.
Critical strengths
- The problem is cleanly factored, and the ablations actually test the two claims. On FrozenLake, a frozen vision-feature target averages 57.8. Training only the pooling module already helps, and letting the task loss update the vision encoder reaches 72.0. Fine-tuning the same vision encoder with plain SFT moves the baseline only from 52.0 to 56.5, so the gain is not explained by extra parameters.
- The FrozenLake comparison is controlled. The authors state that the latent baselines use the same helper images, so the +9.5 cannot be attributed to extra privileged information. Table 5 further shows that a frozen encoder cannot exploit the value heatmap, it is even worse than a sparse red-arrow overlay, whereas the scaffolding encoder turns this non-natural image into a useful target. The contribution is therefore task alignment of the target, not merely a different choice of helper image.
Critical weaknesses
- The +5.6 on the nine visual benchmarks is not controlled to the same degree as FrozenLake. Table 2 states that unmarked numbers are taken from the original papers, and only some baselines are retrained or reimplemented under the authors’ setup. The controlled comparison in Appendix F is more credible: on the same Zebra-CoT data, a frozen target scores 66.3, the scaffolding encoder 70.9, and Scaffolding RL 73.9, only 1.2 above VLPO. The stage-1 conclusion still holds, but the stage-2 gain on real visual tasks is small and does not match the abstract’s equal emphasis on both stages. The main tables also lack multi-seed error bars. The standard deviations in Table 3 are about 0.5–0.7, so a +1.2 gain needs a clearer statistical statement.
- At inference, is the latent the Gaussian mean or a sample from the learned variance? If it is sampled, are the main-table numbers a single sample or an average over samples? Table 1 has no error bars, and the standard deviations in Table 3 are already comparable to some of the stage-2 gains.
Questions for authors
See weaknesses aboved.
6 comments
Dear Reviewer,
As the discussion period is now more than halfway through, we wanted to kindly follow up and ask whether our rebuttal has addressed your concerns. We would be happy to clarify any remaining questions.
Thank you again for your time and consideration.
Dear Reviewers,
We would like to kindly follow up on our previous message. We would greatly appreciate it if you could take a moment to review our responses and share any further feedback or remaining concerns.
Thank you for your time and consideration!
Response to Reviewer vo31
Thank you for the detailed and constructive review, and for recognizing our controlled FrozenLake comparison and parameter-matched ablation. We have followed each of the reviewer's suggestions below.
W1: Is Table 2 controlled, and how reliable are the gains?
(a) Controlled baselines. Thank you for raising this. We'd like to clarify that the strongest baselines in Table 2, SkiLa§ (68.3), VaLR§ (68.1), and DeepEyes§ (67.7), are already retrained under our exact setup (§: same Zebra-CoT data and Qwen2.5-VL-7B backbone), and the +5.6 margin is measured against SkiLa§. During the rebuttal, we further retrained the remaining latent-reasoning baselines with public code (Mirage, CoVT, LVR, and Monet) under the same setup:
Each retrained baseline stays slightly below (within 0.5 points of) its Table 2 number and below SkiLa§, so the +5.6 margin and our conclusions are unchanged. We are adding these rows to Table 2.
(b) Stage-2 gain and multi-seed results. Thank you for this suggestion. We'd like to confirm that the Stage-2 gain over VLPO is statistically significant, and Scaffolding RL improves the Stage-1 checkpoint 1.7–3.3× more than VLPO does.
Significance of the Stage-2 gain. Both RL recipes start from the same Stage-1 checkpoint. We report 3 seeds (mean ± SD), a two-sided Welch's t-test, and the RL gain over that checkpoint:
Scaffolding RL is significantly better than VLPO on both benchmarks (p < 0.05), and its RL gain is 1.7× (nine benchmarks) and 3.3× (FrozenLake) that of VLPO.
Multi-seed results per benchmark. Our Table 2 row equals the Scaffolding RL row of Table 6:
The gain is largest on fine-grained and high-resolution perception (MMVP, HRB-4K, MME, Jigsaw: +1.4 to +1.6), and is within seed noise on MMStar and CVBench (+0.6 to +0.8). Scaffolding RL also has smaller seed variance on every benchmark.
W2: Is the inference latent the Gaussian mean or a sample?
Thank you for pointing out this ambiguity. We'd like to clarify that the latent is sampled at inference (Δz ~ N(μ, σ²)), and all main-table numbers are avg@8, i.e., the mean accuracy over 8 sampled runs per query. We now report the SD over the 8 runs on every benchmark:
FrozenLake (Table 1)
Nine benchmarks (Table 2)
The SDs on the averages are small (≤0.3), well below the margins over all baselines, so the results are stable under sampling. We are stating this inference rule in Section 3.3 and adding these SDs to Tables 1 and 2 in the revised PDF.