acceptodds
Under review as a conference paper at ICLR 2027

Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning

Abstract

Latent reasoning has advanced multimodal reasoning through a two-stage training paradigm: (1) a helper image is encoded into latent tokens to teach visual chain-of-thought during a supervised fine-tuning (SFT) stage, and (2) these latent tokens are further refined with reward feedback during a reinforcement learning (RL) stage. In this paper, we identify two key limitations of this framework, one in each stage. First, the SFT stage typically relies on an off-the-shelf vision encoder to encode the helper image, yielding suboptimal latent representations that may not be well aligned with the downstream reasoning task. Second, existing RL methods treat the latent component only through deterministic regularization, which constrains policy drift but does not create alternative latent trajectories for exploration. To address these limitations, we propose Scaffolding Minds. Our approach learns a dedicated scaffolding encoder that provides an optimized target in latent space, and learns both the mean and variance of the RL sampler. We further show that these two improvements are complementary, together yielding substantial gains over strong baselines. Empirically, our method improves over the strongest latent-reasoning baseline by +9.5 points on FrozenLake spatial planning, with the gain widening to +19 points on the 32x32 grids, and by +5.6 points on average across nine visual-centric reasoning benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal reasoning, latent reasoning, visual reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vo31OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies two concrete limitations in the standard two-stage pipeline for visual latent reasoning and addresses each one. In stage 1, prior methods encode a training-only helper image with a frozen general-purpose vision encoder to obtain the latent target. The authors instead train a scaffolding encoder whose target is optimized directly by the downstream answer cross-entropy, then train the VLM to regress that target from the original input alone. In stage 2, they argue that VLPO only penalizes latent drift with a fixed-variance Gaussian and does not sample latent actions. Scaffolding RL therefore learns an input-adaptive mean and variance over residual actions around the generated latent, and inserts the sampled action into GRPO together with the text tokens. At inference, both the scaffolding encoder and the helper image are discarded.

Critical strengths
  1. The problem is cleanly factored, and the ablations actually test the two claims. On FrozenLake, a frozen vision-feature target averages 57.8. Training only the pooling module already helps, and letting the task loss update the vision encoder reaches 72.0. Fine-tuning the same vision encoder with plain SFT moves the baseline only from 52.0 to 56.5, so the gain is not explained by extra parameters.
  2. The FrozenLake comparison is controlled. The authors state that the latent baselines use the same helper images, so the +9.5 cannot be attributed to extra privileged information. Table 5 further shows that a frozen encoder cannot exploit the value heatmap, it is even worse than a sparse red-arrow overlay, whereas the scaffolding encoder turns this non-natural image into a useful target. The contribution is therefore task alignment of the target, not merely a different choice of helper image.
Critical weaknesses
  1. The +5.6 on the nine visual benchmarks is not controlled to the same degree as FrozenLake. Table 2 states that unmarked numbers are taken from the original papers, and only some baselines are retrained or reimplemented under the authors’ setup. The controlled comparison in Appendix F is more credible: on the same Zebra-CoT data, a frozen target scores 66.3, the scaffolding encoder 70.9, and Scaffolding RL 73.9, only 1.2 above VLPO. The stage-1 conclusion still holds, but the stage-2 gain on real visual tasks is small and does not match the abstract’s equal emphasis on both stages. The main tables also lack multi-seed error bars. The standard deviations in Table 3 are about 0.5–0.7, so a +1.2 gain needs a clearer statistical statement.
  2. At inference, is the latent the Gaussian mean or a sample from the learned variance? If it is sampled, are the main-table numbers a single sample or an average over samples? Table 1 has no error bars, and the standard deviations in Table 3 are already comparable to some of the stage-2 gains.
Questions for authors

See weaknesses aboved.

6 comments
AuthorsOpenReview ↗

Response to Reviewer vo31

Thank you for the detailed and constructive review, and for recognizing our controlled FrozenLake comparison and parameter-matched ablation. We have followed each of the reviewer's suggestions below.

W1: Is Table 2 controlled, and how reliable are the gains?

(a) Controlled baselines. Thank you for raising this. We'd like to clarify that the strongest baselines in Table 2, SkiLa§ (68.3), VaLR§ (68.1), and DeepEyes§ (67.7), are already retrained under our exact setup (§: same Zebra-CoT data and Qwen2.5-VL-7B backbone), and the +5.6 margin is measured against SkiLa§. During the rebuttal, we further retrained the remaining latent-reasoning baselines with public code (Mirage, CoVT, LVR, and Monet) under the same setup:

MethodV*BLINKMMVPMMStarCVB.HRB-4KHRB-8KMMEJig.Avg.Table 2 Avg.
Mirage§ (new)80.357.057.165.476.069.366.141.853.162.963.3
CoVT§ (new)77.655.758.268.579.672.569.051.052.064.965.3
LVR§ (new)81.256.371.066.877.870.967.143.851.965.265.7
Monet§ (new)82.757.259.867.979.270.567.655.054.166.066.5
VaLR§ (Table 2)84.662.370.269.783.270.668.750.153.868.168.1
SkiLa§ (Table 2)84.857.575.765.482.172.867.354.654.368.368.3
Ours90.667.276.773.588.477.574.157.359.773.973.9

Each retrained baseline stays slightly below (within 0.5 points of) its Table 2 number and below SkiLa§, so the +5.6 margin and our conclusions are unchanged. We are adding these rows to Table 2.

(b) Stage-2 gain and multi-seed results. Thank you for this suggestion. We'd like to confirm that the Stage-2 gain over VLPO is statistically significant, and Scaffolding RL improves the Stage-1 checkpoint 1.7–3.3× more than VLPO does.

Significance of the Stage-2 gain. Both RL recipes start from the same Stage-1 checkpoint. We report 3 seeds (mean ± SD), a two-sided Welch's t-test, and the RL gain over that checkpoint:

BenchmarkStage-1 checkpointVLPO (RL gain)Scaffolding RL (RL gain)p-value
9-bench (Table 6)70.972.7±0.5 (+1.8)73.9±0.2 (+3.0)0.039
FrozenLake (Table 3)72.0±0.672.9±0.6 (+0.9)75.0±0.7 (+3.0)0.018

Scaffolding RL is significantly better than VLPO on both benchmarks (p < 0.05), and its RL gain is 1.7× (nine benchmarks) and 3.3× (FrozenLake) that of VLPO.

Multi-seed results per benchmark. Our Table 2 row equals the Scaffolding RL row of Table 6:

Stage 2V*BLINKMMVPMMStarCVB.HRB-4KHRB-8KMMEJig.Avg.
VLPO89.3±1.166.2±0.875.3±1.272.9±0.687.6±0.675.9±0.972.9±1.055.7±0.858.2±1.072.7±0.5
Scaffolding RL90.6±0.667.2±0.576.7±0.773.5±0.588.4±0.577.5±0.574.1±0.657.3±0.459.7±0.673.9±0.2

The gain is largest on fine-grained and high-resolution perception (MMVP, HRB-4K, MME, Jigsaw: +1.4 to +1.6), and is within seed noise on MMStar and CVBench (+0.6 to +0.8). Scaffolding RL also has smaller seed variance on every benchmark.

W2: Is the inference latent the Gaussian mean or a sample?

Thank you for pointing out this ambiguity. We'd like to clarify that the latent is sampled at inference (Δz ~ N(μ, σ²)), and all main-table numbers are avg@8, i.e., the mean accuracy over 8 sampled runs per query. We now report the SD over the 8 runs on every benchmark:

FrozenLake (Table 1)

MethodLv. 8Lv. 16Lv. 24Lv. 32Avg.
Scaffolding Encoder+RL96.0±0.782.0±1.167.0±1.355.0±1.675.0±0.3

Nine benchmarks (Table 2)

MethodV*BLINKMMVPMMStarCVB.HRB-4KHRB-8KMMEJig.Avg.
Scaffolding Encoder+RL90.6±1.067.2±0.676.7±1.173.5±0.488.4±0.377.5±0.774.1±0.857.3±0.759.7±0.973.9±0.2

The SDs on the averages are small (≤0.3), well below the margins over all baselines, so the results are stable under sampling. We are stating this inference rule in Section 3.3 and adding these SDs to Tables 1 and 2 in the revised PDF.

Dear Reviewer,

As the discussion period is now more than halfway through, we wanted to kindly follow up and ask whether our rebuttal has addressed your concerns. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Dear Reviewers,

We would like to kindly follow up on our previous message. We would greatly appreciate it if you could take a moment to review our responses and share any further feedback or remaining concerns.

Thank you for your time and consideration!

Reviewer_KFTVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies supervision targets and reinforcement learning for multimodal latent reasoning. It first trains a scaffolding encoder through downstream answer loss to encode training-time helper images into target latent representations, then trains a VLM to predict these representations without helper images. A learned Gaussian residual policy subsequently explores the latent space through reinforcement learning. Experiments cover FrozenLake spatial planning and nine visual reasoning benchmarks, reporting gains of 9.5 and 5.6 percentage points over the strongest latent-reasoning baselines, respectively, alongside component ablations and inference-efficiency analysis.

Critical strengths
  1. Under the same backbone and training data, the target-representation ablations show improvements over mean-pooled frozen visual features on both FrozenLake and the visual-centric benchmarks. This provides empirical evidence for task-supervised target learning and shows that the gains do not arise solely from the subsequent RL stage.
  2. Evaluation covers both structured spatial planning and real-image understanding, with component gains verified in both settings. Additional experiments examine teacher-forced versus self-predicted latent contexts, smaller models, and LoRA training. Removing the scaffolding encoder at inference yields measured latency overheads of approximately 4.6%–5.9%, giving the improvements practical relevance.
Critical weaknesses
  1. Monet [1] already learns helper-image-conditioned target latent representations in its second SFT stage and subsequently trains a model to predict those targets without auxiliary images. It does not simply regress onto raw frozen vision-encoder features. Grouping Monet with fixed generic-feature targets therefore overstates the distinction motivating Stage 1. The dedicated encoder and answer-loss supervision are concrete differences, but the target ablations in Figure 6 and Table 6 primarily compare against mean-pooled frozen features. They do not sufficiently isolate the additional benefit over an existing learned-target construction.
  2. GTS [2] already explores continuous latent states through a learned Gaussian residual distribution optimized with GRPO. DLR [3] also samples stochastic latent actions in VLMs and jointly optimizes text and latent policies. These precedents limit both the claim that prior methods lack latent exploration and the novelty of Scaffolding RL.
Questions for authors
  1. Does the FrozenLake implementation of Monet retain its complete target-learning and distillation pipeline? Can you compare its target construction against the scaffolding encoder under matched training data, latent-token counts, and training budgets to isolate the contribution of Stage 1?
  2. Starting from the same scaffolding-trained checkpoint and using the same rollout budget, does Scaffolding RL outperform a GTS-style Gaussian residual policy or DLR’s stochastic latent policy? Which specific design choices account for any remaining advantage?
7 comments
AuthorsOpenReview ↗

Response to Reviewer KFTV

Thank you for the careful and constructive review, and for recognizing our controlled target ablations and efficiency analysis.

W1/Q1: Monet's learned target

Thank you for the helpful clarification of Monet's pipeline. We'd like to note that Monet indeed learns its latent target rather than regressing onto frozen vision features, and we are refining its description in Sections 2 and 4 to make this explicit. We'd like to clarify the key difference in how the target is learned:

  • Monet: frozen vision encoder + trainable VLM as the latent target encoder.
  • Ours: trainable vision encoder + frozen VLM as the latent target encoder.

Our method has three advantages:

  1. Fewer trainable parameters. We train only a copy of the vision tower (~0.7B) instead of the full 7B VLM.
  2. No shortcut. Since our VLM is frozen, the only way to lower the answer loss is to improve the latent. In Monet, the trainable VLM can instead learn to answer directly from the question and image, bypassing the latent, where we show in an experiment below. Our encoder also sees only the helper image, which decouples the latent from the input.
  3. Faster training. Monet generates latents autoregressively, one forward step per token, while we produce all K latents in parallel from learnable embeddings.

Experiments. Our paper's Monet baseline runs the official code with the full pipeline (three SFT stages + VLPO, K = 8) on our data. To isolate the contribution of Stage 1, during the rebuttal we also train our method on Monet's official training set, Monet-SFT-125K, so that the data are identical (K = 8, no RL) and only the latent target construction differs. To test the shortcut (advantage 2), we zero out the latent at inference; to test speed (advantage 3), we report training time on 8×H100:

Stage 1 on Monet-SFT-125KV*HRB-4KHRB-8KMME-RWVisualPuzzlesAvg.Avg. with zeroed latentTrain time
Monet-SFT (official numbers, Monet Table 5)82.2068.5066.0052.6830.4859.9757.85 (−2.12)~125 h
Scaffolding encoder (ours)85.3471.2569.0054.8732.5362.6037.32 (−25.28)~68 h
  • Better target: on the same data, ours outperforms Monet-SFT on every benchmark (+2.63 on average), without needing Monet's observation-span labels.
  • Shortcut confirmed: zeroing the latent costs Monet only 2.12 points, so its answers barely depend on the latent; ours drops by 25.28 points, so the latent carries the answer-relevant information.
  • Faster: our Stage 1 trains about 1.8× faster.

W2/Q2: GTS and DLR

Thank you for pointing us to these interesting works; we will cite and discuss both in the revised PDF, and refine our Stage-2 claim to prior latent visual reasoning RL (e.g., VLPO), which does not sample latent actions. We'd like to note that both are contemporaneous under the ICLR 2027 Reviewer Guide: "if a paper was published (i.e., at a peer-reviewed venue) on or after July 17, 2026, authors are not required to compare their own work to that paper. Note that arXiv is not considered a peer-reviewed venue."

  • GTS is available only on arXiv (Feb 2026).
  • DLR was accepted to EMNLP 2026, with notification on August 20, 2026, after the July 17 cutoff.

Nevertheless, we are happy to provide an apples-to-apples comparison.

Method differences.

  • GTS: text-only latent CoT (COCONUT/CODI on GSM8K). The backbone is frozen; only two 2-layer MLP heads predicting μ and σ of a latent residual are trained.
  • DLR: a VLM in which a separate grounder outputs a mean direction on the unit sphere, with a fixed variance σ.
  • Ours: a VLM with input-adaptive μ and σ, trained jointly with the whole VLM, around the task-optimized latent prior from Stage 1.

Experimental comparison. On FrozenLake, all rows start from the same Stage-1 checkpoint (72.0) with the same rollout budget (3 seeds):

Stage 2 recipeWhat RL trainsAvg.
VLPO (Monet), Table 3VLM, latent not sampled72.9±0.6
Learned mean, fixed σ (DLR-like), Table 3VLM + μ head73.6±0.5
GTS-style (new)μ, σ heads only; VLM frozen72.8±0.6
Ours, Table 3VLM + μ, σ heads75.0±0.7

The results show that both of our design choices matter: making σ input-adaptive adds +1.4 over the DLR-like row, and training the VLM jointly adds +2.2 over the GTS-style row.

Dear Reviewer,

As the discussion period is now more than halfway through, we wanted to kindly follow up and ask whether our rebuttal has addressed your concerns. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Dear Reviewers,

We would like to kindly follow up on our previous message. We would greatly appreciate it if you could take a moment to review our responses and share any further feedback or remaining concerns.

Thank you for your time and consideration!

Reviewer_TWYxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes training an auxiliary visual encoder during SFT to learn better latent features for reasoning, showing clear improvements over using an off-the-shelf encoder. It then further optimizes the latent representations through a Gaussian latent policy that samples residual actions around the predicted latent. The method shows promising results across several visual-centric reasoning benchmarks.

Critical strengths
  • Training a reasoning-oriented visual encoder makes sense, and it is reasonable to expect the policy model to benefit from better latent representations.

  • The experiments and ablation studies are comprehensive.

  • The paper is well written and easy to follow.

Critical weaknesses
  • The experiments only cover a single model, which makes it difficult to assess how well the approach generalizes to other modes.

  • The method relies on helper images, which may limit its applicability to more realistic scenarios, which is also acknowledged in the limitations.

Questions for authors
  • It would be great to see whether the approach also works with a different model family such as Gemma.

  • Can the trained model generalize to OOD settings, such as larger map sizes or other grid-based tasks like Maze?

7 comments
AuthorsOpenReview ↗

Response to Reviewer TWYx

We sincerely thank the reviewer for the thoughtful feedback and for recognizing the motivation for a reasoning-oriented target encoder and the breadth of our experiments and ablations.

W1/Q1: Generalization to other models

Thank you for suggesting another model family. We'd like to note that both of our components, the scaffolding encoder and Scaffolding RL, consistently outperform the baselines across model scales, training regimes, and model families.

  • Other scales and training regimes (Appendix F.6, Table 10). On Qwen2.5-VL-3B, the scaffolding encoder alone already exceeds VaLR (61.3 vs. 58.3), and adding RL reaches 64.0 (+5.7). With 7B LoRA (r = 32), the scaffolding encoder reaches 57.0 vs. 53.3, and adding RL reaches 58.8 (+5.5).
  • A different model family (new). During the rebuttal, we train Gemma-3-4B-it with the same data, helpers, K = 4, and hyperparameters (Appendix D), using a trainable copy of Gemma's vision tower as the scaffolding encoder.
Gemma-3-4B-it (FrozenLake)Lv. 8Lv. 16Lv. 24Lv. 32Avg.
SFT8063401750.0
VaLR8774533061.0
Scaffolding Encoder9076583865.5
Scaffolding Encoder+RL9279614068.0

The scaffolding encoder alone outperforms VaLR at every level (+4.5 on average), and Scaffolding RL adds a further +2.5, giving +7.0 over VaLR (vs. +9.5 on Qwen2.5-VL-7B). We are adding the Gemma results to Appendix F.6.

W2: Helper images and applicability to realistic scenarios

Thank you for raising this practical point. We'd like to clarify that our method does not need a separately constructed helper image: using the input image itself as the helper retains nearly all of the gain.

Helper images are a shared training-time setting of latent visual reasoning (Mirage, LVR, Monet, VaLR, SkiLa; Section 2), and, like these methods, we discard the helper at inference (Section 3.2). To test the setting without any helper, we set the helper to the input image itself (h = x) for both VaLR and our method, keeping everything else unchanged (nine benchmarks, Qwen2.5-VL-7B, no RL):

Method (h = input image)V*BLINKMMVPMMStarCVB.HRB-4KHRB-8KMMEJig.Avg.Avg. with helper (Table 2)
Vanilla SFT81.357.258.766.077.070.566.842.052.563.663.6
VaLR83.561.069.068.682.069.567.649.052.867.068.1
Scaffolding Encoder86.564.072.471.285.572.770.352.654.870.070.9

Without any helper, the scaffolding encoder drops by only 0.9 points, still outperforms VaLR on every benchmark (+3.0), and improves over SFT by +6.4. This makes the method directly applicable to domains without helper images, while helpers, when available, add a further gain. We are adding this result to Appendix F and updating the limitation discussion in Appendix A.

Q2: Out-of-distribution generalization

Thank you for this suggestion. We'd like to address three levels of OOD: unseen sizes, larger sizes, and a new task (Maze).

1. Unseen sizes (in the paper). Training uses only even levels (8, 10, …, 32), and testing covers all 12 odd levels (9, 11, …, 31), never seen during training (Section 4.1, Figure 5). On these 1,200 unseen layouts, ours reaches 74.8 vs. 66.2 for VaLR (+8.7 before rounding).

2. Larger sizes (new). Without retraining, we test 100 layouts each at Levels 36 and 40, resizing the full grid to the Level-32 input resolution for both methods:

MethodLv. 36Lv. 40
VaLR2516
Scaffolding Encoder4331

3. A new task: Maze (new). We apply the FrozenLake-trained checkpoints zero-shot to Maze, with no Maze training data or helpers. We use the Maze environment of VPRL (Xu et al., 2025; built on Ivanitskiy et al., 2023) at larger sizes, with 100 mazes per size. The model outputs an action sequence, and we report VPRL's Exact Match (EM): the sequence must be a valid shortest path from start to goal.

Method (EM, %)8×816×1624×24Avg.
SFT52301432.0
VaLR57362238.3
Scaffolding Encoder66513350.0

Our model transfers to Maze with +11.7 EM over VaLR on average and leads at every size. This suggests the scaffolding encoder learns a general spatial-planning representation rather than FrozenLake-specific features. We will add both new evaluations to the revised PDF.

Dear Reviewer,

As the discussion period is now more than halfway through, we wanted to kindly follow up and ask whether our rebuttal has addressed your concerns. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Dear Reviewers,

We would like to kindly follow up on our previous message. We would greatly appreciate it if you could take a moment to review our responses and share any further feedback or remaining concerns.

Thank you for your time and consideration!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Part 1/2)

We thank all reviewers for their constructive feedback.

What reviewers found strong. All three reviewers recognize our core idea, a latent target learned for the task:

  • Problem (vo31). The paper clearly identifies one limitation in each training stage.
  • Method (TWYx, vo31). Training a visual encoder for reasoning makes sense. It even turns a non-natural helper image (a value heatmap) into a useful target, where a frozen encoder fails.
  • Experiments (all three).
    • Controlled: all latent baselines use the same helper images, and plain SFT of the same encoder does not explain the gain (vo31). The target ablations show gains on both FrozenLake and real images, not only from RL (KFTV).
    • Broad: spatial planning and real images, plus smaller models, LoRA, and teacher-forced vs. self-predicted latents (KFTV); comprehensive ablations (TWYx).
    • Practical: inference is only 4.6–5.9% slower (KFTV).
  • Presentation (TWYx). The paper is well written and easy to follow.

Overall, our method beats the strongest latent baseline by +9.5 points on FrozenLake (+19 on the largest 32×32 grids) and by +5.6 points on average across nine visual benchmarks.

Main concerns and how we addressed them

1. Difference with the Monet paper (KFTV) (KFTV: resolved)

Method differences.

  • Frozen VLM vs. fine-tuned VLM. Monet fine-tunes the VLM to build its target, so the VLM can answer from the question and image and bypass the latent (a shortcut). We keep the VLM frozen and train a dedicated encoder (~0.76B) through the answer loss, so the loss can only drop by improving the latent.
  • Parallel vs. autoregressive latents. Monet generates latents one forward step per token; we produce all K latents in parallel.

Results (new). As KFTV suggested, we train our Stage 1 on Monet's own data (Monet-SFT-125K) with the same K = 8:

  • Better target: ours averages 62.60 vs. 59.97. On FrozenLake, Monet's Stage 1 reaches 60.5 vs. 72.0 for ours (Figure 6).
  • No shortcut: zeroing the latent costs ours 25.28 points but Monet's only 2.12.
  • Faster training: ~68 h vs. ~125 h (1.8×).

Our Monet baseline runs its full official pipeline (three SFT stages + VLPO).

2. RL concurrent works GTS and DLR also sample latent actions (KFTV). (KFTV: resolved) Both are contemporaneous (ICLR 2027 Reviewer Guide); we still discuss both and narrowed our Stage-2 claim to prior RL for latent visual reasoning.

Method differences.

  • GTS: text-only latent CoT with a frozen backbone; only the mean and std heads are trained. We train the heads jointly with the whole VLM.
  • DLR: a separate head predicts a mean direction with a fixed σ, a design already compared in our original paper (learned mean, fixed σ; Table 3). We learn an input-adaptive mean and σ around the Stage-1 latent.

Results. From the same Stage-1 checkpoint and rollout budget on FrozenLake, ours reaches 75.0±0.7:

  • Learning σ helps: +1.4 over DLR (73.6±0.5, Table 3).
  • Training the VLM jointly helps: +2.2 over GTS (72.8±0.6, new).
AuthorsOpenReview ↗

General Response (Part 2/2)

Main concerns (continued)

3. Controlled baselines and significance (vo31). (vo31: resolved)

  • Beyond SkiLa§, VaLR§, and DeepEyes§, we retrained Mirage, CoVT, LVR, and Monet under our setup (62.9 / 64.9 / 65.2 / 66.0); the +5.6 margin is unchanged.
  • Over 3 seeds, Scaffolding RL beats VLPO on the nine benchmarks (73.9±0.2 vs. 72.7±0.5, p = 0.039) and on FrozenLake (75.0±0.7 vs. 72.9±0.6, p = 0.018), with 1.7× and 3.3× the RL gain of VLPO.

4. Inference rule (vo31). (vo31: resolved) The latent is sampled at inference; all main-table numbers are avg@8. SD over 8 runs: ±0.3 (FrozenLake), ±0.2 (nine benchmarks).

5. Generalization beyond one model and one task (TWYx). (TWYx: resolved) We added new experiments during the rebuttal:

  • Another model family. On Gemma-3-4B-it, our full method reaches 68.0 on FrozenLake, vs. 61.0 for VaLR.
  • Larger, unseen grids. Without retraining, our scaffolding encoder reaches 43 and 31 at Levels 36 and 40, vs. 25 and 16 for VaLR.
  • A new task. Applied zero-shot to Maze, our FrozenLake-trained scaffolding encoder reaches 50.0 EM, vs. 38.3 for VaLR.

6. Dependence on helper images (TWYx). (TWYx: resolved) We'd like to note that:

  • Helpers are needed only for training. As in prior latent visual reasoning papers, the helper only builds the training target; at inference, the model sees just the input image and question.
  • New experiment: no extra helper image. We use the input image itself as the helper (h = x). On the nine benchmarks, the scaffolding encoder still reaches 70.0, only 0.9 below using helpers (70.9), and beats VaLR (67.0) and SFT (63.6) in the same setting. So the method works without helper images, and helpers add a further gain when available.

Revised PDF

All changes are coloured by reviewer:

  • KFTV (blue): Monet description (Secs. 2, 4, 5); narrowed Stage-2 claim (Abstract, Sec. 1); Monet in Figure 6 (Sec. 4.3); GTS and DLR in Related Work; App. B and F.8.
  • vo31 (orange): inference rule (Sec. 3.3); SDs and four retrained baselines (Tables 1–2); significance tests (Sec. 4.3, Table 8); App. F.9.
  • TWYx (green): Gemma and Maze (Sec. 4.4, Table 4); input image as helper and limitation (App. F.10, A); larger grids (App. F.11).

Discussion status

  • Reviewer KFTV: replied that their concerns "have, for the most part, been well addressed" and raised the score to weak acceptance.
  • Reviewer vo31: replied that our response "addresses my concerns well" and maintains the rating of weak acceptance.
  • Reviewer TWYx: replied that they maintains the positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.