acceptodds
Under review as a conference paper at ICLR 2027

Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning

Abstract

Multimodal Large Language Models (MLLMs) have made remarkable progress on vision-language reasoning, yet most methods still compress visual evidence into discrete textual thoughts, creating an information bottleneck for fine-grained perception. Recent latent visual reasoning methods attempt to reason in continuous hidden states, but we find that they suffer from insufficient manifold compatibility: latent trajectories drift away from pretrained reasoning circuits, collapse into instance-agnostic patterns, and are often bypassed during answer generation. To address these issues, we propose RIS (Retrieve, Integrate, and Synthesize), a spatial-semantic grounded framework that develops latent reasoning as a compatible extension of pretrained MLLM computation. We first construct a step-wise grounded reasoning dataset with bounding boxes and region-specific semantic descriptions. Built on this supervision, RIS anchors latent tokens to both spatial and semantic evidence, enforces their causal role through a progressive attention bottleneck, and introduces language transition tokens to bridge synthesized latent states back to vocabulary-aligned decoding. Experiments on V* , HRBench4K, HRBench8K, MMVP, and BLINK show consistent improvements over closed/open-source and latent reasoning baselines. Further analyses demonstrate that RIS learns more diverse, interpretable, and progressively integrated latent trajectories, offering a practical path toward faithful internal visual reasoning in MLLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent visual reasoning, MLLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2zq4OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RIS (Retrieve, Integrate, and Synthesize), a spatial-semantic grounded framework for latent visual reasoning in multimodal large language models. The paper identifies three challenges in prior latent reasoning methods: weak grounding of latent representations, bypassing of latent states during answer generation, and incompatibility between latent reasoning states and vocabulary-aligned decoding. To address these issues, the authors construct a 96K-sample grounded reasoning dataset and introduce spatial-semantic supervision, a progressive attention bottleneck, and language transition tokens. Experiments on five visual reasoning benchmarks show improvements over several latent reasoning baselines. Additional analyses investigate latent diversity, attention patterns, and the functional influence of intermediate latent representations.

Critical strengths
  1. The method addresses an important limitation of latent visual reasoning. Rather than treating latent tokens as implicitly useful merely because they can be generated, the paper explicitly considers visual grounding and the possibility that the answer decoder bypasses intermediate states. The proposed spatial-semantic supervision and progressive attention mechanism provide a coherent response to these problems. This is particularly valuable given the growing evidence that latent token generation does not necessarily imply effective latent reasoning.

  2. The empirical evaluation extends beyond standard accuracy comparisons. The paper includes comparisons with representative latent reasoning approaches, supervision ablations, latent representation analyses, and inference-time interventions. These experiments provide more evidence about the behavior of latent states than a conventional benchmark-only evaluation. The 96K grounded reasoning dataset is also a potentially useful contribution.

Critical weaknesses
  1. The methodological novelty is somewhat incremental relative to recent grounded latent reasoning approaches. Prior work, including LVR, Monet, LDPVR, and ReaLVR, has already explored latent visual states, visual evidence grounding, intermediate supervision, and the relationship between latent trajectories and answer generation. RIS combines these ideas in a structured manner, but the paper does not fully establish which individual mechanism represents a qualitatively new advance beyond these developments. A sharper comparison with closely related visual-evidence supervision methods would strengthen the novelty claim.

  2. The proposed manifold-compatibility explanation is not yet fully isolated from the effects of additional supervision and information-flow restrictions. Spatial-semantic labels, attention constraints, and language transition tokens may independently improve optimization and answer generation without necessarily establishing the proposed manifold-level interpretation. Although the paper provides useful intervention analyses, a stronger controlled comparison with explicit grounded reasoning under matched supervision and information accessibility is needed to distinguish the claimed latent mechanism from these alternative explanations.

Questions for authors
  1. Can the authors compare RIS against an explicit grounded reasoning baseline trained with the same GLSD examples, region annotations, and intermediate semantic targets, while controlling the visibility of visual evidence during answer generation? This would help separate the benefit of latent reasoning from the benefit of additional grounding supervision.

  2. How much of the improvement remains when the progressive attention bottleneck is replaced by a less restrictive attention configuration while keeping the same supervised latent slots? In particular, can the authors distinguish genuine information integration from reliance induced by architectural information restrictions?

  3. What quantitative evidence establishes manifold compatibility beyond latent similarity and entropy measurements? For example, can the authors show that the transition tokens improve latent-to-language consistency independently of their additional token-level supervision?

  4. How sensitive are the results to the correctness of the automatically generated reasoning steps and bounding boxes, particularly for multi-object or relational questions?

5 comments
AuthorsOpenReview ↗

W1. Novelty relative to related works

We thank the reviewer for the positive assessment and the precise questions. We first address the novelty concern, then each question.

We agree that latent visual states, evidence grounding and intermediate supervision have each been explored. What RIS adds is explicitly controlling how evidence reaches the answer, which prior methods leave unconstrained, and grounding each latent slot at the step level:

  • LVR reconstructs key visual tokens from latents, and Monet distills latents toward visual embeddings. Neither provides step-level spatial or semantic targets, and neither prevents the answer from bypassing the latents (Fig. 1(c): LVR assigns 2% of answer attention to latents).
  • LDPVR uses latents to guide the refinement of explicit textual reasoning states, so the reasoning carrier remains text.
  • ReaLVR (concurrent work) supervises free-running latent trajectories to preserve answer-relevant visual evidence, contrasting correct with wrong answers and relevant with mismatched evidence. It strengthens what the latents encode but does not control whether the answer actually reads them.

RIS differs in three respects:

  1. Each grounded slot is anchored to a specific region (box regression) and to its semantic content (description alignment), while free slots are left for synthesis.
  2. A progressive bottleneck makes the latents the route by which input evidence reaches the answer.
  3. Transition tokens map the synthesized latents back to language-aligned decoding.

The controlled experiments below isolate each of these: grounding vs. no grounding at matched data (Q1), latent vs. explicit reasoning at matched supervision and matched answer-stage access (Q1), the strength of the bottleneck (Q2), and the transition pathway at matched supervision (Q3). We will add LDPVR and ReaLVR to the related work with this comparison.

AuthorsOpenReview ↗

Q1. Explicit grounded reasoning under matched supervision and matched visibility

Qwen2.5-VL-7B+GLSD is trained on the same 80K GLSD examples, with the same bounding boxes, region descriptions and visual information, written as explicit text steps and learned by next-token prediction. To also match answer-stage visibility, we add Qwen+GLSD+Bottleneck. Its training sequences follow the GLSD format used in Phase 1: the explicit grounded trace T\mathcal T, then the full answer AsA_s, then the final answer AA. The trace is generated with full access to the image and question. We then apply the same bottleneck as RIS to AsA_s and AA: they can attend to the trace but not to the image or question tokens. This baseline therefore matches RIS in supervision content and in what the answer can see, and differs only in whether the intermediate reasoning is carried by explicit text or by continuous latents. All models use the same backbone, data and number of optimization steps.

Reasoning carrierGroundingAnswer-stage bottleneckV*HR4KHR8KMMVPBLINKAvg.
Explicit text✓✗78.2570.5864.7066.8056.2567.32
Explicit text✓✓80.2771.1666.2369.5858.0469.06
Continuous latents✗✓74.0664.2460.8766.5350.5663.25
Continuous latents (RIS)✓✓83.7573.2368.5273.5560.6071.93

The third row is the joint grounding ablation reported to Reviewer LsiH (Q3).

The bottleneck alone improves explicit reasoning by 1.74, so part of RIS's gain over vanilla SFT comes from restricting answer-stage access. With supervision and visibility both matched, however, RIS still exceeds the explicit baseline by 2.87, on all five benchmarks. Conversely, latents without grounding fall 5.81 below the explicit baseline. Latent reasoning therefore helps only when it is grounded, and with grounding it outperforms the same supervision expressed as text.

AuthorsOpenReview ↗

Q2. Effect of bottleneck strength on accuracy and latent dependence

We keep all slots, supervision and training settings fixed and only lower the final masking probability ρ in Phase 3 (originally annealed from 0.3 to 1.0) to 0.75 and 0.50. At inference, each variant applies a stochastic mask with its final ρ; results are averaged over 3 runs. w/o Mask corresponds to ρ = 0 throughout.

Final ρV*HR4KHR8KMMVPBLINKAvg.Gain over Qwen+GLSD
1.0 (RIS)83.7573.2368.5273.5560.6071.93+4.61
0.7582.6072.4567.3572.1558.9570.70+3.38
0.5081.4571.3565.9570.7557.8069.46+2.14
0 (w/o Mask)68.1465.8359.3667.7451.7662.57-4.75

Relaxing the bottleneck retains 73% and 46% of RIS's gain over Qwen+GLSD. Part of the improvement is therefore induced by the restriction, and we will state this explicitly. The sharp drop between ρ = 0.5 and ρ = 0 indicates that even partial masking is enough to force the latents to carry information during training, whereas never-masked latents remain uncalibrated and interfere with prediction (see our response to Reviewer 6T3c, Q1 and the per-phase ablation).

To separate genuine integration from induced reliance, we repeat the answer-level interventions of Table 6 on the relaxed models. Entries are accuracy drops relative to each model's own unperturbed accuracy.

Final ρUnperturbedRemove latentsLatent-path maskingCross-example shuffleWithin-example shuffle
1.071.93−5.64−5.26−7.53−7.08
0.7570.70−2.72−2.28−4.79−3.92
0.5069.46−0.90−0.82−2.64−2.52

Two patterns emerge:

  • Dependence on the presence of latents is largely induced by the bottleneck. As ρ decreases, removing the latents costs progressively less (5.64 to 2.72 to 0.90), since the answer can increasingly draw on the inputs directly.
  • Sensitivity to latent content persists. At ρ = 0.5, replacing the latents with another example's costs about three times as much as removing them (2.64 vs. 0.90), and shuffling their order costs nearly as much. If the latents were merely tolerated by a model that could bypass them, wrong latents would be no more harmful than absent ones. Instead, the answer actively reads and integrates instance-specific, ordered latent content even when it is not forced to.

The bottleneck thus determines how much the answer relies on the latents, while the content effect shows that what it relies on is genuinely integrated evidence.

Reviewer_6T3cOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work points out three common flaws in existing latent visual reasoning approaches (LVR, Monet): their latent states deviate substantially from the pretrained token embedding space, lack input-specific variation, and receive little attention from answer tokens. To tackle these issues, the authors construct GLSD, a dataset of 96k GQA examples paired with 2–4 step reasoning traces. Each trace step includes a bounding box and region description, generated via MLLM prompting and verified with Grounding DINO. They fine-tune Qwen2.5-VL-7B through four stages: explicit grounded chain-of-thought, side-head warm-up (box regression plus cosine alignment against cached description embeddings), Coconut-style progressive substitution of textual steps with latent states, and a final stage using a progressive attention mask to block answer tokens from accessing the original image and question. The model inserts brief transition tokens before outputting the final answer. It uses K=5 latent slots; slots unused by annotated reasoning steps are optimized only via answer loss. Evaluations on V*, HRBench 4K/8K, MMVP and BLINK outperform the base model as well as LVR, Monet and CoVT. The paper also includes PCA visualizations, attention statistics, similarity and entropy curves, latent intervention tests at inference time, and ablation studies for analysis.

Critical strengths

Critical Strengths

  1. The paper correctly identifies meaningful failure modes in prior latent visual reasoning work: latent representations are ignored by answer tokens and tend to collapse across different inputs. Its design directly targets these problems. Instead of only relying on attention visualizations, the authors run latent-pathway intervention tests (Appendix E) to verify functional dependence. All six tested perturbations hurt performance across the five benchmarks, and they openly acknowledge distribution shift as a confounding factor.

  2. The method has a complex multi-component pipeline, but the documentation is thorough. It includes dataset statistics and schema, hyperparameters for each training stage, GPU compute cost, and a candid limitations section pointing out concrete drawbacks such as dependence on one teacher model and a fixed latent slot count K. This makes the work easier for others to reproduce.

Critical weaknesses
  1. The mechanistic claim (manifold compatibility, latents as the information pathway) is mainly backed by qualitative or design-built evidence and contains internal inconsistencies. (1) Figure 1(c)’s comparison is circular: RIS has 0% attention on images and text simply because the mask blocks them. This does not prove stronger reasoning than LVR. (2) Figure 3(c) only plots RIS’s curve. The claim that transition tokens smooth baselines’ sharp entropy drop lacks baseline and no-transition-token ablations; the plotted metric is also not standard entropy but mass-weighted sensitivity dispersion.

  2. The empirical comparison does not isolate what the paper credits. (1) GLSD supervision is used only by RIS. The baselines (LVR, Monet, CoVT) appear to use their own data and recipes, and the “+GLSD” baseline has neither side heads nor the mask. Different data and a four-phase, ~20K-step schedule are confounded with the architectural claims. (2) There is no per-phase ablation (Phase 1 init, 2a warm-up, 2b curriculum vs. going straight to Phase 3), and no test of the claimed “emergent” role of unsupervised slots, for example K=3 or 4 versus 5.

Questions for authors
  1. Which attention mask is used at test time, full or bottleneck? Given Table 6, what is the accuracy of the bottleneck model with latents removed, and of the model trained without the mask?

  2. The text says 96k samples, but Appendix C says 80k, and 2,500 steps per pass is 80k/32 (96k would be 3,000). Table 3 gives epochs as “1-2” while Table 5 assumes one pass. Which numbers were actually used?

  3. Section 5.3 says most training samples have three to four supervised steps. Table 11 says 49.4% have two steps and the mean is 2.67. Why K=5 rather than 3 or 4? Was K chosen using the same benchmarks reported in Table 1?

3 comments
AuthorsOpenReview ↗

Q1. Test-time attention mask and functional use of latent tokens

We thank the reviewer for the careful reading and the positive assessment. We address all concerns below in four parts. All new results and clarifications will be incorporated into the main text and the appendix of our paper, and all inconsistencies will be corrected in the revision.

RIS uses the bottleneck mask at test time. The caption of Fig. 2 ("Full attention mask used during inference") is a typo and will be corrected to "Bottleneck attention mask used during inference." Under this mask, transition and answer tokens cannot attend to image or question tokens. They attend only to latent tokens, boundary tokens, and previously generated tokens, as in Eq. (7). Accordingly, Table 7 shows 0% attention to the inputs; this follows from the mask by design.

Setting of the interventions in Table 6. All interventions use the same trained RIS model under the bottleneck mask, and all are applied at the answer level. The latents and transition tokens are first generated normally. The intervention is then applied only to the latent states as read by the final-answer tokens. We will state this setting explicitly in Appendix E.

Results for the requested settings.

SettingV*HR4KHR8KMMVPBLINKAvg.Δ
RIS (full)83.7573.2368.5273.5560.6071.93–
Remove latents (inference)77.8467.9263.0869.0353.5766.29−5.64
Latent-path masking (inference)78.3168.4563.4969.5853.5166.67−5.26
w/o Mask (training)68.1465.8359.3667.7451.7662.57−9.36
  • Bottleneck model with latents removed. Removing the latents from the answer's context lowers accuracy from 71.93 to 66.29, and blocking the answer's attention to them gives a similar 66.67. Under the bottleneck, the transition tokens can obtain input information only through the latents. Even the remaining accuracy is therefore carried by the latent pathway, and the drop measures the additional, direct contribution of the latents to the final answer beyond what is already summarized in the transition tokens.
  • Model trained without the mask. The w/o Mask model is trained and evaluated with full attention, and it reaches only 62.57. Its latents are present but never forced to carry information. This drop reflects two effects together: answers can bypass the latents, and the uncalibrated latents can interfere with prediction.

The answer uses latent content, not merely its presence. The other interventions in Table 6 show that what matters is the specific content of the latents:

  • Replacing the latents with another example's latents (64.40) or shuffling their order (64.85) hurts more than removing them.
  • Moment-matched noise hurts least (67.13).

Plausible but wrong latents mislead the answer, whereas clearly anomalous ones are down-weighted in favor of the transition tokens.

Why we do not perturb the latents before transition generation. Under the bottleneck, such pathway-level interventions would cut off the only route from the inputs to the answer. The resulting collapse would be determined by the mask itself and would say nothing about whether the model has learned to use its latents. The answer-level setting is the more demanding test. A clean alternative source of evidence remains available: the transition tokens, which receive 70% of answer attention in Table 7. Any drop therefore shows that the answer actively relies on the latents rather than bypassing them, which is the failure mode reported for prior latent methods by a recent causal mediation study (Li et al., 2026, Imagination Helps Visual Reasoning, But Not Yet in Latent Space).

Together, these results show that latents contribute only when they are both grounded and placed on the answer pathway, consistent with our joint grounding ablation (response to Reviewer LsiH, Q3).

AuthorsOpenReview ↗

Q2. Dataset size and training schedule & Q3. Choice of K = 5

Q2. Dataset size and training schedule

GLSD contains 96K samples in total. Of these, 80K form the training split. The remaining 16K are held out for validation and further analysis and are never used for training. All models we trained (RIS, Qwen+GLSD and all ablations) use the same 80K split.

With a global batch size of 32, one pass over the training split is 2,500 steps (80K/32). Every stage uses exactly one pass:

PhasePassesSteps
Phase 112.5K
Phase 2a12.5K
Phase 2b1 per curriculum stage × 5 stages12.5K
Phase 312.5K
Total20K

The “1–2 epochs” in Table 3 is inaccurate and will be changed to 1 epoch. For Phase 2b, this means one epoch per curriculum stage, i.e., five passes in total for K = 5. In Table 5 and Appendix C, we will check and remove all ambiguities. We will also replace "96k training samples" in Sec. 4.1 and the introduction with the 96K/80K split above.

Q3. Choice of K = 5

Wording in Sec. 5.3. The reviewer is correct. GLSD samples contain 2–4 supervised steps (49.4% / 34.1% / 16.5% for 2 / 3 / 4 steps; mean 2.67). We will replace "most samples contain three to four steps" with these exact statistics.

Design rationale. K was fixed a priori from the data structure, not tuned on benchmarks. Each supervised step occupies one grounded slot, and the remaining slots are free synthesis slots trained only by the answer objective.

  • K = 3 cannot hold 4-step traces.
  • K = 4 holds all steps but leaves 4-step samples with no synthesis slot.
  • K = 5 is the smallest budget that covers every trace and leaves at least one synthesis slot for every sample.

Smaller budgets. We add K = 3 and K = 4. For 4-step samples under K = 3, only the first three steps keep their grounding targets; the answer supervision is unchanged. With one pass per curriculum stage, Phase 2b has K stages, so K = 3 and K = 4 would use 15K and 17.5K steps in total. To match the 20K-step budget of K = 5, we extend Phase 3 by the missing steps. All other settings are identical.

KV*HR4KHR8KMMVPBLINKAvg.
382.5271.4366.5171.5259.3270.26
483.0372.2467.5272.4360.0171.05
583.7573.2368.5273.5560.6071.93

Accuracy rises monotonically on all five benchmarks: +0.79 from K = 3 to 4, and +0.88 from K = 4 to 5. Since K = 3 also truncates supervision for 4-step samples, this comparison reflects step coverage and synthesis capacity together.

Were the Table 1 benchmarks used to select K? No. K = 5 was set by the rationale above before any evaluation. The Table 1 benchmarks were used only afterwards, for the sensitivity analysis in Fig. 4(c) (K = 5–12) and the K = 3/4 results above. We note that Fig. 4(c) shows higher accuracy at other budgets on some benchmarks (e.g., V* 83.97 and BLINK 61.79). Nevertheless, we report K = 5 for all benchmarks in Table 1, rather than the per-benchmark best values.

AuthorsOpenReview ↗

Weaknesses

Fig. 1(c) is circular for RIS. We agree. RIS's 0% attention to the inputs follows from the mask and is not evidence of better reasoning. Fig. 1(c) is meant to diagnose bypassing in LVR, whose answers give only 2% attention to latents although nothing prevents them from using them. The evidence that RIS uses its latents is functional: answer-level interventions hurt even though the transition tokens remain a clean alternative route (Q1 above), and with a relaxed mask that lets answers see the inputs, wrong latents still cost about three times more than absent ones (response to Reviewer 2zq4, Q2).

Fig. 3(c). We agree that the plotted quantity is a mass-weighted dispersion of visual-evidence sensitivity (Appendix D.3) rather than standard entropy, and would consider renaming it "evidence dispersion." The requested no-transition control is provided quantitatively in our response to Reviewer 2zq4 (Q3): under matched AsA_s supervision, the final answer is closest to the language reference only when the transition tokens lie on the answer path. We will also add the w/o Transition curve to Fig. 3(c).

Isolating the credited components. Table 1 is a system-level comparison, and we will label it as such. All models we trained share the same 80K data and 20K-step budget (Q2), and the following controls isolate each component:

  • the mask applied to explicit grounded reasoning (2zq4, Q1);
  • latents without grounding (LsiH, Q3);
  • the strength of the mask (2zq4, Q2);
  • the transition pathway (2zq4, Q3).

The explicit baseline has no side heads because its boxes and descriptions are already direct next-token targets, a stronger, token-level form of the same supervision. The role of the unsupervised slots is also tested by the K = 3 / 4 / 5 comparison the reviewer suggested (Q3).

Per-phase ablation. We evaluate the training schedule cumulatively by measuring performance at the end of each phase.

Training stagesStepsV*HR4KHR8KMMVPBLINKAvg.
Qwen2.5-VL-7B (Base)—76.6568.3064.3363.4954.9465.54
Phase 12.5K78.2070.1764.5266.4056.0067.06
Phase 1 + 2a + 2b17.5K77.0571.5065.4166.2555.6267.17
Full (+ Phase 3)20K83.7573.2368.5273.5560.6071.93

Note: Due to compute constraints, all results except the full model are from a single run (seed 42). We report cumulative rather than phase-removal ablations because the phases are sequentially dependent: removing one would change the initialization of all later phases.

Phase 1 alone reaches 67.06, close to Qwen+GLSD (67.32), which uses the same explicit supervision for 20K steps. After Phase 2, the average is nearly unchanged (67.17): HR4K and HR8K improve, while the other three benchmarks drop slightly. We attribute this to the transition itself. As the explicit trace is replaced by latents, evidence that was previously verbalized must be re-encoded in continuous states, while the answer can still read the inputs directly and has little incentive to rely on the latents. Phase 3 resolves this by routing the evidence through the latents, adding 4.76 points with gains on all five benchmarks.

Interestingly, continuing from the same Phase-2 checkpoint without the mask (w/o Mask) lowers accuracy to 62.57 (−4.60). We hypothesize that, with the inputs directly accessible, longer training leaves the free latent slots without useful gradient. They then drift toward generic states (Fig. 4(b)) and, since the answer still reads them, interfere with prediction.

We are very grateful to the reviewer for pointing out this issue, which has enabled us to further analyze the dynamics of latent training; these findings are highly valuable. However, due to limited compute, we could not verify this hypothesis thoroughly during the rebuttal period, and we sincerely regret this. We will continue investigating it in the following period and update this analysis in the revision, please follow the update.

Reviewer_LsiHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies current failure patterns in latent visual reasoning: latent tokens are weakly grounded and ignored. The authors proposes RIS to fix this. They build GLSD, a large scale data set for visual grounding and reasoning, and finetuned a VLM on this dataset with 3 phases: explicit grounded cot, latent alignment with spatial and semantic head and progressive attention mask to force meaningful visual latents. Results on several public benchmarks beat previous latent reasoning baselines.

Critical strengths
  1. The method is built upon the diagnosis: Spatial and semantic side heads address grounding, the bottleneck mask addresses bypassing, and transition tokens address the mismatch with the vocabulary manifold.
  2. Clear experimental design and gains: The authors show clear improvement compared to baselines, and the inference-time interventions support the claim that RIS help the problem of visual latents being ignored.
Critical weaknesses
  1. More details about the diagnosis:

This paper observes 3 problems with visual latents: manifold divergence, collapse and bypassing. For divergence, the evidence is a single PCA plot with missing details on how each kind of tokens are collected. The pca visualization may be affected by many factors (e.g. task types and collected numbers). The claim that successful paths stay on the manifold and failed ones leave it has no number behind it. I think some quantitative metrics and more detailed description on the setting would enhance the conclusion.

  1. Details on fair baseline comparison:

"Same backbone" doesn't settle whether LVR, Monet and CoVT were retrained on GLSD with a comparable budget, or taken from released checkpoints. RIS gets 96k grounded traces and four training phases.

Also, Table 1 has no seeds or standard deviations. Over the strongest latent baseline, the margins are small: +1.85 on V* over Monet, +0.12 on HRBench8K over CoVT, and +2.35 on HRBench4K over LVR.

Further, since the paper primarily tackles grounding related problems, a comparison with some related tool use VLMs could better support the claim and the necessity of forcing latent reasoning in such settings.

Questions for authors

See above。

I have an additional questions about ablation experiments: There's no "w/o bbox + w/o description" variant, i.e. plain Coconut-style latents on the same data. This is the key control for "grounding is what matters".

I find the problems identified in this paper real and meaningful, and the proposed solution is interesting. However, the analysis would benefit from more convincing evidence, and the soundness of the method needs further experimental support. I would be willing to raise my score if these concerns are adequately addressed.

5 comments
AuthorsOpenReview ↗

Q1. Details and quantitative evidence for the manifold diagnosis

We thank the reviewer for the careful reading and detailed comments. We address each point below and will incorporate the additions into the revision.

Setup of Fig. 1(a). We sample 100 examples from each of V*, HRBench4K, HRBench8K, MMVP and BLINK (500 in total). The same examples are used for all methods. We collect three sources of representations:

  • Hbase\mathcal H_{\text{base}}: last-layer hidden states of answer tokens from the frozen Qwen2.5-VL-7B (the Vocabulary Manifold).
  • Hm\mathcal H_m: last-layer states of the latent and answer phases of each method m∈m \in {LVR, Monet, RIS}, all with K=5K=5 latent tokens.
  • E\mathbf E: the vocabulary embedding matrix (the Embedding Manifold), subsampled to match the other sources.

PCA is fitted separately for each method on benchmark-stratified, source-balanced samples. We therefore only interpret relative geometry within a subplot, never across subplots. Each polyline is one example, colored by final-answer correctness.

High-dimensional metrics. Since a 2-D projection cannot establish distributional claims, we compute three metrics in the original hidden space on latent-phase states only, using Hbase\mathcal H_{\text{base}} as a shared reference:

  • MMD² with an RBF kernel. The bandwidth σ is set by the median heuristic, i.e., the median pairwise Euclidean distance among the L2-normalized reference states Hbase\mathcal H_{\text{base}}. It is computed once per benchmark and shared by all methods.
  • Mean cosine distance to the 10 nearest base states, computed leave-one-example-out.
  • Off-manifold Ratio: the fraction of latent states whose 10-NN distance exceeds the 95th percentile (τ95\tau_{95}) of the base states' own leave-one-out distances.

All metrics are computed per benchmark and macro-averaged.

MethodMMD² ↓10-NN dist. ↓Off-manifold ↓
Base (self-reference)––5.0% (by construction)
LVR0.1490.22645.0%
Monet0.3000.35580.0%
RIS0.0580.13923.3%

The ranking Monet > LVR > RIS in distributional shift holds on all five benchmarks (per-benchmark results will be added to the appendix). This matches Fig. 1(a).

These metrics measure compatibility with the pretrained representation regime rather than quality; Monet, for instance, aligns latents to a visual space by design. We therefore test the link between compatibility and correctness within a single method.

Successful vs. failed trajectories. On the same 500 examples, we split RIS trajectories by correctness (359 successful / 141 failed) and compute the metrics per trajectory over its 5 latent states.

10-NN dist. ↓Off-manifold ↓
Successful0.12519.2%
Failed0.17634.0%

Empirically, failed trajectories lie farther from the reference and fall off-manifold more often on every benchmark, supporting the observation in Sec. 3.1 that successful reasoning paths remain more compatible with the pretrained representation regime. We will add this analysis to our paper.

AuthorsOpenReview ↗

Q2. Fairness of baseline comparison and statistical reliability

Baseline protocol. LVR, Monet, and CoVT use their publicly released checkpoints with their original training recipes. They were not retrained on GLSD. Table 1 is therefore a system-level comparison, and we will state this explicitly in its caption.

Regarding data size, GLSD contains 96K samples in total. Of these, 80K form the training split. The remaining 16K are held out for validation and further analysis and are never used for training. RIS uses 20K optimization steps in total (Table 5). We will correct the wording in Sec. 4.1 accordingly.

Controlled comparison under identical data and budget. Qwen2.5-VL-7B+GLSD uses the same 80K samples, the same BBox+Description content, and the same number of steps as RIS, trained with standard next-token prediction. Table 1 reports the mean over three seeds (42, 123, 2026). We previously omitted the standard deviations and now add them.

MethodV*HR4KHR8KMMVPBLINKAvg.
Qwen2.5-VL-7B76.6568.3064.3363.4954.9465.54
LVR†80.6070.8863.5071.0356.7968.56
Monet†81.9069.9767.0170.0056.7069.12
CoVT†79.1071.9068.4058.7057.4067.10
Qwen+GLSD78.25±0.2570.58±0.2764.70±0.3066.80±0.3056.25±0.3567.32±0.21
RIS83.75±0.1373.23±0.2068.52±0.3373.55±0.3360.60±0.3071.93±0.17

† Public checkpoint, single evaluation.

Grounded supervision alone yields +1.78 over the backbone. Under identical supervision, RIS adds a further +4.61, consistently across all five benchmarks. The Coconut-style latent variant trained on the same data (Q3) provides a third matched-data reference point.

Margins over the strongest latent baseline. Per benchmark, the margins are:

  • V*: +1.85 over Monet.
  • HRBench4K: +1.33 over CoVT. (We note respectfully that CoVT, not LVR, is the strongest baseline here.)
  • HRBench8K: +0.12 over CoVT.
  • MMVP: +2.52 over LVR.
  • BLINK: +3.20 over CoVT.

Beyond individual margins, the strongest baseline changes across benchmarks: Monet on V*, CoVT on HRBench4K, HRBench8K and BLINK, and LVR on MMVP. Each baseline is also weak elsewhere. CoVT falls 4.79 points below the backbone on MMVP, and LVR falls below it on HRBench8K. RIS is the only method that sustains leading performance on all five benchmarks. In average accuracy, it exceeds LVR by 3.37, Monet by 2.81, and CoVT by 4.83, showing a consistent advantage rather than benchmark-specific gains.

AuthorsOpenReview ↗

Q3. Joint grounding ablation

We add the requested w/o BBox + w/o Description variant. It uses the same backbone, the same 80K training data, K=5K=5 and the same training budget as RIS. Both side-head losses (Lreg\mathcal L_{\text{reg}} and Ldesc\mathcal L_{\text{desc}}) are removed, so the latent states receive no direct spatial or semantic supervision. The progressive mask and transition tokens are kept to isolate the effect of grounding.

VariantV*HR4KHR8KMMVPBLINKAvg.
RIS (full)83.7573.2368.5273.5560.6071.93
w/o BBox78.0467.4364.9669.2752.5866.46
w/o Description79.7370.3462.7669.9457.1267.98
w/o BBox + w/o Desc.74.0664.2460.8766.5350.5663.25
w/o Mask68.1465.8359.3667.7451.7662.57
w/o Transition81.5370.9465.3271.5657.9369.46

Removing both forms of grounding costs 8.68 points and is worse than either single ablation on every benchmark. This indicates that spatial and semantic grounding are complementary.

Notably, this variant also falls below the backbone (65.54). Since the bottleneck forces answers to route through latents that carry no grounded evidence, the latent pathway becomes a liability. Symmetrically, keeping grounding but removing the bottleneck (w/o Mask, 62.57) also fails. Continuous latents help only when they are both grounded and on the answer pathway.

We will add these additional ablation results to our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.