acceptodds
Under review as a conference paper at ICLR 2027

Construct, Replay, and Reason: Visual-Semantic Latent Block for Multi-modal Reasoning

Abstract

Latent Visual Reasoning (LVR) performs intermediate computation in a visual latent space, allowing reasoning states to carry visual information without verbalizing it. It constructs this space by aligning hidden-state-derived latents with selected visual tokens. However, our analysis identifies two limitations: direct alignment overlooks the distinct representational roles of high-level hidden states and local visual tokens, while using ground-truth RoI visual tokens during training but self-generated latents during inference introduces a substantial training–inference gap. We propose **CR**, a two-pass framework that **C**onstructs and **R**eplays visual-semantic latent block for multi-modal **R**easoning. In Pass-I, a latent head transforms sampled seeds into a multi-token latent block, with auxiliary supervision from RoI visual features and answer-related content. The constructed latent block, serves as the reasoning state and is replayed in Pass-II for answer generation. CR employs the same construct-and-replay procedure during training and inference, thereby reducing their discrepancy. Experiments with 3B and 7B backbones across six multimodal benchmarks demonstrate consistent improvements. At the 7B scale, our method outperforms the strongest baselines by 3.3% and 3.2% on RealWorldQA and OCRBench, respectively, and by 6.4% and 3.5% on BLINK Counting and Spatial Relation. Our project is available at https://anonymous.4open.science/r/CR2.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
latent reasoning, multimodal reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kH7nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CR², a framework for multimodal latent reasoning that first constructs a visual-semantic latent block and then reuses it as context for answer generation. Separate visual and semantic branches receive supervision from RoI features and answer-related concepts. Training and inference follow the same latent construction procedure. Experiments with Qwen2.5-VL 3B and 7B across six benchmarks demonstrate improvements over standard SFT and several latent reasoning baselines.

Critical strengths
  1. The paper identifies two relevant limitations of existing latent visual reasoning methods: conflicting representational objectives and differences between training and inference.

  2. Evaluation covers diverse multimodal capabilities, with particularly strong improvements in counting, spatial relations, OCR, and real-world visual understanding at the 7B scale.

  3. Latent replacement experiments and representation analyses support the constructed latents contribute to answer prediction.

Critical weaknesses
  1. The training–inference gap comparison is not directly matched. The 12.8-point gap for LVR compares ground-truth RoI features with predicted latents, whereas the 2.02-point gap for CR² compares RoI-matched and fixed latent lengths. LVR also uses eight latents, while CR² reports its best fixed-length result with sixteen. These differences make it difficult to attribute the smaller gap to the proposed method. A comparison under a unified evaluation protocol would strengthen this claim.

  2. Baseline for 7B is limited. The 7B baselines are limited. Beyond the original backbone and SFT, only LVR is evaluated. Additional recent latent reasoning baselines would strengthen the comparison.

  3. The ablations conflate architecture and supervision. Removing each auxiliary loss while retaining both branches would better isolate the contribution of visual and semantic supervision.

  4. Inference costs are not sufficiently reported. Latency, memory usage, and FLOPs comparisons would clarify the overhead of the two-pass framework.

Questions for authors

see the above weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer kH7n(1/2)

Weakness-I
The training–inference gap comparison is not directly matched. The 12.8-point gap for LVR compares ground-truth RoI features with predicted latents, whereas the 2.02-point gap for CR² compares RoI-matched and fixed latent lengths. LVR also uses eight latents, while CR² reports its best fixed-length result with sixteen. These differences make it difficult to attribute the smaller gap to the proposed method. A comparison under a unified evaluation protocol would strengthen this claim.

Thank you for raising this point. The LVR comparison can involve two sources of mismatch: the latent features used during training and inference, and the number of latent steps.

Controlling latent length in LVR. We additionally evaluated LVR with 8, 12, and 16 fixed latent steps, as well as RoI-matched steps (Table 1). Even when the number of predicted latents matches the RoI token count, a 10.4% gap remains between using ground-truth RoI features and predicted latents.

Distinguishing the remaining gap in CR². CR² instead uses the same latent construction-and-replay procedure during training and inference. This removes the specific train–test substitution of ground-truth RoI features with predicted latents. A length mismatch can still remain: under our CR² evaluation, it differs by 2.02%.

Table 1: LVR accuracy gaps under fixed and RoI-matched latent lengths.

SettingTraining-Inference gap %
8 steps12.8
12 steps13.6
16 steps15.3
RoI-matched steps10.4

We have revised the manuscript to distinguish feature-substitution and latent-length effects, rather than presenting the two numbers as a matched reduction of the same gap.

Action taken: We have added relevant discussion in Appendix C.

Weakness-II
Baseline for 7B is limited. The 7B baselines are limited. Beyond the original backbone and SFT, only LVR is evaluated. Additional recent latent reasoning baselines would strengthen the comparison.

Thank you for the suggestion. We have added Monet-7B, LIVR-7B, and CoVT-7B alongside the original baselines, and expanded evaluation to V, CV-Bench, HRBench, MMVP*. As shown in Table 2, CR²-7B achieves the highest reported scores among the listed 7B models on MME, OCRBench, V*, CVBench-2D, HRBench-4K/8K, and BLINK Spatial Relation. In particular, it exceeds the strongest listed baselines by 2.5 percentage points on OCRBench and 3.5 points on BLINK Spatial Relation.

Table 2: Benchmark comparison with additional 7B latent-reasoning baselines.

MethodHallusionMMEMathVistaOCRBenchRWQAV*CV-2DCV-3DHR-4KHR-8KMMVPBLINK-CBLINK-SBLINK-Total
Qwen2.5-VL-7B37.81961.457.382.060.473.873.775.264.655.863.350.488.848.5
SFT-7B54.22084.861.582.861.968.074.679.765.654.261.060.588.552.5
LVR-7B58.62185.063.980.960.480.174.678.168.462.569.762.186.752.7
Monet-7B34.51841.9655.480.459.760.760.666.156.853.065.655.067.144.2
LIVR-7B—————80.1————75.3———
CoVT-7B56.32202.4465.883.568.771.775.284.163.555.173.671.684.659.4
CR²-7B58.32258.464.986.065.280.676.180.669.364.168.368.592.353.4

Action taken: We have revised in Section 5.2.

AuthorsOpenReview ↗

Response to Reviewer kH7n(2/2)

Weakness-III
The ablations conflate architecture and supervision. Removing each auxiliary loss while retaining both branches would better isolate the contribution of visual and semantic supervision.

Thank you for the suggestion. We added ablations that retain both branches while removing the visual loss, the semantic loss, or both. As shown in Table 3, full supervision outperforms all three settings across the six reported metrics.

Table 3: Ablations of latent-head architecture and auxiliary supervision.

VariantHallusionMMERWQAV*HR-4KHR-8K
CR²61.12189.362.273.865.160.3
VisualHeadOnly57.72133.957.972.262.159.0
SemanticHeadOnly56.92146.460.771.763.659.6
SharedHead55.72138.659.571.263.259.3
SharedHead w/o Sup.56.62130.2560.569.662.358.3
VisualSup.60.32110.357.970.662.658.5
SemanticSup.59.72124.656.771.264.659.2
NoSup.58.12093.556.370.162.859.6

Action taken: We have revised in Section 5.3.

Weakness-IV
Inference costs are not sufficiently reported. Latency, memory usage, and FLOPs comparisons would clarify the overhead of the two-pass framework.

We evaluate inference efficiency on an NVIDIA A100 80GB GPU, using 100 samples for latency and memory and 20 samples for profiler-supported FLOPs. CR² maintains stable latency as latent steps increase, achieving a 1.28×–2.02× speedup over LVR with a modest increase in memory and profiler-supported FLOPs.

Table 4: Inference latency, peak memory, and profiler-supported FLOPs.

SettingEnd-to-end latencyPeak GPU memoryFLOPs
qwen2.5-vl-3B1246.44 ms7.05 GiB2.63 T
LVR-3B (step 4)762.58 ms7.10 GiB2.66 T
LVR-3B (step 8)986.66 ms7.10 GiB2.68 T
LVR-3B (step 16)1240.20 ms7.10 GiB2.73 T
CR_2 (step 4)594.59 ms (1.28x)7.18 GiB2.68 T
CR_2 (step 8)605.67 ms (1.63x)7.18 GiB2.72 T
CR_2 (step 16)612.92 ms (2.02x)7.18 GiB2.81 T

Action taken: We have added relevant discussion in Appendix D.

AuthorsOpenReview ↗

A Friendly Reminder to Review Our Rebuttal

Dear Reviewer kH7n,

Thank you for your thoughtful review and constructive suggestions. We would like to briefly follow up on our rebuttal, in which we have addressed your concerns and updated the manuscript.

To clarify the training–inference gap comparison, we evaluated LVR under multiple latent lengths, including RoI-matched lengths, and distinguished the effects of feature substitution from those of latent length (Weakness-I). We also added recent 7B baselines and vision-centric benchmarks (Weakness-II), as well as controlled ablations that retain both branches while removing the auxiliary losses (Weakness-III). Finally, we reported inference latency, peak memory, and profiler-supported FLOPs to characterize the overhead of our two-pass framework (Weakness-IV).

We hope that these clarifications and additional experiments are helpful. We would be grateful if you could consider our response, and we would be happy to discuss any remaining questions.

Thank you again for your time and consideration.

Best regards,
All Authors

Reviewer_v7NhOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CR2, a two-pass latent reasoning framework improving the original LVR by two key mechanisms:

  1. In Pass-I, the model encodes the image, the question, and randomly sampled seeds in one forward pass. A latent head splits each seed’s hidden state into two streams: a visual stream aligned to RoI visual tokens, and a semantic stream that predicts answer-related keywords through the LM head. The two streams are fused into a residual added to the seed.

  2. In Pass-II, the constructed latent block is replayed as a prefix for answer generation.

Because training and inference use the same construct-and-replay procedure, the authors argue that CR2 closes LVR’s train–inference gap. On Qwen2.5-VL 3B and 7B, CR2 improves over SFT and LVR on six benchmarks.

Critical strengths
  1. The paper identifies a real train–inference mismatch in LVR and addresses it with a procedure that is the same at training and inference.
  2. The gains over SFT trained on the same data are consistent, and the probing analysis shows that the two streams play distinct roles.
  3. [Open discussion]The prefix-and-generate method reminds me of transfusion. Also, there is some potential for CR2 to be paired with recent works like Representational Auto Encoder for a more broader visual reasoning capability.
Critical weaknesses

1.Gains come mainly from the two-pass structure rather than the proposed supervision. The variant without both auxiliary losses already reaches MME 2130 vs. 1939 for SFT, suggesting the extra depth given to latent positions matters more than visual–semantic construction. Moreover, the semantic stream predicts keywords generated from the ground-truth answer, so the latents act more as an answer draft than visual reasoning.

  1. The evaluation is limited. BLINK is reported for only two subtasks. Vision-centric benchmarks such as V*, HRBench, CV-Bench and MMVP are missing. The 7B model is compared only against LVR, and the ablations are run on 3B with three benchmarks.
Questions for authors
  1. What is the inference overhead of the two passes?
  2. Is the image–question KV cache reused between the two passes?
8 comments
AuthorsOpenReview ↗

Response to Reviewer v7Nh (1/3)

Weakness-I
Gains come mainly from the two-pass structure rather than the proposed supervision. The variant without both auxiliary losses already reaches MME 2130 vs. 1939 for SFT, suggesting the extra depth given to latent positions matters more than visual–semantic construction. Moreover, the semantic stream predicts keywords generated from the ground-truth answer, so the latents act more as an answer draft than visual reasoning.

Thank you for raising this point. The construct-and-replay framework and auxiliary supervision make distinct contributions, which we examine separately below.

Contribution beyond two-pass computation. We agree that the two-pass framework is beneficial because it reduces the training–inference mismatch in latent construction. The visual and semantic auxiliary objectives further improve latent learning. To isolate their contribution, we retain the dual-branch architecture and two-pass computation while removing either or both auxiliary losses; answer cross-entropy is retained throughout. We also extend the ablations to V* and HRBench-4K/8K. As shown in Table 3 (see W2), full supervision improves over the no-auxiliary-loss variant by 3.0% on HallusionBench, 95.8 MME score, 5.9% on RWQA, and 3.7% on V*, with gains across all six metrics. Our branch analysis (Section 5.5) and case studies (Figure 6) further suggest distinct roles for the two branches.

Semantic content beyond the answer label. Our semantic supervision emphasizes task-related visual concepts in addition to the answer. To examine this distinction, we conducted an Answer–Semantic Evidence Differentiation Analysis using Yes/No questions. For each image–question pair, we constructed latents and decoded high-scoring tokens from the semantic branch. Since the final answer is binary, we examined whether the decoded tokens also contained concepts related to the task. These cases provide qualitative evidence that it can represent information beyond the answer.

For “Is there a red object?”, two images both answered “No” yield different scene-related concepts: bird, rock, tree for a bird image, versus glass, cup, table for a beverage image. With both the question and answer unchanged, these differences provide qualitative evidence that the semantic stream captures contents beyond the final answer label.

Table 1-1: Decoded semantic concepts for Yes/No questions.

Question 1
Is there a red object?
Image DescriptionAnswerDecoded Semantic Token
A striking dark blue bird perches on a mossy rock, highlighted against a softly blurred background of green vegetation and blue mountains.Nogreen,write,bird,grass,rock,tree
A chilled, golden-green beverage in a glass filled with ice sits on a wooden table, with vibrant green leaves softly blurred in the background.Nogreen,plate,table,glass,cup,wooden
A vibrant male Northern Cardinal, distinguished by its brilliant red plumage and striking black facial mask, perches on a bare wooden branch against a soft, blurred beige background.Yesred,animal,branch,bird,tree,plant
Three vibrant iced cocktails—a red strawberry-mint drink in the foreground and two yellow pineapple-garnished drinks behind—sit on a wooden table illuminated by warm, dramatic sunlight.Yescup,plate,drink,table,red,wine
Question 2
Are there some flowers?
Image DescriptionAnswerDecoded Semantic Token
A vibrant cluster of bright red salvia flowers blooms among lush, dark green leaves, illuminated by dappled sunlight against a softly blurred and shadowy forest background.Yesbird,flowers,green,grass,plant,leaves
A vibrant carpet of spring crocuses in shades of purple, white, and yellow blooms beautifully across a lush green lawn.Yesgreen,flowers,purple,grass,bloom,animal
A nearly bare tree with spreading branches stands alone on a sandy beach, framed by calm water and a clear, hazy blue sky.Noblue,water,sea,beach,tree,leaves
A picturesque hillside town features densely packed blue and white buildings cascading down the slope, backed by a green-covered mountain under a blue sky.Nowrite,green,town,tree,building,mountain
AuthorsOpenReview ↗

Response to Reviewer v7Nh (2/3)

Weakness-II
The evaluation is limited. BLINK is reported for only two subtasks. Vision-centric benchmarks such as V*, HRBench, CV-Bench and MMVP are missing. The 7B model is compared only against LVR, and the ablations are run on 3B with three benchmarks.

Thank you for the suggestions. We have enriched our evaluation.

Benchmark Evaluation. We add V, CV-Bench, HRBench, and MMVP* for evaluation.

Additional 7B baselines. We add comparisons with Monet[1], LIVR[2], and CoVT[3]. CR² remains competitive, with strengths on MME, OCRBench, V*, HRBench, and BLINK Spatial Relation.

Broader ablations. We extend the ablations from 3 to 6 evaluation metrics by adding V* and HRBench-4/8K. The new ablations retain both branches, isolating auxiliary supervision from architectural changes, as discussed in response to Weakness 1.

We evaluate CR² on LLaVA-OneVision-1.5-4B, providing evidence of applicability beyond the Qwen family.

Table 2: Expanded benchmark comparisons.

MethodHallusionMMEMathVistaOCRBenchRWQAV*CV-2DCV-3DHR-4KHR-8KMMVPBLINK-CBLINK-SBLINK-Total
SFT-3B57.31938.852.277.452.967.066.171.262.655.059.661.582.545.4
DMLR41.41839.250.074.454.247.060.462.152.443.556.845.879.040.4
LVR-3B53.32142.657.580.357.668.070.670.859.052.263.060.583.946.5
LaViT-3B51.12178.357.378.459.672.568.774.264.059.761.660.080.446.8
LIVR-3B———————————63.6—67.8
CR²-3B61.12189.358.782.262.273.867.073.665.160.362.062.583.247.7
SFT-7B54.22084.861.582.861.968.074.679.765.654.261.060.588.552.5
LVR-7B58.62185.063.980.960.480.174.678.168.462.569.762.186.752.7
Monet-7B34.51841.9655.480.459.760.760.666.156.853.065.655.067.144.2
LIVR-7B—————80.1————75.3———
CoVT-7B56.32202.4465.883.568.771.775.284.163.555.173.671.684.659.4
CR²-7B58.32258.464.986.065.280.676.180.669.364.168.368.592.353.4
LLaVA38.41867.1957.776.161.263.862.067.762.656.570.660.076.944.9
LLaVA-SFT53.52096.0258.577.764.968.569.067.167.658.569.062.883.947.5
LLaVA-CR²57.82164.6659.578.766.472.768.071.865.257.371.762.086.047.7

Table 3: Ablations of latent-head.

VariantHallusionMMERWQAV*HR-4KHR-8K
CR²61.12189.362.273.865.160.3
VisualHeadOnly57.72133.957.972.262.159.0
SemanticHeadOnly56.92146.460.771.763.659.6
SharedHead55.72138.659.571.263.259.3
SharedHead w/o Sup.56.62130.2560.569.662.358.3
w/ VisualSup.60.32110.357.970.662.658.5
w/ SemanticSup.59.72124.656.771.264.659.2
w/o Sup.58.12093.556.370.162.859.6

Action taken: We have updated the Section 5.2, Section 5.3 and Appendix E.

Reference:

[1] Monet: Reasoning in Latent Visual Space Beyond Images and Language.

[2] Latent Implicit Visual Reasoning.

[3] Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens.

AuthorsOpenReview ↗

Response to Reviewer v7Nh (3/3)

Question-1
What is the inference overhead of the two passes?

Thank you for the question. CR² adds a latent-construction pass, but constructs the entire block in parallel rather than generating latents autoregressively. We measure inference cost on an NVIDIA A100 80GB GPU, using 100 samples for latency and memory and 20 samples for profiler-supported FLOPs.

As shown in Table 4, at 16 latent positions, CR² uses 2.81 T FLOPs and 7.18 GiB peak memory, compared with 2.73 T and 7.10 GiB for LVR—approximately 2.9% more FLOPs and 1.1% more memory. Despite this additional computation, latency decreases from 1240.20 to 612.92 ms, yielding a 2.02× speedup. Across budgets of 4–16 positions, CR²’s latency increases by only 3.1%.

Thus, the two-pass design incurs modest additional computation and memory relative to LVR, while avoiding its sequential latent-generation latency.

Table 4: Inference latency, peak memory, and profiler-supported FLOPs.

SettingEnd-to-end latencyPeak GPU memoryFLOPs
qwen2.5-vl-3B1246.44 ms7.05 GiB2.63 T
LVR-3B (step 4)762.58 ms7.10 GiB2.66 T
LVR-3B (step 8)986.66 ms7.10 GiB2.68 T
LVR-3B (step 16)1240.20 ms7.10 GiB2.73 T
CR_2 (step 4)594.59 ms (1.28x)7.18 GiB2.68 T
CR_2 (step 8)605.67 ms (1.63x)7.18 GiB2.72 T
CR_2 (step 16)612.92 ms (2.02x)7.18 GiB2.81 T

Action taken: We have added in Appendix D.

Question-2
Is the image–question KV cache reused between the two passes?

Yes. We prefill the image–question prefix once and reuse its KV cache in Pass II. Only the constructed latent block is processed again, followed by autoregressive answer generation, avoiding redundant computation of the shared prefix. As shown in Table 5, at 16 latent positions, cache reuse reduces Pass-II latency from 75.42 to 42.26 ms, a 44.0% reduction.

Table 5: Impact of image–question prefix KV-cache reuse on inference latency.

SettingPass I latencyPass II latencyGeneration latencyTotal latency
CR_2 w/o kv-cache reuse150.13 ms75.42 ms432.60 ms658.15 ms
CR_2 w/ kv-cache reuse147.76 ms42.26 ms421.50 ms611.52 ms

Action taken: We have added in Appendix D.

Reviewer_Yc94OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The authors addresses the hidden-state/visual token gap in Latent Visual Reasoning by CR^2, the framework which utilizes both hidden-state and visual tokens by generating latent prior to reasoning through an additional architecture(Latent Head)
  • The authors report that they are successful in having performance gain against existing benchmarks
Critical strengths

Problem Setting

  • The authors do not just qualitatively suggest the core problem addressed in the paper, hidden-state and (visual) token discrepancy from its own nature, but show via preliminary experiments (in Figure 1. ) showing the distributional gap in visual

Methodology

  • The main idea of using both visual tokens and hidden states, clearly represented in Line 60-62(As shown in Figure 2, the central idea is to assign the reasoning-state role to a constructed latent block, while visual tokens remain evidence rather than the state on which the answer is conditioned.) , is simple and intuitive, and well implemented with simple/light module, Latent Head,

Ablation

  • The effectiveness of Latent Effectiveness is well shown through empirical results where performance drops as with corrupting latents, which shows that latents are actually used for reasoning.
Critical weaknesses

Claim

  • I believe that the authors do not go through sufficient analysis or provide enough citations to claim that (1) hidden states and token plays different "role" and (2) their roles are "contextual" information and "local" evidence. I enjoyed the authors preliminary analysis for showing the distributional gap (which I also put in strengh), but such distributional gap itself does not tell their own functions. Although this can be justified throughout literatures or partially justified from the performance gain, there is no sufficient causal relation here.

Method

  • The generalizability of Latent Head across domains is not addressed in the paper, which I assume can be vulnerable to overfitting considering the size of the modules.

Evaluation

  • The thorough analysis on computational cost with respect to performance gain is absent, which is necessary to justify the benefit from the CR^2. The additional compute required for building latent blocks and its training, though expected to humble regarding the length of the image relative to CoT, should be suggested in at least the appendix for scientific rigor and efficiency perspective. I also mentioned this in the Question For Authors section.
  • The coverage of the experiments is not sufficient, only covering the Qwen2.5-VL based architectures, which is fundamentally only one model family. Other VLM model families with (presumably) different datasets and architectures should be covered.
  • As with Latent effectiveness analysis in Figure 4, the ablation for visual tokens should be added to justify the one of the main supporting claims of the paper, "visual tokens also matter". I think this can be easily done by corrupting the visual tokens with random value and see the performance drops as in Figure 4. (Also added in the Questions For Authors section)

Writing

  • There are some unclear wordings throughout the paper.
    • In the abstract part line 20-22, In Pass-I, a latent head transforms sampled seeds into a multi-token latent block , sampled seeds come out with no context, so diffcult to cleary follow without reading the whole paper.
    • In the abstract part line 21-22, _ ... a multi-token latent block, with auxiliary supervision from RoI visual features and answer-related content._ ..., _ ... a multi-token latent block, trained with auxiliary supervision from RoI visual features and answer-related content._
    • In the Intro part line 52-53, ...and increasing the alignment strength even degrades performance, possibly by suppressing textual information (Xu et al., 2026) is unclearly described. In the same regard, the term of alignment strength is somewhat unclear in its meaning and measurement.

Misc

  • (Not critical) I suggest making the nuance more clear in the Introduction section, to make the hidden states / visual token discrepancy. For example in line 49-50, stressing the difference of computational process between two before pointing out their own properties can be one way. e.g., This overlooks their distinct roles: hidden states integrate visual, semantic, and contextual information, whereas visual tokens mainly encode local evidence -> This overlooks their distinct roles in their computational histories: hidden states progressively integrate visual, semantic, and contextual information through successive decoder layers, whereas visual tokens originate from the vision encoder and primarily capture local visual evidence prior to contextualization from the decoder.

  • (Not critical) Figure 1 and 2 are distantly located to the pages it is mentioned first time. Figure 1 is first mentioned in the page 1, but located at the page 3, Figure 2 mentioned in page 2 but located at the page 5.

  • In upper left part of Figure 1, the location of Images and Questions are different.

Questions for authors
  1. Could you show the hidden-state / token discrepancy specific or enhance in the multimodal setting (VLM and visual tokens), compared to text setting? I do think this will be concluded in a manner that CR^2 is also compute-efficient regarding the first pass will take place only once for the relatively short part, user prompt, so adding this will not only help gain the scientific rigor of the work but also increase its strength.
  2. Regarding the weakness I pointed out in evaluation section, could you add the additional analysis regarding the efficiency such as justifies computation cost from training and inference can / performance gain compared to the baselines, such as visual-token reasoning, and exsiting latent reasoning?
  3. Could you do ablations on visual tokens as you did for latents (Figure 4)? I mentioned in the weakness section as well.
  4. How well does Latent Head generalize? Does this generalize to domains other than the trained ones?
8 comments
AuthorsOpenReview ↗

Response to Reviewer Yc94 (1/4)

Weaknesses(Claim) & Questiones-1
Could you show the hidden-state / token discrepancy specific or enhance in the multimodal setting (VLM and visual tokens), compared to text setting? I do think this will be concluded in a manner that CR^2 is also compute-efficient regarding the first pass will take place only once for the relatively short part, user prompt, so adding this will not only help gain the scientific rigor of the work but also increase its strength.

Thank you for your constructive comment. Our distinction concerns the computational roles of RoI visual targets and decoder hidden states, rather than mutually exclusive types of information. The RoI tokens used as alignment targets provide region-level visual evidence. Decoder hidden states, by contrast, progressively integrate visual evidence with question semantics and preceding context through successive decoder layers.

Existing studies support the contextual-role interpretation. Ethayarajh [1] notes that in text-only models, hidden representations evolve based on the surrounding context—a departure from static token embeddings. Long et al. [2] analyze how visual information influences hidden representations across various layers of a VLM. Furthermore, Krojer et al. [3] demonstrate that visual token representations can be mapped to interpretable semantic descriptions, thereby clarifying their connection to visual content.

To examine this contextual dependence in our multimodal backbone, we compare image-based and description-based inputs, and vary the image and question separately. Using Qwen2.5-VL-3B, we extract the last-layer hidden state at the final prompt position before answer generation:

  • Image vs. textual description. Keeping the question fixed, we replace each image with its textual description. Across 200 pairs, the resulting states have an average cosine similarity of 96.14% and a relative L2 distance of 27.80%. This suggests that the image-induced contextual state is not fully reproduced by its description, although the difference may reflect both modality and information preserved by the description.
  • Changing one modality: Changing the question while fixing the image yields an average relative L2 distance of 29.20% over 60 pairs; changing the image while fixing the question yields 30.99% over 60 pairs. This indicates that decoder states represent visual evidence in a question-dependent context, rather than image content alone.

We will clarify that these findings support our role interpretation, but do not establish functional incompatibility or prove that alignment causes semantic information loss.

Table 1: Decoder-state comparison under image and textual-description inputs.

SettingCosine Similarity %L2 DistanceRelative L2 %
Image input vs. textual-description input96.14 (+-0.98)64.40 (+-8.27)27.80 (+-3.61)

Table 2: Decoder-state sensitivity to image and question changes.

SettingCosine Similarity %L2 DistanceRelative L2 %
Same image, different questions95.45 (+-2.32)66.88 (+-18.00)29.20 (+-8.02)
Same question, different images95.13 (+-1.85)71.79 (+-12.98)30.99 (+-5.62)

Action taken: We have added relevant discussion to Section 3 and included this experiment in Appendix B.

Reference:

[1] Kawin Ethayarajh. How Contextual are Contextualized Word Representations?

[2] Lin Long, Changdae Oh, Seongheon Park, Sharon Li. Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding.

[3] Benno Krojer, Shravan Nayak, Oscar Mañas, Vaibhav Adlakha, Desmond Elliott, Siva Reddy, Marius Mosbach. LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs.

AuthorsOpenReview ↗

Response to Reviewer Yc94 (2/4)

Weakness(Evaluation-I) & Question-2
Regarding the weakness I pointed out in evaluation section, could you add the additional analysis regarding the efficiency such as justifies computation cost from training and inference can / performance gain compared to the baselines, such as visual-token reasoning, and exsiting latent reasoning?

Thank you for the suggestion. We have added an analysis of both training and inference costs.

Training cost. Using Qwen2.5-VL-3B for 1,600 steps, SFT, LVR, and CR² require 11.61, 18.63, and 34.12 GPU-hours, respectively (Table 3). CR² incurs 1.83× the training cost of LVR through its construct-and-replay procedure.

Inference cost. On an NVIDIA A100 80GB GPU, CR² achieves a 1.28×–2.02× speedup over LVR at matched latent budgets (Table 4). At 16 positions, latency decreases from 1240.20 to 612.92 ms, while memory increases from 7.10 to 7.18 GiB and profiler-supported FLOPs from 2.73 to 2.81 T.

Latency breakdown. CR² constructs latents in parallel and reuses the image–question prefix KV cache in Pass II. Answer generation dominates latency at approximately 420 ms (Table 5), while increasing the latent budget from 4 to 16 raises total latency by only 3.1%.

These results characterize the trade-off accompanying our reported performance gains: stronger performance, higher training cost, but lower measured inference latency than LVR, with modest additional memory and FLOPs.

Table 3: Training cost comparison on Qwen2.5-VL-3B.

Method (based on Qwen2.5-vl-3B)GPUsStepsGPU-hoursThroughput
SFTNVIDIA A100 80GB160011.610.612 samples/s
LVRNVIDIA A100 80GB160018.630.382 samples/s
CR_2NVIDIA A100 80GB160034.120.208 samples/s

Table 4: Inference latency, peak memory, and profiler-supported FLOPs.

SettingEnd-to-end latencyPeak GPU memoryFLOPs
qwen2.5-vl-3B1246.44 ms7.05 GiB2.63 T
LVR-3B (step 4)762.58 ms7.10 GiB2.66 T
LVR-3B (step 8)986.66 ms7.10 GiB2.68 T
LVR-3B (step 16)1240.20 ms7.10 GiB2.73 T
CR_2 (step 4)594.59 ms (1.28x)7.18 GiB2.68 T
CR_2 (step 8)605.67 ms (1.63x)7.18 GiB2.72 T
CR_2 (step 16)612.92 ms (2.02x)7.18 GiB2.81 T

Table 5: Latency breakdown of latent construction, replay, and answer generation.

SettingPass I latencyPass II latencyGeneration latencyTotal latency
CR_2 (step 4)131.97 ms42.09 ms419.17 ms593.23 ms
CR_2 (step 8)140.13 ms42.58 ms420.60 ms603.31 ms
CR_2 (step 16)147.76 ms42.26 ms421.50 ms611.52 ms

Action taken: We have added this experiment in Appendix D.

AuthorsOpenReview ↗

Response to Reviewer Yc94 (3/4)

Weakness(Evaluation-III) & Question-3
Could you do ablations on visual tokens as you did for latents (Figure 4)? I mentioned in the weakness section as well.

Thank you for the kind comment. As both passes have access to visual tokens, we perturb them in each pass separately while keeping the other pass’s visual tokens intact. Perturbing "Pass I" affects the construction of latent representations, whereas perturbing "Pass II" is intended to examine the influence of visual tokens on latent replay and the process of answer generation.

Pass I ablation. Perturbing visual tokens reduces accuracy from 65.4% to 53.2–53.5%, despite intact visual access in Pass II. This supports the contribution of visual evidence during latent construction.

Pass II ablation. Perturbing Pass II leads to a more pronounced drop in performance (by 43.8%–45.0%). This indicates that both latent representations and visual tokens provide critical information for the final reasoning process.

Table 6: Effects of visual-token perturbations in Pass I and Pass II.

SettingPass I Ablation Acc.Pass II Ablation Acc.
Real visual token65.465.4
Zero visual token53.320.4
Random visual token53.521.6
Match random token53.221.6

Together, these results show that visual evidence contributes to both latent construction and answer generation: the constructed latents complement, rather than replace, direct visual access.

Action taken: We have added this experiment in Appendix F.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.