acceptodds
Under review as a conference paper at ICLR 2027

Interlude: Reasoning Between Tokens in Vision-Language Models

Abstract

Multimodal reasoning mixes two kinds of intermediate steps: symbolic relations that language expresses precisely and continuous visual judgments that text can only approximate. Approaches using a single medium face a trade-off: explicit chain-of-thought can discard fine visual details, while purely latent reasoning can lose the readable scaffold. We propose **Interlude**, retaining language as the backbone while punctuating it with latent segments: short, undecoded spans of recurrent hidden-state computation used by subsequent text. We seek states that remain useful to subsequent text while retaining visual evidence that is difficult to verbalize. Motivated by information under-saturation and the modality gap, we combine semantic guidance on the language-head readout with a geometric margin favoring an intermediate reference over either modal pole. Interlude improves over Qwen3-VL-8B-Instruct on all six evaluated benchmarks, with an average gain of 4.72%, and achieves the highest Overall score among the compared explicit and latent reasoning baselines. Its Overall score also exceeds that of Qwen3-VL-8B-Thinking while using just 7.6% of its decoded tokens in our efficiency evaluation. Geometric analysis shows a pronounced increase in visual alignment for the component orthogonal to the word-embedding principal subspace, with little change within that subspace.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, latent reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wbcnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Interlude, a method that inserts recurrent hidden-state computation into the textual reasoning process of Qwen3-VL-8B-Instruct. The model switches between textual and latent segments using boundary tokens, allowing subsequent text to access latent states through self-attention. Training uses 50K teacher-generated examples constructed from VCR and VIRL-39K, combining textual cross-entropy loss, stage-dependent semantic-window supervision, and a geometric margin constraint defined relative to the visual and textual centroids. Across six multimodal benchmarks, Interlude achieves an Overall score of 68.7, outperforming the original backbone (63.9), standard CoT fine-tuning on the same data (65.1), and SwimBird (66.5). Ablation studies show that removing the intermediate-modality constraint or semantic guidance, or disabling latent segments at inference time, reduces the Overall score to 66.9, 66.2, and 66.4, respectively.

Critical strengths
  1. Table 2 shows that Interlude outperforms the original backbone across all six benchmarks, with improvements of 10.7 and 5.5 percentage points on HallusionBench and ERQA, respectively, demonstrating broad and substantial performance gains.
  2. The ablation studies further validate the practical contributions of both supervision objectives and the latent segments: the full model consistently outperforms variants without the intermediate-modality constraint, without semantic guidance, or with latent segments disabled during inference.
Critical weaknesses
  1. The geometric analysis does not sufficiently support the proposed mechanism. The intermediate-modality constraint explicitly introduces a margin loss that encourages latent states h to align more closely with the intermediate reference r than with either modality centroid. Consequently, the observed increase in effective dimensionality and changes in the orthogonal component may simply be optimization artifacts induced by L_{imc}, rather than evidence that the model meaningfully exploits under-saturated representational capacity to preserve useful visual information. The authors should demonstrate that these additional dimensions encode task-relevant visual information that contributes to downstream performance, rather than merely reflecting noise or geometric redistribution induced by the loss function.
  2. The experimental comparisons lack sufficient fairness. Interlude is trained on a specially constructed dataset of 50K high-quality teacher-supervised examples, whereas the competing baselines rely on their respective training corpora, introducing differences in data quality and supervision strategies. Although the SFT experiment using the same training data provides partial supporting evidence, I recommend reproducing representative baselines under a unified backbone, training dataset, and computational budget to better isolate and validate the advantages of the proposed method itself.
Questions for authors

Under the same backbone, 50K training examples, training budget, and number of latent reasoning steps, how does the proposed geometric constraint compare with direct visual alignment and a fixed midpoint target?

4 comments

Thank you for your positive assessment of our method and your valuable feedback. We address each of your comments below.


W1. Insufficient evidence that the additional dimensions encode useful visual information

Thank you for highlighting the need for a more direct connection between geometric changes and effective reasoning. We agree that an increase in effective dimensionality alone does not establish whether the additional representational capacity is useful. We therefore investigate this interpretation through visual-alignment analyses, interventions on latent content, and controlled comparisons of geometric objectives.

Section 6 provides two relevant observations. Figure 5(b) shows that the component orthogonal to the principal word-embedding subspace exhibits a 0.202 increase in cosine similarity with visual representations, whereas the change for the component within that subspace is −0.002. Figure 7(b) further shows that the analyzed post-training directions align more strongly with the visual subspace than with the principal word-embedding subspace. Thus, the expansion of the representation distribution is accompanied by stronger visual alignment, supporting the use of representational space beyond the dominant textual directions.

To examine whether subsequent reasoning actually uses latent content, we conduct intervention experiments on V*Bench, replacing only the states at the specified positions.

Intervention positionCross-sample swappingDistance-matched random perturbation
First latent segment80.6 (−6.3)83.8 (−3.1)
Last latent segment81.2 (−5.7)84.3 (−2.6)
First and last latent segments77.0 (−9.9)82.7 (−4.2)

Cross-sample swapping replaces the specified latent segment with states from the corresponding positions of another question. At each latent step, the random-perturbation control matches both the resulting state norm and the perturbation distance induced by the corresponding swap. All conditions preserve the text prefix preceding the intervention, the latent slots, and the number of reasoning steps per segment.

Perturbing latent segments at different positions consistently reduces performance, with a larger degradation when both the first and last segments are perturbed. These results indicate that the model’s reasoning relies on information encoded in the latent states. Cross-sample swapping also causes larger performance drops than matched random perturbations, further suggesting that the latent states encode input-specific information that influences subsequent predictions.

The analyses in Section 6 characterize the visual structure of the expanded representations, while the intervention experiments establish a connection between latent content and task performance. Together, these complementary findings support the interpretation that intermediate-modality supervision helps the model learn latent representations useful for visual reasoning, beyond merely inducing geometric changes.


W2. Fairness of comparisons across different training conditions

Thank you for pointing out this limitation. Our cross-method comparisons do not control for differences in training data and computational budgets. A key consideration is that reproducing methods such as LVR, Monet, and SwimBird requires constructing method-specific supervision, including region annotations, intermediate auxiliary images, and specialized reasoning sequences. These data-construction procedures are themselves integral components of the respective methods. Even when starting from the same raw samples, differences in teacher models, annotation granularity, and filtering criteria can affect the quality of the reconstructed supervision and, consequently, the comparison results.

We therefore retain each method’s data-construction and training protocol for an overall comparison, rather than reconstructing its specialized supervision ourselves. Among the latent-reasoning methods compared, SwimBird and Interlude share the same Qwen3-VL-8B backbone, providing a more direct comparison with respect to the underlying model. Nevertheless, a shared backbone alone does not eliminate differences in training data or compute.

Q1. Controlled comparisons with direct visual alignment and a fixed midpoint

Experimental setup. All variants use the same backbone, 50K training samples, training budget, and number of latent reasoning steps. We compare five geometric-supervision configurations:

  • Interlude: Uses the two-sided margin in Eq. (7), with the stage-dependent τ\tau in Eq. (6): 0.75 for the abstract stage, 0.50 for the bridging or unified stage, and 0.25 for the concrete stage.
  • Without IMC: Removes geometric supervision while retaining the latent architecture and semantic supervision.
  • Direct visual alignment: Directly minimizes the cosine distance between latent states and the retrieved visual anchors.
  • Fixed-midpoint target: Fixes τ=0.5\tau=0.5 in Eq. (6) and directly minimizes the cosine distance between latent states and the text–visual midpoint.
  • Fixed-midpoint two-sided margin: Also fixes τ=0.5\tau=0.5, but applies the two-sided margin in Eq. (7) instead of directly pulling latent states toward the midpoint.
ConfigurationMMStarHallusionBenchV*BenchERQA
Interlude72.171.886.948.3
Without IMC72.171.283.844.0
Direct visual alignment70.570.084.346.5
Fixed-midpoint target69.869.383.845.5
Fixed-midpoint two-sided margin70.169.884.846.0

Removing IMC leaves only semantic-anchor supervision and degrades performance overall, particularly on the two visually intensive benchmarks, ERQA and V*Bench. Directly aligning latent states with visual anchors or a fixed text–visual midpoint also underperforms the full IMC objective. These results suggest that simply introducing a fixed alignment target is insufficient to obtain consistent gains and may interfere with semantic supervision.

Using a two-sided margin at the same fixed midpoint improves performance over direct midpoint alignment, but still falls short of Interlude with its stage-dependent schedule. Together, these comparisons support the proposed design as a more effective way to balance semantic supervision and geometric manifold supervision among the configurations tested.


Thank you again for the time and effort you have devoted to reviewing our work. If our responses have addressed your concerns, we would sincerely appreciate your reconsideration of the score. We would be grateful for the opportunity to share this work with the community and inspire future research.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. With two days remaining in the rebuttal period, we would greatly appreciate it if you could review our responses and additional experimental results when you have a chance. We hope these address your concerns, and we would be happy to clarify any remaining questions.

Thank you for your consideration.

AuthorsOpenReview ↗

[11 hours Remaining] Could you confirm whether our responses have addressed your concern? Thanks!

Dear Reviewer wbcn,

As the rebuttal discussion phase will end in 11 hours, we would like to express our sincere gratitude for your engagement so far. We kindly remind you that after November 18 (AoE), we will no longer be able to respond to any further questions.

We hope that our responses have addressed your concerns during the author-reviewer discussion phase. If there are any remaining issues or clarifications needed, please let us know at your earliest convenience.

Thank you again for your time and consideration!

Best regards,

Authors of paper #45355

Reviewer_tBSjOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Interlude alternates textual reasoning with short recurrent hidden-state segments in Qwen3-VL-8B. It is trained on 50K synthetic traces with latent boundaries and ordered concept sets. A semantic loss places language-head probability on relevant concepts, and a two-sided geometric margin favors a reference interpolated between textual and visual centroids. Experiments cover six benchmarks, ablate the latent segments and both objectives, measure inference cost, and analyze latent representation geometry.

Critical strengths
  1. The same-data controls support a contribution from the method beyond the synthetic traces. In Table 2, ordinary CoT SFT reaches 65.1 Overall compared with Interlude's 68.7. Suppressing latent segments at test time lowers the score to 66.4, and removing either latent objective also reduces performance. Gains over the original backbone appear on all six benchmarks, including image-to-video transfer to VSI-Bench.

  2. Interlude improves the accuracy/compute trade-off while retaining readable text between latent segments. Figure 4 measures latency and FLOPs as well as decoded tokens, including the overhead relative to Instruct. The geometric analysis is appropriately cautious. It distinguishes the word-embedding principal subspace from the language-head null space and presents the geometry as supporting evidence, without claiming a proved causal decomposition.

Critical weaknesses
  1. The intermediate-modality ablation removes both visual supervision and the proposed geometry. It therefore does not show why the interpolated reference and two-sided margin are preferable to simpler visual alignment with the same data and latent architecture. The before/after cosine and PCA measurements cannot settle this comparison. I find the overall performance gains better supported than the specific geometric design.

  2. The experimental recipe is incomplete. Appendix D gives latent-loss parameters and specifies greedy evaluation, but leaves out benchmark-specific generation budgets and details of the efficiency measurements. The anonymous README provides a Stage 2 template without the paper-matched Stage 1 recipe. Its inference defaults use sampling and allow a second direct-answer attempt. These defaults may have been overridden for the reported experiments; the full training recipe and actual evaluation settings are needed to reproduce the accuracy and latency results.

Questions for authors
  1. With the same latent architecture, synthetic traces, and compute budget, how does the intermediate-modality loss compare with direct alignment to the retrieved visual anchor or a fixed mixed anchor? This would isolate the value of the proposed geometry.

  2. Please provide the complete training stages and initialization, plus the evaluation overrides for Table 2 and Figure 4. Was direct-answer fallback disabled? What generation budgets, evaluation items, batch size, and hardware were used for the efficiency measurements, and did timing include all latent steps and retries?

  3. How stable are the gains from the geometric objective and latent segments? Repeated training runs or paired uncertainty estimates would help assess the smaller component-level gains.

4 comments

Thank you for recognizing the methodological value of Interlude, its performance gains, and its inference efficiency. We respond below to your comments on the geometric design, reproducibility details, and stability of the gains.


W1. The contribution of the geometric design relative to simple visual supervision

Thank you for pointing out that removing the intermediate-modality constraint (IMC) removes both visual supervision and the geometric constraint. To distinguish their contributions, we added the following controlled experiments. All settings use the same Qwen3-VL-8B-Instruct backbone, 50K training examples, training budget, latent architecture, semantic supervision, and number of latent reasoning steps; only the geometric objective changes:

  • Interlude: uses the two-sided margin in Eq. (7) and adjusts τ\tau in Eq. (6) according to the concept stage: 0.75 for abstract, 0.50 for bridge or unified, and 0.25 for concrete.
  • Direct visual alignment: directly minimizes the cosine distance between the latent state and the retrieved visual anchor.
  • Fixed midpoint target: fixes τ=0.5\tau=0.5 and directly minimizes the cosine distance between the latent state and the textual–visual midpoint.
  • Fixed midpoint with two-sided margin: also fixes τ=0.5\tau=0.5, but retains the two-sided margin in Eq. (7).

“Fixed” refers to the interpolation coefficient; the anchors still vary with the example and concept stage. Both settings that use a margin set δ=0.05\delta=0.05.

SettingMMStarHallusionBenchV*BenchERQA
Interlude72.171.886.948.3
Direct visual alignment70.570.084.346.5
Fixed midpoint target69.869.383.845.5
Fixed midpoint with two-sided margin70.169.884.846.0

The fixed midpoint target scores lower than direct visual alignment on all four benchmarks, showing that a mixed anchor alone does not automatically yield gains. Using the two-sided margin at the same midpoint improves all four scores. Adding stage-dependent scheduling further improves Interlude's scores on all four benchmarks, with Interlude also outperforming direct visual alignment on each. Thus, the combination of an intermediate-modality reference, a two-sided margin, and stage-dependent scheduling performs best among these controls, and its advantage cannot be attributed solely to visual supervision.

W2. Completeness of the training and evaluation documentation

Thank you for identifying the ambiguity in the release documentation. “Stage 1 / Stage 2” is legacy naming. The paper uses a single joint training procedure initialized from the backbone. The “Stage 2” script jointly optimizes textual cross-entropy, semantic supervision, and the intermediate-modality constraint, implementing the complete method without depending on an additional Stage 1. In the formal code release, we will standardize the script names, correct the outdated README descriptions, and provide the run configurations corresponding to Table 2 and Figure 4. The complete training parameters and actual evaluation settings are given in Q2.

Q1. Comparison with visual and fixed mixed anchors under matched conditions

The four settings in W1 examine anchor selection, loss formulation, and stage-dependent scheduling in turn. The results show that Interlude outperforms direct visual alignment and both fixed midpoint variants under matched conditions.

Eq. (7) requires the latent state's similarity to the intermediate reference to exceed its similarity to each of the textual and visual anchors by at least δ\delta. Compared with continuously reducing the distance between the state and a target, this relative constraint leaves room for the state to adapt to the semantic and task objectives. Combined with stage-dependent scheduling, it achieves the best results reported above.

Q2. Initialization, training parameters, and actual evaluation settings

The model is initialized from Qwen3-VL-8B-Instruct, and latent computation reuses the language-model stack. The added step embeddings use Gaussian initialization with a standard deviation of 0.02, and latent outputs use RMSNorm. During training, we freeze the vision encoder and jointly update the language model, language head, and added latent parameters with the following settings:

ItemSetting
Training data50K examples
Training duration and hardware2 epochs; 16 H20 GPUs
OptimizerAdamW; β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, ϵ=10−8\epsilon=10^{-8}
Learning ratesTrainable backbone parameters: 1e−51e-5; latent parameters: 3e−53e-5
Schedule and regularizationCosine schedule; 50 warmup steps; weight decay 0.01; gradient clipping 1.0
Batch size and sequence lengthEffective batch size 64 (16 GPUs × 1 example per GPU × 4 gradient accumulation steps); maximum sequence length 8192
Latent supervision1–3 steps per segment according to the concept annotations; semantic and geometric loss weights both set to 1

Table 2 and Figure 4 both use greedy decoding. The generation budget is max_new_tokens=2048 across all six benchmarks. Each latent segment ends when the model predicts its end marker, the cosine similarity between consecutive states exceeds 0.99, or the segment reaches three steps. Concept annotations are not used at inference. Direct-answer fallback remains enabled: the model falls back to a direct answer only in the very few cases where pathological repetition prevents normal completion. The efficiency measurements in Figure 4 use the full ERQA evaluation set of 400 questions on H20 hardware, with the same generation budget. Timing includes the initial generation, all latent steps, and all retries.

Q3. Stability of the gains from the geometric objective and latent segments

Following your suggestion, we use paired bootstrap analysis to compare Interlude with the variant without IMC and the variant with latent segments disabled at inference. We perform 10,000 bootstrap replicates, resampling questions with replacement within each benchmark while keeping the two outcomes for each question paired. We preserve each benchmark's original scoring rule and compute the Overall gain as the equally weighted mean of the six benchmark score differences. The 2.5th and 97.5th percentiles of the resulting distribution form the 95% confidence interval. This analysis measures the stability of the gains under resampling of the evaluation examples, with model checkpoints held fixed.

SettingOverallInterlude − setting95% paired confidence interval for the difference
Interlude68.7——
IMC removed66.9+1.8[+0.9, +2.7]
Latent segments disabled66.4+2.3[+1.3, +3.2]

Both 95% confidence intervals lie entirely above zero, providing statistical support from paired comparisons for the positive contributions of the geometric objective and latent segments. The control with latent segments disabled keeps the model weights unchanged, further clarifying the role of latent computation at inference.


Thank you again for your recognition of our work and your constructive feedback. We hope the additional experiments and clarifications address your questions and concerns.

Thank you for the clarifications. Your response resolves the concerns I raised in my review, and I will raise my overall score accordingly.

Thank you for taking the time to review our rebuttal and update your score! We sincerely appreciate your constructive feedback, which has helped us clarify and strengthen our work.

Reviewer_rnVzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method for latent visual reasoning. It proposes to interleave textual reasoning with multiple short latent reasoning sequences. The model is trained with cross-entropy on the text portions and directly passes hidden states from one time step to the next in the latent segments. On the latent segments, there are two auxiliary losses: a semantic/textual one, aligning the hidden states with semantic keywords (arranged from general to more specific); and an intermodality objective (textual-visual), encouraging similarity between the hidden states and a mixture of the embeddings of the semantic keywords and the visual embeddings most closely associated with the semantic keywords. The more specific the semantic keywords become in the latent segment, the more the loss focuses on matching visual (rather than textual) information.

The paper collects data that contain the necessary annotations of keywords and reports evaluations on six benchmarks with a focus on perceptual and spatial tasks.

Critical strengths
  • The intermodality constraint is interesting, seems intuitively well-justified and new in this context.
  • The reported performance gains against SFT on the same data are consistent across a diverse set of benchmarks, and quite noticeable in some cases (e.g. on ERQA).
Critical weaknesses
  • The main innovation, the intermodality constraint, has a relatively large design space that is not very well explored. This makes it hard to tell, e.g. how sensitive the results are on the various hyperparameters and what the impact of the design decisions is (margin, schedule of tau, why a spherical interpolation).
  • The causal role of the latent tokens in the overall decision making process of the model is not very clearly established. Recent work ("What’s Holding Back Latent Visual Reasoning?" by Viveiros et al. 2026, "Imagination helps visual reasoning, but not yet in latent space" by Li et al. 2026) has shown for several latent visual reasoning methods that latent tokens can be replaced by dummy values without materially impacting performance. As I understand, the ablation "w/o Latent Segment" presented here tests a limited form of this (similar to "Skip Latents" intervention in Viveiros et al. but Skip latents on its own does not show the full picture). Additional interventions (e.g. swapping latents across examples or replacing them by noise) are needed to make a stronger claim that the latent tokens contribute to the overall performance.
Questions for authors

Questions

  • Does the SFT baseline use the semantic keywords in the training data or remove the information related to latent segments entirely? Both comparisons seem relevant in principle.

  • In section 6, in 409/410 what do you mean by "reasoning positions" for extracting representations H before training? Why are these reasoning positions before training a good reference to compare against for representations of latent segments after training?

  • For the intermediate-modality constraint, what is the impact of the τ\tau schedule on final performance? Would a well-chosen single τ\tau value work just as well?

Comments on presentation

The comparison with related work is incomplete: In particular, this work builds strongly on Laser for the semantic guidance objective but this is only indicated in the appendix. In addition, related work on modality alignment and mixing modalities is not discussed, e.g. "Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval" by Jang et al. 2024 seems relevant.

Figure 1a doesn't make a very clear point to me. The example mentions "waist height", which sounds like a good way to capture height in language, although it may not be the correct description here. The ground-truth answer (C) doesn't seem obvious to me either. Answer A sounds reasonable as well. Perhaps a different choice of example can make the point better.

L 135: the softmax bottleneck refers to the issue that the vocabulary is much larger than the hidden state dimension, such that the output layer has to be low rank and couple logits, i.e. a limit of expressivity. It's unclear how the argument arrives at the hidden states being more expressive than what is (linearly) encoded from this.

L144: it harms the flow of the paper to have to jump all the way to Section 6 to understand the methodology in Section 3. As a result, section 3 is hard to follow.

Table 1: unclear why the discussion switches between angles in degrees and cosine similarity.

The τ\tau schedule seems undefined in Eq 6, unclear which values are used in experiments. The appendix refers back to the equation.

6 comments

Thank you for recognizing the value of Interlude and for your detailed suggestions. We address each point below.


W1. Design choices for the intermediate-modality constraint

Thank you for highlighting the need to validate these design choices more thoroughly. We added controlled comparisons of anchor selection, loss formulation, and the stage-dependent schedule. All configurations use the same Qwen3-VL-8B-Instruct backbone, 50K training examples, training budget, semantic supervision, and number of latent reasoning steps. Only the geometric-supervision setting changes:

  • Interlude: Uses the two-sided margin in Eq. (7), with the stage-dependent τ\tau in Eq. (6): 0.75 for the abstract stage, 0.50 for the bridging or unified stage, and 0.25 for the concrete stage.
  • Direct visual alignment: Directly minimizes the cosine distance between latent states and the retrieved visual anchors, without constructing a mixed text–visual reference.
  • Fixed-midpoint target: Fixes τ=0.5\tau=0.5 and directly minimizes the cosine distance between latent states and the text–visual midpoint.
  • Fixed-midpoint two-sided margin: Also fixes τ=0.5\tau=0.5, but retains the two-sided margin in Eq. (7).

Here, “fixed” refers to the interpolation coefficient; the textual and visual anchors still vary with the example and concept stage. Both configurations using a margin set δ=0.05\delta=0.05, as in the paper.

ConfigurationMMStarHallusionBenchV*BenchERQA
Interlude72.171.886.948.3
Direct visual alignment70.570.084.346.5
Fixed-midpoint target69.869.383.845.5
Fixed-midpoint two-sided margin70.169.884.846.0

The fixed-midpoint target underperforms direct visual alignment on all four benchmarks, showing that simply replacing the visual anchor with a mixed anchor does not yield a gain. Using a two-sided margin at the same midpoint improves all four scores. Adding the stage-dependent schedule improves them further, with Interlude also outperforming direct visual alignment throughout. These results support the combined design of an intermediate reference, a two-sided margin, and stage-dependent scheduling over the simplified alternatives tested.

Eq. (7) imposes a relative constraint: once a state's cosine similarity to the intermediate reference exceeds its similarity to each modality anchor by at least δ\delta, the geometric loss becomes zero and no longer continually pulls the state toward a single target. By holding the margin value fixed, these comparisons examine the loss formulation and its combination with stage-dependent scheduling.

We choose SLERP for consistency with the cosine metric: spherical interpolation between normalized anchors makes τ\tau directly control the angular proportion between them. We use it to construct the reference direction; our contribution lies in combining this reference with the stage-dependent schedule and the two-sided margin.

W2. Whether latent content contributes to subsequent decisions

In the paper, disabling latent segments reduces Overall from 68.7 to 66.4, demonstrating the benefit of retaining latent computation. To address your question about whether the state content contributes to decisions, we added cross-sample swapping and matched random-perturbation controls on V*Bench. These interventions directly test the role of content while preserving the computational structure.

Experimental setup. Cross-sample swapping replaces the current latent states with states from the corresponding positions of another question. We intervene on the first segment, the last segment, or both. At each step, the random control matches the swapped state's norm and its distance from the original state. All conditions preserve the text prefix preceding the intervention, the latent slots, and the number of steps per segment, and use the same decoding and answer-completion rules. Examples with only one latent segment are intervened on once. The unmodified model achieves 86.9% accuracy. Table entries are accuracies (%); parentheses show changes in percentage points.

Intervention positionCross-sample swappingDistance-matched random perturbation
First latent segment80.6 (−6.3)83.8 (−3.1)
Last latent segment81.2 (−5.7)84.3 (−2.6)
First and last latent segments77.0 (−9.9)82.7 (−4.2)

Changing state content consistently reduces performance despite preserving the latent computational structure, with a larger effect when both the first and last segments are perturbed. Cross-sample swapping causes larger drops than distance-matched random perturbations at all three intervention positions, showing that state content and its correspondence to the current question affect predictions. These results address your concern about dummy latents: Interlude's subsequent reasoning uses information carried by the latent states.

Q1. Use of semantic keywords in the SFT baseline

The SFT baseline in the paper uses the complete teacher-generated textual CoT and final answers for the same examples. It does not additionally include the ordered semantic keywords constructed for latent segments, nor does it use latent boundaries or auxiliary losses. It retains the teacher's full textual reasoning; the ordered keywords are used for Interlude's latent supervision. We will clarify this distinction and specify the SFT inputs and training objective in the baseline description.

Q2. Meaning and comparability of “reasoning positions” before training

Thank you for pointing out this ambiguity. “Reasoning positions” refers to the textual reasoning token positions generated normally by the backbone. Before training, HH is the mean of the final-layer hidden states at all textual reasoning token positions; after training, HH is the mean of the final-layer hidden states at all latent positions.

This reference serves the purpose of Section 6: characterizing the modality alignment of Interlude's latent representations relative to the textual reasoning representations actually used by the backbone. Both conditions use the same ERQA and VSI-Bench questions and extract hidden states from the same architecture, layer, and dimensionality. Figure 5(a) compares centroid angles, Figure 5(b) compares visual alignment within and outside the principal word-embedding subspace, and Figure 6 compares per-example cosine similarities. The same statistical procedures are applied before and after training to measure the relationship between HH and the visual and textual references within each model.

Q3. Effect of the stage-dependent schedule versus a fixed τ\tau

In Eq. (6), τ=1\tau=1 corresponds to the textual anchor and τ=0\tau=0 to the visual anchor. Interlude uses 0.75, 0.50, and 0.25 as concepts progress from abstract to concrete, gradually shifting the reference direction from the textual side toward visual evidence.

The fixed-midpoint two-sided margin in W1 retains the same loss and only fixes τ=0.5\tau=0.5 across all stages. Interlude outperforms this configuration on all four benchmarks. Thus, this fixed value does not match the stage-dependent schedule: adapting the reference direction to the concept stage is more effective than always using the symmetric text–visual midpoint.

Writing and presentation


P1. Relation to Laser and work on modality mixing

Thank you for bringing these related works to our attention. We will explicitly cite Laser when first introducing semantic guidance in the main text, acknowledge that the dynamic semantic window and self-refined soft targets follow its approach, and explain how we construct stage-specific windows from each interlude's ordered concept set. We will also discuss Jang et al. (2024), which applies SLERP and text anchoring to composed image retrieval. Our method uses a stage-dependent intermediate reference and a two-sided margin to train recurrent latent states.

P2. Ambiguity of the example and answer in Figure 1a

Your point is well taken. “Waist height” can express relative height, and the current example does not clearly establish the evidence for option C or distinguish it from option A. We will replace it with an example that has clear visual evidence, an unambiguous answer, and verifiable model outputs. We will also highlight the local visual relationship needed to answer the question, so that Figure 1a more clearly illustrates the task-relevant visual details that textual reasoning may omit.

P3. Softmax bottleneck and hidden-state capacity

Our intention is to distinguish textual readout from the organization of visual evidence. “Information under-saturation” is our empirical hypothesis about representational capacity: continuous hidden states can carry richer visual evidence while maintaining semantic readability. Interlude accordingly combines semantic guidance with the intermediate-modality constraint, directly passes hidden states within latent segments, and allows subsequent text to use these states through self-attention.

Thank you for pointing out the need to distinguish background from supporting evidence. The softmax bottleneck concerns the expressivity of the output-distribution family induced by a linear language head. The empirical basis for our hypothesis is the concentrated representations and weak visual alignment observed in the paper. Section 6 characterizes the broader representational spread and changes in visual alignment after training, while task experiments and state interventions demonstrate the role of latent representations in reasoning. We will revise the discussion to make this logic explicit.

P4. Dependence of Section 3 on definitions in Section 6

We agree that the method discussion should be self-contained. We will define HH (reasoning hidden states), VV (projected visual-token representations), TT (input word embeddings), and the principal word-embedding subspace when they first appear in Section 3, and briefly explain how they motivate the semantic and geometric objectives. Section 6 will focus on measurement details and experimental analysis, so that understanding the method does not require looking ahead for definitions.

P5. Use of angles and cosine similarities in Table 1

We intend to characterize weak visual alignment through both mean directions and per-example correspondence. The angle of 89.4° indicates near-orthogonality between the centroids of the reasoning hidden states and visual representations; 0.008 is the mean per-example cosine similarity. Together, these measurements show weak direct geometric alignment between the backbone's textual reasoning states and visual representations, providing empirical motivation for visual constraints.

Angles provide an intuitive view of directional relationships, while cosine similarities facilitate the presentation of per-example distributions. We will explicitly label the quantities as “centroid angle (°)” and “mean per-example cosine similarity” and explain their respective aggregation procedures.

P6. Explicit values of the τ\tau schedule in Eq. (6)

Thank you for identifying this gap in the explanation. Eq. (6) specifies the monotonic relation for τ\tau, while the numerical values are listed in Table 3 of Appendix D. We will place the stage-to-value mapping given in W1 directly after Eq. (6) and clarify that the schedule follows concept stages.


Thank you again for your careful review and constructive feedback; we hope the additional experiments and clarifications address your concerns and would be grateful for your reconsideration of our work in light of this evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.