acceptodds
Under review as a conference paper at ICLR 2027

Unifying Heterogeneous Visual Reasoning in Expressive Latent Space

Abstract

Visual reasoning increasingly relies on heterogeneous intermediate representations, including textual rationales, regions, scene graphs, segmentation, depth, and visual manipulations. Because these representations differ substantially in structure and modality, they are typically modeled with separate reasoning interfaces, which scale poorly as more reasoning forms are incorporated. This raises a fundamental question: can heterogeneous forms of visual reasoning be unified in a single shared latent space? We investigate this question through HVLR (Heterogeneous Visual Latent Reasoning), which unifies eight distinct reasoning forms within a common autoregressive latent interface. To reconcile heterogeneous supervision, we first render all reasoning representations into a shared pixel space and encode them with the same vision encoder. We then use a two-stage distillation procedure to translate these representations into autoregressive latent states and internalize them into a closed-book model that requires no explicit intermediate representation at inference time. We conduct a systematic analysis with HVLR and uncover three main findings. First, heterogeneous reasoning forms can be effectively unified: combining all eight improves Qwen2.5-VL-7B from 64.66% to 69.97% average accuracy across nine benchmarks. Second, the shared latent space consistently benefits from reasoning diversity. Under a fixed 40K-example budget, increasing the number of reasoning forms from 1 to 3, 5, and 8 improves performance from 66.65% to 67.67%, 68.61%, and 69.97%, respectively, with positive trends across all nine benchmarks. Third, latent reasoning outperforms explicit intermediate decoding while reducing intermediate computation from roughly 177 generated tokens to only 8 latent steps, suggesting that continuous latent representations provide a more expressive and efficient medium for heterogeneous visual reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Visual Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PMSgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes HVLR, which unifies eight distinct reasoning forms within the common autoregressive latent interface, to tackle visual reasoning. Its core idea is to render all reasoning forms into the common pixel space and encoding them with the same vision encoder, then use a two-stage distillation procedure to transfer these representations into latent states that can be generated without explicit intermediate supervision at inference time. Experiments on nine benchmarks show that HVLR improves Qwen2.5-VL-7B from 64.66% to 69.97% in average performance, with greater reasoning-form diversity yielding consistent gains and latent reasoning being more efficient than explicit intermediate decoding.

Critical strengths
  • The paper identifies an important limitation of existing visual reasoning methods: different intermediate representations are typically modeled through separate reasoning interfaces, making it difficult to combine complementary reasoning signals.
  • The proposed framework provides a simple and effective way to unify heterogeneous reasoning signals, while avoiding representation-specific decoders and explicit intermediate reasoning at inference time.
  • Experiments demonstrate that HVLR improves Qwen2.5-VL-7B from 64.66% to 69.97% in average performance, while achieving the promising accuracy–efficiency tradeoff.
Critical weaknesses
  • The novelty is limited, as the core idea of rendering reasoning content into images and internalizing the resulting representations as autoregressive latent states has already been explored by Render-of-Thought [1]. While HVLR extends this paradigm to eight heterogeneous reasoning forms, the paper neither discusses this closely related work nor clearly establishes the additional technical novelty beyond using broader heterogeneous supervision.
  • The comparison with latent-reasoning baselines (e.g., LVR and SkiLa) is not fair. HVLR is trained on the newly curated heterogeneous reasoning data, whereas these competing latent-reasoning baselines directly use their official checkpoints. Therefore, the reported performance gains cannot be attributed solely to the proposed method.
  • The modality-diversity experiment does not isolate the effect of reasoning-form diversity. The eight reasoning forms are generally collected from different datasets rather than being alternative representations of the same examples. Therefore, increasing the number of reasoning forms also increases source and task diversity.

[1] Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning

Questions for authors
  • Can the authors retrain some strong latent-reasoning baselines, such as LVR or SkiLa, using the same training setting as HVLR? This comparison is necessary to determine whether the reported gains arise from the proposed method.
  • What is the total training cost of HVLR, including both training stages and the offline extraction of teacher targets, and how does it compare with existing latent-reasoning baselines?
  • How well does HVLR generalize across VLM architectures beyond the Qwen-VL family?
6 comments

We thank the reviewer for recognizing the motivation and practical value of unifying heterogeneous reasoning representations. We appreciate the suggestion to discuss RoT and clarify our technical contribution. We address this comparison below, together with the concerns about baseline fairness, training cost, and generalization beyond the Qwen family.


W1

The novelty is limited, as the core idea of rendering reasoning content into images and internalizing the resulting representations as autoregressive latent states has already been explored by Render-of-Thought [1]. While HVLR extends this paradigm to eight heterogeneous reasoning forms, the paper neither discusses this closely related work nor clearly establishes the additional technical novelty beyond using broader heterogeneous supervision.

[1] Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning

Response

We will cite Render-of-Thought (RoT) and add a detailed comparison in the Introduction and Related Work. HVLR differs from RoT in its goal, method and outcome.

  1. Goal: compressing one form vs. unifying many. RoT renders a single form, textual CoT, into images in order to compress text reasoning. It is evaluated only on text-only math benchmarks (trained mainly on GSM8k-Aug-NL and evaluated on GSM8k-Aug, GSM-Hard, SVAMP, MultiArith and MATH).

    In HVLR, rendering is a unification device. 7/8 of our forms (crops, highlights, masks, depth, edges, scene graphs, interleaved images) are natively visual or structural and have no textual chain to compress. Our central question is whether heterogeneous forms can share one latent space, and how capability scales with the number of forms (Sec. 4.1). This question cannot be posed in a single-form setting.

  2. Method: a projection-head regression vs. two-stage translation and distillation. RoT attaches a projection head to the LLM. In Stage I, the backbone and vision encoder are frozen and only the projection head is trained, with an MSE alignment loss against the vision-encoder features of the rendered CoT, plus a cross-entropy term on the answer. In Stage II, the vision encoder and projection head are frozen and the LLM is fine-tuned with LoRA using cross-entropy only, with no visual alignment loss. The latents are therefore regressed directly onto the encoder features of the rendered text, and once the LLM itself is trained there is no alignment signal at all. Our Table 5 shows that direct alignment to encoded helper features is suboptimal in the heterogeneous setting: the "w/o distillation" variant, which aligns the student directly to the pooled helper features Z, is 1.95% worse.

    HVLR differs in three ways:

    • (i) a Stage-1 teacher first translates helper features into autoregressive latent targets, and its answer positions are masked from the helpers, so information must flow through the latents
    • (ii) these targets are distilled into a closed-book student with layer-wise alignment at all layers, which stays active while the model is trained
    • (iii) there is no projection or prediction head, because the latents re-enter directly as input embeddings.
  3. Outcome: a large accuracy loss vs. a gain in both accuracy and efficiency. By RoT's own results, its latent reasoning comes at a large accuracy cost relative to explicit CoT: RoT scores below explicit CoT in every cell of its main table. Numbers reported in the RoT paper (Pass@1, %):

Backbone (RoT paper)BenchmarkExplicit CoT (SFT-CoT)RoT (latent)Δ
Qwen3-VL-2B4-benchmark avg. (Table 1)63.837.0−26.8
Qwen3-VL-4B4-benchmark avg. (Table 1)79.355.4−23.9
LLaVA-1.6-Mistral-7B4-benchmark avg. (Table 1)48.334.3−14.0
Qwen3-VL-4BGSM8k-Aug (Table 1)81.237.8−43.4
Qwen3-VL-4BMATH (Table 3)55.833.2−22.6

RoT decreases 14–27% on the four-benchmark average across three backbones, and up to 43% on GSM8k-Aug, in exchange for 3–4× shorter reasoning.

HVLR shows the opposite pattern. With the same model, data and weights, keeping reasoning in the latent space is more accurate than decoding it as text: forced-latent beats forced-text by 1.16 and 1.24 points, and no explicit-text variant reaches the all-latent model (Sec. 4.2, Table 11). It does so with 8 latent steps instead of ~177 generated tokens.

In short, RoT shows that rendered text can be compressed into latents at a substantial accuracy cost, while HVLR shows that a shared latent space can outperform explicit decoding while unifying eight heterogeneous forms.

W2 & Q1 (baseline fairness)

W2. The comparison with latent-reasoning baselines (e.g., LVR and SkiLa) is not fair. HVLR is trained on the newly curated heterogeneous reasoning data, whereas these competing latent-reasoning baselines directly use their official checkpoints. Therefore, the reported performance gains cannot be attributed solely to the proposed method.

Q1. Can the authors retrain some strong latent-reasoning baselines, such as LVR or SkiLa, using the same training setting as HVLR? This comparison is necessary to determine whether the reported gains arise from the proposed method.

Response

The paper already includes a data-controlled re-implementation of LVR (Sec. 4.4, Table 4, Finding 3), which separates the effect of the data from that of the method.

We adapted LVR's method (a single teacher-forced stage, a prediction head that predicts each latent slot, and alignment + answer cross-entropy + contrastive losses) and trained it on our corpus:

Model (7 benchmarks of Table 4)Training dataAvg.
LVR-7B (official checkpoint)LVR's own data (~430K)62.41
LVR-style method + our data500K examples of our corpus67.44
HVLR40K examples of our corpus69.58
  • Method effect with the data source held fixed: HVLR beats the LVR recipe by +2.14 while using 12.5× less data from the same corpus.
  • Data effect with the recipe held fixed: our data lifts the LVR recipe by +5.03 over its official checkpoint.

Both factors therefore contribute, and we see this as complementary rather than as a weakness. The heterogeneous corpus is part of our contribution, and it can only be exploited through a shared interface: LVR's original pipeline supervises a single form of helper. Further evidence for the method on identical data comes from the stage ablations (Table 5: −1.95 / −1.47) and a new plain-SFT control. We trained plain SFT (image + question → answer, no helper images and no latent slots) on the same 40K data and the same 25 mixtures as Sec. 4.1, and compared each model with the HVLR model trained on the same mixture:

KPlain SFTHVLRΔ (HVLR − SFT)HVLR wins (same mixture)
166.6366.65+0.025 / 8
366.8367.67+0.837 / 8
566.9568.61+1.668 / 8
868.7969.97+1.181 / 1
All 25 mixtures+0.8521 / 25 (Wilcoxon p = 1.9×10⁻⁴)

HVLR is better on 21 of 25 paired mixtures (+0.85 on average, Wilcoxon p = 1.9×10⁻⁴) and reaches 69.97 vs. 68.79 on the full 8-form mixture.

SkiLa is designed for interleaved text–sketch reasoning: its training traces interleave textual reasoning steps with latent sketches. 7/8 of our forms provide a helper representation without such an interleaved textual trace, so SkiLa's recipe cannot be applied to our corpus without being redesigned.

W3

The modality-diversity experiment does not isolate the effect of reasoning-form diversity. The eight reasoning forms are generally collected from different datasets rather than being alternative representations of the same examples. Therefore, increasing the number of reasoning forms also increases source and task diversity.

Response

We ran a source-controlled experiment. We selected 10K images from Visual Genome. For every image, we built all four of bbox crop, text CoT, segmentation and depth: each form has a question that depends on it and a newly rendered helper of that form. Every K condition therefore draws from the same image pool, with the same training budget (10K examples per condition) and schedule. Only the number of reasoning forms changes. Since all four forms exist for every image, K ranges from 1 to 4. We report the tier mean over all C(4,K) form subsets.

K (same images)Avg. (HVLR)Δ vs. K=1
165.25–
266.18+0.93
367.33+2.08
468.41+3.16

Accuracy still rises monotonically with fixed sources: each added form helps (+0.93, +1.15, +1.08), at about +1.06% per form. If the original trend came mainly from source or task diversity, holding the images fixed should make it smaller. It does not: the total gain from K=1 to K=4 here (+3.16) is similar to that from K=1 to K=8 in Sec. 4.1 (+3.32, 66.65 → 69.97). The two setups differ in data scale, so we compare the direction and rough size of the trend, not exact values.

In addition, we trained plain SFT (image + question → answer, no helper images and no latent slots) on the same 25 mixtures as Sec. 4.1, under the same 40K budget:

Plain SFTHVLR (latent)
K=1 tier mean (std)66.63 (1.18)66.65 (1.09)
K=3 tier mean (std)66.83 (0.45)67.67 (0.45)
K=5 tier mean (std)66.95 (0.20)68.61 (0.42)
K=868.7969.97
OLS slope per form+0.166 ± 0.086 (p = 0.066)+0.483 ± 0.079 (p = 2.9×10⁻⁶)
R²14.0%62.1%
Spearman ρ0.17 (p = 0.41)0.88 (p = 7.4×10⁻⁹)
Jonckheere–Terpstra concordant pairs126 / 216 (chance: 108)205 / 216
Mann–Whitney, K=5 > K=336 / 64 pairs64 / 64 pairs
Benchmarks with positive slope8 / 99 / 9

Under plain SFT, the trend is much weaker and not significant (+0.166 vs. +0.483 per form, Spearman ρ = 0.17 vs. 0.88). The Sec. 4.1 trend is therefore not explained by mixing more data sources alone. It requires the shared latent interface.

Because of the rebuttal timeline, the source-controlled study uses 10K images, four forms and one source. In the revision we will extend it to more forms and a second image pool (e.g., Visual CoT images), and add it to Sec. 4.1.

Reviewer_a7PmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether heterogeneous forms of visual reasoning can be unified within a shared continuous latent space. The proposed HVLR framework converts eight types of intermediate reasoning representations into a common visual encoding interface and employs a two-stage teacher-student distillation procedure. The teacher learns latent states from auxiliary reasoning representations, while the student internalizes these states and performs inference without explicit intermediate representations. Experiments on nine benchmarks demonstrate improvements over the base multimodal model and several latent reasoning approaches. The paper further studies the effects of increasing reasoning-form diversity and compares latent reasoning with explicit intermediate decoding in terms of accuracy and efficiency.

Critical strengths
  1. The paper investigates a meaningful research question through a relatively systematic empirical study. The central question is not simply whether latent reasoning improves accuracy, but whether diverse intermediate reasoning forms can be internalized within a single shared latent interface. The fixed-data-budget experiments over multiple combinations of reasoning forms provide interesting evidence that heterogeneous supervision can be beneficial.

  2. The framework offers a practical and potentially extensible approach to multimodal reasoning. Rendering heterogeneous intermediate representations through a common visual encoder avoids the need to design specialized prediction heads for each reasoning form. The two-stage distillation procedure also removes explicit auxiliary representations at inference time. The evaluation across nine benchmarks, together with inference-efficiency measurements, makes the empirical findings potentially useful to the community.

Critical weaknesses
  1. A training-budget inconsistency weakens the paper's central diversity-scaling claim. Section 4.1 presents the comparison across K=1, 3, 5, and 8 reasoning forms as a controlled study with a fixed 40K-example budget. However, the appendix specifies that the main K=8 model uses two epochs in Stage 1 and one epoch in Stage 2, whereas the K<=5 ablations use one epoch per stage with different learning-rate settings. Consequently, the full comparison does not vary only the number of reasoning forms. The additional optimization budget and schedule differences for K=8 could partially explain its higher performance. This issue is particularly consequential because the monotonic diversity-scaling trend is one of the paper's principal claims.

  2. The algorithmic novelty relative to existing visual-reasoning distillation methods is moderate. Prior work has studied interleaved multimodal reasoning, continuous latent visual representations, and distillation of explicit visual reasoning into implicit model computation. HVLR presents an interesting combination and a broader empirical investigation, but the paper does not fully demonstrate that the proposed shared latent interface provides a fundamentally different mechanism rather than an effective multi-task distillation strategy. The evidence for latent-space expressiveness is also largely based on downstream accuracy rather than direct measurements of what information the latent states preserve from each reasoning form.

Questions for authors
  1. Can the authors rerun the K=8 diversity experiment using exactly the same number of optimization steps, learning-rate schedule, and stage-wise training budget as the K=1, 3, and 5 experiments? This is important for establishing the claimed relationship between reasoning diversity and performance.

  2. Under the same 40K-example budget, how does HVLR compare with a conventional multi-task SFT or teacher-student distillation baseline that uses the same heterogeneous supervision but does not employ autoregressive latent steps? This would help isolate the contribution of the latent interface.

  3. Can the authors quantitatively assess whether the final latent states retain information corresponding to each of the eight reasoning forms? For example, could lightweight probes or modality-specific interventions test the extent of genuinely shared versus form-specific information?

  4. How does the claimed efficiency advantage change when controlling for inference accuracy and accounting for auxiliary visual encoding and the additional training cost of the two-stage distillation pipeline?

5 comments

We thank the reviewer for recognizing the contributions and the practical value of the unified latent interface. We particularly appreciate the careful examination of the training schedules and the suggestions for analyzing the latent representations. These comments guided the additional controls and analyses presented below.


W1 & Q1 (training-budget inconsistency)

W1. A training-budget inconsistency weakens the paper's central diversity-scaling claim. Section 4.1 presents the comparison across K=1, 3, 5, and 8 reasoning forms as a controlled study with a fixed 40K-example budget. However, the appendix specifies that the main K=8 model uses two epochs in Stage 1 and one epoch in Stage 2, whereas the K<=5 ablations use one epoch per stage with different learning-rate settings. Consequently, the full comparison does not vary only the number of reasoning forms. The additional optimization budget and schedule differences for K=8 could partially explain its higher performance. This issue is particularly consequential because the monotonic diversity-scaling trend is one of the paper's principal claims.

Q1. Can the authors rerun the K=8 diversity experiment using exactly the same number of optimization steps, learning-rate schedule, and stage-wise training budget as the K=1, 3, and 5 experiments? This is important for establishing the claimed relationship between reasoning diversity and performance.

Response

The K=8 point in Sec. 4.1 is the Table 2 model, whose schedule (Stage 1: 2 epochs at 1×10⁻⁵; Stage 2: 1 epoch at 5×10⁻⁶) was chosen in a tuning pass after the K≤5 ablations were completed.

We have now trained K=8 with exactly the K≤5 schedule: one epoch per stage (313 steps each), peak learning rate 1×10⁻⁵ in both stages, and the same warmup, effective batch size (128), seed and data.

ModelV⋆HR4KHR8KMME-RWBLINKPOPERWQACV-BQSp++Avg.
K=5 tier mean (8 models)80.1671.6468.6651.2856.0688.0771.0677.6952.8768.61
Best K=5 mixture {4,5,6,7,8}81.1572.0069.3853.3153.0289.1371.6377.9458.4269.55
K=8, matched schedule (new)81.7071.8869.3153.3156.1089.1072.0377.5157.5669.83
K=8, tuned schedule (Table 2)81.7071.7570.8854.3056.8589.1972.0378.5354.4669.97
  • The schedule accounts for only 0.14% of the K=8 result (69.83 vs. 69.97).
  • Under the matched schedule, K=8 is still above all 24 K≤5 models, including the best K=5 mixture (69.55), and above the K=5 tier mean on 8 of 9 benchmarks.
  • Refitting the trend with the matched point: OLS slope +0.476 per form (was +0.483), Spearman ρ = 0.879 (unchanged), and the per-benchmark slope remains positive on all nine benchmarks.

In the revision, we will add the matched K=8 run to Sec. 4.1, Fig. 3 and Tables 8–10 as a schedule-controlled check, and state the schedule of every model explicitly. The main result in Table 2 (69.97) is unchanged.

W2

The algorithmic novelty relative to existing visual-reasoning distillation methods is moderate. Prior work has studied interleaved multimodal reasoning, continuous latent visual representations, and distillation of explicit visual reasoning into implicit model computation. HVLR presents an interesting combination and a broader empirical investigation, but the paper does not fully demonstrate that the proposed shared latent interface provides a fundamentally different mechanism rather than an effective multi-task distillation strategy. The evidence for latent-space expressiveness is also largely based on downstream accuracy rather than direct measurements of what information the latent states preserve from each reasoning form.

Response

We argue that there is conceptional difference between the two. Conventional multi-task distillation typically transfers knowledge from different tasks into a shared representation with multiple output heads, one per task. However, different tasks usually have competing objectives, leading to negative transfer or gradient interference [1, 2]. In contrast, HVLR distills different latent reasoning scaffolds that are coordinated toward the same underlying task and objective. Rather than learning potentially conflicting task-specific features, these scaffolds provide complementary reasoning strategies toward a common goal. This distinction is also reflected in our empirical findings: while multi-task learning can suffer from negative transfer [1, 2], we observe that different reasoning scaffolds mutually benefit from joint distillation, yielding consistent improvements when combined. Furthermore, HVLR uses a single set of latent slots, a unified target type, and no form-specific output heads. At inference, the model receives no form-identity information. To examine whether this produces a genuinely shared latent space rather than form-specific subspaces, we directly analyze the learned representations on 9,600 held-out samples.

Retention: do the final latents keep each form's helper content? For each form, we compute objective content targets from the helper image (e.g., depth layout, edge distribution, box location, mask area, scene-graph size). Linear probes (cross-validated R²) then read these targets from the Stage-1 teacher latents, which see the helper, and from the final closed-book latents. Retention = final R² / teacher R².

FormTeacher R²Final latent R²RetentionBefore training (same positions)
Interleaved helper0.750.75100%82%
BBox crop0.470.4698%2%
Segmentation0.420.3890%2%
BBox highlight0.480.3573%13%
Depth0.540.3769%25%
Edge0.410.2561%24%
Text CoT0.760.4154%21%
Scene graph0.590.3153%0%
  • The final latents retain 53–100% of the helper content that is readable from the teacher, for all eight forms. Five of the eight retain at least 69%.
  • Retention is far above the pre-training level for every form except the interleaved helper. That form is already partly readable before training (82%), because its intermediate images are close to the input image.
  • Per-sample retrieval. Each final latent retrieves its own teacher latent among ~850 same-form samples with 13–62% top-1 accuracy, against 1.5–10% before training and ~0.1% at chance.

Sharing: how much of the latent space is shared vs. form-specific? We decompose latent variance into a form-specific part (between-form differences in mean position) and a shared remainder. To avoid confounding form with image source, we compare forms only within the same data source. The table reports the form-specific share; the shared share is the remainder.

Same-source groupTeacherFinal latentInput onlyBefore training
BBox crop vs. highlight (similar questions)12.3%11.8%0.9%0.1%
Depth / edge / segmentation (same images)47.8%31.9%3.7%0.6%
Depth / edge / scene graph (same images)55.6%45.7%42.7%9.8%
Text CoT / interleaved51.9%37.4%27.6%5.9%

"Input only" is the final model's state after reading the question, before it generates any latent. All values are significantly above a form-label permutation baseline.

  • The teacher is form-partitioned; the final latent is mostly shared. In every group, the form-specific share is lower in the final latents than in the teacher (by up to 15.9%, for depth/edge/segmentation). As a result, 54–88% of final-latent variance is shared across forms. In the bbox group the share is already low in the teacher and stays low (12.3% → 11.8%).

In short: the teacher keeps different forms in separate regions of latent space. Closed-book distillation merges them into a predominantly shared space while preserving most of each form's helper content. We take this as evidence of a genuinely shared interface rather than parallel per-task channels.

[1] Standley, Trevor, et al. "Which tasks should be learned together in multi-task learning?." International conference on machine learning. PMLR, 2020.

[2] Yu, Tianhe, et al. "Gradient surgery for multi-task learning." Advances in neural information processing systems 33 (2020): 5824-5836.

Q2

Under the same 40K-example budget, how does HVLR compare with a conventional multi-task SFT or teacher-student distillation baseline that uses the same heterogeneous supervision but does not employ autoregressive latent steps? This would help isolate the contribution of the latent interface.

Response

Thanks for the questions. We will answer with two controlled comparisons.

(i) Plain multi-task SFT on the identical 40K data, without helpers or latent steps (image + question → answer, cross-entropy on the answer tokens only). We trained it on the same 25 mixtures as Sec. 4.1 and compared each model with the HVLR model trained on the same mixture:

KPlain SFTHVLRΔ (HVLR − SFT)HVLR wins (same mixture)
166.6366.65+0.025 / 8
366.8367.67+0.837 / 8
566.9568.61+1.668 / 8
868.7969.97+1.181 / 1
All 25 mixtures+0.8521 / 25 (Wilcoxon p = 1.9×10⁻⁴)

HVLR is better on 21 of 25 paired mixtures (+0.85 on average, Wilcoxon p = 1.9×10⁻⁴) and reaches 69.97 vs. 68.79 on the full 8-form mixture. The gap grows with the number of forms, from +0.02 at K=1 to +1.66 at K=5, where HVLR wins all 8 mixtures.

(ii) Explicit decoded targets instead of latents (Sec. 4.2, Table 11), on the same 40K data and backbone. Latent wins in every variant: forced-latent beats forced-text by 1.16 and 1.24 points on the same weights, and no explicit variant reaches the all-latent model.

Q3

Can the authors quantitatively assess whether the final latent states retain information corresponding to each of the eight reasoning forms? For example, could lightweight probes or modality-specific interventions test the extent of genuinely shared versus form-specific information?

Response

Thanks for the advice. On 9,600 held-out samples, we used linear probes to measure how much of each form's helper content the final latents retain, and how much of the latent variance is shared vs. form-specific.

Retention. For each form, we compute objective content targets from the helper image (e.g., depth layout, edge distribution, box location, mask area, scene-graph size). We summarize each helper image as a few numbers, train a linear regressor to predict these numbers from the latents, and compare how well it works on the final model’s latents against the teacher’s latents, which were produced with the helper visible. Linear probes (cross-validated R²) read these targets from the Stage-1 teacher latents, which see the helper, and from the final closed-book latents. Retention = final R² / teacher R².

FormTeacher R²Final latent R²RetentionBefore training (same positions)
Interleaved helper0.750.75100%82%
BBox crop0.470.4698%2%
Segmentation0.420.3890%2%
BBox highlight0.480.3573%13%
Depth0.540.3769%25%
Edge0.410.2561%24%
Text CoT0.760.4154%21%
Scene graph0.590.3153%0%

The final latents retain 53–100% of the teacher-readable helper content for all eight forms, far above the pre-training level for every form except the interleaved helper, whose intermediate images are close to the input image. Each final latent also retrieves its own teacher latent among ~850 same-form samples with 13–62% top-1 accuracy, against 1.5–10% before training and ~0.1% at chance.

Shared vs. form-specific. This test asks whether the eight forms share one latent space or occupy separate regions of it. We decompose latent variance into a form-specific part (between-form differences in mean position) and a shared remainder. To avoid confounding form with image source, we compare forms only within the same data source. The table reports the form-specific share; the shared share is the remainder.

Same-source groupTeacherFinal latentInput onlyBefore training
BBox crop vs. highlight (similar questions)12.3%11.8%0.9%0.1%
Depth / edge / segmentation (same images)47.8%31.9%3.7%0.6%
Depth / edge / scene graph (same images)55.6%45.7%42.7%9.8%
Text CoT / interleaved51.9%37.4%27.6%5.9%
Reviewer_bi41OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

To solve the problem of different visual feature reasoning increasingly relying on heterogeneous intermediate representations, this paper uses the Heterogeneous Visual Latent Reasoning to unify eight distinct resasongin forms within a common autoregressive latent interface. The authors get three findings. First, heterogeneous reasoning forms can be effectively unified. Second, the reasoning diversity is goog for the shared latent space consistently. Third, latent reasoning could reduce the intermediate computation.

Critical strengths
  1. The motivation of this paper is interesting. The authors want to investigate an approach that is widely accepted within the field. This unified feature representation method could serves as a unified framework, which can advance the representation of visual features derived from different domains.

2.The describtion of paper is clear to understand.

Critical weaknesses
  1. This paper never shows up where the gain comes from. The 5.31 point improvement is measured against the untuned Qwen2.5-VL-7B, while HVLR is fine uned on 40K curated examples. This is so weird and several experiment comparisons are not suffient and solid to support the paper's novelty.

  2. The paper lacks a control experiment involving SFT without latent variables, conducted using the same dataset. I think this is very important for your experiment comparision.

  3. The diversity result is confounded with data diversity. Each reasoning form brings its own sources and question types (Table 1)

Questions for authors
  1. If you run plain SFT on the K-form mixtures, do you still get the trend in Section 4.1?

  2. How did you check the overlap with the test sets? For example, did you check whether any Visual Genome images also show up in POPE. This makes the results very hard to believe.

  3. Some of the baselines did a lot worse than the Qwen2.5-VL-7B. for example, monet gets 41.56 on BLINK and LVR gets 24.75 on QSpatial++. That’s hard to believe. Did you run each model with its own prompt and answer format?

7 comments

We thank the reviewer for recognizing the motivation of our work and for carefully examining the experimental comparisons. We appreciate the questions about performance attribution, data overlap, and evaluation protocols. Below, we present the additional controls and evaluation checks that address these concerns.

W1 & W2

This paper never shows up where the gain comes from. The 5.31 point improvement is measured against the untuned Qwen2.5-VL-7B, while HVLR is finetuned on 40K curated examples. This is so weird and several experiment comparisons are not suffient and solid to support the paper's novelty.

The paper lacks a control experiment involving SFT without latent variables, conducted using the same dataset. I think this is very important for your experiment comparision.

If you run plain SFT on the K-form mixtures, do you still get the trend in Section 4.1?

Response

(A) Does the diversity trend survive under plain SFT?

We trained plain SFT on exactly the same 25 mixtures as in Sec. 4.1, including the 8-form mixture, under the same 40K budget. Plain SFT here means image + question → answer, with cross-entropy on the answer tokens only, no helper images and no latent slots.

Plain SFTHVLR (latent)
K=1 tier mean (std)66.63 (1.18)66.65 (1.09)
K=3 tier mean (std)66.83 (0.45)67.67 (0.45)
K=5 tier mean (std)66.95 (0.20)68.61 (0.42)
K=868.7969.97
OLS slope per form+0.166 ± 0.086 (p = 0.066)+0.483 ± 0.079 (p = 2.9×10⁻⁶)
R²14.0%62.1%
Spearman ρ0.17 (p = 0.41)0.88 (p = 7.4×10⁻⁹)
Jonckheere–Terpstra concordant pairs126 / 216 (chance: 108)205 / 216
Mann–Whitney, K=5 > K=336 / 64 pairs64 / 64 pairs
Benchmarks with positive slope8 / 99 / 9

The direction of the trend is preserved under plain SFT. The tier means rise monotonically from K=1 to K=8 (66.63 → 66.83 → 66.95 → 68.79), and the 8-form mixture is the best of all 25 SFT models. The within-tier spread shrinks as K grows (std 1.18 → 0.45 → 0.20), as it does for HVLR.

However, the trend is much weaker and not statistically significant. The slope is about one third of HVLR's (+0.166 vs. +0.483 per form, p = 0.066), and the rank correlation is close to zero (Spearman ρ = 0.17, p = 0.41). The ordered-group tests are near chance: 126 of 216 Jonckheere–Terpstra pairs are concordant (108 expected by chance), and K=5 beats K=3 in only 36 of 64 pairs. Diverse data alone therefore gives a small benefit, but the consistent and statistically robust scaling reported in Sec. 4.1 requires the shared latent interface. The Sec. 4.1 trend is not merely an artifact of mixing more data sources.

(B) Plain SFT vs. latent reasoning on identical data

KPlain SFTHVLRΔ (HVLR − SFT)HVLR wins (same mixture)
166.6366.65+0.025 / 8
366.8367.67+0.837 / 8
566.9568.61+1.668 / 8
868.7969.97+1.181 / 1
All 25 mixtures+0.8521 / 25 (Wilcoxon p = 1.9×10⁻⁴)

On identical data and mixtures, HVLR outperforms plain SFT on 21 of 25 paired mixtures, with a mean gain of +0.85% (Wilcoxon signed-rank test, p = 1.9×10⁻⁴). On the full 8-form mixture, HVLR reaches 69.97 against 68.79 for plain SFT (+1.18).

The advantage of the latent interface grows with reasoning diversity. With a single form the two are essentially tied (+0.02; HVLR wins 5 of 8). The gap widens to +0.83 at K=3 and to +1.66 at K=5, where HVLR wins all 8 mixtures. Overall it grows by +0.32% per added form. Every HVLR model at K=5 (≥ 68.24) outperforms every plain-SFT model at K ≤ 5 (≤ 67.51).

W3

The diversity result is confounded with data diversity. Each reasoning form brings its own sources and question types (Table 1)

Response

We ran a new controlled experiment that removes the source confound, and the diversity trend still holds.

Existing evidence. In our original setup, the number of image sources grows much more slowly than K: Visual CoT underlies five of the eight forms and Visual Genome three (Table 1), so many mixtures add forms without adding new sources. The nested F-test in Sec. 4.1 also finds no detectable effect of which forms are included beyond how many (F(7,16)=0.85, p=0.57). Neither result fully removes the confound, so we ran the experiment below.

New controlled experiment. We selected 10K images from Visual Genome. For every image, we built all four of bbox crop, text CoT, segmentation and depth: each form has a question that depends on it and a newly rendered helper of that form. Every K condition therefore draws from the same image pool, with the same training budget (10K examples per condition) and schedule. Only the number of reasoning forms changes. Since all four forms exist for every image, K ranges from 1 to 4. We report the tier mean over all C(4,K) form subsets.

K (same images)Avg. (HVLR)Δ vs. K=1
165.25–
266.18+0.93
367.33+2.08
468.41+3.16

Analysis.

  1. The trend is monotonic with fixed sources. Each added form helps (+0.93, +1.15, +1.08), and the trend is close to linear, at about +1.06% per form. Source diversity is held constant here, so it cannot explain these gains.
  2. Fixing the sources does not weaken the effect. If the original trend came mainly from source or task diversity, holding the images fixed should make it smaller. It does not. In Sec. 4.1, going from K=1 to K=8 gives +3.32 (66.65 → 69.97), with an OLS slope of +0.48 per form. Here, going from K=1 to K=4 on a single image pool gives a similar total gain (+3.16), about +1.06 per form. The two setups differ in data scale, so we compare the direction and rough size of the trend, not exact values.
  3. The gain comes from more forms on the same images. With one form, the model is only 0.59 above the base Qwen2.5-VL-7B (64.66). With all four forms on the same 10K images, it is 3.75 above. The images and questions come from one pool, so the extra benefit comes from supervising the same content through more reasoning forms.

Scope. Because of the rebuttal timeline, this study uses 10K images, four forms and one source. In the revision we will extend it to more forms and a second image pool (e.g., Visual CoT images), and add it to Sec. 4.1.

Q2

How did you check the overlap with the test sets? For example, did you check whether any Visual Genome images also show up in POPE. This makes the results very hard to believe.

Response

We examined potential overlap between our training data and the evaluation test sets at three levels.

  1. Source level. For each of the nine benchmarks, we traced the image sources stated in its paper. All our training data come from the public training splits of the source datasets (Table 1), and the benchmarks are built from the val/test splits of their sources. Evaluation-overlapping samples are removed during curation (Sec. 2.1, Appendix A.1).
  2. Question level. We compared every training question with every benchmark question verbatim (after normalization) and found 0 duplicates.
  3. Image level. We compared all 737,132,130 image pairs exhaustively (46,065 training images × 16,002 benchmark image files). Benchmarks usually re-encode or resize images, so we used four detectors of increasing robustness:
MethodWhat it catchesUse / result
File MD5Byte-identical copies0 pairs (expected, since benchmarks re-encode images)
Perceptual hash (pHash, dHash)The same image after resizing or recompressionPair flagged when pHash ≤ 6 and dHash ≤ 8
SSCD (Meta copy-detection model)The same image after cropping, color changes or overlaid boxesScore > 0.5 flagged as a suspected copy
CLIP similarityImages with similar contentUsed only to widen the candidate pool

Every pair flagged by both hashes or by SSCD was inspected with CLIP and manually: 196 candidate pairs, none of which was a true copy. In addition, Visual Genome provides the original coco_id for every COCO-derived image; matching these IDs against the POPE image list gives no overlap. We will add this audit to the appendix and release the script.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal

We thank Reviewers bi41, a7Pm and PMSg for their careful reading and constructive questions. We thank the Area Chair for handling our submission. The reviewers' concerns primarily involve the following points.

(i) Novelty relative to Render-of-Thought (PMSg) and to multi-task distillation (a7Pm)

(a) RoT. HVLR differs from RoT in its goal, method and outcome. RoT renders a single form, textual CoT, into images in order to compress text reasoning. In HVLR, rendering is a unification device: 7/8 of our forms are natively visual or structural and have no textual chain to compress. RoT regresses latents directly onto the encoder features through a projection head. Our Table 5 shows that direct alignment to encoded helper features is suboptimal in the heterogeneous setting: the "w/o distillation" variant is 1.95% worse. RoT decreases 14–27% on the four-benchmark average across three backbones. HVLR shows the opposite pattern: forced-latent beats forced-text by 1.16 and 1.24 points, and no explicit-text variant reaches the all-latent model (Sec. 4.2, Table 11).

(b) Multi-task distillation. Conventional multi-task distillation typically transfers knowledge from different tasks into a shared representation with multiple output heads, one per task. Different tasks usually have competing objectives, leading to negative transfer or gradient interference.

In contrast, HVLR distills different latent reasoning scaffolds that are coordinated toward the same underlying task and objective. HVLR uses a single set of latent slots, a unified target type, and no form-specific output heads. At inference, the model receives no form-identity information. On 9,600 held-out samples, the final latents retain 53–100% of the helper content that is readable from the teacher, for all eight forms. In every same-source group, the form-specific share is lower in the final latents than in the teacher (by up to 15.9%), so 54–88% of final-latent variance is shared across forms.

(ii) Is the diversity trend of Sec. 4.1 confounded with data sources or the training schedule? (bi41, a7Pm, PMSg)

We ran additional experiment that removes the source confound, and the diversity trend still holds. We selected 10K images from Visual Genome and for every image, we built all four of bbox crop, text CoT, segmentation and depth with only the number of reasoning forms changes. Accuracy still rises monotonically with fixed sources (65.25, 66.18, 67.33, 68.41 for K=1 to 4). Under plain SFT, the trend is much weaker and not significant (+0.166 vs. +0.483 per form, Spearman ρ\rho = 0.17 vs. 0.88). The trend in our Section 4.1 is therefore not explained by mixing more data sources alone. We have also trained K=8 with exactly the K≤5 schedule. The schedule accounts for only 0.14% of the K=8 result (69.83 vs. 69.97). Under the matched schedule, K=8 is still above all 24 K≤5 models, including the best K=5 mixture (69.55).

(iii) Where does the gain come from, and is the baseline comparison fair? (bi41, a7Pm, PMSg)

On identical data and mixtures, HVLR outperforms plain SFT on 21 of 25 paired mixtures, with a mean gain of +0.85% (Wilcoxon signed-rank test, p = 1.9×10⁻⁴). On the full 8-form mixture, HVLR reaches 69.97 against 68.79 for plain SFT (+1.18). The advantage of the latent interface grows with reasoning diversity (+0.02 at K=1 to +1.66 at K=5). Our paper already includes a data-controlled re-implementation of LVR (Sec. 4.4, Table 4). HVLR beats the LVR recipe by +2.14 while using 12.5× less data from the same corpus. SkiLa's recipe cannot be applied to our corpus without being redesigned.

(iv) Test-set overlap and low baseline scores (bi41)

We compared every training question with every benchmark question verbatim (after normalization) and found 0 duplicates. We compared all 737,132,130 image pairs exhaustively: 196 candidate pairs, none of which was a true copy. All models are evaluated through a single VLMEvalKit pipeline, with system prompt, reasoning template and answer format copied from each baseline's official evaluation script. Monet-7B on BLINK gives a mean of 43.48 ± 1.83 over four runs. When we train the LVR recipe on our data (Table 4), QSpatial++ rises from 24.75 to 47.52.

(v) Training cost and generalization beyond the Qwen family (PMSg, a7Pm)

Across four full runs on a single node with four H100 GPUs, training takes 38.7 ± 2.1 H100 GPU-hours, including offline teacher-target extraction. HVLR improves GLM-4.1V-9B on all four benchmarks, with an average gain of 6.70%, demonstrating that the method generalizes beyond the Qwen model family.

We will include these results in the revised paper. We will add the RoT comparison to the Introduction and Related Work, and the matched K=8 run and the source-controlled study to Sec. 4.1.

As the discussion period has only a few days remaining, we would really appreciate it if you could help encourage further discussion around our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.