acceptodds
Under review as a conference paper at ICLR 2027

Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning

Abstract

Multimodal models increasingly think with different modalities such as images, 3D point clouds, and robot states, not just text. Yet each modality is still encoded into its own representation space, so every step that switches modality must do two things: reason one step further, and translate between the formats of the two modalities. We call this cost the modality-switching gap. In this paper, we introduce Uni-LaDiR (Unified Latent Diffusion Reasoner), a framework that unifies different modalities into a shared latent space for multimodal reasoning. A unified encoder maps teacher reasoning steps from different modalities into latent thought tokens in a shared space, trained to preserve the information needed for later reasoning steps and the final output. A diffusion reasoner, trained jointly with the encoder, generates these tokens at inference without teacher observations. Across eleven vision-language model (VLM) benchmarks and two vision-language-action (VLA) suites, Uni-LaDiR achieves relative gains over the strongest baselines of 7.3% on four mathematical and logical VLM benchmarks and 6.1% on RLBench manipulation tasks. Controlled comparisons show increasing gains as more teacher modalities are unified. These results suggest that unification improves multimodal reasoning by weaving it into a single thread, where the model advances from thought to thought without translating between formats.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal reasoning, Vision Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bnqcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Uni-LaDiR with 3 pieces: unified encoder built from a single LLM with latent encoding, continuation prediction where tokens are trained to predict later steps and final output, and diffusion reasoner, a flow-matching head generating thought blocks at inference time. Evaluation across VLM reasoning and robotics simulation benchmark demonstrate the effectiveness of proposed method.

Critical strengths
  1. A general framework across domains and modality:

Uni-LaDiR maps reasoning steps of any modality into a shared latent space learned through continuation prediction, so one formulation covers both VLM reasoning and VLA manipulation. The detailed ablations support the design.

  1. Broad evaluation with evidence that the latents are used:

Gains hold across 11 VLM benchmarks, two VLA suites, and four backbones. The latent-intervention study shows large drops when the chain is zeroed or shuffled, confirming the model does not bypass its latents.

Critical weaknesses
  1. Missing controlled baseline and questionable comparisons:

Table 1 has no Qwen2.5-VL baseline fine-tuned on the same Zebra-CoT data without latents. Every baseline uses its own data and recipe, so the gains may largely come from training data close to the evaluation tasks. Some results also need verification. Several baseline scores fall below chance, e.g. Monet at 10.6 and Vision-R1 at 15.2 on VisuLogic. Zebra-cot has subset very similiar to questions VisuLogic while other visual latent baselines may not access during training. I believe a detailed clarification on data usage or comparison under same data budget would strengthen the paper.

  1. Weak support for the "more modalities, larger gains" claim:

On the VLM side only two modalities (text and image) are unified, so Fig. 3a gives a single comparison, not a trend. On RLBench, the two-modality variants reach 83.5–84.5 and all three reach 87.0 (Fig. 3b). These 1.5–3 point differences fall within the seed std of 3.0–4.0 reported in Table 2, and LIBERO is not included. The abstract's claim of "increasing gains as more teacher modalities are unified" is therefore not well supported.

Questions for authors

See weakness above.

I have some additional questions for authors:

Prior work on visual latent reasoning has shown that latent visual tokens can collapse and be largely ignored by the subsequent reasoning chain [1]. Could the authors provide an explanation or empirical evidence on why the unified formulation avoids this failure mode?

[1] Imagination Helps Visual Reasoning, But Not Yet in Latent Space, ICML 26.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bnqc (Part 2)

W2: "More modalities, larger gains"

Thank you for this careful analysis. Since Zebra-CoT provides only text and image, the multi-step trend can only be tested on VLA benchmarks. We add the same grouping experiment on LIBERO and test every step with 8 seeds and paired significance tests (mean ± SD):

Unified modalitiesSeparateVisual + 3DVisual + State3D + StateAll three
RLBench81.5±1.883.3±1.984.0±1.884.5±1.986.8±1.5
LIBERO (mean)95.5±0.496.8±0.498.2±0.397.1±0.499.0±0.2
LIBERO-Long88.4±1.291.2±1.196.0±0.792.0±1.097.4±0.6

The SD in Table 2 measures how one method varies across seeds, not the noise of a difference between variants. Since all variants share seeds and initial scenes, we use a paired t-test over the 8 seeds:

Paired comparisonRLBenchLIBEROLIBERO-Long
Worst pair vs. separate+1.8 (p = 0.012)+1.3 (p < 0.001)+2.8 (p < 0.001)
All three vs. best pair+2.3 (p = 0.002)+0.8 (p < 0.001)+1.4 (p < 0.001)

These results show that, on both benchmarks, every pair beats separate encoding and all three beat every pair, with every step statistically significant (p < 0.05). We will add these results to Fig. 3 and Section 4.3.

Q: Latent collapse

Thank you for this insightful question. With separate encoding, each modality's latents live in their own space, so the model can lower its loss by relying on the representation it is most familiar with and skipping the rest. In a unified space, all latents are equally familiar, so this shortcut disappears.

To test this, we also apply the interventions of Table 7 to the separate variant:

Latent chainV* (separate)V* (unified)RLBench (separate)RLBench (unified)
Intact62.889.582.087.0
Zeroed59.0 (−3.8)55.8 (−33.7)74.5 (−7.5)55.0 (−32.0)
Shuffled61.3 (−1.5)65.8 (−23.7)79.0 (−3.0)65.0 (−22.0)
Cross-example57.4 (−5.4)50.8 (−38.7)72.5 (−9.5)51.0 (−36.0)

These results show that the separate variant's latents are largely skipped (it barely changes when they are removed), whereas Uni-LaDiR drops by 22–39 points, so its latents are used.

Thank you for pointing us to this work. The cited ICML 2026 paper's proceedings (PMLR vol. 306) were published on September 29, 2026, after the ICLR full-paper deadline; we will cite and discuss it in the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer bnqc (Part 1)

Thank you for the constructive review, and for recognizing the general cross-domain formulation, the breadth of evaluation, and the latent-intervention evidence.

W1(a): Same-data comparison

Thank you for this valuable suggestion. We add two kinds of same-data baselines on Qwen2.5-VL-7B and Zebra-CoT (mean ± SD over 3 seeds): (i) SFT without latents (direct-answer and text-CoT), and (ii) every Table 1 baseline with public code, retrained on Zebra-CoT (§), since several (e.g., Vision-R1, Monet) were originally trained on other data.

(a) Vision-centricVisPuz.ChartQAV*BLINK-JMMVPSATCV-BAvg.Table 1 Avg.
Qwen2.5-VL (base)34.878.176.459.377.351.375.264.664.6
Direct-answer SFT38.6±1.178.8±0.477.5±1.061.3±1.176.0±0.855.3±1.277.3±0.566.4–
Text-CoT SFT41.2±1.279.6±0.479.1±1.163.3±1.277.7±0.857.3±1.178.5±0.568.1–
Monet§37.4±1.179.8±0.582.9±1.265.3±1.261.0±1.057.3±1.376.9±0.665.862.7
LVR§44.8±1.280.3±0.480.5±1.151.3±1.270.3±0.958.7±1.282.4±0.566.968.1
CoVT§42.6±1.280.3±0.479.4±1.167.3±1.163.0±1.062.7±1.181.4±0.568.166.7
SLVR§36.9±1.178.1±0.482.9±1.169.3±1.175.0±0.964.0±1.282.4±0.569.868.7
Mirage§43.2±1.179.7±0.482.2±1.066.0±1.172.3±0.970.0±1.182.2±0.570.872.2
PixelReasoner§45.3±1.182.0±0.482.9±1.071.3±1.277.0±0.861.3±1.283.5±0.571.970.7
Mull-Tokens§47.1±1.180.6±0.483.2±1.073.3±1.275.3±0.875.3±1.183.9±0.574.174.9
DeepEyes§48.9±1.083.1±0.386.0±0.972.7±1.279.0±0.865.3±1.185.1±0.474.373.4
Uni-LaDiR57.4±1.084.2±0.389.5±0.975.3±1.182.0±0.772.0±1.288.8±0.478.578.5
(b) Math and logicMathVistaMathVisionVisuLogicEMMAAvg.Table 1 Avg.
Qwen2.5-VL (base)68.225.426.026.036.436.4
Direct-answer SFT67.5±2.227.8±0.925.4±1.227.3±2.237.0–
Text-CoT SFT69.4±2.131.6±1.027.2±1.128.6±2.139.2–
Monet§63.7±2.133.1±1.127.2±1.122.4±2.236.632.3
PixelReasoner§68.1±2.132.0±1.025.2±1.221.1±2.336.637.5
DeepEyes§70.5±2.128.9±0.927.6±1.130.2±2.239.338.4
Vision-R1§72.6±2.240.1±1.027.8±1.225.1±2.141.438.5
Mirage§69.5±2.141.2±0.927.4±1.128.3±2.141.640.7
ILVR§70.2±2.047.6±0.828.4±1.031.4±2.244.445.7
Uni-LaDiR73.2±2.054.3±0.833.0±1.035.6±2.249.049.0

These results show that:

  • Same-data SFT gives only modest gains (+0.6 to +3.5 on average), and barely changes VisuLogic (25.4/27.2 vs. 26.0), so the similarity between Zebra-CoT and VisuLogic does not explain our 33.0.
  • Retrained baselines stay close to their Table 1 results, and Uni-LaDiR remains best on both groups (+4.2 over DeepEyes§; +4.6 over ILVR§).

W1(b): Below-chance baseline scores

Thank you for checking these numbers carefully. We'd like to clarify that the floor is 0, not 25%. Under our shared protocol (Appendix B.1), models generate free-form text rather than selecting an option, and outputs without a valid option (e.g., truncated or off-format) are scored as incorrect. Retrained on Zebra-CoT, Vision-R1§ and Monet§ reach 27.8 and 27.2 on VisuLogic, and the conclusions are unchanged.

Dear Reviewers,

We would like to kindly follow up on our previous message. We would greatly appreciate it if you could take a moment to review our responses and share any further feedback or remaining concerns.

Thank you for your time and consideration!

Reviewer_vU3eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Uni-LaDiR, which maps intermediate teacher steps from different modalities into shared latent thought tokens. The encoder is trained to preserve information for predicting later steps and the final output. A flow-matching reasoner shares backbone weights with the encoder and generates these tokens at inference, without teacher traces. The method is evaluated on eleven VLM benchmarks and two manipulation suites. Average accuracy on four mathematical and logical benchmarks improves from 45.7% with ILVR to 49.0%. RLBench success improves from 82.0% with LaST0 to 87.0%.

Critical strengths
  1. The representation learning objective is sensible. Continuation prediction encourages the latent tokens to retain information needed for subsequent steps, rather than reconstruct source details. The comparison with reconstruction in Figure 5 supports this choice on average.

  2. The evaluation is comprehensive and includes useful ablations. Results cover visual reasoning, robotic manipulation, and several VLM backbones. The controlled experiments keep training data and teacher supervision fixed while examining encoder sharing, joint training, and latent generation objectives.

  3. The paper tests whether the model really uses its latent chain. Zeroing, shuffling, and cross-example replacement substantially reduce performance in Table 7. These results support the use of input-specific and order-sensitive latent information, rather than latent tokens being ignored.

Critical weaknesses
  1. The explanation based on a shared latent space remains incomplete. Comparing shared and separate encoders also changes parameter sharing and cross-modal training. The experiments do not isolate these effects from representation alignment. Moreover, unified encoding reduces MathVista from 75.5% to 73.2% and EMMA from 36.2% to 35.6% under continuation supervision in Figure 5. These exceptions deserve discussion.

  2. The diffusion comparison lacks a matched-compute control. Flow matching uses iterative generation, while the L2 and cosine baselines directly predict latent blocks. The paper does not specify the latent integration solver or step count. The ten steps reported for the action head refer to a different component. It is therefore unclear how much of the gain comes from the prediction objective versus additional computation.

Questions for authors
  1. Can you provide a control with comparable parameter sharing but modality-specific latent interfaces? What explains the MathVista and EMMA regressions under unified encoding, and are they consistent across seeds?

  2. Which solver and how many network evaluations are used per latent block? Please report average chain length and inference latency, and compare flow matching with direct regression under similar inference budgets.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vU3e

Thank you for the constructive review, and for recognizing the continuation-prediction objective, the controlled ablations that fix data and teacher supervision, and the latent-intervention test.

W1/Q1: Shared latent space vs. parameter sharing

(a) Control with matched parameter sharing. Thank you for this thoughtful question. We'd like to clarify that our separate vs. unified comparison (Fig. 5) is exactly this control. Both variants use the same LLM backbone as the encoder, so parameter sharing and cross-modal training are matched; the separate variant differs only in its modality-specific latent interfaces (queries and output projections). We rerun both with 8 seeds (mean ± SD):

VariantV*MMVPMathVistaEMMALIBERORLBenchAvg.
Separate, recon.66.3±0.970.2±0.764.2±2.030.8±2.188.1±0.573.4±1.865.5
Unified, recon.71.9±0.873.8±0.667.2±1.933.1±2.090.1±0.477.3±1.768.9
Separate, cont.63.4±1.077.0±0.775.0±2.135.4±2.295.5±0.481.5±1.871.3
Unified, cont.89.3±0.682.0±0.573.4±1.936.9±2.099.0±0.286.8±1.577.9

These results show that, with matched parameter sharing, the shared latent interface improves the average by +3.4 (reconstruction) and +6.6 (continuation).

(b) MathVista and EMMA. Thank you for pointing out these exceptions. We'd like to note that these two benchmarks have the largest variance. With 8 seeds, the unified–separate difference is within one SD on both and flips sign (MathVista 73.4±1.9 vs. 75.0±2.1; EMMA 36.9±2.0 vs. 35.4±2.2), so the earlier −2.3/−0.6 is not consistent across seeds. Unification helps most on tasks that switch modalities often (e.g., V*) and least on text-heavy MathVista and EMMA. We will update Fig. 5 and Section 4.4 accordingly.

W2/Q2: Solver, inference budget, and matched-compute comparison

Thank you for this careful observation. We'd like to clarify that the latent sampler uses Euler integration with 10 steps, i.e., 10 NFE per latent block; the ten steps in Table 5 refer to the action head. For a matched budget, we add 1-step flow matching, which uses the same single NFE per block as regression (accuracy: mean ± SD over 3 seeds; latency on one RTX 4090):

ObjectiveNFE / blockV*MMVPMathVistaEMMALIBERORLBenchAvg.Chain lengthV* latencyRLBench latency
Cosine regression158.6±1.472.3±1.169.8±2.127.4±2.385.9±0.971.5±2.864.34.90.57 s42.9 ms
L2 regression161.3±1.374.0±1.071.2±2.029.3±2.288.7±0.876.7±2.766.95.00.57 s42.9 ms
Flow matching (new)172.3±1.177.0±0.971.9±2.131.2±2.192.6±0.781.0±2.571.05.40.58 s43.2 ms
Flow matching (default)1089.5±0.982.0±0.773.2±2.035.6±2.299.2±0.387.0±3.077.85.21.42 s60.6 ms

These results show that the gain comes from the objective rather than extra compute: at the same NFE and latency, 1-step flow matching already outperforms L2 regression by +4.1 points. Using 10 steps adds a further 6.8 points at higher latency (1.42 vs. 0.58 s on V*; 60.6 vs. 43.2 ms on RLBench, still 16.5 Hz). We will add the solver, NFE, chain length, and latency to Table 5.

Thank you for your detailed feedback!

I apologize for my misunderstanding of the setting in Figure 5. The results under different random seeds better support the effectiveness of the shared latent interface.

The results under a matched computational budget also address my second concern.

I decide to maintain my ratings.

Thank you very much for taking the time to carefully reconsider our results. We sincerely appreciate your constructive feedback and are glad that our clarifications have addressed your concerns.

Given that your main concerns have been resolved, we would be very grateful if you could consider a slightly higher rating to reflect your updated assessment. We believe this would be a meaningful encouragement for our work and its potential contributions to the research community.

Thank you again for your time, consideration, and valuable feedback!

Reviewer_W8jTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Uni-LaDiR (Unified Latent Diffusion Reasoner) to unify different modalities into a shared latent space for multimodal reasoning. Uni-LaDiR uses a shared encoder to map reasoning steps from different modalities into a common latent space. Such latent thought tokens are trained to predict subsequent reasoning steps and the final output to maintain information needed for future reasoning. Moreover, a diffusion reasoner is utilized to generate latent thought tokens at inference without teacher reasoning steps. Extensive experiments have been done across various multimodal benchmarks (VLM reasoning and VLA manipulation) and Uni-LaDiR outperforms other baselines on average.

Critical strengths
  1. Good motivation: It is an interesting approach to map different modalities into a shared latent space for multimodal reasoning. And it is also a general framework that applies to both VLM and VLA.

  2. Thorough experiments: Extensive experiments (including various ablation studies) have been done to evaluate the core idea. And Uni-LaDiR outperforms other baselines on average across multiple benchmarks.

Critical weaknesses
  1. In figure 5, it seems that the benefits from unified encoding vary a lot across different datasets as compared to the separate encoding. For example, for V*, the improvement is huge (62.8 vs. 89.5) but on MathVista and EMMA the unified encoding does not improve the performance as compared to separate encoding (75.5/36.2 vs. 73.2/35.6). This makes it difficult to establish whether unification consistently improves multimodal reasoning or whether the benefit is dataset dependent.
Questions for authors
  1. To separate the gain from the proposed method and the training data, it would be helpful to also include the performance of the base model (Qwen2.5-VL) trained on the same dataset (Zebra-CoT) with direct answer or CoT supervision.

  2. In fig 5, the benefit from unified encoding varies a lot and on MathVista or EMMA, unified encoding fails to improve the performance. Can the authors provide any insights on these differences, particularly the low V* baseline? Also it would be helpful to report std across multiple runs for the fig 3&5 comparisons.

  3. It would be helpful to see the performance with z* (teacher thought tokens) and compare it to diffusion-generated z_hat. This gap would show whether representation or generation is the bottleneck.

4 comments
AuthorsOpenReview ↗

Response to Reviewer W8jT

Thank you for the constructive review, and for recognizing the motivation for a shared latent space, its generality across VLMs and VLAs, and the breadth of our ablations.

W1/Q2: Dataset dependence in Fig. 5

(a) MathVista and EMMA. Thank you for this careful observation. We'd like to note that these two benchmarks have the largest variance. We rerun all Fig. 3 and Fig. 5 variants with 8 seeds (full results in (c)). Under continuation supervision:

Mean ± SD (8 seeds)MathVistaEMMA
Separate75.0±2.135.4±2.2
Unified73.4±1.936.9±2.0

These results show that the unified–separate difference is within one SD on both benchmarks and flips sign (−1.6 / +1.5), so the earlier −2.3/−0.6 is run-to-run variance, not a loss from unification.

(b) Why the separate V* baseline is low. We appreciate this insightful question. V* requires locating an object visually and then reading it out in text, so almost every step crosses modalities. Separate latent spaces make these switches error-prone, dropping V* below the base model; the unified space removes this gap. The gain of unification thus grows with how often a task switches modalities: largest on V*, smallest on text-heavy MathVista and EMMA. We will add this discussion to Section 4.4.

(c) Full 8-seed results for Figs. 3 and 5. We appreciate the reviewer requesting these error bars (mean ± SD):

Fig. 5V*MMVPMathVistaEMMALIBERORLBenchAvg.
Frozen targets61.4±1.066.8±0.860.1±2.124.6±2.286.5±0.571.4±1.961.8
Separate, recon.66.3±0.970.2±0.764.2±2.030.8±2.188.1±0.573.4±1.865.5
Unified, recon.71.9±0.873.8±0.667.2±1.933.1±2.090.1±0.477.3±1.768.9
Separate, cont.63.4±1.077.0±0.775.0±2.135.4±2.295.5±0.481.5±1.871.3
Unified, cont.89.3±0.682.0±0.573.4±1.936.9±2.099.0±0.286.8±1.577.9
Fig. 3(a) VLM mean(b) RLBench
VariantSeparateText + ImageSeparateVisual + 3DVisual + State3D + StateAll three
Score62.7±0.670.4±0.481.5±1.883.3±1.984.0±1.684.5±1.386.8±1.2

These results show that all conclusions hold: unification improves the average under both reconstruction (+3.4) and continuation (+6.6), and gains grow as more modalities are unified. We will replace Figs. 3 and 5 with these results.

Q1: Same-data baseline

Thank you for this helpful suggestion. We fine-tune Qwen2.5-VL-7B on the same Zebra-CoT data with direct-answer and text-CoT supervision (same schedule as Appendix B.2; mean ± SD over 3 seeds, base model evaluated once):

(a) Vision-centricVisPuz.ChartQAV*BLINK-JMMVPSATCV-BAvg.
Qwen2.5-VL (base)34.878.176.459.377.351.375.264.6
Direct-answer SFT38.6±1.178.8±0.477.5±1.061.3±1.376.0±0.855.3±1.477.3±0.566.4
Text-CoT SFT41.2±1.279.6±0.479.1±1.163.3±1.277.7±0.857.3±1.378.5±0.568.1
Uni-LaDiR57.4±1.084.2±0.389.5±0.975.3±1.182.0±0.772.0±1.288.8±0.478.5
(b) Math and logicMathVistaMathVisionVisuLogicEMMAAvg.
Qwen2.5-VL (base)68.225.426.026.036.4
Direct-answer SFT67.5±2.227.8±0.925.4±1.227.3±2.437.0
Text-CoT SFT69.4±2.131.6±1.027.2±1.128.6±2.339.2
Uni-LaDiR73.2±2.054.3±0.833.0±1.035.6±2.249.0

These results show that Zebra-CoT alone gives only modest gains (+0.6 to +3.5 on average), while Uni-LaDiR improves over text-CoT SFT by +10.4 and +9.8, so the gains come from shared latent reasoning rather than the data.

Q3: Teacher z* vs. generated ẑ

Thank you for this insightful suggestion. Teacher steps require ground-truth reasoning traces, so we evaluate on held-out data unseen in training: the 600-question Zebra-CoT validation set and the LIBERO and RLBench evaluation scenes, where expert demonstrations provide the teacher steps. We replace the generated ẑ with teacher thought tokens z* from the unified encoder, keeping everything else fixed (mean ± SD over 3 seeds):

Latent chainZebra-CoT (val.)LIBERORLBench
Teacher z* (oracle)77.0±0.999.8±0.193.0±1.8
Generated ẑ (Uni-LaDiR)71.4±1.299.2±0.387.0±3.0

These results show that the representation is the main bottleneck. Even with ground-truth teacher steps, the gain over generated latents (≤ 6.0 points) is much smaller than the gain from improving the representation (up to 16.1 points across Fig. 5 variants). We will add this analysis to Appendix B.4.

Dear Reviewers,

We would like to kindly follow up on our previous message. We would greatly appreciate it if you could take a moment to review our responses and share any further feedback or remaining concerns.

Thank you for your time and consideration!

Thanks for the detailed response and additional experiments. The multiple seeds & same-data baseline results help strengthen the evidence for the proposed method. Overall, I will maintain my positive assessment of the paper.

Thank you very much for your positive feedback! We are glad that our additional experiments have helped strengthen the evidence for our method and addressed your concerns.

We sincerely appreciate your recognition of our work. If you feel it'd be appropriate, we would be truly grateful if you could kindly consider a slight increase in your rating to reflect your updated assessment of our work.

Thank you again for your time and thoughtful consideration!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Part 1/2)

We thank all reviewers for their constructive feedback.

What reviewers found strong

  • Problem. Mapping different modalities into one shared latent space for multimodal reasoning is a well-motivated and interesting approach (W8jT).
  • Method. One general formulation covers both VLM reasoning and VLA manipulation (bnqc, W8jT). Continuation prediction is a sensible objective, because it keeps the information needed for later steps instead of reconstructing source details (vU3e). The detailed ablations support the design (bnqc).
  • Experiments. The evaluation is broad: 11 VLM benchmarks, two VLA suites, and four backbones (bnqc, vU3e, W8jT). The controlled ablations fix training data and teacher supervision (vU3e). Zeroing, shuffling, or swapping the latent chain causes large drops, which shows that the model uses its latents (bnqc, vU3e).

In the paper, Uni-LaDiR improves over the strongest baselines by 7.3% on four mathematical and logical VLM benchmarks and 6.1% on RLBench (relative gains).

Main concerns and how we addressed them

1. Do the gains come from the Zebra-CoT training data? (bnqc, W8jT) (W8jT: resolved)

Results.

  • New experiment: we fine-tuned Qwen2.5-VL-7B on Zebra-CoT without latents. Direct-answer and text-CoT SFT reach 66.4 and 68.1 on the vision-centric average, and 37.0 and 39.2 on math and logic. Uni-LaDiR reaches 78.5 and 49.0. These rows are now in Table 1.
  • New experiment: we retrained every Table 1 baseline with public code on Zebra-CoT. Each stays close to its Table 1 result. Uni-LaDiR remains best on both groups: +4.2 over DeepEyes (74.3) and +4.6 over ILVR (44.4).
  • Same-data SFT barely changes VisuLogic (25.4 and 27.2, vs. 26.0 for the base model). Uni-LaDiR reaches 33.0.
  • Scores below chance are expected under our protocol. Models answer in free form, and outputs without a valid option count as wrong, so the floor is 0, not chance. Retrained on Zebra-CoT, Vision-R1 and Monet reach 27.8 and 27.2 on VisuLogic.

2. Do gains really grow as more modalities are unified? (bnqc, W8jT) (W8jT: resolved)

Results.

  • New experiment: we reran every Figure 3 and Figure 5 variant with 8 seeds and added the same grouping study on LIBERO, which also has three teacher modalities (new Figure 3c).
  • On RLBench, every pair beats separate encoding (81.5 to 83.3–84.5), and all three reach 86.8. On LIBERO-Long, the same steps go from 88.4 to 97.4.
  • Because all variants share seeds and initial scenes, we used paired t-tests. Every step is significant (p < 0.05) on RLBench and LIBERO.

3. Is the benefit from the shared latent space, or from parameter sharing? Why do MathVista and EMMA not improve? (vU3e, W8jT) (vU3e and W8jT: resolved)

Method differences.

  • The separate and unified variants already use the same LLM backbone. They differ only in whether the learnable queries and output projections are shared. So parameter sharing and cross-modal training are matched. We corrected the Appendix B.3 wording that suggested otherwise.

Results.

  • New experiment (8 seeds): the shared interface improves the six-benchmark average by +3.4 (reconstruction) and +6.6 (continuation).
  • On MathVista and EMMA, the two benchmarks with the largest variance, the two encodings perform on par over 8 seeds (73.4 vs. 75.0; 36.9 vs. 35.4, both within one SD). Both reviewers found the multi-seed results convincing.
  • The gain grows with how often a task switches modalities. It is largest on V, where the model must locate an object visually and then read it out in text.
AuthorsOpenReview ↗

General Response (Part 2/2)

4. Is the diffusion gain just extra compute? (vU3e) (vU3e: resolved)

Results.

  • The latent sampler uses Euler integration with 10 steps (10 NFE) per latent block. We now report this, the chain length, and the latency in Table 5.
  • New experiment: one-step flow matching uses the same single NFE and latency as regression. It still beats L2 regression by +4.1 points (71.0 vs. 66.9), so the gain comes from the objective. Ten steps add 6.8 more points, at 60.6 ms per RLBench step (16.5 Hz).

5. Do the latent tokens collapse and get ignored, as reported by Li et al. (ICML 2026)? (bnqc)

Method differences.

  • Li et al. find that latent visual tokens in existing latent reasoning methods barely affect the final answer.
  • Uni-LaDiR learns one shared latent space through continuation prediction. With separate spaces, the model can lean on the space it knows best and skip the rest. A shared space removes this shortcut.

Results.

  • Already in the paper (now Figure 6): zeroing, shuffling, or swapping the chain lowers Uni-LaDiR by 22.0–38.7 points.
  • New experiment: the same interventions change the separate-encoding variant by only 1.5–9.5 points. Its latents are largely skipped; ours are used.

Revised PDF

New in the main text:

  • Figure 3c: LIBERO-Long with the same modality groupings (8 seeds).
  • Figures 3 and 5: re-plotted with 8-seed means and SD error bars.
  • Figure 6: latent interventions on V and RLBench, now also for the separate-encoding variant (moved from the appendix table).
  • Figure 7: generated vs. teacher thought tokens (new experiment).
  • Table 1: two new rows, Qwen2.5-VL-7B fine-tuned on the same Zebra-CoT data without latents (direct-answer and text-CoT SFT).

New in the appendix: Table 7 (matched inference budget), Table 8 (all baselines retrained on the same data), and Table 9 (full 8-seed results and paired t-tests).

All changes are coloured by reviewer:

  • bnqc (blue): LIBERO teacher modalities corrected to include point clouds (Section 4.1, Table 5, Appendix B.2); Figure 3c and significance sentence (Section 4.3); "Causal role of latent thoughts" paragraph and Figure 6 (Section 4.4); Li et al. (2026) in Section 5; scoring protocol in Appendix B.1.
  • vU3e (green): corrected description of separate encoding (Sections 4.3–4.4, Appendix B.3); sampler and matched-NFE result (Section 4.4); solver, NFE, chain length, and latency (Table 5, Appendix B.2); Table 7.
  • W8jT (orange): why the separate V score is low, and the "Teacher versus generated thought tokens" paragraph with Figure 7 (Section 4.4).
  • Several reviewers (purple): 8-seed Figures 3 and 5 and the MathVista/EMMA discussion (Sections 4.3–4.4); same-data rows in Table 1; Tables 8 and 9.

The main text stays within 10 pages. Because Figures 6–7 are new, the generation-objective figure is now Figure 8, and some appendix tables are renumbered.

Discussion status

  • Reviewer vU3e: replied that the 8-seed results "better support the effectiveness of the shared latent interface" and that the matched-compute results "address my second concern"; maintains the positive rating.
  • Reviewer W8jT: replied that "the multiple seeds & same-data baseline results help strengthen the evidence for the proposed method" and maintains a positive assessment.
  • Reviewer bnqc: no reply yet.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.