- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes DLWM, a module on top of a frozen Qwen2.5-VL-7B that keeps several latent hypotheses about an ambiguous image. An orthogonality loss keeps the hypotheses distinct, and an RL controller decides at each step whether to expand, stop, merge, or answer under a compute budget. Trained on GQA, VQAv2, and ScienceQA, it reports 2–5 point gains over latent-reasoning baselines and about 24% lower memory than LVR.
Critical strengths
- Reasonable motivation and broad ablations. Keeping several interpretations of an ambiguous image and spending compute adaptively is a sensible goal. The ablations cover the number of hypotheses, the compute budget, each controller action, the diversity loss, and an analysis split by question ambiguity.
Critical weaknesses
1. Several reported numbers are inconsistent or impossible. In the per-seed results (Table 9), three separately trained methods vary across seeds in exactly the same way on ScienceQA, MMVP, and VQAv2, so their standard deviations are identical. MMVP has 300 questions, and the paper states that evaluation is deterministic (temperature 0, greedy decoding; Appendix. B.1.2). Under that setup, every evaluation of a given trained model returns the same answers, so each baseline's score should be a multiple of 1/3 point (e.g., 66.0, 66.3, 66.7, same as the numbers reported in the original paper) in single-run or even after "averaging over repeated evaluations" (App. B.1.3). Yet several are not, the single-run baselines in Table 1a (e.g., 57.2, 59.8, 60.5) and most per-seed values in Table 9. The stated architecture (six Transformer blocks at width 4096) cannot fit in the reported parameter budget. The reported peak memory is lower than the memory needed just to hold the frozen backbone's weights. This is consequential because, until these are explained with raw logs, I cannot rely on any of the accuracy or efficiency comparisons.
2. The method is under-specified, and its claimed mechanism is not separated from simple ensembling. The paper never says how the input features are extracted from the frozen model or how an answer is produced. Because each "hypothesis" is a single vector, the "Transformer" reasoning module works like an MLP. There is no frozen-backbone baseline, and the CoT baseline looks badly under-configured (ScienceQA 68.1 vs. about 80+ typically reported for this model). There is also no ensemble or self-consistency control at matched compute. Close prior work on multi-path latent reasoning (e.g., SoftCoT++, Parallel-R1, Mirage) is not cited. This is consequential because the central claim, multi-hypothesis reasoning, cannot be evaluated when the model is undefined and the gain could simply come from ensembling small heads on frozen features.
Questions for authors
- Please provide raw per-seed logs for Table 9. Please also explain: the identical seed offsets across methods; the impossible MMVP scores; the per-module parameter counts; and whether peak memory includes the frozen backbone.
- How exactly are the input features extracted and the answers produced? How does DLWM compare with a frozen-backbone baseline and an ensemble of independent heads at matched compute?
Thank you for checking the numbers closely and for raising the ensemble comparison.
Q1. Can you provide the raw Table 9 logs and explain the seed offsets, MMVP scores, parameter counts, and peak memory?
Seeds. We cannot provide the original per-seed logs or per-question predictions with this response. We recomputed all 12 means and sample standard deviations in Table 7 from the displayed Table 9 values; they agree to the reported precision. All three methods show the same successive changes: +0.2/+0.1 points on VQAv2, +0.2/+0.2 on ScienceQA, and +0.3/+0.2 on MMVP. This arithmetic does not explain why the three methods vary in lockstep. Nor do the three seed averages verify the paired-bootstrap intervals, which depend on paired example-level scores.
MMVP. The official evaluator credits a pair only when both answers are correct. On the full set, one pair changes accuracy by 100/150 ≈ 0.667 points; even scoring the 300 questions individually gives increments of about 0.333. The single-run values 57.2, 59.8, and 60.5 in Table 1a cannot arise under either convention, rounded to one decimal. Nor can a single full run yield 65.8 or 66.1 in Table 9. We described the latter as averages of repeated evaluations, but repeating identical predictions with deterministic grading does not change the score. Without the original denominators and grader outputs, we cannot substantiate the MMVP gain under the protocol we reported.
Parameters. In the supplementary 7B configuration, the six width-4096 reasoning blocks share weights across worlds and steps, not across layers. Their attention projections alone contain 402,653,184 weights. Counting shared weights once gives:
We withdraw the reported trainable total of 126M (1.8% of the backbone) and the “parameter-matched” label for the K=1 row listed as 126M in Table 12. The 1.481B counts added weights in the released model, not trainable weights measured from the benchmark checkpoints.
Peak memory. Appendix B.1.7 describes bf16 inference on one A100 at batch size eight, using PyTorch’s maximum allocated GPU tensor memory. Resetting the peak counter does not subtract weights already on the GPU. The Qwen checkpoint index lists 16.58 GB (15.45 GiB) of weights alone. The reported 14.2–14.8 GB therefore cannot include the entire bf16 backbone resident on that GPU. We cannot verify which components were resident in the original measurements, and we do not claim the reported 24% difference as a verified end-to-end memory saving.
Q2. How are features extracted and answers produced? How does DLWM compare with frozen Qwen and a compute-matched ensemble?
In the supplementary implementation, frozen Qwen processes the image and question together. We project its final-layer state at the last prompt token from 3584 to 4096 dimensions to obtain z. A shared generator combines z with a learned embedding for each world. Expand updates one world token: its self-attention has only one position to attend to, while the value/output projections and feed-forward layers still transform its state. Attention across worlds occurs in the controller. At each answer position we project each surviving world’s difference from z back to Qwen’s width, add it to the shared decoder state, normalize, and apply the same vocabulary head. The final next-token distribution is a learned weighted average of the world distributions.
Table 6 tests fusion on the same DLWM candidates: on GQA, mean pooling gives 67.8% and learned fusion 69.3%. It does not test independent heads or self-consistency at matched compute. Our ScienceQA score of 68.1% is for generated CoT, not direct answering. Li et al. report 80.76% with direct option scoring and 75.24% with generated CoT in their evaluation. Neither is a result under our protocol. Our comparisons do not establish an advantage over direct frozen Qwen or a matched ensemble.
We will revise the Related Work section to cite SoftCoT++, Parallel-R1, and Mirage and explain how DLWM differs from each. They diversify latent paths, learn parallel text reasoning, and interleave visual latent tokens with text, respectively. Our contribution is the policy that selectively updates, prunes, or merges concurrent multimodal latent states before fusion, not the idea of multiple reasoning paths itself.