acceptodds
Withdrawn from ICLR 2027

LatentWeave: Test-Time Latent Optimization for Multimodal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in test-time optimization have shown that the multimodal reasoning performance of large multimodal language models (MLLMs) can be improved during inference without modifying backbone model parameters. However, existing approaches primarily operate at the prompt or output level, leaving the latent reasoning process largely uncontrolled and insufficient to prevent error accumulation during autoregressive generation. To address this limitation, we propose LatentWeave, a test-time framework that treats multimodal reasoning as a controllable latent trajectory. LatentWeave performs structured Latent Weaving, progressively refining intermediate hidden states segment-wise to correct emerging deviations, and further introduces Reasoning Anchors via lightweight WEAVE tokens to stabilize reasoning structure and guide answer-oriented generation. These mechanisms are designed to enable fine-grained, step-aware intervention in the latent reasoning process while keeping the main MLLM backbone parameters fixed. Extensive experiments across diverse multimodal reasoning benchmarks report improvements over the evaluated baselines; the contribution of latent optimization, anchor training, and additional test-time computation is analyzed separately in the ablation and efficiency studies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Scaling, Multimodal Reasoning, Latent Space

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xWuKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatentWeave, a frozen-backbone test-time optimization framework for multimodal reasoning. The method has two components. First, Latent Weaving divides the autoregressive reasoning trajectory into contiguous segments and optimizes the hidden states of the active segment using verifier-based policy-gradient updates, while keeping previously accepted segments fixed and regenerating the suffix. Second, Reasoning Anchors uses an auxiliary model, WEAVEGen, trained on offline optimized trajectories to produce structured textual anchors delimited by <WEAVE> tokens. These anchors are inserted into the evaluated backbone’s textual context before latent refinement. The method is evaluated on seven multimodal reasoning benchmarks, including mathematical reasoning, visual question answering, and diagram understanding, using five MLLM backbones ranging from 4B to 27B parameters. The results show consistent improvements over vanilla decoding, with gains of up to 8.73 percentage points on individual benchmarks. The paper also reports component ablations, sensitivity studies, verifier audits, and a comparison with same-verifier Best-of-N under a 96-candidate evaluation budget.

Critical strengths
  1. The segment-wise freezing and suffix regeneration strategy provides a plausible mechanism for correcting early reasoning errors while preserving previously accepted latent states. This is a meaningful design choice beyond simple answer-level self-refinement.
  2. The method is evaluated across five backbones, seven benchmarks, and several task categories. The paper also separates the effects of Latent Weaving and Reasoning Anchors and reports resource costs, which makes the empirical claims easier to assess.
Critical weaknesses
  1. What model is used to implement each verifier? Are the verifier models independent of the evaluated backbone and WEAVEGen? How do the results change when using a different verifier or human-annotated correctness signals?

  2. How many examples are used for offline trajectory collection and WEAVEGen training? Are any benchmark questions, images, or near-duplicate examples included in the offline data or used for hyperparameter selection?

  3. Could the authors clarify how gradients are computed with respect to hidden states in Equation (5), particularly after sampling and autoregressive suffix regeneration? An explicit algorithm would improve the technical clarity and reproducibility of the method.

  4. Since the verifier provides the main test-time optimization signal, how do the authors rule out optimization toward verifier-specific preferences, such as output length or formatting, rather than genuine reasoning improvements? Results with independent verifiers or human evaluation would help address this concern.

  5. The experiments focus on relatively short benchmark instances. How would LatentWeave extend to long-horizon tasks involving hundreds of interaction turns or many hours of execution? In particular, how would the authors control the growing test-time computation cost while preserving context continuity over long trajectories?

Questions for authors

See Weaknesses

Reviewer_FByoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes LatentWeave, a test-time method for multimodal reasoning with MLLMs, consisting of two components.

  • Latent Weaving splits the generated trajectory into segments and updates each segment's hidden states in turn with a policy-gradient objective, then re-decodes the suffix. The reward comes from four prompted verifier checks, plus a ground-truth term used only during offline collection.
  • Reasoning Anchors use WEAVEGen, a 2B auxiliary model trained on optimized trajectories, to generate <WEAVE>-delimited textual anchors. These are inserted into the frozen backbone's context before latent weaving. Experiments cover seven benchmarks (MathVision, MathVerse, MathVista, MMMU, RealWorldQA, MME, AI2D) and five backbones (4B–27B). They report gains over vanilla decoding and over baselines including CoT, SFT, GRPO, Self-Consistency, Self-Refine and TTRV, along with component ablations, a same-verifier Best-of-N comparison and sensitivity analyses. An anonymous code repository is provided.
Critical strengths

None of substance. There are two minor points, neither of which bears on my recommendation:

  • The paper is well written and easy to follow. Bold run-in headings, numbered contributions and consistent highlighting make the key points clear at a glance.
  • The experimental protocol, as described, is carefully designed. However, it is unclear whether the experiments were carried out as described
Critical weaknesses

W1. Apart from WEAVEGen, the method is identical to LatentSeek, which the paper cites only in passing.

  • Apart from WEAVEGen, the method does not differ substantively from LatentSeek (Li et al., 2025a). Both apply test-time policy-gradient updates to hidden states, guided by self-verification rewards. Yet LatentSeek is cited only in passing (L050, L135).
  • The released code is clearly adapted from LatentSeek's implementation, including its verifier prompts, reward model and latent update. Yet LatentSeek is not compared, and I see no justification for this omission.
  • Nothing in the method or the code depends on the visual modality, and nothing prevents LatentSeek from being applied to an MLLM as is. The same holds for Zhang et al. (2026), a frozen-backbone test-time latent optimization method for MLLMs, which is not compared either.

W2. The WEAVEGen component is unmotivated, its effectiveness is not supported, and the paper's own case study contradicts the described pipeline.

  • The paper does not explain why anchors from a separate 2B model, trained on optimized trajectories, should help a frozen backbone of up to 27B.
  • The results do not establish its effectiveness. In Table 9, adding anchors gains +0.51 on MathVista, which is below the reported standard deviation (±0.55). The latent-only row has no seed statistics, so the gains on MMMU and AI2D cannot be assessed either.
  • More seriously, the case study contradicts the described pipeline. They claim WEAVEGen only produces anchors that are inserted into the backbone's context, and the frozen backbone then performs latent weaving and produces the answer (Table 7). In the case study (Fig. 3), the final answer instead follows directly from the <WEAVE>-delimited sequence, with no backbone reasoning conditioned on it.

W3. The released code does not match the paper at all; I suspect the wrong version was uploaded.

  • The training pipeline, evaluation pipeline, loaded models and datasets in the released code all differ entirely from what the paper describes. The reported results could not have been produced with this code.
  • In particular, WEAVEGen, the only addition over LatentSeek, works totally differently in the code than in the paper, but it just matches that wrong case study discussed in W2.
  • Independently of the code, the reported numbers are inconsistent with the stated sample sizes (Appendix A.6). Under the paper's own setup (one shared split, with the stated number of seeds), a sweep over every sample count up to the stated sizes finds no denominator that explains the Table 1 entries on most benchmarks. Most plainly, MathVista has 1,000 questions, so deterministic Vanilla decoding cannot score 66.15.
Questions for authors
  1. Please compare against LatentSeek and Zhang et al. (2026) on the same backbones and benchmarks, under the same verifier-call budget. Given that the method is modality-agnostic, what does LatentWeave offer beyond LatentSeek run on an MLLM?
  2. Please release the code actually used for the experiments, covering all backbones, benchmarks and baselines in Tables 1-2, and explain the discrepancies listed in W3.
  3. For each entry in Tables 1-2, how many samples were evaluated, and how many seeds were averaged? How do the reported values follow from the sample sizes in Appendix A.6?
  4. Which model serves as the verifier, both during offline collection and at test time? Is it trained or only prompted, and how does it relate to the backbone and to WEAVEGen?
Reviewer_jvrTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LatentWeave improves multimodal generation through test-time optimization of instance-specific hidden states while keeping the backbone parameters fixed. It uses verifier feedback to refine successive segments, preserves accepted prefixes, and regenerates suffixes. An offline-trained WEAVEGen supplies textual reasoning anchors. The paper reports gains across multiple backbones and benchmarks and includes component ablations, a same-verifier Best-of-N comparison, resource measurements, and a verifier reliability audit.

Critical strengths
  1. A concrete framework for directing test-time computation with feedback. Sequential refinement of hidden-state segments, combined with explicit textual anchors, is a plausible approach to correcting generation trajectories. The paper separates offline auxiliary training from test-time hidden-state optimization and disables ground-truth reward during evaluation. Extra computation is not a drawback by itself if this procedure obtains improvements that comparably resourced alternatives cannot achieve.

  2. The component study and same-verifier control provide meaningful positive evidence. On MathVista, Best-of-N reaches 69.44%, latent-only reaches 71.52%, and the full method reaches 72.03%, with the latter three conditions sharing 96 candidate-level verifier evaluations (Table 11). The anchor ablation helps distinguish auxiliary textual guidance from latent updates. The cost measurements and verifier audit are also useful qualifications. These experiments support a benefit over vanilla decoding and independent candidate selection, although they do not fully isolate the benefit of editing hidden states.

Critical weaknesses
  1. The independent value of hidden-state optimization over iterative text refinement is not established. The same-verifier Best-of-N control selects among independent candidates; it does not use feedback to adapt subsequent revisions. Conversely, Self-Refine uses self-critique and only two refinement rounds (Table 13), unlike the feedback source and budget of the latent procedure. Thus, neither comparison isolates the editing space while controlling feedback-driven search. The gain could partly arise from iterative feedback and prefix-preserving refinement rather than a distinctive advantage of hidden-state updates. A text-level refinement or resampling baseline with the same verifier, comparable candidate-evaluation budget, segmentation, and anchor access would directly address this question. Equal verifier calls need not imply equal execution cost, but higher cost alone is not the concern: the issue is whether latent optimization achieves a better attainable result than a comparably resourced alternative.

  2. Verifier reliability under repeated optimization remains insufficiently characterized. Table 12 reports a useful audit on a disjoint gold set, so verifier reliability is not ignored. However, static audit performance does not establish that the verifier remains reliable on candidates repeatedly optimized to improve its score. It is unclear whether increasing verifier reward throughout refinement consistently tracks true answer correctness or increasingly favors confidently scored errors. This matters because the verifier actively guides the optimization, rather than only evaluating a fixed collection of outputs. I am not claiming reward exploitation occurs; the missing evidence limits confidence in the robustness of the refinement procedure.

Questions for authors
  1. [Decision-relevant evidence] How does latent refinement compare with iterative text-level revision or segment resampling using the same backbone, verifier feedback, prefix-preservation policy, and comparable evaluation budget? Anchors should be provided to both methods or removed from both. Please distinguish use of identical scalar/check feedback from any richer textual critique, and report the execution cost alongside accuracy so the benefit of latent editing can be separated from feedback and resource differences.

  2. [Clarification and decision-relevant evidence] How do verifier reward, benchmark answer accuracy, and the frequency of high-reward incorrect outputs evolve over refinement iterations? Does the verifier audit cover the optimized-candidate distribution, or is independent re-evaluation available? Ground-truth labels could be used strictly for this post-hoc analysis, without entering the test-time optimization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.