acceptodds
Withdrawn from ICLR 2027

Calibrating Perception and Temporal Trajectories for Test-Time Latent Alignment in Multimodal LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While existing MLLMs enhance logical reasoning, they still suffer from severe confidence miscalibration. Moreover, current explicit and implicit reasoning mechanisms are constrained by discrete tokens and autoregressive mapping, which easily induces "premature semantic collapse" and "tunnel vision," failing to meet the holistic and hierarchical demands of visual perception. To address these challenges, we propose CHARM, a progressive three-stage architecture: First, Metacognitive Visual Learning (MVL) uses reinforcement learning to jointly optimize accuracy, calibration, and sensitivity, thereby aligning perception with confidence. Second, Chronological Trajectory Refinement (CTR) introduces temporal perturbation penalties to reinforce explicit temporal reasoning and keyframe grounding. Finally, Multi-Horizon Latent Alignment (MHLA) operates during test-time inference, leveraging pre-calibrated uncertainty to dynamically mix future multi-step semantic superposition distributions with hard labels, optimizing latent reasoning to decode the final explicit textual answer. Built on Qwen2.5-VL-7B, our method achieves 80.5% (OE) and 87.3% (MC) on Math-Vista, as well as 72.8% (STEM) and 80.6% (HASS) on MMMU. When scaled to Qwen3-VL-8B-Instruct, it reaches 69.7 ([email protected]) on the ActivityNet benchmark and sets new state-of-the-art records on QVHighlights and TVGBench with 90.3 and 65.1 ([email protected]).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Confidence Calibration, Metacognitive Visual Learning, Visual Logical Reasoning, Multi-Horizon Latent Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_98WzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CHARM, a three-stage framework for multimodal large language models. Metacognitive Visual Learning uses GRPO to jointly optimize task accuracy, confidence calibration, and sensitivity to visual perturbations. Chronological Trajectory Refinement introduces temporal frame-shuffling rewards to improve temporally grounded reasoning. Multi-Horizon Latent Alignment then combines future semantic distributions with hard targets through entropy-based gating, purportedly during test-time inference. The paper reports improvements across visual mathematical reasoning.

Critical strengths
  1. Overconfidence under degraded visual input and insufficient reliance on temporal evidence are meaningful limitations of multimodal models. Incorporating visual and temporal perturbations into optimization provides a potentially useful direction for improving reliability and temporal understanding.

  2. The reported evaluation covers a broad range of tasks and model sizes. The experiments span image reasoning, general multimodal understanding, and temporal video grounding.

Critical weaknesses
  1. The proposed test-time procedure depends on ground-truth answers that are unavailable at inference. Section 3.3 explicitly describes MHLA as a test-time procedure, yet Equation (4) defines (y_{\mathrm{hard}}) as a ground-truth hard-label distribution, and Equation (6) computes cross-entropy against ground-truth answer tokens (a_j). Algorithm 3 on page 15 explicitly requires the ground-truth answer sequence as input, computes the supervised loss, updates model parameters in Steps 14–16, and only then generates the answer in Step 17. Consequently, the algorithm as written cannot operate under the standard evaluation setting in which only the visual input and question are available.

  2. Section 4.1 lists MMMU-Pro as training data while evaluating on MMMU. However, MMMU-Pro is constructed from MMMU questions through filtering, option augmentation, and changes in presentation, rather than from an independent source. The submission does not specify the relevant splits, sample identifiers, or cross-dataset deduplication procedure. This creates a concrete risk of overlapping training and evaluation content, although the available evidence does not establish that overlap actually occurred in the authors’ selected subsets. Separately, Table 4 explicitly states that all models are evaluated using Pass@1 and reports 87.3% on MathVista MC. Section 4.3 and Figure 3 associate the same 87.3% result with 32 samples. The paper does not explain whether these results use different checkpoints, subsets, or aggregation procedures. Moreover, because MHLA involves additional optimization, sample counts and scaling slopes alone do not establish computational efficiency.

Questions for authors
  1. Where do (y_{\mathrm{hard}}) and the answer tokens (a_j) in Equation (6) come from during evaluation? Was Algorithm 3 used to obtain the results in Tables 1–7?

  2. Which MMMU-Pro samples were used for training, and which MMMU split and samples were used for evaluation?

  3. How should the 87.3% Pass@1 result in Table 4 be reconciled with the 87.3% result at 32 samples in Figure 3?

  4. What are the exact definitions of (R_{\mathrm{calib}}) and (R_{\mathrm{sensitivity}})? Can you provide direct calibration measurements to establish that Stage 1 improves calibration and that this improvement makes Stage 3 entropy gating more effective?

Reviewer_Bj1qOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CHARM, a three-stage framework for multimodal reasoning and video temporal grounding. Metacognitive Visual Learning (MVL) uses reinforcement learning with accuracy, confidence-calibration, and perception-sensitivity rewards. Chronological Trajectory Refinement (CTR) adds a temporal perturbation signal to encourage reasoning grounded in frame order. Multi-Horizon Latent Alignment (MHLA) is presented as a test-time procedure that combines future semantic distributions with entropy-dependent hard targets before generating an answer. The paper reports improvements across image reasoning and video grounding benchmarks using several Qwen backbones, with component ablations, hyperparameter studies, and qualitative examples.

Critical strengths
  1. The overall research direction is worthwhile. Connecting uncertainty about visual perception with temporal reasoning and latent inference is a plausible way to improve multimodal reasoning. The three-stage design provides an understandable hypothesis about how these capabilities might support one another, addressing tasks beyond static image question answering.

  2. The evaluation has useful breadth and includes substantive ablations. Tables 1–3 cover image reasoning, video grounding, and multiple backbones. Tables 4–6 remove individual components, and Table 7 varies the shuffle window, entropy threshold, and mixing weight. The reported gains could be meaningful if the evaluation protocol is substantiated: for example, Table 2 reports QVHighlights [email protected] of 73.8 for CHARM versus 66.2 for Time-R1 on the Qwen2.5-VL-7B backbone. These results and ablations provide a promising starting point for assessing the framework.

Critical weaknesses

W1. The metacognitive contribution and core method are insufficiently specified.

  • The operational meaning of “metacognition” and its connection to confidence estimation and entropy gating are unclear. The distinction from [1], which already combines perceptual-sensitivity/calibration rewards with confidence-guided test-time reasoning, needs explanation.
  • Neither Section 3.1 nor Appendix A, Algorithm 1, gives explicit definitions of RcalibR_{\mathrm{calib}} and RsensitivityR_{\mathrm{sensitivity}}. The MHLA window construction, weighting, and latent-state updates are also underspecified.
  • Equations (4) and (6) and Algorithm 3 require ground-truth labels or answer tokens during the purported test-time procedure. The source of these targets must be clarified before the evaluation can be assessed; the description alone does not establish actual test-label use.
  • Figure 1 is a low-resolution raster image, and Figures 2(b) and 5 contain small text. Vector graphics and larger labels would improve readability.

W2. The experiments do not establish calibrated uncertainty or a consistent evaluation protocol.

  • Accuracy/recall ablations do not directly demonstrate calibration. The paper lacks calibration measurements and evidence connecting MVL’s confidence signal to MHLA’s entropy gate.
  • Table 4 labels the MathVista MC result of 87.3 as Pass@1, whereas Figure 3 associates 87.3 with 32 samples. The settings need reconciliation. Sample-count scaling slopes also do not establish compute efficiency without accounting for MHLA’s additional cost.
  • Section 4.1 uses MMMU-Pro for training and MMMU for evaluation. Since MMMU-Pro derives from MMMU, question-level overlap checks are needed. The current description does not establish whether overlapping items were excluded.

[1] Linking Perception, Confidence and Accuracy in MLLMs, CVPR 2026

Questions for authors
  1. Metacognition: What does “metacognition” mean operationally in CHARM, and how do the proposed components implement it? How does the confidence learned by MVL relate to H(Qt)H(Q_t) in MHLA, and what evidence validates this connection?

  2. Reward definitions: Could you provide explicit mathematical definitions of RcalibR_{\mathrm{calib}} and RsensitivityR_{\mathrm{sensitivity}}, including the confidence computation and visual perturbations? I could not find these definitions in the main text or appendix; Section 3.1 and Algorithm 1 describe their intended roles.

  3. Novelty Clarification: Could you clarify the main conceptual and technical advances of the proposed framework over existing work on confidence calibration and confidence-guided reasoning?

  4. Test-time supervision: Where do the hard targets and answer tokens in Equations (4) and (6) and Algorithm 3 come from during evaluation? If they are pseudo-labels, please specify their generation and the parameter-update/reset procedure without reference-answer access.

  5. Sampling and computation: How should Table 4’s Pass@1 result of 87.3 be reconciled with Figure 3’s 32-sample result? Please specify the checkpoints, subsets, aggregation rules, and actual inference costs used in these comparisons.

  6. Data separation: Which MMMU-Pro examples and MMMU evaluation split were used, and how was overlap excluded at the original-question level, including transformed variants?

Reviewer_8Ra8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

CHARM is a three-stage framework addressing confidence miscalibration and latent-space reasoning problems in MLLMs. It uses GRPO to optimize accuracy, calibration, and perturbation sensitivity (Stage 1), frame-shuffling logit changes to strengthen temporal reasoning (Stage 2), and entropy gating at decoding time to mix a soft target with the hard label (Stage 3). The authors report large gains on MathVista, MMMU, and QVHighlights.

Critical strengths
  1. The paper evaluates on several benchmarks, covering both image reasoning and video temporal grounding, and provides stage-wise ablations and hyperparameter sensitivity analyses.

  2. The reported results show consistent improvements over the baselines across the evaluated benchmarks.

Critical weaknesses
  1. The presentation is hard to follow. Figure 1 is a little blurry. And Its caption is "Overall Structure", with no explanation of the individual modules.

  2. The motivation is unclear. "Premature semantic collapse" and "tunnel vision" are used repeatedly as motivation, with no intuitive examples or empirical evidence. The paper does not explain how they relate to confidence miscalibration.

  3. Method details are partly missing. Stages 1 and 2 are mostly a list of formulas. RcalibR_{calib} and RsensitivityR_{sensitivity} are not clearly defined.

  4. The "test-time" setting of MHLA is unclear. The method relies on the ground-truth hard label yhardy_{hard}, but no ground truth is available at test time. The paper does not explain how this works.

  5. The evaluation setting is unclear. Table 4 is stated to use Pass@1, but in Section 4.3 the 87.3% corresponds to the result when the number of samples is increased to 32. It is unclear which setting the main results use.

Questions for authors
  1. How does MHLA use the ground-truth hard label yhardy_{hard} at test time, when no ground truth is available?

  2. How many samples are used for "Ours" in the main table? Is the comparison with baselines done under the same setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.