- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CHARM, a three-stage framework for multimodal large language models. Metacognitive Visual Learning uses GRPO to jointly optimize task accuracy, confidence calibration, and sensitivity to visual perturbations. Chronological Trajectory Refinement introduces temporal frame-shuffling rewards to improve temporally grounded reasoning. Multi-Horizon Latent Alignment then combines future semantic distributions with hard targets through entropy-based gating, purportedly during test-time inference. The paper reports improvements across visual mathematical reasoning.
Critical strengths
-
Overconfidence under degraded visual input and insufficient reliance on temporal evidence are meaningful limitations of multimodal models. Incorporating visual and temporal perturbations into optimization provides a potentially useful direction for improving reliability and temporal understanding.
-
The reported evaluation covers a broad range of tasks and model sizes. The experiments span image reasoning, general multimodal understanding, and temporal video grounding.
Critical weaknesses
-
The proposed test-time procedure depends on ground-truth answers that are unavailable at inference. Section 3.3 explicitly describes MHLA as a test-time procedure, yet Equation (4) defines (y_{\mathrm{hard}}) as a ground-truth hard-label distribution, and Equation (6) computes cross-entropy against ground-truth answer tokens (a_j). Algorithm 3 on page 15 explicitly requires the ground-truth answer sequence as input, computes the supervised loss, updates model parameters in Steps 14–16, and only then generates the answer in Step 17. Consequently, the algorithm as written cannot operate under the standard evaluation setting in which only the visual input and question are available.
-
Section 4.1 lists MMMU-Pro as training data while evaluating on MMMU. However, MMMU-Pro is constructed from MMMU questions through filtering, option augmentation, and changes in presentation, rather than from an independent source. The submission does not specify the relevant splits, sample identifiers, or cross-dataset deduplication procedure. This creates a concrete risk of overlapping training and evaluation content, although the available evidence does not establish that overlap actually occurred in the authors’ selected subsets. Separately, Table 4 explicitly states that all models are evaluated using Pass@1 and reports 87.3% on MathVista MC. Section 4.3 and Figure 3 associate the same 87.3% result with 32 samples. The paper does not explain whether these results use different checkpoints, subsets, or aggregation procedures. Moreover, because MHLA involves additional optimization, sample counts and scaling slopes alone do not establish computational efficiency.
Questions for authors
-
Where do (y_{\mathrm{hard}}) and the answer tokens (a_j) in Equation (6) come from during evaluation? Was Algorithm 3 used to obtain the results in Tables 1–7?
-
Which MMMU-Pro samples were used for training, and which MMMU split and samples were used for evaluation?
-
How should the 87.3% Pass@1 result in Table 4 be reconciled with the 87.3% result at 32 samples in Figure 3?
-
What are the exact definitions of (R_{\mathrm{calib}}) and (R_{\mathrm{sensitivity}})? Can you provide direct calibration measurements to establish that Stage 1 improves calibration and that this improvement makes Stage 3 entropy gating more effective?