- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Meta-CogGran represents multimodal reasoning as updates to a memory-augmented latent state with global, entity, layout, relation, and uncertainty components. A discrete policy chooses inspection, verification, revision, or stopping actions. Gated updates combine visual and memory evidence, while an energy criterion and a learned stopping head determine when to stop. A language model then generates an answer from the final state. The models are trained on FLUX-Reason and evaluated across visual encoders, language backbones, and held-out datasets using GPT-4o judging, text-similarity metrics, and qualitative examples.
Critical strengths
-
The paper offers a useful formulation of multimodal reasoning as the evolution of a structured state. Separating global, entity, layout, relation, and uncertainty information gives visual evidence and semantic memory explicit roles. The inspection, verification, and revision actions make the intended reasoning process concrete. This is an interesting system-level contribution, even though the benefit of each component remains to be isolated.
-
The reported DINOv3 gains across several language backbones and downstream datasets are a main reason for my weak-accept recommendation. The evaluation extends to out-of-distribution surgical imagery, with downstream benchmark data and prototype statistics excluded from memory construction according to the appendix. Text-based metrics, runtime/memory measurements, and a human check on 100 outputs complement the model-judged scores. This breadth provides initial evidence that the combined framework transfers across settings.
Critical weaknesses
-
The training description needs to be more complete. Sections 3.1-3.3 introduce memory banks inherited from Meta-MemGran, discrete actions, an uncertainty component, a contradiction estimator, and a stopping head. The full objective, policy-training targets or gradient estimator, contradiction/stopping supervision and calibration, and memory construction/update rules are not specified clearly enough. Appendix B gives optimization settings but does not resolve these points. The energy weights, stopping thresholds, maximum steps, and handling of non-convergent cases also need to be documented. These are important reproducibility questions; my positive assessment of the framework does not remove the need for substantive clarification.
-
The evidence is stronger for the combined system than for the proposed adaptive reasoning mechanism. Section 4.3 mainly varies encoders and model configurations, leaving the roles of online state evolution, action selection, and semantic-equilibrium stopping unresolved. The GPTscore protocol also needs clarification: Appendix E shows an image and model output without an explicit question or reference answer, while Appendix B.3 mentions task-adapted criteria without giving the VQA-specific prompts. Close mean human/GPT hallucination scores do not establish instance-level agreement or reliable system rankings. These limitations constrain how strongly the results can be interpreted as gains in reasoning or hallucination reduction.
Questions for authors
-
Could you provide the full objective and training/inference algorithm, including memory construction, state dimensions, policy training, contradiction/stopping supervision, energy weights, thresholds, and maximum steps? Is the OR stopping rule intentional, and what prevents termination on a confident but contradictory state?
-
What task-specific judge inputs and prompts were used, and which comparisons and aggregation produce the abstract's approximately 45% accuracy improvement and 30% hallucination reduction? Please clarify the conventional VQA answer-correctness results, instance-level human/judge agreement, and invalid-judgment counts per system. These details would help assess the reported gains.
-
To assess the adaptive components, could you report any available comparisons with static memory retrieval, action-free iterative updates, and fixed-step stopping, using the same encoder, backbone, data, memory, and comparable inference budget? Accuracy/hallucination and iteration-count distributions would be informative. Further matched ablations would strengthen the mechanism claim; my current weak-accept recommendation rests primarily on the combined framework and its reported transfer results.
5 comments
Response to Reviewer rnLM on training details, stopping behavior, and mechanism validation.
Question 2:
What task-specific judge inputs and prompts were used, and which comparisons and aggregation produce the abstract's approximately 45% accuracy improvement and 30% hallucination reduction? Please clarify the conventional VQA answer-correctness results, instance-level human/judge agreement, and invalid-judgment counts per system. These details would help assess the reported gains.
Authors’ Response:
-
Judge protocol. The recovered evaluator receives image + generated prediction, without the question, options, or reference answer. It scores image-grounded response quality on a 0–100 scale and hallucination severity on a 0–80 scale, using gpt-4o, temperature 0.1, high image detail, and a 300-token response limit.
- This is not conventional VQA answer correctness. We will correct Appendix B.3’s task-adapted wording and distinguish GPTscore, lexical Recall, and benchmark-native answer accuracy.
-
Headline gains. The approximately 45-point figure refers to SurgVLM phase-recognition Recall: 78.11% versus 33.33% (+44.78 percentage points), not an overall accuracy improvement.
- The original 30% summary lacks a traceable aggregation. We will replace it with an explicit Fig. 3a comparison: FLUX-Reason/Qwen2.5-7B hallucination severity decreases from 65.95 for CLIP to 33.32, or 32.63 score points. This is a severity-score difference, not a reduction in hallucination-event incidence.
-
Instance-level human agreement.
-
We now supplement the mean scores with paired analysis on the same 100 FLUX-Reason cases:
Metrics Result Bootstrap 95% CI Human mean 69.70 — GPT mean 70.00 — MAE ↓ 2.76 [2.44, 3.09] Spearman’s ρ ↑ 0.769 [0.673, 0.838] - Confidence intervals were calculated using 20,000 paired bootstrap resamples.
-
Question 3:
To assess the adaptive components, could you report any available comparisons with static memory retrieval, action-free iterative updates, and fixed-step stopping, using the same encoder, backbone, data, memory, and comparable inference budget? Accuracy/hallucination and iteration-count distributions would be informative. Further matched ablations would strengthen the mechanism claim; my current weak-accept recommendation rests primarily on the combined framework and its reported transfer results.
Authors’ Response:
-
We agree that encoder comparisons alone cannot isolate the adaptive components.
-
We therefore report controlled comparisons using the same encoder, LLM, training data, memory banks, and visual-token interface:
-
Configuration SEED Recall (%) A-OKVQA Recall (%) POPE F1 Mean steps Static retrieval 19.12 18.95 30.77 0 One transition 45.82 44.02 46.63 1 Action-free updates; four steps 42.63 28.15 62.84 4 Random actions; four steps 44.02 28.69 52.83 4 Without halting; last-state readout 19.28 26.33 65.93 4 Full adaptive model 53.19 64.76 69.08 3.27 - The full configuration exceeds the one-transition variant by 7.37/20.74/22.45 points, respectively, and outperforms both fixed-step policy alternatives.
-
Response to Reviewer rnLM on training details, stopping behavior, and mechanism validation.
Weaknesses 2:
The evidence is stronger for the combined system than for the proposed adaptive reasoning mechanism. Section 4.3 mainly varies encoders and model configurations, leaving the roles of online state evolution, action selection, and semantic-equilibrium stopping unresolved. The GPTscore protocol also needs clarification: Appendix E shows an image and model output without an explicit question or reference answer, while Appendix B.3 mentions task-adapted criteria without giving the VQA-specific prompts. Close mean human/GPT hallucination scores do not establish instance-level agreement or reliable system rankings. These limitations constrain how strongly the results can be interpreted as gains in reasoning or hallucination reduction.
Authors’ Response:
Thank you for highlighting the distinction.
-
1. Mechanism-level ablations.
-
Configuration SEED-BENCH RECALL (%) ↑ A-OKVQA RECALL (%) ↑ POPE F1 ↑ Action entropy (nats) Mean steps Static four-bank retrieval 19.12 18.95 30.77 - 0 No loop: one transition 45.82 44.02 46.63 0.043 1 Action-free updates; fixed 4 steps 42.63 28.15 62.84 - 4 Random actions; fixed 4 steps 44.02 28.69 52.83 - 4 Without halting; last-state readout (4 steps) 19.28 26.33 65.93 0.091 4 Full adaptive model 53.19 64.76 69.08 0.084 3.27 - Iteration: The full model exceeds one transition by 7.37/20.74/22.45 points.
- Action selection: It exceeds uniform-action and random-action updates by 10.56/36.61/6.24 and 9.17/36.07/16.25 points, respectively.
- Control and readout: Replacing adaptive control and weighted aggregation with a four-step last-state readout reduces performance by 33.91/38.43/3.15 points. The full model executes 3.27 transitions on average.
Differences are percentage points in SEED-Bench Recall, A-OKVQA Recall, and POPE F1, respectively.
-
-
2. GPTscore evaluation protocol.
- The released GPT-4o prompt evaluates image-description quality using only the image and model prediction, without an explicit question or reference answer.
- Official benchmark metrics will serve as the primary evidence for VQA performance. If supplementary GPT-based VQA evaluation is included, the evaluator will receive the image, question, candidate answer, and reference answer/options, using a separately specified task-aware prompt.
- The released GPT-4o prompt evaluates image-description quality using only the image and model prediction, without an explicit question or reference answer.
-
3. Human agreement and system ranking.
-
Thank you for distinguishing aggregate agreement from instance-level consistency.
-
We now supplement the mean scores with paired analysis on the same 100 FLUX-Reason cases:
-
Metrics Result Bootstrap 95% CI Human mean 69.70 — GPT mean 70.00 — MAE ↓ 2.76 [2.44, 3.09] Spearman’s ρ ↑ 0.769 [0.673, 0.838] - Confidence intervals were calculated using 20,000 paired bootstrap resamples.
-
-
Response to Reviewer rnLM on training details, stopping behavior, and mechanism validation.
Question 1 & Weakness 1:
Authors’ Response:
Thank you for raising these important questions regarding reproducibility.
1. Objective and policy training.
2. State dimensions and control signals.
**3. Energy and readout.**The energy function is
4. Memory construction and updates.
5. Training/inference settings and termination.
The main implementation settings are summarized below:
Training always executes four transitions. Inference evaluates the following stopping condition after each transition, starting from the second:
6. Clarification of the OR stopping rule and contradiction handling.
We will clarify these implementation details.