acceptodds
Under review as a conference paper at ICLR 2027

Meta-CogGran: Reasoning as Online Latent Cognitive State Evolution for MLLMs

Abstract

Recent progress in multimodal large language models (MLLMs) has largely depended on CLIP-based visual encoders. While these models excel at capturing high-level semantic alignment, they often struggle to preserve fine-grained visual evidence. By comparison, the recent DINOv3 has exhibited emergent capabilities in fine-grained pixel-level perception, but still lacks the higher-level semantic abstraction required for alignment with LLMs and complex multimodal reasoning. These complementary limitations reveal a fundamental bottleneck: existing MLLMs lack a mechanism to dynamically reconcile fine-grained perception with high-level semantics during reasoning. We argue that the key challenge lies not merely in obtaining richer visual representations, but in enabling perceptual evidence and semantic knowledge to continuously interact, update, and converge during reasoning. This raises a central question: How can fine-grained perception and high-level semantics be dynamically integrated into evolving cognitive states for multimodal reasoning? To address this challenge, we propose **Meta-CogGran**, which formulates multimodal reasoning as online latent cognitive state evolution. It iteratively updates structured cognitive states through interactions with visual evidence and semantic memory, while a semantic equilibrium mechanism terminates reasoning once the state becomes stable and self-consistent. Extensive experiments demonstrate approximately **45% ↑** improvement in accuracy and **30% ↓** reduction in hallucination. Code is available at [Meta-CogGran](https://anonymous.4open.science/r/Meta-CogGran-5331/).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Reasoning, Cognitive State Evolution, Perception-Semantics Integration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rnLMOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Meta-CogGran represents multimodal reasoning as updates to a memory-augmented latent state with global, entity, layout, relation, and uncertainty components. A discrete policy chooses inspection, verification, revision, or stopping actions. Gated updates combine visual and memory evidence, while an energy criterion and a learned stopping head determine when to stop. A language model then generates an answer from the final state. The models are trained on FLUX-Reason and evaluated across visual encoders, language backbones, and held-out datasets using GPT-4o judging, text-similarity metrics, and qualitative examples.

Critical strengths
  1. The paper offers a useful formulation of multimodal reasoning as the evolution of a structured state. Separating global, entity, layout, relation, and uncertainty information gives visual evidence and semantic memory explicit roles. The inspection, verification, and revision actions make the intended reasoning process concrete. This is an interesting system-level contribution, even though the benefit of each component remains to be isolated.

  2. The reported DINOv3 gains across several language backbones and downstream datasets are a main reason for my weak-accept recommendation. The evaluation extends to out-of-distribution surgical imagery, with downstream benchmark data and prototype statistics excluded from memory construction according to the appendix. Text-based metrics, runtime/memory measurements, and a human check on 100 outputs complement the model-judged scores. This breadth provides initial evidence that the combined framework transfers across settings.

Critical weaknesses
  1. The training description needs to be more complete. Sections 3.1-3.3 introduce memory banks inherited from Meta-MemGran, discrete actions, an uncertainty component, a contradiction estimator, and a stopping head. The full objective, policy-training targets or gradient estimator, contradiction/stopping supervision and calibration, and memory construction/update rules are not specified clearly enough. Appendix B gives optimization settings but does not resolve these points. The energy weights, stopping thresholds, maximum steps, and handling of non-convergent cases also need to be documented. These are important reproducibility questions; my positive assessment of the framework does not remove the need for substantive clarification.

  2. The evidence is stronger for the combined system than for the proposed adaptive reasoning mechanism. Section 4.3 mainly varies encoders and model configurations, leaving the roles of online state evolution, action selection, and semantic-equilibrium stopping unresolved. The GPTscore protocol also needs clarification: Appendix E shows an image and model output without an explicit question or reference answer, while Appendix B.3 mentions task-adapted criteria without giving the VQA-specific prompts. Close mean human/GPT hallucination scores do not establish instance-level agreement or reliable system rankings. These limitations constrain how strongly the results can be interpreted as gains in reasoning or hallucination reduction.

Questions for authors
  1. Could you provide the full objective and training/inference algorithm, including memory construction, state dimensions, policy training, contradiction/stopping supervision, energy weights, thresholds, and maximum steps? Is the OR stopping rule intentional, and what prevents termination on a confident but contradictory state?

  2. What task-specific judge inputs and prompts were used, and which comparisons and aggregation produce the abstract's approximately 45% accuracy improvement and 30% hallucination reduction? Please clarify the conventional VQA answer-correctness results, instance-level human/judge agreement, and invalid-judgment counts per system. These details would help assess the reported gains.

  3. To assess the adaptive components, could you report any available comparisons with static memory retrieval, action-free iterative updates, and fixed-step stopping, using the same encoder, backbone, data, memory, and comparable inference budget? Accuracy/hallucination and iteration-count distributions would be informative. Further matched ablations would strengthen the mechanism claim; my current weak-accept recommendation rests primarily on the combined framework and its reported transfer results.

5 comments
AuthorsOpenReview ↗

Response to Reviewer rnLM on training details, stopping behavior, and mechanism validation.

Question 1 & Weakness 1:

Could you provide the full objective and training/inference algorithm, including memory construction, state dimensions, policy training, contradiction/stopping supervision, energy weights, thresholds, and maximum steps? Is the OR stopping rule intentional, and what prevents termination on a confident but contradictory state?

Authors’ Response:

  • Thank you for raising these important questions regarding reproducibility.

    • 1. Objective and policy training.

      • Training minimizes answer/EOS-token cross-entropy, with prompt and padding labels masked:
        • $\frac{1}{|\mathcal{T}|} \sum_{i\in\mathcal{T}} \log P(y_i\mid y_{<i},x,z_{\mathrm{out}}), $
      • where T\mathcal{T} denotes answer and EOS token positions. The vision encoder and LLM remain frozen, while the projector and cognitive modules are trainable.
      • The action policy computes
        • $ p_t=\operatorname{softmax} \left(\operatorname{MLP}_{\mathrm{act}}([\bar z_t;\bar F])/\tau\right), $
        • $ e_t=\sum_{a=1}^{8}p_{t,a}E_a,\qquad m_t=\sum_{a=1}^{8}p_{t,a}b_{t,a}, $
      • where τ=1\tau=1. Both training and inference use differentiable soft mixtures over eight action embeddings and corresponding memory contexts, followed by a shared gated transition.
    • 2. State dimensions and control signals.

      • The visual input contains 196 projected patches and 14 state tokens.
      • The uncertainty, halt, and contradiction signals are computed as
        • $ u_t=\sigma(U(\bar z_{t+1})),\quad h_t=\sigma(H(\bar z_{t+1})), $
        • $ c_t=\sigma(C([\bar z_{t+1};\bar m_t;\bar F])) \frac{1-\cos(\bar z_{t+1},\bar F)}{2}. $
      • The mean token-wise L2 state change is
        • $ D_t=\frac{1}{N_s}\sum_{i=1}^{N_s} |z_{t+1,i}-z_{t,i}|_2. $
      • These control heads receive no separate supervision labels or auxiliary losses.
    • **3. Energy and readout.**The energy function is

      • $ E_t=D_t+0.05\operatorname{Entropy}(p_t) +0.20c_t+0.10u_t, $
      • where Entropy⁡(pt)=−∑a=18pt,alog⁡pt,a\operatorname{Entropy}(p_t)=-\sum_{a=1}^{8}p_{t,a}\log p_{t,a}. Starting from r0=1r_0=1, the readout weights are
        • $ w_t=r_th_te^{-E_t},\qquad r_{t+1}=r_t(1-h_te^{-E_t}), $
        • $ z_{\mathrm{out}}= \sum_{t=0}^{K-1}w_tz_{t+1}+r_Kz_K. $
    • 4. Memory construction and updates.

      • The offline memory builder clusters global image features, entity-associated pooled local features, grid-conditioned local features, and typed directed-pair features into four banks: Global, Entity, Layout, and Relation.
      • At runtime, K-means constructs 10 local visual anchors (maximum 100 iterations; tolerance (10^{-6})). Retrieval retains 4/6/2/2 tokens from the respective banks using global, local/state, grid-cell, and relation-type queries.
    • 5. Training/inference settings and termination.

      • The main implementation settings are summarized below:

      • SettingValue
        Visual input196 patches + 14 state tokens
        State dimensionLLM hidden dimension
        Layout grid(4×4)
        Relation thresholds (near/overlap/direction)0.22/0.10/0.08
        Maximum relation-query pairs8
        Action space / temperature8 / 1.0
        Training/inference maximum steps4 / 4
        Earliest inference exitAfter 2 transitions
        Energy weights (shift/entropy/contradiction/uncertainty)1.0/0.05/0.20/0.10
        Halt/energy/state-shift thresholds0.78/0.16/0.015
      • Training always executes four transitions. Inference evaluates the following stopping condition after each transition, starting from the second:

        • $ k\geq2\land D_t\leq0.015 \land(h_t\geq0.78\lor E_t\leq0.16). $
        • Any remaining readout mass is assigned to the final state, ensuring a valid output even without convergence.
    • 6. Clarification of the OR stopping rule and contradiction handling.

      • The OR condition is intentional in the current implementation: after at least two transitions and a sufficiently small state change, inference terminates when either the halt score is sufficiently high or the energy is sufficiently low.
    • We will clarify these implementation details.

AuthorsOpenReview ↗

Response to Reviewer rnLM on training details, stopping behavior, and mechanism validation.

Question 2:

What task-specific judge inputs and prompts were used, and which comparisons and aggregation produce the abstract's approximately 45% accuracy improvement and 30% hallucination reduction? Please clarify the conventional VQA answer-correctness results, instance-level human/judge agreement, and invalid-judgment counts per system. These details would help assess the reported gains.

Authors’ Response:

  • Judge protocol. The recovered evaluator receives image + generated prediction, without the question, options, or reference answer. It scores image-grounded response quality on a 0–100 scale and hallucination severity on a 0–80 scale, using gpt-4o, temperature 0.1, high image detail, and a 300-token response limit.

    • This is not conventional VQA answer correctness. We will correct Appendix B.3’s task-adapted wording and distinguish GPTscore, lexical Recall, and benchmark-native answer accuracy.
  • Headline gains. The approximately 45-point figure refers to SurgVLM phase-recognition Recall: 78.11% versus 33.33% (+44.78 percentage points), not an overall accuracy improvement.

    • The original 30% summary lacks a traceable aggregation. We will replace it with an explicit Fig. 3a comparison: FLUX-Reason/Qwen2.5-7B hallucination severity decreases from 65.95 for CLIP to 33.32, or 32.63 score points. This is a severity-score difference, not a reduction in hallucination-event incidence.
  • Instance-level human agreement.

    • We now supplement the mean scores with paired analysis on the same 100 FLUX-Reason cases:

      MetricsResultBootstrap 95% CI
      Human mean69.70—
      GPT mean70.00—
      MAE ↓2.76[2.44, 3.09]
      Spearman’s ρ ↑0.769[0.673, 0.838]
      • Confidence intervals were calculated using 20,000 paired bootstrap resamples.

Question 3:

To assess the adaptive components, could you report any available comparisons with static memory retrieval, action-free iterative updates, and fixed-step stopping, using the same encoder, backbone, data, memory, and comparable inference budget? Accuracy/hallucination and iteration-count distributions would be informative. Further matched ablations would strengthen the mechanism claim; my current weak-accept recommendation rests primarily on the combined framework and its reported transfer results.

Authors’ Response:

  • We agree that encoder comparisons alone cannot isolate the adaptive components.

    • We therefore report controlled comparisons using the same encoder, LLM, training data, memory banks, and visual-token interface:

    • ConfigurationSEED Recall (%)A-OKVQA Recall (%)POPE F1Mean steps
      Static retrieval19.1218.9530.770
      One transition45.8244.0246.631
      Action-free updates; four steps42.6328.1562.844
      Random actions; four steps44.0228.6952.834
      Without halting; last-state readout19.2826.3365.934
      Full adaptive model53.1964.7669.083.27
      • The full configuration exceeds the one-transition variant by 7.37/20.74/22.45 points, respectively, and outperforms both fixed-step policy alternatives.
AuthorsOpenReview ↗

Response to Reviewer rnLM on training details, stopping behavior, and mechanism validation.

Weaknesses 2:

The evidence is stronger for the combined system than for the proposed adaptive reasoning mechanism. Section 4.3 mainly varies encoders and model configurations, leaving the roles of online state evolution, action selection, and semantic-equilibrium stopping unresolved. The GPTscore protocol also needs clarification: Appendix E shows an image and model output without an explicit question or reference answer, while Appendix B.3 mentions task-adapted criteria without giving the VQA-specific prompts. Close mean human/GPT hallucination scores do not establish instance-level agreement or reliable system rankings. These limitations constrain how strongly the results can be interpreted as gains in reasoning or hallucination reduction.

Authors’ Response:

Thank you for highlighting the distinction.

  • 1. Mechanism-level ablations.

    • ConfigurationSEED-BENCH RECALL (%) ↑A-OKVQA RECALL (%) ↑POPE F1 ↑Action entropy (nats)Mean steps
      Static four-bank retrieval19.1218.9530.77-0
      No loop: one transition45.8244.0246.630.0431
      Action-free updates; fixed 4 steps42.6328.1562.84-4
      Random actions; fixed 4 steps44.0228.6952.83-4
      Without halting; last-state readout (4 steps)19.2826.3365.930.0914
      Full adaptive model53.1964.7669.080.0843.27
      • Iteration: The full model exceeds one transition by 7.37/20.74/22.45 points.
      • Action selection: It exceeds uniform-action and random-action updates by 10.56/36.61/6.24 and 9.17/36.07/16.25 points, respectively.
      • Control and readout: Replacing adaptive control and weighted aggregation with a four-step last-state readout reduces performance by 33.91/38.43/3.15 points. The full model executes 3.27 transitions on average.

      Differences are percentage points in SEED-Bench Recall, A-OKVQA Recall, and POPE F1, respectively.

  • 2. GPTscore evaluation protocol.

    • The released GPT-4o prompt evaluates image-description quality using only the image and model prediction, without an explicit question or reference answer.
      • Official benchmark metrics will serve as the primary evidence for VQA performance. If supplementary GPT-based VQA evaluation is included, the evaluator will receive the image, question, candidate answer, and reference answer/options, using a separately specified task-aware prompt.
  • 3. Human agreement and system ranking.

    • Thank you for distinguishing aggregate agreement from instance-level consistency.

      • We now supplement the mean scores with paired analysis on the same 100 FLUX-Reason cases:

      • MetricsResultBootstrap 95% CI
        Human mean69.70—
        GPT mean70.00—
        MAE ↓2.76[2.44, 3.09]
        Spearman’s ρ ↑0.769[0.673, 0.838]
        • Confidence intervals were calculated using 20,000 paired bootstrap resamples.
Reviewer_WCmpOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Existing multimodal large language models mostly rely on CLIP based visual encoders and struggle to dynamically integrate fine grained visual evidence and high level semantics during reasoning To address this bottleneck this work introduces Meta CogGran which formulates multimodal reasoning as online latent cognitive state evolution The framework maintains structured cognitive states and iteratively updates them with visual evidence and multiple types of semantic memory It determines reasoning termination via semantic equilibrium and feeds only stabilized states into large language models for decoding Evaluations across diverse visual encoders large language model backbones and multimodal datasets demonstrate that this framework improves multimodal reasoning ability and mitigates model hallucination.

Critical strengths

1.The paper reformulates multimodal reasoning from autoregressive token generation into online latent cognitive state evolution. It explicitly tracks the reasoning progress through structured states covering scene entity layout relation beliefs plus uncertainty.

2.The stopping criterion is designed reasonably. The stopping criteria take into account the state change amount, action distribution entropy, and semantic contradiction degree, supplemented by a learnable stopping strategy, replacing a fixed number of iteration steps, and exhibiting a certain degree of interpretability.

3.Comprehensive experiments. The method is validated across multiple LLM backbones and diverse tasks.

Critical weaknesses

1.The relation memory bank is directly inherited from Meta‑MemGran. The transfer operator essentially combines attention retrieval and gated recurrent update. Clarification is needed on why this setup can form a cognitive framework.

2.Even though your method is highlighted in bold. Table 6 shows that applying your method to CLIP brings overall performance drops. It works effectively only with DINOv3. This indicates its benefits heavily depend on fine‑grained features from the encoder rather than the general cognitive evolution mechanism.

3.Primary metrics rely on GPT‑4o scoring. This introduces preference and circular bias from LLM‑as‑judge. Improvements on objective metrics are modest. There is a minor discrepancy with the forty‑five percent improvement claimed in the abstract.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer WCmp on novelty, encoder dependence, and evaluation reliability.

Weakness 1:

The relation memory bank is directly inherited from Meta‑MemGran. The transfer operator essentially combines attention retrieval and gated recurrent update. Clarification is needed on why this setup can form a cognitive framework.

Authors’ Response:

We thank the reviewer for raising this important concern.

  • We do not claim the relation bank or attention/gating primitives as new contributions.

    • The four-bank memory is inherited from Meta-MemGran.

      • The proposed distinction is how it is used:

        • state-dependent retrieval

        • action-conditioned recurrent refinement,

        • with a learned readout and an inference-time stopping criterion.

        • AspectStatic four-bank retrievalMeta-CogGran implementation
          Memory contentGlobal / Entity / Layout / RelationSame inherited banks
          Retrieval queryInitial visual representationUpdated state at each step
          State updateNo recurrent refinementSoft-action-conditioned gated update
          ReadoutRetrieved tokensHalting-weighted state aggregation
  • Here, “cognitive” denotes this explicit state/action/update/control organization.

    • It is an architectural interpretation, not evidence of human-like cognition or a convergence guarantee.
      • We will sharpen this distinction and use controlled static-memory, action-free, and fixed-step comparisons to test the claimed mechanism.

Weakness 2:

Even though your method is highlighted in bold. Table 6 shows that applying your method to CLIP brings overall performance drops. It works effectively only with DINOv3. This indicates its benefits heavily depend on fine‑grained features from the encoder rather than the general cognitive evolution mechanism.

Authors’ Response:

We thank the reviewer for this insightful comment.

  • We acknowledge that Meta-CogGran exhibits encoder-dependent effectiveness rather than universal compatibility.

  • We argue that this behavior reflects the complementarity between visual representations and cognitive evolution.

    • CLIP primarily captures high-level semantic alignment, potentially providing insufficient fine-grained spatial evidence for operations such as entity inspection, relation verification, and uncertainty refinement.
    • DINOv3 preserves richer local visual structures, enabling more effective cognitive state evolution.
  • Importantly, with the visual encoder fixed to DINOv3, Meta-CogGran still yields substantial improvements over the baseline (e.g., +21.69 and +15.50 GPTscore points on SEED-Bench and A-OKVQA with Qwen2.5).

    • BenchmarkDINOv3DINOv3 + Meta-CogGranGain
      SEED-Bench15.4037.09+21.69
      A-OKVQA25.6041.10+15.50
      Image Caption23.9751.02+27.05
      Reasoning45.3151.41+6.10

Weakness 3:

Primary metrics rely on GPT‑4o scoring. This introduces preference and circular bias from LLM‑as‑judge. Improvements on objective metrics are modest. There is a minor discrepancy with the forty‑five percent improvement claimed in the abstract.

Authors’ Response:

Thank you for raising these concerns.

  • Human validation.

    • We would like to clarify that we have already conducted independent human evaluation, as reported in Appendix B.5 (Table 9).
      • On 100 randomly sampled FLUX-Reason cases, human and GPT-based evaluators obtain average hallucination scores of 69.70 and 70.00, respectively, with only a 0.30 difference.
  • Clarification of the 45% improvement.

    • The reported 45% refers specifically to the absolute gain in phase-recognition Recall on SurgVLM (Table 3): 78.11% versus 33.33% for DINOv2, i.e., +44.78 percentage points.
      • It is not an average improvement across benchmarks.
        • We agree that the abstract lacks this specificity and will revise it to avoid ambiguity.

Dear Reviewer WCmp,

We sincerely appreciate your time and effort in providing such thoughtful reviews and insightful comments.

Could you please kindly let us know if our rebuttal has addressed your concerns?

If you have any further questions regarding our work, we would be delighted to address them.

If our responses have addressed your concerns, we would greatly appreciate it if you would consider whether the paper merits a higher overall rating.

Best regards,

Authors

Thank you to the authors for their detailed responses, which have addressed some of my questions. They have also acknowledged some limitations, so I will maintain my rating.

Reviewer_4E8gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Meta-CogGran, which models reasoning as a latent cognitive state (scene, entity, layout, relation, uncertainty) updated over several steps by discrete actions such as inspect, verify, revise and stop, using DINOv3 features and memory banks. Updates stop when an equilibrium energy is low or a learned stop head fires, and the final state is passed to the LLM.

Critical strengths
  • Combining pixel-level DINOv3 features with semantic memory in a multi-step process is a reasonable direction.
  • Table 2 and Table 6 show consistent gains over the plain DINOv3 baseline across LLM backbones, and the method transfers zero-shot to a medical domain.
Critical weaknesses
  • Insufficient ablation. No experiment removes the action policy, the equilibrium stop or the loop, and no step counts or action statistics are reported. The gains therefore cannot be attributed to cognitive state evolution rather than to extra modules and parameters. Nothing shows that the modules behave as their names suggest.
  • Training is under-specified. The paper gives no losses for the action policy, stop head or contradiction estimator. It does not state which parts are trained or frozen, how the final state is fed to the LLM, or what the connector is for each row of Table 2.
  • The comparison may be unfair. The baseline rows follow the Granulon setup, where only the projector is trained on frozen models, while Meta-CogGran adds many trained modules. The method is effectively a new connector, but it is not compared with other connectors such as Q-Former or a larger MLP at matched parameter count.
  • Off-the-shelf VLMs are evaluated zero-shot, while the proposed model is trained on FLUX-Reason, which is also a test set. Even so, Qwen2.5-VL-7B matches it on SEED-Bench and A-OKVQA recall, as Table 4 shows. Qwen3-VL and InternVL have no results on these two benchmarks.
  • The tasks are mostly captioning. FLUX-Reason is a text-to-image prompt dataset, and the reasoning prompt asks the model to describe the image in detail. Multi-step reasoning benchmarks (MathVista, MMMU, BLINK), hallucination benchmarks (POPE, HallusionBench), and multi-turn or video settings, where updating beliefs should matter most, are missing.
Questions for authors
  • What is the connector for each row of Table 2? Please give the number of visual tokens, the trainable parameters and the frozen parts. How many parameters does Meta-CogGran add?
  • What losses train the action policy, stop head and contradiction estimator? Without action or stop labels, how do these modules learn their intended roles?
  • Can you ablate (a) no loop, (b) a fixed number of steps, (c) no stop head and (d) random actions, and report action and step statistics?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 4E8g on mechanism attribution, reproducibility, and fair evaluation. Thank you for your constructive feedback. We address each concern below and distinguish existing evidence from the additional checks needed to support the revised claims.

Question 1:

What is the connector for each row of Table 2? Please give the number of visual tokens, the trainable parameters and the frozen parts. How many parameters does Meta-CogGran add?

Authors’ response:

  • Thank you for raising this important question.

    • We clarify the connector architectures, visual-token configurations, and parameter accounting for Meta-CogGran below.

    • Connector / visual interface
      CLIP / DINOv2 / DINOv3 baselinesTwo-layer LLaVA MLP
      GranulonLLaVA MLP + cluster-token aggregation
      Meta-CogGran (Ours)LLaVA MLP + cognitive refinement (196 patch + 14 state tokens)
      Qwen3-VL-8B-ThinkingSpatial patch merger: LayerNorm + two-layer GELU MLP; DeepStack mergers
      InternVL-3.5-8BPixel shuffle + LayerNorm + two-layer MLP
    • Visual tokens: 210 = 196 projected patch tokens + 14 aggregated cognitive-state tokens.

    • Trainable: The multimodal projector and all cognitive modules, including attention, action policy, transition, gate, halt, uncertainty, and contradiction heads.

    • Frozen: The vision encoder and LLM. Offline memory prototypes remain fixed.

      • BackboneProjector (M)Added cognitive modules (M)Total trainable (M)
        Qwen2.5-1.5B3.9450.7954.73
        Llama3.2-3B12.59203.02215.61
        Qwen3-8B20.98360.87381.85
        • “Added” excludes the existing projector. All rollout steps share parameters; intermediate states are aggregated into 14 output tokens.

Question 2:

What losses train the action policy, stop head and contradiction estimator? Without action or stop labels, how do these modules learn their intended roles?

Authors’ response:

Thank you for raising this important question.

The action policy, halt head, and contradiction estimator are jointly optimized using answer-token cross-entropy loss, with prompt and padding tokens masked. No explicit action, stopping, or contradiction labels, or auxiliary losses, are used.

These modules receive indirect supervision through differentiable state evolution and aggregation:

  • Action policy: Differentiable soft action mixtures influence state transitions, allowing answer-loss gradients to update the policy.
  • Halt head: The predicted halt probability hth_t determines each intermediate state's contribution to the final representation.
  • Contradiction estimator: The predicted energy EtE_t modulates state aggregation through: wt=rthtexp⁡(−Et)w_t=r_t h_t\exp(-E_t) Thus, answer-token loss propagates through the aggregation weights to jointly optimize all three modules, encouraging task-useful actions, state weighting.

Question 3:

Can you ablate (a) no loop, (b) a fixed number of steps, (c) no stop head and (d) random actions, and report action and step statistics?

Authors’ response:

  • We ablate iterative reasoning (single transition) and action selection (action-free or random actions).

    • ConfigurationSEED-BENCH RECALL (%) ↑A-OKVQA RECALL (%) ↑POPE F1 ↑Action entropy (nats)Mean steps
      Static four-bank retrieval19.1218.9530.77-0
      No loop: one transition45.8244.0246.630.0431
      Action-free updates; fixed 4 steps42.6328.1562.84-4
      Random actions; fixed 4 steps44.0228.6952.83-4
      Without halting; last-state readout (4 steps)19.2826.3365.930.0914
      Full adaptive model53.1964.7669.080.0843.27
    • The full model consistently outperforms the strongest ablation by 7.37 / 20.74 / 3.15 points, respectively,

      • supporting the benefit of adaptive multi-step reasoning and learned action selection.
AuthorsOpenReview ↗

Response to Reviewer 4E8g on mechanism attribution, reproducibility, and fair evaluation.

Weakness 1:

Insufficient ablation. No experiment removes the action policy, the equilibrium stop or the loop, and no step counts or action statistics are reported. The gains therefore cannot be attributed to cognitive state evolution rather than to extra modules and parameters. Nothing shows that the modules behave as their names suggest.

Authors’ Response:

  • Thank you for highlighting this attribution gap.

    • We conduct controlled ablations with identical encoders, LLMs, training data, memory banks, and output-token budgets.

    • ConfigurationSEED-BENCH RECALL (%) ↑A-OKVQA RECALL (%) ↑POPE F1 ↑Action entropy (nats)Mean steps
      Static four-bank retrieval19.1218.9530.77-0
      No loop: one transition45.8244.0246.630.0431
      Action-free updates; fixed 4 steps42.6328.1562.84-4
      Random actions; fixed 4 steps44.0228.6952.83-4
      Without halting; last-state readout (4 steps)19.2826.3365.930.0914
      Full adaptive model53.1964.7669.080.0843.27
      • Iteration: The full model exceeds one transition by 7.37/20.74/22.45 points.
      • Action selection: It exceeds uniform-action and random-action updates by 10.56/36.61/6.24 and 9.17/36.07/16.25 points, respectively.
      • Control and readout: Replacing adaptive control and weighted aggregation with a four-step last-state readout reduces performance by 33.91/38.43/3.15 points. The full model executes 3.27 transitions on average.

      Differences are percentage points in SEED-Bench Recall, A-OKVQA Recall, and POPE F1, respectively.


Weakness 2:

Training is under-specified. The paper gives no losses for the action policy, stop head or contradiction estimator. It does not state which parts are trained or frozen, how the final state is fed to the LLM, or what the connector is for each row of Table 2.

Authors’ Response:

Thank you for identifying the missing implementation details.

  • Training objective.

    • The action policy, halt head, and contradiction estimator are jointly optimized through answer-token cross-entropy, with prompt/padding tokens masked.
      • There are no separate action, stopping, or contradiction labels or auxiliary supervision losses.
      • Actions use differentiable soft mixtures. The halt probability and contradiction-dependent energy determine the state-aggregation weights, wt=rthtexp⁡(−Et)w_t=r_t h_t\exp(-E_t),allowing the answer loss to backpropagate into these heads.
      • Energy controls the readout; it is not an additional loss. Hard early exit is disabled during training.
  • Trainable and frozen components.

    • Freeze the vision encoder and LLM and
    • Train the multimodal projector and all cognitive modules,
      • including attention, action embeddings/policy, state transitions, gates, and halt/uncertainty/contradiction heads.
      • Offline memory prototypes remain fixed.
  • State-to-LLM interface.

    • The model concatenates 196 projected patch tokens with 14 halting-weighted state tokens in the LLM embedding space.
      • Thus, the LLM receives 210 visual tokens alongside the text input.
      • Intermediate states are aggregated rather than separately appended.
  • Connectors in Table 2. Rows sharing a connector family are grouped below.

    • Table 2 rowsConnector / visual interface
      CLIP / DINOv2 / DINOv3 baselinesTwo-layer LLaVA MLP
      GranulonLLaVA MLP + cluster-token aggregation
      Meta-CogGran (Ours)LLaVA MLP + cognitive refinement (196 patch + 14 state tokens)
      Qwen3-VL-8B-ThinkingSpatial patch merger: LayerNorm + two-layer GELU MLP; DeepStack mergers
      InternVL-3.5-8BPixel shuffle + LayerNorm + two-layer MLP
AuthorsOpenReview ↗

Response to Reviewer 4E8g on mechanism attribution, reproducibility, and fair evaluation.

Weakness 3:

The comparison may be unfair. The baseline rows follow the Granulon setup, where only the projector is trained on frozen models, while Meta-CogGran adds many trained modules. The method is effectively a new connector, but it is not compared with other connectors such as Q-Former or a larger MLP at matched parameter count.

Authors’ response:

  • We agree that additional capacity must be controlled explicitly.

    • We compared these connectors in a controlled pilot.

      • MLP and Q-Former provide approximately parameter-matched connector controls.

      • ConnectorTrainable params (M)Visual tokensSEED-BENCH RECALL (%) ↑Image-to-MC latency (ms) ↓
        Larger MLP + shared memory215.60021045.17145.3
        Q-Former + shared memory215.51321050.82149.6
        Meta-CogGran ( DINOv3/Llama3.2-3B)215.61121053.19366.4
    • Latency retains the separate image-to-MC scoring benchmark: preprocessing, frozen vision, connector and one LLM forward pass, excluding subsequent autoregressive decoding.


Weakness 4:

Off-the-shelf VLMs are evaluated zero-shot, while the proposed model is trained on FLUX-Reason, which is also a test set. Even so, Qwen2.5-VL-7B matches it on SEED-Bench and A-OKVQA recall, as Table 4 shows. Qwen3-VL and InternVL have no results on these two benchmarks.

Authors’ response:

Thank you for raising these concerns. We clarify two points:

  • No train–test overlap.

    • As specified in Appendix B.1, FLUX-Reason contains 200K training samples and 1K disjoint test samples. We will provide split manifests, duplicate checks, and the exact memory-construction subset to verify this separation.
  • Stronger baseline comparisons.

    • EncodersModelsSEED-Bench Recall(%)↑SEED-Bench GPTScore(%)↑A-OKVQA Recall(%)↑A-OKVQA GPTScore(%)↑
      Qwen3-VL-8B-Thinking–95.2043.9060.8048.60
      InternVL-3.5-8B–93.7041.7057.4045.10
      OursQwen2.5-1.5B69.4837.0964.4741.10
      OursQwen3-8B97.1445.8150.7754.23
      OursLlama3.2-3B53.1959.1964.7665.00
    • We have added Qwen3-VL-8B-Thinking and InternVL-3.5-8B results to Table 4. The comparisons show:

      • SEED-Bench: Our Qwen3-8B variant outperforms both Qwen3-VL and InternVL in Recall (97.14% vs. 95.20% and 93.70%) and GPTScore (45.81% vs. 43.90% and 41.70%).
      • A-OKVQA: Our Llama3.2-3B variant achieves higher Recall (64.76%) and GPTScore (65.00%) than both Qwen3-VL (60.80% / 48.60%) and InternVL (57.40% / 45.10%).

Weakness 5:

The tasks are mostly captioning. FLUX-Reason is a text-to-image prompt dataset, and the reasoning prompt asks the model to describe the image in detail. Multi-step reasoning benchmarks (MathVista, MMMU, BLINK), hallucination benchmarks (POPE, HallusionBench), and multi-turn or video settings, where updating beliefs should matter most, are missing.

Authors’ response:

  • We conducted an objective pilot evaluation using the same frozen DINOv3/Llama-3.2-3B backbone and training budget

    • ConfigurationBLINK Acc. (%)POPE F1 (%)
      DINOv3 + projector33.8529.56
      Static four-bank retrieval34.0230.77
      Meta-CogGran43.0769.08
  • Evaluation protocol.

    • BLINK uses all 1,901 validation questions, with accuracy macro-averaged over 14 tasks.
    • POPE uses all 9,000 test questions: 3,000 each for Random, Popular and Adversarial sampling.
      • We report their mean positive-class F1. Neither evaluation uses an LLM judge.
  • What these results establish.

    • Meta-CogGran achieves higher scores than both baselines on the two reported benchmarks.
      • BLINK accuracy improves to 43.07% (vs. 33.85% and 34.02%).
      • On POPE, positive-class F1 increases to 69.08% (vs. 29.56% and 30.77%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.