acceptodds
Under review as a conference paper at ICLR 2027

Towards Multimodal Agents with Cognitive Modality Planning and Reflective Reasoning

Abstract

Multimodal agents must decide which perceptual modality to read, when to stop refining their belief, and how to keep an answer honest about its own uncertainty. Existing pipelines either run a single fixed toolchain (no planning), or invoke every modality and then reflect (no budget), or rely on a heuristic LLM-as-judge for confidence (no calibration). We intro duce CoMPR (Cognitive Modality Planning + Reflective Reasoning), a training-friendly framework that formulates modality selection as a bud geted submodular maximization, runs a short chain of reflective revise and-verify passes with a temperature-calibrated verifier, and is post-trained with GRPO under a composite reward that explicitly rewards evidence grounded, calibrated, budget-efficient behaviour. CoMPR + a GRPO warm-start (CoMPR+RL) is evaluated on five multimodal QA benchmarks and two agentic benchmarks across four backbone LLMs, and compared against eleven strong baselines including CoT, self-consistency, ReAct, Re flexion, Self-Refine, Critic, Set-of-Mark, random planning, all-modalities, and the recent CogPlanner and DeepEyes agents. On Qwen2.5-VL-7B CoMPR+RL improves mean accuracy by +5 .0 over the strongest base line, reduces expected calibration error from 0.148 to 0.031, and shortens latency by ∼17%, while remaining backbone-agnostic and competitive with the much larger GPT-4o.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal agents, Modality planning, Submodular maximization, Reflective reasoning, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_QHjyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoMPR selects perception tools under a budget, checks the resulting answer with a calibrated verifier, and uses reflection to revise the answer or gather more evidence. The authors also train the policy with GRPO using rewards for correctness, evidence citation, calibration, and cost. On Qwen2.5-VL-7B, Table 2 reports average accuracy of 81.02% for CoMPR and 82.06% for CoMPR+RL, compared with 78.40% for CRITIC. The paper also evaluates the approach on ALFWorld and MMSearch.

Critical strengths
  1. The method addresses a practical problem: an agent needs to decide both what evidence to collect and when it has collected enough. I find the connection between reflection and tool selection useful. A failed check can trigger another observation instead of merely asking the model to rewrite its answer.

  2. The reported gains are consistent across the five QA datasets in Table 2. Table 3 also shows lower token use and latency than CRITIC. These results are encouraging, and the component ablations help explain which parts of the system contribute.

Critical weaknesses
  1. The planner’s theoretical guarantee is not justified. Section 3.2 argues that the objective is submodular because each tool addresses one facet of the answer. Tools can instead be complementary: reading an axis label may make an extracted chart value more useful. The paper does not give assumptions that rule this out. It also mentions both partial enumeration and a gain-per-cost greedy rule without specifying the algorithm sufficiently to establish the claimed (1-1/e) bound. This matters because the guarantee is presented as a main contribution.

  2. The reported improvements are difficult to reconcile, and key implementation details are missing. The abstract claims a +5.0 accuracy gain over the strongest baseline, while Table 2 gives +3.66. The comparison behind the claimed 17% latency reduction is unclear. Table 3 calls CoMPR+RL the most calibrated method, although CoMPR has lower ECE. I also could not determine how tool utilities and evidence-support labels are obtained. Together, these issues make it difficult to assess what was implemented and how much it improves over the baselines.

Questions for authors
  1. What assumptions ensure that the planner’s objective is submodular? Please specify the implemented selection algorithm and explain whether its guarantee applies to the learned utility estimates.

  2. How are tool utilities predicted before execution, and how are the verifier’s evidence-support labels produced? Which components are trained, including in the GPT-4o experiments?

  3. Which results should be treated as authoritative where the abstract and tables disagree? Please also explain the relationship between the 1,000 ALFWorld trajectories in Section 4.1 and the 134 unseen tasks in Figure 6.

  4. The verifier is trained to predict evidence support, but the calibration reward uses answer correctness. Why are these treated as the same target? Since absolute-error rewards do not generally favor calibrated probabilities, what supports this choice?

Reviewer_9vL5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CoMPR, a multimodal agent framework with three parts. (i) A “cognitive planner” picks a subset of nine perception operators (OCR, chart parser, crop, retrieval, etc.) by budgeted submodular maximization, with a claimed (1−1/e) guarantee. (ii) A temperature-calibrated verifier decides when an iterative critique→revise loop stops. (iii) GRPO post-training uses a composite reward covering accuracy, evidence grounding, calibration and cost. Results are reported on five multimodal QA benchmarks and two agentic benchmarks, mostly with Qwen2.5-VL-7B. The paper claims +5.0 mean accuracy over the strongest baseline, ECE dropping from 0.148 to 0.031, and ~17% lower latency.

Critical strengths
  1. The motivation is relevant. Choosing which perception tools to call under a cost budget, and deciding when to stop reflecting, are real practical problems for multimodal agents.
  2. Bringing budget, calibration and grounding into a single training objective is a reasonable idea in principle.
Critical weaknesses
  1. The theory doesn’t hold. Submodularity is asserted without proof, and MI is not submodular when operators are correlated, as they are here (e.g. document_structure depends on OCR). Cost-benefit greedy has no 1−1/e guarantee under a knapsack constraint. With only 9 operators (512 subsets), exact search is trivial anyway, and re-planning during reflection breaks the one-shot bound.
  2. The reward and verifier are flawed or underspecified. r_cal uses absolute error, which is an improper scoring rule and pushes predictions toward 0/1, against calibration. r_gr can be gamed by citing everything. It’s never said where the verifier’s evidence-support labels come from, and the SFT/GRPO training data isn’t described
  3. The comparisons aren’t controlled. Hyperparameters are tuned on ScienceQA, which is also a test set. The baselines get no training while CoMPR gets SFT, GRPO and a trained verifier, and ECE is compared against uncalibrated baselines. The paper needs accuracy-only GRPO and tool-equipped baselines. ALFWorld is text-only, so the visual operators don’t apply.
  4. Numbers contradict each other throughout. The abstract’s +5.0 is really +3.66 over CRITIC; +5.0 is the gap to Random Plan. The abstract’s ECE of 0.031 belongs to CoMPR, while CoMPR+RL is 0.034. The “full” model scores differently in Tables 2 and 4. Several numbers in the text (MathVista +5.83, the ablation deltas, “2–3× more calls”) don’t match the tables. AURC results and the referenced appendix are missing.
  5. the citations and presentation aren’t reliable. CogPlanner, MMSearch and DeepEyes are cited with the wrong authors or venue, there is a broken “(?)” citation, and several references are never cited. Figures are unreadable, and some captions don’t match their axes. The compute claim (240 GPU-hours for the full grid) is implausible.
Questions for authors
  1. Please give a proof or empirical check of submodularity for the f you actually optimize, and correct the knapsack guarantee. Why not enumerate all 512 subsets exactly?
  2. Where do the verifier’s evidence-support labels come from? Is the verifier updated during GRPO?
  3. Please add accuracy-only GRPO and tool-equipped baselines on the same data, and calibrate baseline confidences before comparing ECE.
  4. please report results on all backbones in a table with standard deviations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.