- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CoMPR selects perception tools under a budget, checks the resulting answer with a calibrated verifier, and uses reflection to revise the answer or gather more evidence. The authors also train the policy with GRPO using rewards for correctness, evidence citation, calibration, and cost. On Qwen2.5-VL-7B, Table 2 reports average accuracy of 81.02% for CoMPR and 82.06% for CoMPR+RL, compared with 78.40% for CRITIC. The paper also evaluates the approach on ALFWorld and MMSearch.
Critical strengths
-
The method addresses a practical problem: an agent needs to decide both what evidence to collect and when it has collected enough. I find the connection between reflection and tool selection useful. A failed check can trigger another observation instead of merely asking the model to rewrite its answer.
-
The reported gains are consistent across the five QA datasets in Table 2. Table 3 also shows lower token use and latency than CRITIC. These results are encouraging, and the component ablations help explain which parts of the system contribute.
Critical weaknesses
-
The planner’s theoretical guarantee is not justified. Section 3.2 argues that the objective is submodular because each tool addresses one facet of the answer. Tools can instead be complementary: reading an axis label may make an extracted chart value more useful. The paper does not give assumptions that rule this out. It also mentions both partial enumeration and a gain-per-cost greedy rule without specifying the algorithm sufficiently to establish the claimed (1-1/e) bound. This matters because the guarantee is presented as a main contribution.
-
The reported improvements are difficult to reconcile, and key implementation details are missing. The abstract claims a +5.0 accuracy gain over the strongest baseline, while Table 2 gives +3.66. The comparison behind the claimed 17% latency reduction is unclear. Table 3 calls CoMPR+RL the most calibrated method, although CoMPR has lower ECE. I also could not determine how tool utilities and evidence-support labels are obtained. Together, these issues make it difficult to assess what was implemented and how much it improves over the baselines.
Questions for authors
-
What assumptions ensure that the planner’s objective is submodular? Please specify the implemented selection algorithm and explain whether its guarantee applies to the learned utility estimates.
-
How are tool utilities predicted before execution, and how are the verifier’s evidence-support labels produced? Which components are trained, including in the GPT-4o experiments?
-
Which results should be treated as authoritative where the abstract and tables disagree? Please also explain the relationship between the 1,000 ALFWorld trajectories in Section 4.1 and the 134 unseen tasks in Figure 6.
-
The verifier is trained to predict evidence support, but the calibration reward uses answer correctness. Why are these treated as the same target? Since absolute-error rewards do not generally favor calibrated probabilities, what supports this choice?