- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a framework for multimodal multi-agent debate that combines dynamic resolution assignment with uncertainty-guided answer rollback. It uses token-level uncertainty to select visual resolutions and recover earlier responses during debate. Experiments on several multimodal benchmarks report improvements over existing debate frameworks.
Critical strengths
The framework is training-free and can be integrated into existing debate pipelines. The evaluation covers multiple models and multimodal reasoning benchmarks.
Critical weaknesses
-
The methodological novelty is rather limited. Both resolution assignment and answer rollback rely on relatively simple heuristics. More importantly, the assumption that lower token-level uncertainty indicates better visual grounding or more reliable answers is not sufficiently validated.
-
The accuracy improvements are relatively modest and come with considerable additional token consumption. Without compute-matched baselines, it is difficult to determine whether the gains come from the proposed mechanisms or simply from additional inference. Some results also contradict the claimed universal improvements.
Questions for authors
-
How strongly does ANLL correlate with actual answer correctness across different visual resolutions?
-
How often does the rollback mechanism restore incorrect answers or discard valid final predictions?
-
How does DREAM compare with stronger baselines under matched inference budgets?
We thank the reviewer. We respond to each question in turn.
ANLL and correctness across resolutions (Q1). ANLL is validated end to end in the paper. Average accuracy increases at each step of the component ablation (DRA, then URA with the adaptive trigger, then URA with rollback), and the adaptive threshold outperforms the best tuned static threshold on every model (50.20 vs. 49.74). In the revision we will add the AUROC of ANLL against correctness per resolution and benchmark, the accuracy of agents inside versus outside the confident set, and a comparison of ANLL values across the three resolutions.
Rollback errors (Q2). The module ablation shows that adding URA with rollback raises average accuracy, and the sensitivity sweep peaks at alpha = 0.5. In the revision we will report, over debates in which rollback was triggered, the number of cases in which it restored a correct answer, restored an incorrect one, or had no effect on the final answer.
Matched inference budgets (Q3). On Qwen2.5-VL-7B, DREAM requires roughly 10% (DMAD) to 30% (LLM Debate) more tokens for gains of +2.2 to +3.2 points. We will add a matched-budget comparison under the main configuration.
Gains and generality. DREAM improves average accuracy in all 12 model-by-backbone combinations (four models, three backbones), by +0.9 to +3.2 points. On LLaVA-Next-7B (appendix), the corrected differences are +0.9 (LLM Debate), +1.5 (DMAD) and -0.2 (Div-MAD). We have scoped the wording in Sec. 5 and in the introduction to the settings the tables cover. The reported figures are means over three seeds, and we will add standard deviations and a paired significance test.
Novelty. DREAM is training-free and lightweight by design, and its contribution lies in what it varies. The closest multimodal debate work keeps the visual input fixed: Perceiver isolates perception in a dedicated agent, and M3MAD-Bench standardizes multimodal debate and documents its token inefficiency. In both, all agents receive the same visual input. DREAM instead treats resolution as a per-agent, per-sample decision made from a probe round, and uses each agent’s uncertainty trajectory to restore earlier answers. The ablations above evaluate each of these components.