acceptodds
Withdrawn from ICLR 2027

DREAM: Dynamic Resolution Assignment Multimodal Multi-Agent Debate

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-agent debate (MAD) has emerged as an effective paradigm to improve the reasoning capabilities of large language models (LLMs) and is increasingly being extended to multimodal settings. However, existing multimodal MAD frameworks typically expose agents to the same fixed visual input, ignoring substantial variation in the visual scale needed across samples and agents. In addition, these frameworks frequently suffer from groupthink—a phenomenon where agents prematurely abandon correct deductions to conform with confident but hallucinated peer responses. To address these bottlenecks, we introduce DREAM (Dynamic Resolution Assignment For Multimodal Multi-Agent Debate), which operates via two core components: (1) Dynamic Resolution Assignment, a zero-shot “probe round” where agents test multiple resolutions, quantify uncertainty using Average Normalized Log-Likelihood (ANLL), and use an adaptive threshold to assign each agent to its empirically optimal resolution; (2) Uncertainty-Guided Rollback Aggregation counters groupthink by tracking each agent’s uncertainty over rounds and restoring early low-uncertainty answers overridden by group pressure. On six multimodal datasets, DREAM improves the accuracy–token trade-off over multi-agent debate baselines by 1.5–3.2% accuracy without dataset-specific tuning.

Primary area:
interpretability and explainable AI
Keywords:
LLMs, MLLMs, multi-agent, debate

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_acF6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a framework for multimodal multi-agent debate that combines dynamic resolution assignment with uncertainty-guided answer rollback. It uses token-level uncertainty to select visual resolutions and recover earlier responses during debate. Experiments on several multimodal benchmarks report improvements over existing debate frameworks.

Critical strengths

The framework is training-free and can be integrated into existing debate pipelines. The evaluation covers multiple models and multimodal reasoning benchmarks.

Critical weaknesses
  1. The methodological novelty is rather limited. Both resolution assignment and answer rollback rely on relatively simple heuristics. More importantly, the assumption that lower token-level uncertainty indicates better visual grounding or more reliable answers is not sufficiently validated.

  2. The accuracy improvements are relatively modest and come with considerable additional token consumption. Without compute-matched baselines, it is difficult to determine whether the gains come from the proposed mechanisms or simply from additional inference. Some results also contradict the claimed universal improvements.

Questions for authors
  1. How strongly does ANLL correlate with actual answer correctness across different visual resolutions?

  2. How often does the rollback mechanism restore incorrect answers or discard valid final predictions?

  3. How does DREAM compare with stronger baselines under matched inference budgets?

1 comment

We thank the reviewer. We respond to each question in turn.

ANLL and correctness across resolutions (Q1). ANLL is validated end to end in the paper. Average accuracy increases at each step of the component ablation (DRA, then URA with the adaptive trigger, then URA with rollback), and the adaptive threshold outperforms the best tuned static threshold on every model (50.20 vs. 49.74). In the revision we will add the AUROC of ANLL against correctness per resolution and benchmark, the accuracy of agents inside versus outside the confident set, and a comparison of ANLL values across the three resolutions.

Rollback errors (Q2). The module ablation shows that adding URA with rollback raises average accuracy, and the sensitivity sweep peaks at alpha = 0.5. In the revision we will report, over debates in which rollback was triggered, the number of cases in which it restored a correct answer, restored an incorrect one, or had no effect on the final answer.

Matched inference budgets (Q3). On Qwen2.5-VL-7B, DREAM requires roughly 10% (DMAD) to 30% (LLM Debate) more tokens for gains of +2.2 to +3.2 points. We will add a matched-budget comparison under the main configuration.

Gains and generality. DREAM improves average accuracy in all 12 model-by-backbone combinations (four models, three backbones), by +0.9 to +3.2 points. On LLaVA-Next-7B (appendix), the corrected differences are +0.9 (LLM Debate), +1.5 (DMAD) and -0.2 (Div-MAD). We have scoped the wording in Sec. 5 and in the introduction to the settings the tables cover. The reported figures are means over three seeds, and we will add standard deviations and a paired significance test.

Novelty. DREAM is training-free and lightweight by design, and its contribution lies in what it varies. The closest multimodal debate work keeps the visual input fixed: Perceiver isolates perception in a dedicated agent, and M3MAD-Bench standardizes multimodal debate and documents its token inefficiency. In both, all agents receive the same visual input. DREAM instead treats resolution as a per-agent, per-sample decision made from a probe round, and uses each agent’s uncertainty trajectory to restore earlier answers. The ablations above evaluate each of these components.

Reviewer_nNbTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes DREAM, which uses generation uncertainty to dynamically select image resolutions and selectively restore earlier high-confidence answers, aiming to mitigate erroneous consensus in multimodal multi-agent debate. The framework is evaluated across multiple models and six multimodal benchmarks, with reported accuracy improvements accompanied by additional token costs.

Critical strengths
  1. The problem is practically relevant. The paper addresses resolution selection and erroneous consensus in multimodal debate, connecting visual input configuration with answer aggregation through a clear practical motivation.

  2. The design is lightweight and modular. It uses token probabilities already available during generation to select resolutions and restore earlier answers, requiring no additional training and potentially supporting different models and debate frameworks.

Critical weaknesses
  1. The routing rule contradicts the reported branch statistics. With three agents, the μ-σ threshold cannot select exactly two confident agents, yet the experiments report frequent activation of a branch requiring this condition.

  2. The reported averages contain numerical inconsistencies. In Table 5, some averages do not match the individual benchmark scores; for example, an average of 30.83 is reported as 30.1, undermining confidence in the results.

  3. A key baseline is mischaracterized. The paper describes the Perceiver baseline as compressing images into a fixed number of latent tokens, although it communicates visual information through textual dialogue, making the associated performance attribution unsupported.

  4. The efficiency claim lacks budget-matched comparisons. DREAM increases total token consumption by approximately 45%, but comparisons against equally costly additional sampling or debate rounds are missing, leaving the contribution of extra computation unresolved.

Questions for authors
  1. How can SubsetStaticMixed occur when the stated μ-σ rule cannot select exactly two confident agents out of three?

  2. Can you reconcile the inconsistent averages in Table 5 and aggregate accuracy in Figures 5–6, and provide per-sample results and aggregation code for verification?

1 comment

We thank the reviewer for a careful and precise reading. The points raised have been addressed as follows.

Routing rule and SubsetStaticMixed (Q1). Sec. 3.2 describes a static threshold (default 0.2) alongside the adaptive rule and a fallback, and it did not specify how they interact or how ties are handled. We will state the exact rule once, use it consistently in Sec. 3.2, Algorithm 1 and the appendix with unified notation for tau and C, and label each branch figure with the rule that produced it. The threshold choice is validated end to end: the adaptive rule outperforms the best static threshold by +0.46 on average and on every model, and the pooled branch accuracy (50.2) matches the alpha = 0.5 row of the sensitivity sweep.

Table 5 averages. We thank the reviewer for identifying this. Recomputed from the six benchmark cells, the averages are 30.83 for LLM Debate w/ DREAM (previously 30.1) and 24.12 for Div-MAD w/ DREAM (previously 23.6). Relative to the bare baselines (29.97 and 24.33), the corrected differences are +0.9 for LLM Debate (previously +0.1), -0.2 for Div-MAD (previously -0.7) and +1.5 for DMAD (unchanged). We also rechecked the Avg. cell of every row in the main, generalization and appendix tables against its six benchmark cells and found no other mismatch.

Perceiver. Huang et al. use a dedicated perceiver agent that relays visual information to the debaters through dialogue. We have corrected the description in the results section accordingly, and we removed the explanation of its accuracy that referred to spatial compression. The reported Perceiver numbers are unchanged.

Budget-matched comparison. DREAM is presented as a trade-off (roughly +10% to +30% tokens for +2.2 to +3.2 points on Qwen2.5-VL-7B). We will add a matched-cost comparison under the Table 1 configuration.

Per-sample results and code (Q2). We will extend this to the per-sample outputs and the aggregation script, so that every Avg. cell and branch statistic can be recomputed.

Reviewer_EZxYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose DREAM, it adds two components to multimodal debate. A preliminary probe uses response log-likelihood to choose image resolutions, and a rollback rule replaces an agent's final answer with an earlier answer, when its uncertainty has increased enough. The paper evaluates these additions across several models, debate backbones, and several benchmarks.

Critical strengths
  1. Adjusting image resolution to the question makes sense, some questions need only a broad view, while others depend on fine details. The method also works without retraining the model.

  2. The method is tested on several model families and three debate frameworks. The average gains across these settings provide stronger evidence than results from a single model or task.

  3. The ablations, threshold tests, and breakdown of routing costs help show how the method works beyond the overall accuracy numbers.

Critical weaknesses
  1. Each agent is probed at only one resolution, so the effects of agent role and image scale are hard to separate. Rotating the scale assignments or testing each agent at multiple scales would help show whether the selected resolution suits the question.

  2. The accuracy gains come with higher token costs. A fairer comparison would give the baselines the same total budget for additional samples, rounds, or higher-resolution inputs. Costs should include probing, generation, and judging, alongside visual and text tokens.

  3. The adaptive threshold does not seem consistent with the reported branch frequencies. With three agents and nonzero variance, at most one ANLL score can fall below mean minus standard deviation. The paper needs to clarify which threshold and probe setup produced the results.

Questions for authors

see weakness

1 comment

We thank the reviewer for the constructive review and address the three points in turn.

Threshold and branch frequencies. Sec. 3.2 presents a static default (0.2), the adaptive rule and a fallback without specifying how ties and the fallback interact. We will state a single rule used throughout and label branch figures with it. The headline results are unaffected: the adaptive rule outperforms the best tuned static threshold on average (50.20 vs. 49.74) and on every model.

Matched total budget. The paper frames DREAM as an accuracy-versus-token trade-off: on Qwen2.5-VL-7B it adds roughly 10% to 30% more tokens for gains of +2.2 to +3.2 points. We will add the matched-budget comparison under the main configuration.

Agent role versus image scale. The fixed-resolution ablation (0.5x: 49.74; 2x: 49.52; 4x: 49.94) shows that image scale has a measurable effect on accuracy. To separate scale from agent identity directly, we will add a rotation of the probe-resolution assignment across agents and compare the routed choices against the best fixed resolution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.