- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
MACR coordinates heterogeneous multimodal models by letting them continue a shared reasoning trajectory. It standardizes local mean token log probabilities using model-specific historical statistics, selects an initial model through short trial generations, and triggers new bidding rounds when the active model's confidence falls below a threshold. The winning model's continuation is appended to the shared context. Experiments with three model families on MathVista, LogicVista, MMStar, and HallusionBench show improvements over the evaluated single-model strategies and lower reported token usage than the multi-agent baselines. Ablations examine initialization, model selection, trigger timing, and model heterogeneity, while additional analyses study the confidence signal and relay patterns.
Critical strengths
- A useful accuracy–token tradeoff. On MMStar and HallusionBench, MACR uses approximately one-sixth of H-DMAD's reported tokens, with accuracy lower by 1.20 and 0.84 percentage points, respectively. It also reports higher mean accuracy and fewer tokens than majority voting on all four benchmarks.
- The ablations support the main design choices. On the MMStar subset, fixed selection and fixed triggering reduce accuracy from 72.4% to 69.0% and 69.8%, respectively. The study also reports disjoint parameter tuning and three-seed evaluations.
Critical weaknesses
- The methodological novelty is limited. The paper replaces the parallel MAC topology with a sequential one and adds a confidence-based selection mechanism, but this does not appear to constitute a fundamental methodological advance. Many existing approaches already use sequential collaboration or integrate agents’ answers based on confidence.
- Figure 3 does not clearly convey the proposed method. It presents a basic workflow for selecting the next agent, but provides limited information about the method’s key components. In particular, it is unclear whether the “Scale Calibration” module is part of the proposed method, since it appears inconsistent with Figure 2.
- Algorithm 1 largely repeats the method’s workflow. From a presentation perspective, including it in the main text seems unnecessary, especially given the substantial space it occupies.
- The design of the main results table is difficult to interpret. Since the paper proposes a MAC method, its effectiveness should be demonstrated primarily through comparisons with other MAC approaches. A substantial portion of the baselines are single-model methods. Although the proposed method achieves a high IBC, its accuracy gains over the baselines appear modest.
Questions for authors
Just see weaknesses.
2 comments
Response to Reviewer wfKu (2/2)
Q4: Lack of MAC baselines.
Thanks for your valuable comment!
We added four new MAC baselines:
- Diverse debate (H-DMAD): Agents debate using different reasoning strategies to break fixed mental sets. [ref4]
- Memory-masked debate (MAD-M2): Agents collaborate while masking out likely-wrong responses before the next round. [ref5]
- Graph-based collaboration (GoA): Multi-agent communication structured as a graph for collaborative problem-solving. [ref6]
- Critic-and-refine framework (Table-Critic): A multi-agent framework for collaborative criticism and iterative refinement. [ref2]
Table R1: Performance of MLLMs in terms of accuracy (Acc, %), token overhead (Tok) and IBC ().
| Method | MathVista (Acc / Tok / IBC) | LogicVista | MMStar | Hallusion |
|---|---|---|---|---|
| Best Single-Model | 74.70 / 1,736 / 13.85 | 54.24 / 1,564 / 3.22 | 68.13 / 2,155 / 9.70 | 73.61 / 1,424 / 6.32 |
| Majority Vote (MV) | 77.80 / 7,699 / 7.80 | 55.13 / 7,943 / 4.54 | 71.00 / 7,508 / 7.19 | 74.13 / 8,072 / 2.45 |
| H-DMAD [ref4] | 78.80 / 23,780 / 2.63 | 55.58 / 27,574 / 1.27 | 73.20 / 24,677 / 2.83 | 75.50 / 26,375 / 1.19 |
| MAD-M2 [ref5] | 79.20 / 25,294 / 2.60 | 57.60 / 27,672 / 1.89 | 71.33 / 22,446 / 1.92 | 74.24 / 21,574 / 0.49 |
| GoA [ref6] | 74.3/14,893/0.84 | 52.46/14,002/0.12 | 68.2/12,795/1.40 | 75.71/14,839/2.59 |
| Table-Critic [ref2] | 76.30/6,365/6.81 | 53.14/6,385/1.71 | 71.53/5,642/11.40 | 74.45/5,939/4.11 |
| MACR (Ours) | 78.00 / 4,811 / 15.38 | 55.80 / 6,149 / 7.89 | 72.00 / 4,150 / 21.35 | 74.66 / 4,414 / 7.71 |
MACR outperforms MV, GoA, and Table-Critic in both accuracy and IBC, and remains competitive with H-DMAD and MAD-M2. Notably, MACR achieves the highest IBC across all benchmarks while requiring fewer tokens than these methods (e.g., 4,150 vs. 24,677 tokens on MMStar).
We will include the results in the revision.
To demonstrate the generalizability of MACR, we further evaluate it on text-only LLMs (Llama3.1-8B-Instruct, Qwen3-8B, MiniCPM4-8B) using CommonsenseQA [ref7], ARC-Challenge [ref8] and MMLU-Pro [ref9] (500 randomly sampled instances per dataset). IBC is calculated using L2M as the baseline:
Table R2: Performance of LLMs in terms of accuracy (Acc, %), token overhead (Tok) and IBC ().
| Benchmarks | Methods | Llama3.1-8B-Instruct (Acc/Tok/IBC) | Qwen3-8B | MiniCPM4-8B |
|---|---|---|---|---|
| CommonsenseQA [ref7] | L2M [ref10] | 71.40 / 847 | 78.60 / 618 | 72.80 / 625 |
| CoT [ref11] | 79.00 / 340 | 79.80 / 357 | 74.40 / 245 | |
| SBP [ref12] | 75.80 / 815 | 79.80 / 663 | 73.20 / 818 | |
| DMAD | 81.60 / 8659/9.21 | |||
| MACR | 82.00 / 1450/102.65 | |||
| ARC-Challenge [ref8] | L2M | 84.60 / 985 | 93.80 / 702 | 90.00 / 691 |
| CoT | 88.00 / 392 | 93.80 / 420 | 90.60 / 271 | |
| SBP | 85.00 / 777 | 93.00 / 709 | 91.20 / 669 | |
| DMAD | 96.00 / 8688/8.27 | |||
| MACR | 95.80 / 1584/80.03 | |||
| MMLU-Pro [ref9] | L2M | 47.20 / 1666 | 63.40 / 1173 | 57.40 / 1989 |
| CoT | 51.60 / 868 | 69.80 / 861 | 60.60 / 783 | |
| SBP | 50.00 / 1354 | 69.80 / 1172 | 53.40 / 1320 | |
| DMAD | 72.00 / 16980/10.41 | |||
| MACR | 72.00 / 2697/147.10 |
MACR outperforms L2M, CoT, and SBP in both accuracy and IBC across text-only benchmarks. Compared with DMAD, MACR achieves comparable or higher accuracy with up to higher IBC (e.g., 147.10 vs. 10.41 on MMLU-Pro), while requiring fewer tokens.
We will include the results in the revision.
[ref1] MetaGPT: Meta programming for a multi-agent collaborative framework, ICLR 2024
[ref2] Table-critic: A multi-agent framework for collaborative criticism and refinement in table reasoning, ACL 2025
[ref3] Dynamic Weighted Consensus Framework for LLM Multi-agent Debate, KSEM 2025
[ref4] Breaking Mental Set to Improve Reasoning through Diverse Multi-Agent Debate, ICLR 2025
[ref5] Multi-Agent Debate with Memory Masking, ICLR 2026
[ref6] Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration, ICLR 2026
[ref7] Commonsenseqa: A question answering challenge targeting commonsense knowledge, ACL 2019
[ref8] Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
[ref9] Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, NeurIPS 2024
[ref10] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR 2026
[ref11] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022
[ref12] Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models, ICLR 2024
Response to Reviewer wfKu (1/2)
We thank reviewer wfKu for the valuable time and constructive feedback. We provide point-to-point response below.
Q1: Clarification of our contributions and novelty.
Good question!
We clarify the differences between existing sequential/confidence methods and MACR:
Existing sequential pipelines (e.g., MetaGPT [ref1] or iterative critique-and-refine [ref2]) execute predefined roles across fixed turns, while confidence-integration methods [ref3] assume raw confidence scores across heterogeneous models are comparable. Both operate at the complete response (turn-level) granularity.
As stated in lines 82-88, our contributions are:
Q2: Clarity of Figure 3 and consistency.
Thanks for your feedback!
Figure 3 is designed as an overview of the framework. To maintain conciseness while providing access to key components, we embedded clickable references to specific sections and equations directly within the figure. Furthermore, the calibration module is a core component of our proposed framework, which is why it is explicitly introduced in Section 2.1 under "Methodology" (lines 130-179).
Regarding the visual consistency, the figures serve different purposes: Figure 2 demonstrates the statistical effect of calibration (density shifts before and after), whereas Figure 3 (Calibration part) illustrates the operational process of this calibration.
However, we fully agree that the overview diagram should be clear and consistent. We will redraw the overview more clearly in the revision.
Q3: Location of Algorithm 1.
Good suggestion!
We agree that Algorithm 1 occupies valuable space in the main text. We will move it to Appendix and use the freed space to integrate our new experiments into the revised manuscript.