acceptodds
Withdrawn from ICLR 2027

MACR: Multi-Agent Confidence-Based Bidding Relay for Collaborative Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing multi-agent collaboration (MAC) methods have primarily explored a group thinking collaboration topology, where agents independently generate solutions and aggregate their outputs. In this work, we investigate an alternative relay thinking topology, where reasoning ownership is dynamically transferred across agents, allowing subsequent agents to inherit and refine an evolving reasoning trajectory. We introduce Multi-Agent Confidence-guided Relay (MACR), a framework that enables dynamical relay collaboration among agents. The key challenge is determining when reasoning should be relayed and which agent should continue the process. MACR addresses this challenge through confidence-guided relay triggering and a bidding mechanism that selects the next agent based on its confidence. Experiments on four benchmarks show that MACR outperforms the strongest single-model strategies in accuracy, e.g., +3.3% on MathVista and +3.9% on MMStar, and remains on par with multi-agent strategies while consuming 6.0X less token overhead on HallusionBench and MMStar. Further analysis demonstrates that relay collaboration enables heterogeneous agents to progressively integrate complementary insights, combining heterogeneous capabilities across reasoning stages. Our source code is available at https://anonymous.4open.science/r/iclr2027macr-C822.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multimodal Large Language Models, Multi-agent Collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wfKuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

MACR coordinates heterogeneous multimodal models by letting them continue a shared reasoning trajectory. It standardizes local mean token log probabilities using model-specific historical statistics, selects an initial model through short trial generations, and triggers new bidding rounds when the active model's confidence falls below a threshold. The winning model's continuation is appended to the shared context. Experiments with three model families on MathVista, LogicVista, MMStar, and HallusionBench show improvements over the evaluated single-model strategies and lower reported token usage than the multi-agent baselines. Ablations examine initialization, model selection, trigger timing, and model heterogeneity, while additional analyses study the confidence signal and relay patterns.

Critical strengths
  1. A useful accuracy–token tradeoff. On MMStar and HallusionBench, MACR uses approximately one-sixth of H-DMAD's reported tokens, with accuracy lower by 1.20 and 0.84 percentage points, respectively. It also reports higher mean accuracy and fewer tokens than majority voting on all four benchmarks.
  2. The ablations support the main design choices. On the MMStar subset, fixed selection and fixed triggering reduce accuracy from 72.4% to 69.0% and 69.8%, respectively. The study also reports disjoint parameter tuning and three-seed evaluations.
Critical weaknesses
  1. The methodological novelty is limited. The paper replaces the parallel MAC topology with a sequential one and adds a confidence-based selection mechanism, but this does not appear to constitute a fundamental methodological advance. Many existing approaches already use sequential collaboration or integrate agents’ answers based on confidence.
  2. Figure 3 does not clearly convey the proposed method. It presents a basic workflow for selecting the next agent, but provides limited information about the method’s key components. In particular, it is unclear whether the “Scale Calibration” module is part of the proposed method, since it appears inconsistent with Figure 2.
  3. Algorithm 1 largely repeats the method’s workflow. From a presentation perspective, including it in the main text seems unnecessary, especially given the substantial space it occupies.
  4. The design of the main results table is difficult to interpret. Since the paper proposes a MAC method, its effectiveness should be demonstrated primarily through comparisons with other MAC approaches. A substantial portion of the baselines are single-model methods. Although the proposed method achieves a high IBC, its accuracy gains over the baselines appear modest.
Questions for authors

Just see weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer wfKu (1/2)

We thank reviewer wfKu for the valuable time and constructive feedback. We provide point-to-point response below.

Q1: Clarification of our contributions and novelty.

Good question!

We clarify the differences between existing sequential/confidence methods and MACR:

Existing sequential pipelines (e.g., MetaGPT [ref1] or iterative critique-and-refine [ref2]) execute predefined roles across fixed turns, while confidence-integration methods [ref3] assume raw confidence scores across heterogeneous models are comparable. Both operate at the complete response (turn-level) granularity.

As stated in lines 82-88, our contributions are:

  1. Relay thinking framework: Drawing on cognitive relay thinking, we formulate dynamic ownership transfer along a single shared reasoning chain.
  2. Window-level granularity: We shift collaboration from turn-level to intra-trajectory window-level, interrupting bottlenecks mid-generation to avoid wasting tokens on completing responses and letting the path drift further.
  3. Cross-model confidence calibration: We are THE FIRST to reveal that local confidence across heterogeneous models is mathematically incomparable (ANOVA F=13.31F=13.31), and provide an effective calibration method (reducing FF to 1.011.01).
  4. Insights on heterogeneous collaboration: We uncover that models exhibit complementary strengths across reasoning stages (Fig. 6b), while reasoning bottlenecks emerge at different stages across tasks (Fig.6c).
  5. Effectiveness and efficiency: MACR reduces token consumption by up to 6.0×6.0\times and achieves higher IBC than turn-level MAC baselines (Table 1, Table R1 and Table R2).

Q2: Clarity of Figure 3 and consistency.

Thanks for your feedback!

Figure 3 is designed as an overview of the framework. To maintain conciseness while providing access to key components, we embedded clickable references to specific sections and equations directly within the figure. Furthermore, the calibration module is a core component of our proposed framework, which is why it is explicitly introduced in Section 2.1 under "Methodology" (lines 130-179).

Regarding the visual consistency, the figures serve different purposes: Figure 2 demonstrates the statistical effect of calibration (density shifts before and after), whereas Figure 3 (Calibration part) illustrates the operational process of this calibration.

However, we fully agree that the overview diagram should be clear and consistent. We will redraw the overview more clearly in the revision.


Q3: Location of Algorithm 1.

Good suggestion!

We agree that Algorithm 1 occupies valuable space in the main text. We will move it to Appendix and use the freed space to integrate our new experiments into the revised manuscript.


AuthorsOpenReview ↗

Response to Reviewer wfKu (2/2)

Q4: Lack of MAC baselines.

Thanks for your valuable comment!

We added four new MAC baselines:

  1. Diverse debate (H-DMAD): Agents debate using different reasoning strategies to break fixed mental sets. [ref4]
  2. Memory-masked debate (MAD-M2): Agents collaborate while masking out likely-wrong responses before the next round. [ref5]
  3. Graph-based collaboration (GoA): Multi-agent communication structured as a graph for collaborative problem-solving. [ref6]
  4. Critic-and-refine framework (Table-Critic): A multi-agent framework for collaborative criticism and iterative refinement. [ref2]

Table R1: Performance of MLLMs in terms of accuracy (Acc, %), token overhead (Tok) and IBC (×10−6\times10^{−6}).

MethodMathVista (Acc / Tok / IBC)LogicVistaMMStarHallusion
Best Single-Model74.70 / 1,736 / 13.8554.24 / 1,564 / 3.2268.13 / 2,155 / 9.7073.61 / 1,424 / 6.32
Majority Vote (MV)77.80 / 7,699 / 7.8055.13 / 7,943 / 4.5471.00 / 7,508 / 7.1974.13 / 8,072 / 2.45
H-DMAD [ref4]78.80 / 23,780 / 2.6355.58 / 27,574 / 1.2773.20 / 24,677 / 2.8375.50 / 26,375 / 1.19
MAD-M2 [ref5]79.20 / 25,294 / 2.6057.60 / 27,672 / 1.8971.33 / 22,446 / 1.9274.24 / 21,574 / 0.49
GoA [ref6]74.3/14,893/0.8452.46/14,002/0.1268.2/12,795/1.4075.71/14,839/2.59
Table-Critic [ref2]76.30/6,365/6.8153.14/6,385/1.7171.53/5,642/11.4074.45/5,939/4.11
MACR (Ours)78.00 / 4,811 / 15.3855.80 / 6,149 / 7.8972.00 / 4,150 / 21.3574.66 / 4,414 / 7.71

MACR outperforms MV, GoA, and Table-Critic in both accuracy and IBC, and remains competitive with H-DMAD and MAD-M2. Notably, MACR achieves the highest IBC across all benchmarks while requiring 6.0×6.0\times fewer tokens than these methods (e.g., 4,150 vs. 24,677 tokens on MMStar).

We will include the results in the revision.

To demonstrate the generalizability of MACR, we further evaluate it on text-only LLMs (Llama3.1-8B-Instruct, Qwen3-8B, MiniCPM4-8B) using CommonsenseQA [ref7], ARC-Challenge [ref8] and MMLU-Pro [ref9] (500 randomly sampled instances per dataset). IBC is calculated using L2M as the baseline:

Table R2: Performance of LLMs in terms of accuracy (Acc, %), token overhead (Tok) and IBC (×10−6\times10^{−6}).

BenchmarksMethodsLlama3.1-8B-Instruct (Acc/Tok/IBC)Qwen3-8BMiniCPM4-8B
CommonsenseQA [ref7]L2M [ref10]71.40 / 84778.60 / 61872.80 / 625
CoT [ref11]79.00 / 34079.80 / 35774.40 / 245
SBP [ref12]75.80 / 81579.80 / 66373.20 / 818
DMAD81.60 / 8659/9.21
MACR82.00 / 1450/102.65
ARC-Challenge [ref8]L2M84.60 / 98593.80 / 70290.00 / 691
CoT88.00 / 39293.80 / 42090.60 / 271
SBP85.00 / 77793.00 / 70991.20 / 669
DMAD96.00 / 8688/8.27
MACR95.80 / 1584/80.03
MMLU-Pro [ref9]L2M47.20 / 166663.40 / 117357.40 / 1989
CoT51.60 / 86869.80 / 86160.60 / 783
SBP50.00 / 135469.80 / 117253.40 / 1320
DMAD72.00 / 16980/10.41
MACR72.00 / 2697/147.10

MACR outperforms L2M, CoT, and SBP in both accuracy and IBC across text-only benchmarks. Compared with DMAD, MACR achieves comparable or higher accuracy with up to 14.1×14.1\times higher IBC (e.g., 147.10 vs. 10.41 on MMLU-Pro), while requiring 6.3×6.3\times fewer tokens.

We will include the results in the revision.

[ref1] MetaGPT: Meta programming for a multi-agent collaborative framework, ICLR 2024

[ref2] Table-critic: A multi-agent framework for collaborative criticism and refinement in table reasoning, ACL 2025

[ref3] Dynamic Weighted Consensus Framework for LLM Multi-agent Debate, KSEM 2025

[ref4] Breaking Mental Set to Improve Reasoning through Diverse Multi-Agent Debate, ICLR 2025

[ref5] Multi-Agent Debate with Memory Masking, ICLR 2026

[ref6] Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration, ICLR 2026

[ref7] Commonsenseqa: A question answering challenge targeting commonsense knowledge, ACL 2019

[ref8] Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

[ref9] Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, NeurIPS 2024

[ref10] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR 2026

[ref11] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022

[ref12] Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models, ICLR 2024

Reviewer_PrFvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MACR, a multi-agent relay framework that dynamically transfers reasoning between heterogeneous vision-language models. It uses calibrated local token confidence to determine when to trigger a handoff and which model should continue generation. Experiments on four multimodal benchmarks show improved accuracy over single-model baselines and comparable performance to multi-agent baselines with substantially lower reported token usage.

Critical strengths
  1. The paper explores dynamic reasoning handoffs as an alternative to independent generation and answer aggregation, offering a useful perspective on multi-agent collaboration.
  2. MACR combines confidence calibration, online bottleneck detection, and model selection without additional model training. The design is intuitive and supported by relevant ablation studies.
  3. MACR consistently outperforms the evaluated single-model baselines and achieves competitive accuracy with substantially fewer reported tokens than strong multi-agent baselines.
Critical weaknesses
  1. Limited validity of confidence-based routing. Token likelihood is not necessarily indicative of reasoning correctness. The substantial overlap between correct and incorrect trajectories, together with the modest AUROC values (0.59–0.63), raises concerns about its reliability for detecting reasoning bottlenecks and selecting the best continuation model.
  2. Insufficient evidence for model complementarity. The current experiments do not clearly demonstrate whether different models actually resolve reasoning bottlenecks that previous models could not overcome, or whether the improvements mainly come from inheriting a shared reasoning context. Furthermore, the lexical differences in Figure 7 may simply reflect differences in language styles or selection bias in the analyzed tokens, rather than model-specific weaknesses in reasoning capabilities.
  3. Limited calibration and hyperparameter robustness analysis. Confidence statistics are estimated on MMMU, while the window size and threshold are tuned on RealWorldQA. The stability of calibration across different benchmarks and relay-induced contexts is not sufficiently examined. Sensitivity analyses of the window size and threshold are also missing.
  4. Incomplete efficiency evaluation. The paper emphasizes token savings but does not report end-to-end latency or computational cost. It is also unclear how discarded bidding generations and cross-model context processing are accounted for in the reported token usage.
Questions for authors
  1. Starting from identical intermediate reasoning states, does confidence-guided handoff actually improve reasoning performance compared to allowing the current model to continue?
  2. How robust are the confidence calibration and routing decisions across different datasets? Could the authors provide sensitivity analyses for the sliding window size and confidence threshold?
  3. How is token consumption measured? Does it include discarded bidding tokens and cross-model context processing? Additionally, why does MACR's reported token usage on MMStar differ substantially between Table 1 (4,150) and Table 2 (312)? Could the authors clarify whether this difference arises from the evaluation scope, token accounting, or implementation settings?
3 comments
AuthorsOpenReview ↗

Response to Reviewer PrFv (1/3)

We thank reviewer PrFv for the valuable time and constructive feedback. We provide point-to-point response below.


Q1: Validity of confidence routing and model complementarity.

Thanks for your valuable feedback!

In appendix C, the 10-bin curves show a consistent trend: higher confidence correlates with higher accuracy, with accuracy rising from 37% in the lowest bin to 77% in the highest bin (line 918-930), indicating that confidence reliably reflects potential accuracy. Besides, we evaluate pairwise accuracy discrepancies within the triggering interval:

Table R1: Pairwise correctness discrepancies.

Model PairAcc. Diff. (%)Pointwise 95% CI (%)
Qwen3-VL / InternVL 3.5-2.82[-8.66, +3.24]
Qwen3-VL / MiniCPM-V 4.5+3.09[-3.35, +9.39]
InternVL 3.5 / MiniCPM-V 4.5+5.91[-0.92, +12.12]

All pairwise differences are within 6%, indicating that confidence and accuracy near the triggering threshold are comparable across heterogeneous models. This supports using calibrated confidence as a signal for bidding across heterogeneous models.

We also provide a complete relay case in Fig. S1 (line 810-854). In this case, the initial model (MiniCPM-V 4.5) identifies the relevant theorem but struggles with spatial localization, causing a confidence drop. The relay to Qwen3-VL resolves this bottleneck by identifying the alternate segment angle. Finally, InternVL 3.5 provides the visual confirmation of point A's position. This directly shows that different models contribute distinct capabilities at reasoning bottlenecks.

In addition, we conduct two sets of experiments on a randomly selected MMStar subset of 500 instances to verify the validity of confidence routing and model complementarity.

Exp 1: Direct rollout of different confidence-selected models. In these experiments, each selected model generates to completion without further relay.

  1. Breakpoint. We randomly sample intermediate bidding trigger points XtX_t. From each XtX_t, we roll out: (1) the current, (2) a random, (3) the lowest-confidence, (4) the second-lowest-confidence, and (5) the highest-confidence model. (Table R2)
  2. Initiation. Starting from the original input X0X_0, we roll out: (1) a random, (2) the lowest-confidence, (3) the second-lowest-confidence, and (4) the highest-confidence model. (Table R3)

Table R2: Breakpoint rollout results.

Rollout SelectionAcc. (%)
Current64.4
Random65.0
Lowest-confidence64.0
Second-lowest confidence65.6
Highest-confidence65.8
MACR67.4

Table R3: Initiation rollout results.

Rollout SelectionAcc. (%)
Random62.8
Lowest-confidence62.4
Second-lowest confidence64.0
Highest-confidence64.8
MACR67.4

Results show that the random and lowest-confidence selections perform similarly, and accuracy increases as confidence increases, with the highest-confidence selection outperforming all other rollouts. MACR, which combines confidence-guided initiation and relay, achieves the highest accuracy. This indicates that MACR's gain does not come from inheriting a shared context: all rollouts start from identical states, yet single-model rollouts underperform MACR, pointing to complementarity across models as the source of the improvement.

Exp 2: Full MACR relay with controlled relay rules. To isolate the effect of the relay mechanism, we evaluate the full relay pipeline, changing only the relay rule:

  1. Random relay.
  2. Lowest-confidence relay.
  3. Fixed-order relay.
  4. MACR (highest-confidence relay).

Table R4: Relay rule results.

Relay StrategyAcc. (%)
Random relay64.8
Lowest-confidence relay64.0
Fixed-order relay66.2
MACR (Highest-confidence relay)67.4

Results show that MACR performs best. Together, these results indicate that confidence-based bidding is effective for both initiating and continuing the relay, and that the resulting gains reflect complementarity across models.

We will include these results in the revision.

AuthorsOpenReview ↗

Response to Reviewer PrFv (2/3)

Q2: Model complementarity.

Thanks for your insight!

First, we provide a qualitative case study in the main text (line 302-308):

We compare MACR reasoning trajectories against those of single-model CoT. In cases where single-model CoT fails, single models anchor on isolated cues (e.g., local textures or lighting attributes), whereas MACR succeeds by synthesizing these fragmented cues into a more comprehensive global view. .….. This indicates that MACR enhances the generation quality by executing global contextual integration, even when the underlying model pool reaches its capability ceiling.

Besides, we provide a complete example in Fig. S1 (line 810-866). The initial model (MiniCPM-V 4.5) correctly identifies the relevant theorem but struggles with spatial localization, causing a confidence drop. The relay to Qwen3-VL resolves this bottleneck by identifying the alternate segment angle. Finally, InternVL 3.5 provides the visual confirmation of point A's position. This shows that different models contribute distinct capabilities at specific reasoning bottlenecks.

Second, as stated in lines 343-344:

…our analysis reveals that models excel at different stages of generation.

As shown by the Kaplan-Meier survival analysis (Fig. 6b), MiniCPM-V 4.5 shows the highest early survival, InternVL 3.5 maintains a plateau during later stages, whereas Qwen3-VL exhibits the poorest long-horizon retention.

Third, as stated in lines 397-411. Simulated role-play study (RPH) and temperature decoding (TDH) underperform MACR (72.4% vs. 64.8% and 63.6%). This confirms that the performance ceiling is raised by complementarity across different models, which cannot be replaced by single-model sampling or role-playing strategies.

Fourth, as reported in Table 1 (Breakpoint rollouts) and Table 2 (Initiation rollouts) in Q1, all single-model rollouts start from identical states. The fact that the highest-confidence selection outperforms the random and lowest-confidence selections directly rules out the shared-context explanation. Furthermore, MACR achieves the highest accuracy, surpassing the single highest-confidence rollout. All rollouts share the same starting states, so MACR's advantage over single-model rollouts reflects model complementarity, not inherited context.

We will emphasize this complementarity analysis in the revision.


Q3: Calibration stability.

Good suggestion!

We expand the evaluation of confidence calibration stability (ANOVA-FF) to MMStar and MathVista beyond MMMU, applying the same μm(w)\mu_m^{(w)} and σm(w)\sigma_m^{(w)} computed on MMMU:

Table R5: Confidence calibration stability tests.

BenchmarkRaw Window ANOVA-FFCalibrated ANOVA-FF
MMMU13.311.01
MMStar67.641.13
MathVista80.231.45

This confirms that our calibration remains stable across different benchmarks.

We will include the results in the revision.


Q4: Hyperparameter sensitivity.

Thanks for your feedback!

We now report the hyperparameter search results on RealWorldQA, and expand the search over window size ww and threshold τ\tau to MathVista (random 300 instances). We adopt a two-stage grid search: first fixing w=8w=8 to find the optimal τ\tau, then fixing the optimal τ\tau to find the optimal ww.

Table R6: Search for threshold τ\tau (window size w=8w=8).

τ\tau\AccuracyRealWorldQAMathVista
-0.568.0073.67
-0.468.374.00
-0.368.5074.67
-0.268.3073.67
-0.167.6773.33
0.067.1773.67
0.167.5072.67
0.266.6774.00
0.368.1073.33
0.467.0074.00
0.566.8372.67

Table R7: Search for window size ww (threshold τ=−0.3\tau = -0.3)

w\AccuracyRealWorldQAMathVista
w=4w=467.0074.33
w=8w=868.5074.67
w=16w=1669.1776.00
w=32w=3267.1772.67

These results indicate that the hyperparameters exhibit consistency across different benchmarks.

We will include these results in the revision.

AuthorsOpenReview ↗

Response to Reviewer PrFv (3/3)

Q5: Latency report.

Good suggestion!

We test average end-to-end latency (measured on 1,000 randomly sampled MMStar instances). Δ\Delta uses each model's own IO latency for single-model baselines, and the slowest model's (Qwen3-VL) IO latency as the baseline for MAC methods.

Table R8: Latency tests.

MethodLatency (s) ↓Δ\Delta (s) ↓
Qwen3-VL (IO)18.24\
Qwen3-VL (CCoT)20.28+2.04
Qwen3-VL (DDCoT)19.28+1.04
InternVL 3.5 (IO)12.97\
InternVL 3.5 (CCoT)27.37+14.4
InternVL 3.5 (DDCoT)20.44+7.47
MiniCPM-V 4.5 (IO)7.78\
MiniCPM-V 4.5 (CCoT)22.38+14.6
MiniCPM-V 4.5 (DDCoT)11.76+3.98
Majority Vote38.99+20.75
H-DMAD53.53+35.29
MACR18.56+0.32

We will add the results into our revision.


Q6: Clarification on token accounting.

Good question!

  1. Table 1 (4,150 tokens) reports the total token cost, comprising the full input/output tokens plus the discarded tokens.
  2. Table 2 (312 tokens) reported only the output tokens, including total bidding tokens but excluding input tokens.

We clarify that adopted tokens are directly concatenated to the trajectory without regeneration, while each model maintains its own KV cache to avoid repeated prefilling.

We report the raw token breakdown of main table:

Table R9: Token breakdown.

Benchmark & Method (Table 1)Adopted BidDiscarded BidTotal BidTotal Cost
MathVista (1,000)-MACR571141714,811
LogicVista (448)-MACR1012023036,149
HallusionBench (951)-MACR34681024,414
MMStar (1,500)-MACR42841264,150

We will add the results into our revision.

Reviewer_cxj3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MACR for relay thinking, an alternative to group thinking in which heterogeneous agents sequentially extend a shared reasoning trajectory. It analyzes cross-model confidence shifts and standardizes local scores to decide when to relay and which agent continues. Experiments show 3.3/3.9 percentage-point gains on MathVista/MMStar over the strongest single-model baselines, and competitive accuracy with up to 6× lower token usage than H-DMAD.

Critical strengths
  1. MACR provides an intuitive, training-free alternative to group thinking, with confidence guiding relay timing and model selection along a shared reasoning trajectory. This matters because relay-style collaboration is a less explored topology, and the approach requires no additional training.

  2. Results on four multimodal benchmarks show consistent gains over single-model baselines and competitive accuracy at lower token costs than H-DMAD. If the token accounting holds, this efficiency is the main practical argument for the method.

Critical weaknesses
  1. Cross-model candidate ranking is not directly validated. Appendix C does provide AUROC, reliability-curve, threshold-region, and ECE analyses; Table S1 reports AUROC from 0.5903 to 0.6343, with confidence intervals above chance. However, the described protocol labels each candidate window by the correctness of the resulting MACR trajectory. As described, candidates in a bidding round appear to share the executed trajectory's label unless separate candidate-specific rollouts were used. These analyses therefore do not directly establish that the selected continuation is better than the discarded alternatives. Candidate-level comparisons would address the main routing claim more directly.

  2. The empirical attribution and cost reporting need clarification. The paper does not clearly specify whether reported token usage includes all generated but discarded bidding candidates. Table 1 reports 4,150 tokens on the full 1,500-example MMStar set, whereas Table 2 reports 312 on a randomly sampled 500-example ablation subset; the different samples should be acknowledged, but the large difference still warrants an explanation of the accounting and settings. The FI/FS/FT ablations also do not replace an initialization-only baseline or budget-matched random relays. Finally, the 0.8-point difference between MACR and the best FI variant in Table 2 lacks uncertainty estimates, despite the main setup reporting three seeds.

Questions for authors
  1. Were discarded candidate windows labeled using the executed MACR trajectory, or did each candidate receive a separate completion rollout? Can you compare selected and discarded candidates within a bidding round?

  2. Does token accounting include every generated bidding candidate? What explains 4,150 tokens on the full MMStar set versus 312 on the 500-example ablation subset, and were decoding and cost-accounting settings identical?

  3. Can you add initialization-only routing and budget-matched random relay controls, and report uncertainty for the ablation gains?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cxj3 (1/2)

We thank reviewer cxj3 for the valuable time and constructive feedback. We provide point-to-point response below.


Q1: Lack of direct validation of candidate ranking.

Good suggestion!

We clarify that Appendix C only labels adopted windows (sampled 3 times per instance) for the 10-bin reliability curves and parity in the threshold region.

First, the 10-bin curves show a consistent trend: higher confidence correlates with higher accuracy, indicating that confidence reliably indicates potential accuracy. Second, we evaluate pairwise accuracy discrepancies within the triggering interval:

Table R1: Pairwise correctness discrepancies.

Model PairAcc. Diff. (%)Pointwise 95% CI (%)
Qwen3-VL / InternVL 3.5-2.82[-8.66, +3.24]
Qwen3-VL / MiniCPM-V 4.5+3.09[-3.35, +9.39]
InternVL 3.5 / MiniCPM-V 4.5+5.91[-0.92, +12.12]

All pairwise differences are within 6%, indicating that confidence and accuracy near the triggering threshold are comparable across heterogeneous models. Therefore, comparing cross-model confidence to rank candidates is reasonable.

We fully agree with you that comparing selected and discarded candidates is a more direct validation method. Accordingly, we conduct two experiments on a randomly selected MMStar subset of 500 instances. In both experiments, each selected model generates to completion, without any further relay.

  1. Breakpoint: We randomly sample intermediate bidding trigger points XtX_t. From each XtX_t, we roll out: (1) current, (2) a randomly selected, (3) the lowest-confidence, (4) the second-lowest-confidence, and (5) the highest-confidence model. (Table R2)
  2. Initiation: Starting from the original input X0X_0, we roll out: (1) a randomly selected, (2) the lowest-confidence, (3) the second-lowest-confidence, and (4) the highest-confidence initial model. (Table R3)

Table R2: Breakpoint rollout results.

Rollout SelectionAcc. (%)
Current64.4
Random65.0
Lowest-confidence64.0
Second-lowest confidence65.6
Highest-confidence65.8
MACR67.4

Table R3: Initiation rollout results.

Rollout SelectionAcc. (%)
Random62.8
Lowest-confidence62.4
Second-lowest confidence64.0
Highest-confidence64.8
MACR67.4

It shows that accuracy increases with candidate confidence, with the highest-confidence (65.8%) exceeding the second-lowest (65.6%) and lowest-confidence (64.0%). MACR performs best (67.4%).

We also record the window confidence and branch correctness across all rollouts to recompute the ANOVA-FF . The calibrated FF-value drops from 68.26 to 1.44 , confirming that three models are calibrated and comparable to each other.

We will add the clarification to Appendix C and include the new results in the revision.


Q2: Clarification on token accounting.

Good question!

  1. Table 1 (4,150 tokens) reports the total token cost, comprising the full input/output tokens plus the discarded tokens.
  2. Table 2 (312 tokens) reports only the output tokens, including total bidding tokens but excluding input tokens.

We clarify that adopted tokens are directly concatenated to the trajectory without regeneration, while each model maintains its own KV cache to avoid repeated prefilling.

We report the raw token breakdown of the main table:

Table R4: Token breakdown.

Benchmark & Method (Table 1)Adopted BidDiscarded BidTotal BidTotal Cost
MathVista (1,000)-MACR571141714,811
LogicVista (448)-MACR1012023036,149
HallusionBench (951)-MACR34681024,414
MMStar (1,500)-MACR42841264,150

We additionally report average end-to-end latency (measured on 1,000 randomly sampled MMStar instances). Δ\Delta uses each model's own IO latency for single-model baselines, and the slowest model's (Qwen3-VL) IO latency as the baseline for MAC methods.

Table R5: Latency tests.

MethodLatency (s) ↓Δ\Delta (s) ↓
Qwen3-VL (IO)18.24\
Qwen3-VL (CCoT)20.28+2.04
Qwen3-VL (DDCoT)19.28+1.04
InternVL 3.5 (IO)12.97\
InternVL 3.5 (CCoT)27.37+14.4
InternVL 3.5 (DDCoT)20.44+7.47
MiniCPM-V 4.5 (IO)7.78\
MiniCPM-V 4.5 (CCoT)22.38+14.6
MiniCPM-V 4.5 (DDCoT)11.76+3.98
Majority Vote38.99+20.75
H-DMAD53.53+35.29
MACR18.56+0.32

MACR incurs negligible additional latency (+0.32s). We will add the results into our revision.

AuthorsOpenReview ↗

Response to Reviewer cxj3 (2/2)

Q3: Lack of initialization-only baseline or random relays.

Thanks for your feedback!

We evaluate the full relay pipeline on a randomly selected MMStar subset of 500 instances, changing only which model initiates:

Table R6: Initial model selection results.

Relay StrategyAcc. (%)
Random initial model65.4
Lowest-confidence initial model63.0
Fixed Qwen3-VL66.0
Fixed MiniCPM-V-4.564.4
Fixed InternVL-3.565.6
MACR (Highest-confidence)67.4

, and changing only the relay rule:

Table R7: Relay rule results.

Relay StrategyAcc. (%)
Random relay64.8
Lowest-confidence relay64.0
Fixed-order relay66.2
MACR (Highest-confidence relay)67.4

It shows that MACR performs best.

We will include the new results in the revision.


Q4: Ablation study lacks uncertainty estimates.

Thanks for your feedback.

We expand the 3-seed ablation study (mean ± std) on MMStar.

Table R8: 3-seed ablation study.

VariantMMStar (500)
FI (MiniCPM-V 4.5)71.70 ± 0.20
FI (InternVL 3.5)71.40 ± 0.25
FI (Qwen3-VL)70.55 ± 0.30
Fixed Selection (FS)69.20 ± 0.26
Fixed Trigger (FT)69.95 ± 0.22
MACR (Ours)72.65 ± 0.18

We additionally expand the ablation study on MathVista (random 500 instances) and LogicVista.

Table R9: Expanded ablation study.

VariantMathVista (500)LogicVista (448)
FI (MiniCPM-V 4.5)74.0050.67
FI (InternVL 3.5)73.8050.89
FI (Qwen3-VL)72.0050.45
Fixed Selection (FS)73.4049.33
Fixed Trigger (FT)72.6049.78
MACR (Ours)75.6052.90

MACR performs best. We will include the new results in the revision.

Comments on the paper

2 comments
~Linwei_Timothy_Li1OpenReview ↗

Model-Specific Reasoning Prompts in MACR

Hi, authors! I first reproduced the main results with the released code. I was glad to see the results closely match the paper (MathVista 78.3% for MACR in my run). While going through the implementation, I noticed that MACR uses the same prompt for all models in the pool. I assume this is intentional, to keep the comparison fair across baselines. However, I suspect it is not the optimal choice, as the main tables show that each model has its own strongest single-model reasoning recipe. So I tried a small extension: during relay, the selected model continues reasoning with its own best-performing single-model strategy (e.g., on MathVista, Qwen3-VL with DDCoT, MiniCPM-V / InternVL with CCoT), while keeping everything else in MACR unchanged. To my surprise, this improved MACR: accuracy went from 78.3% to 79.6%, performing better than my reproduced H-DMAD (78.9%). This suggests that preserving each model's native reasoning strength inside the relay loop may unlock additional gains beyond a uniform prompt. Have the authors considered incorporating each model's best single-model reasoning configuration into MACR? Separately, what I am most curious about is whether MACR transfers beyond vision-language settings. The relay-and-bid mechanism seems modality-agnostic in principle, and I would very much like to see the authors' results on text-only reasoning benchmarks. If the relay mechanism also works there, it would broaden the impact of the method.

AuthorsOpenReview ↗

Response to Public Reviewer Linwei Timothy Li

Thank you for taking the time to reproduce our results! We are very glad to hear that you successfully reproduced our main results. We provide point-to-point response below.

Q1: Same prompt for all models.

Good question!

Adopting the same prompt for all models is intentional. We deliberately kept the prompt uniform to ensure a fair comparison across all baselines. Otherwise, MACR would be given an unfair advantage over the single-model and multi-agent baselines. However, we find your exploration very interesting and insightful. We will include a detailed discussion of this in the revision.

Q2: Transfer to text-only reasoning.

Good suggestion!

We are also very curious about this. Even though it is the submission period, we have been actively exploring MACR in other modalities. We further evaluate it on text-only LLMs (Llama3.1-8B-Instruct, Qwen3-8B, MiniCPM4-8B) using CommonsenseQA [ref1], ARC-Challenge [ref2] and MMLU-Pro [ref3] (500 randomly sampled instances per dataset). IBC is calculated using L2M as the baseline:

Table R1: Performance of LLMs in terms of accuracy (Acc, %), token overhead (Tok) and IBC (×10−6\times10^{−6}).

BenchmarksMethodsLlama3.1-8B-Instruct (Acc/Tok/IBC)Qwen3-8BMiniCPM4-8B
CommonsenseQA [ref7]L2M [ref10]71.40 / 84778.60 / 61872.80 / 625
CoT [ref11]79.00 / 34079.80 / 35774.40 / 245
SBP [ref12]75.80 / 81579.80 / 66373.20 / 818
DMAD81.60 / 8659/9.21
MACR82.00 / 1450/102.65
ARC-Challenge [ref8]L2M84.60 / 98593.80 / 70290.00 / 691
CoT88.00 / 39293.80 / 42090.60 / 271
SBP85.00 / 77793.00 / 70991.20 / 669
DMAD96.00 / 8688/8.27
MACR95.80 / 1584/80.03
MMLU-Pro [ref9]L2M47.20 / 166663.40 / 117357.40 / 1989
CoT51.60 / 86869.80 / 86160.60 / 783
SBP50.00 / 135469.80 / 117253.40 / 1320
DMAD72.00 / 16980/10.41
MACR72.00 / 2697/147.10

MACR outperforms L2M, CoT, and SBP in both accuracy and IBC across text-only benchmarks. Compared with DMAD, MACR achieves comparable or higher accuracy with up to 14.1×14.1\times higher IBC (e.g., 147.10 vs. 10.41 on MMLU-Pro), while requiring 6.3×6.3\times fewer tokens.

We will include the results in the revision.

[ref1] Commonsenseqa: A question answering challenge targeting commonsense knowledge, ACL 2019

[ref2] Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

[ref3] Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, NeurIPS 2024

[ref4] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR 2026

[ref5] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022

[ref6] Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models, ICLR 2024

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.