acceptodds
Under review as a conference paper at ICLR 2027

Cross-Family Exchange Under Matched Call Budgets: Sampling, Revision, and Tie Resolution

Abstract

Does cross-family exchange improve a language-model panel beyond independent sampling or revisiting its own responses? We compare Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct on 200 GSM8K questions and a 100-question rational-answer MATH subset. For each question, 16 saved generations support four-call conditions that share initial answers, model composition, output ceilings, and aggregation. Both proposer orders are evaluated. Averaging the two orders within each question, exchange trails independent voting by 3.0 percentage points on GSM8K (descriptive 95% paired-bootstrap interval ) and 1.5 points on MATH (). Relative to own-response revision, the differences are and points; neither interval excludes zero. Exchange adds 603 and 1,405 input tokens per question over independent voting. A post hoc order-invariant tie sensitivity reduces the MATH advantage over own-response revision to 1.1 points: answer order affects the fixed vote even without changing generated content. Shared valid initial agreement also fixes the final vote regardless of subsequent answers. These results identify restrictions of the decision procedure, not a general benefit of exchange or cognitive specialization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent language models, debate, self-revision, majority voting, tie breaking, paired evaluation, test-time compute

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1yeoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The main question is, under a fix budget of four calls, if exchanging responses between two model families (qwen2.5 and llama3.1) is better than independent sampling or self-revision over its own answer. The authors use GSM8k and MATH-Hard questions and find that exchange underperforms both. They do further analysis and find that order and tie breaking matters a lot in multi-agent decisions. For example, re-ordering scores seems to change final decisions (4pp).

Critical strengths

Pros:

  • 4.2 audit and 5 uniform tie scoring are novel from what I can tell and have impact; this probably affects most agent debate papers.
  • Careful controls in S3.2
  • Analysis in general are good in 4-6
Critical weaknesses

Cons:

  • Lots of msising answer sin MATH
  • Paper was a little hard to read since there's a lot of hedging
  • Novelty might be limited; they headline a result that has been mostly known
Questions for authors
  • generalizeability: does the tie-rate asymetry hold with different models, larger K, or non-math tasks?
  • roughly 1 out of the 4 answers aren't parseable -- doe sthat change the tie analysis if we could pars eit?
  • 2 of the 4 votes are shared initial answers, which locks outcomes when they agree. Was this allocation chosen on purpose? Would the results change if later calls got more votes, for example?
1 comment

Thank you for recognizing the value of the aggregation audit and for suggesting the additional model, parsing, and vote-allocation checks.

We completed 2,640 new ARC-Challenge generations and reanalyzed all 4,800 original mathematical responses. The additions address the three questions directly.

  1. Different models, larger K, and a non-math task

We selected 60 ARC-Challenge test questions by fixed hash ordering and evaluated two pairs: Qwen2.5-7B-Instruct + Llama-3.1-8B-Instruct, and Qwen3-4B-Instruct-2507 + Llama-3.1-8B-Instruct. Each question has 22 saved paths supporting four- and six-call comparisons, equal calls per family, and both role orders. The extension uses a common revision prompt, temperature 0.7, and a 384-token output ceiling.

Pair / callsIndependent accuracyExchange accuracyOwn-response accuracyIndependent/exchange tie rate
Qwen2.5 + Llama / 490.83%90.83%90.83%1.67% / 0%
Qwen2.5 + Llama / 690.83%91.67%90.83%1.67% / 0.83%
Qwen3 + Llama / 492.50%91.67%90.83%3.33% / 1.67%
Qwen3 + Llama / 692.50%92.50%90.83%3.33% / 5.00%

Entries average the two role orders within each question and use the original parser and earliest-answer tie rule. Exchange-minus-independent paired intervals include zero in all four settings.

The tie asymmetry therefore is not universal: it is smaller on ARC, and its direction reverses for the newer pair at six calls. The diagnostic remains useful precisely because it measures this dependence rather than assuming that exchange always reduces ties.

  1. Unparseable answers

We reproduced every original parsed answer and fixed-vote prediction from the saved text. Of 1,600 MATH paths, 451 are unparsed; 359 of these reach the 1,024-token ceiling and 92 finish below it. GSM8K has 280/3,200 unparsed paths, of which 34 reach the ceiling. Thus missingness and length-cap events can be reported separately.

A label-blind format recovery preserves every existing answer and handles explicit final fields or complete boxes containing supported punctuation, unit wrappers, or currency escapes. It recovers 12 GSM8K and four MATH answers. All primary accuracy contrasts remain unchanged. On MATH, tie rates also remain 32.0%, 9.5%, and 22.5% for independent, exchange, and own-response panels. The exchange-minus-own contrast still falls from +4.5 to +1.125 points under uniform tie resolution. This audit tests recoverable formatting errors; it cannot reconstruct answers absent from the saved text.

We applied an additional explicit-format sensitivity to ARC, covering boxed option-plus-text, explicit answer declarations, and opening option lines. It recovers 100 answers across the two pairs and changes any accuracy in the table by at most 0.83 points. The six-call Qwen3 pair retains the reversed tie ordering: 4.17% for exchange versus 3.33% for independent voting. All capped and unresolved outputs remain in the evaluation.

  1. Why share the initial votes, and what if later calls receive more weight?

Sharing starts was deliberate: it holds initial answer quality fixed while changing how the remaining two calls are used. Keeping those starts as equal votes also creates the lock you identified. Under earliest-answer tie resolution, even two matching corrections cannot overturn two agreeing initial answers.

We therefore recomputed the original outputs using [1,1,2,2] weights and [0,0,1,1] weights, applying the same rule to all three protocols:

Vote weightsGSM8K: exchange − independentMATH: exchange − independent
[1,1,1,1]−3.00 pp−1.50 pp
[1,1,2,2]−4.25 pp−3.00 pp
[0,0,1,1]−4.25 pp−1.00 pp

These post hoc sensitivities remove the original lock without producing a positive point advantage over independent sampling. Uniform tie scoring gives the same direction in all six cells. The new six-call design also removes the structural guarantee of locking, since four later votes can overturn two initial votes.

The paper's central finding is that the decision procedure can both create apparent role effects and suppress later corrections. On unchanged MATH generations, the tie-rule change reduces the exchange-minus-own contrast by 3.375 points; its newly computed question-paired bootstrap 95% interval is [1.25, 5.58]. This gives the aggregation audit a directly measured effect, beyond the broader observation that voting is competitive. We will organize the revision around these findings and shorten the repeated qualifications.

Reviewer_EbUbOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tests whether letting two small AI models (Qwen 7B and Llama 8B) review each other's answers helps. It compares this approach with simply asking each model several times and taking a vote. The experiments use 200 GSM8K and 100 MATH questions. Exchanging answers does not beat simple voting. Some differences depend only on how ties in the vote are broken.

Critical strengths
  • Careful setup
Critical weaknesses

The main findings do not provide a clear conclusion. Most differences have intervals that include zero, so the paper cannot say whether exchange helps, hurts, or does nothing.

Questions for authors

No question.

1 comment

Thank you for highlighting the need to distinguish inconclusive exchange comparisons from the findings the data do establish.

The intervals do not all support the same conclusion. On GSM8K, exchange scores 89.0% versus 92.0% for independent voting: a paired difference of −3.0 percentage points with a descriptive 95% interval of [−5.5, −0.5]. Under uniform tie resolution, the difference remains negative, −3.40 points [−5.90, −0.90]. The MATH comparison is less resolved: −1.5 points [−6.0, 3.0]. We do not interpret intervals spanning zero as evidence of equivalence.

The stronger contribution concerns effects that can be measured while holding generation fixed.

First, the apparent MATH advantage over own-response revision depends substantially on the selector. It is +4.5 points with the original tie rule and +1.125 with uniform tie resolution. The resulting 3.375-point change has a newly computed question-paired bootstrap 95% interval of [1.25, 5.58]. This post hoc calculation uses the same responses for both selectors; it measures the change in the contrast directly. It does not depend on establishing that either exchange-versus-revision contrast is itself nonzero.

Second, response order alone changes independent-vote accuracy from 45% to 41% on MATH. The answer multisets are identical, and correctness changes on six questions. This provides an exact explanation for an observed order effect without attributing it to model specialization.

Third, matching initial answers lock the four-vote decision under the original tie rule. Two later corrections cannot defeat the two initial votes. This occurs on 134/200 GSM8K and 18/100 MATH questions. It explains why improvements in later reasoning need not produce improvements in the reported panel score.

We have now tested alternative vote allocations. Giving the later answers double weight yields exchange-minus-independent differences of −4.25 points on GSM8K and −3.0 on MATH. Counting only the later answers yields −4.25 and −1.0 points. These fixed-output sensitivities address whether the initial-answer lock alone explains the observed ordering.

We also completed 2,640 new generations on 60 fixed-hash ARC-Challenge test questions, using two model pairs and four/six-call budgets. Role-averaged accuracy under the original selector is:

Model pair / callsIndependentExchangeOwn-response
Qwen2.5-7B + Llama-3.1-8B / 490.83%90.83%90.83%
Qwen2.5-7B + Llama-3.1-8B / 690.83%91.67%90.83%
Qwen3-4B-Instruct-2507 + Llama-3.1-8B / 492.50%91.67%90.83%
Qwen3-4B-Instruct-2507 + Llama-3.1-8B / 692.50%92.50%90.83%

The new exchange-versus-independent contrasts are small and their paired intervals include zero. Exchange nevertheless adds 333–474 input tokens per question at four calls and 669–945 at six calls. A separate explicit-format recovery audit changes any displayed accuracy by at most 0.83 points.

These results sharpen the paper's conclusion: modest exchange gains must be assessed against sampling, self-revision, and the selector's ability to use corrections. The paper demonstrates measurable aggregation effects and a structural decision constraint, alongside the empirical exchange comparisons. We will foreground those findings in the abstract and introduction.

Reviewer_gKDSOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates the allocation of calls to LLMs from different families to check whether different orders and dependence could increase performance. Authors compare 3 protocols, each consisting of 4 calls: independent, exchange, own-response and show that not only the strategy influences the final performance, but also the way of aggregation of 4 answers. The authors showed that independent protocol has the highest average accuracy (Figure 1). In Figure 2 authors showed the difference of the final accuracy that depends on the order of models.

Critical strengths
  1. No demonstrated advantage from cross-family exchange. Authors showed that cross-family exchange does not establish an advantage over independent sampling on tested datasets.
  2. Analysis of different information flow protocols. Authors compared how different information flows protocols of choosing the answer and model order influences the accuracy.
Critical weaknesses
  1. Incremental novelty. The authors specified the contribution as a paired comparison of information flow, own-response revision, role order, and numerical aggregation. I don't think it is sufficient for the contribution.
Questions for authors
  1. Novelty. As authors mentioned, the competitiveness of the majority voting is already established. Could authors precisely specify what insights their analysis brings and why it is useful?
1 comment

Thank you for asking us to clarify what the analysis adds beyond the established voting baseline.

The contribution is to identify when an apparent exchange benefit comes from the decision rule, and when that rule prevents later responses from changing the answer. The new analyses quantify both effects on saved generations and extend the experiment to a non-math task, a newer model pair, and six calls.

  1. Changing the selector changes the measured exchange benefit.

On the MATH subset, exchange exceeds own-response revision by 4.5 percentage points under the original earliest-answer tie rule. Uniform selection among tied modal answers reduces this to 1.125 points, with every generated response held fixed. In this post hoc analysis, the change in the contrast is 3.375 points, with a descriptive question-paired bootstrap 95% interval of [1.25, 5.58]. Thus three quarters of the displayed advantage depends on this tie-rule choice in these records.

The role-order comparison provides an even simpler check: permuting the same four independent answers changes correctness on 6/100 MATH questions and accuracy from 45% to 41%. No communication or new generation occurs. An order effect can therefore arise entirely in aggregation.

  1. Shared starts can make the final decision insensitive to subsequent computation.

Under four equal votes and earliest-answer tie resolution, two matching initial answers cannot be overturned by the two later answers. Even two identical corrections only produce a 2–2 tie, which the initial answer wins. This locks the result on 134/200 GSM8K and 18/100 MATH questions.

We tested the reviewer's suggested alternative allocations on the same outputs. With weights [1,1,2,2], exchange minus independent voting is −4.25 points on GSM8K and −3.0 on MATH. With only the later answers voting, the differences are −4.25 and −1.0 points. These post hoc comparisons remove the original lock without revealing a positive point advantage over independent sampling. They answer whether the result is simply an artifact of preserving the two initial votes.

  1. The new experiment tests when these effects matter beyond the original setting.

We completed 2,640 new generations on 60 ARC-Challenge test questions selected by fixed hash ordering. We tested Qwen2.5-7B-Instruct/Llama-3.1-8B-Instruct and Qwen3-4B-Instruct-2507/Llama-3.1-8B-Instruct at four and six calls, with equal family call counts and both role orders. The extension uses a common revision prompt and a 384-token output ceiling.

Tie rates are much lower than on MATH. At four calls, independent/exchange tie rates are 1.67%/0% for the first pair and 3.33%/1.67% for the newer pair. At six calls, the newer pair reverses that ordering: 3.33%/5.0%. This is useful evidence against treating fewer ties as an inherent property of exchange.

Initial-answer locking also appears on ARC: 44/60 and 38/60 questions are locked at four calls under the original parser. At six calls the structural lock no longer follows, since four later votes can overturn two initial votes.

The practical lesson is specific: an exchange evaluation should test whether later corrections can reach the final decision, separate response-order effects from generation effects, and report tie-rule sensitivity on identical outputs. The paper supplies these checks and measures their impact. We will make this the central narrative, rather than headline the already-established competitiveness of voting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.