- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The main question is, under a fix budget of four calls, if exchanging responses between two model families (qwen2.5 and llama3.1) is better than independent sampling or self-revision over its own answer. The authors use GSM8k and MATH-Hard questions and find that exchange underperforms both. They do further analysis and find that order and tie breaking matters a lot in multi-agent decisions. For example, re-ordering scores seems to change final decisions (4pp).
Critical strengths
Pros:
- 4.2 audit and 5 uniform tie scoring are novel from what I can tell and have impact; this probably affects most agent debate papers.
- Careful controls in S3.2
- Analysis in general are good in 4-6
Critical weaknesses
Cons:
- Lots of msising answer sin MATH
- Paper was a little hard to read since there's a lot of hedging
- Novelty might be limited; they headline a result that has been mostly known
Questions for authors
- generalizeability: does the tie-rate asymetry hold with different models, larger K, or non-math tasks?
- roughly 1 out of the 4 answers aren't parseable -- doe sthat change the tie analysis if we could pars eit?
- 2 of the 4 votes are shared initial answers, which locks outcomes when they agree. Was this allocation chosen on purpose? Would the results change if later calls got more votes, for example?
Thank you for recognizing the value of the aggregation audit and for suggesting the additional model, parsing, and vote-allocation checks.
We completed 2,640 new ARC-Challenge generations and reanalyzed all 4,800 original mathematical responses. The additions address the three questions directly.
We selected 60 ARC-Challenge test questions by fixed hash ordering and evaluated two pairs: Qwen2.5-7B-Instruct + Llama-3.1-8B-Instruct, and Qwen3-4B-Instruct-2507 + Llama-3.1-8B-Instruct. Each question has 22 saved paths supporting four- and six-call comparisons, equal calls per family, and both role orders. The extension uses a common revision prompt, temperature 0.7, and a 384-token output ceiling.
Entries average the two role orders within each question and use the original parser and earliest-answer tie rule. Exchange-minus-independent paired intervals include zero in all four settings.
The tie asymmetry therefore is not universal: it is smaller on ARC, and its direction reverses for the newer pair at six calls. The diagnostic remains useful precisely because it measures this dependence rather than assuming that exchange always reduces ties.
We reproduced every original parsed answer and fixed-vote prediction from the saved text. Of 1,600 MATH paths, 451 are unparsed; 359 of these reach the 1,024-token ceiling and 92 finish below it. GSM8K has 280/3,200 unparsed paths, of which 34 reach the ceiling. Thus missingness and length-cap events can be reported separately.
A label-blind format recovery preserves every existing answer and handles explicit final fields or complete boxes containing supported punctuation, unit wrappers, or currency escapes. It recovers 12 GSM8K and four MATH answers. All primary accuracy contrasts remain unchanged. On MATH, tie rates also remain 32.0%, 9.5%, and 22.5% for independent, exchange, and own-response panels. The exchange-minus-own contrast still falls from +4.5 to +1.125 points under uniform tie resolution. This audit tests recoverable formatting errors; it cannot reconstruct answers absent from the saved text.
We applied an additional explicit-format sensitivity to ARC, covering boxed option-plus-text, explicit answer declarations, and opening option lines. It recovers 100 answers across the two pairs and changes any accuracy in the table by at most 0.83 points. The six-call Qwen3 pair retains the reversed tie ordering: 4.17% for exchange versus 3.33% for independent voting. All capped and unresolved outputs remain in the evaluation.
Sharing starts was deliberate: it holds initial answer quality fixed while changing how the remaining two calls are used. Keeping those starts as equal votes also creates the lock you identified. Under earliest-answer tie resolution, even two matching corrections cannot overturn two agreeing initial answers.
We therefore recomputed the original outputs using [1,1,2,2] weights and [0,0,1,1] weights, applying the same rule to all three protocols:
These post hoc sensitivities remove the original lock without producing a positive point advantage over independent sampling. Uniform tie scoring gives the same direction in all six cells. The new six-call design also removes the structural guarantee of locking, since four later votes can overturn two initial votes.
The paper's central finding is that the decision procedure can both create apparent role effects and suppress later corrections. On unchanged MATH generations, the tie-rule change reduces the exchange-minus-own contrast by 3.375 points; its newly computed question-paired bootstrap 95% interval is [1.25, 5.58]. This gives the aggregation audit a directly measured effect, beyond the broader observation that voting is competitive. We will organize the revision around these findings and shorten the repeated qualifications.