- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ARGOT, a training-free framework that organizes heterogeneous LLM collaboration as evidence routing. It selects task-relevant agents, converts their initial responses into distributions, and uses optimal-transport costs and self-reported model confidence to construct support relations and protected challenge relations. Two directed communication phases refine responses, after which a trajectory-level verifier compares evidence cards with a synthesized anchor answer to select the final result. Experiments use a pool of six smaller specialist and generalist models across six reasoning benchmarks spanning science, mathematics, coding, finance, and medicine. ARGOT obtains the highest reported mean score and leads on four benchmarks. Ablations, disagreement-conditioned analysis, minority-answer rescue measurements, sensitivity tests, and qualitative success/failure cases explore routing and verification mechanisms.
Critical strengths
-
Explicit separation of support and challenge, with protected minority edges, targets an important failure mode in majority-dominated collaboration. The framework requires no training and makes final selection more inspectable through trajectory evidence cards.
-
The evaluation covers six benchmarks and provides a controlled comparison with GoA using the same heterogeneous model pool and dynamic agent selection. Disagreement-conditioned analysis and minority-answer rescue analysis are relevant to the stated motivation.
Critical weaknesses
-
Existing ablations do not establish the necessity of optimal transport. For multiple-choice tasks, transport distributions depend only on selected options and self-reported confidence, making them a coarse proxy for claim-level evidence.
-
The evaluation does not sufficiently separate evidence routing from the stronger final-verification stage. Removing verification causes the largest drop on HumanEval, but the replacement is majority voting and GoA lacks the same trajectory verifier.
-
Computational trade-offs remain unclear because end-to-end token, model-call, and latency comparisons are missing. ARGOT involves selection, two directed refinement stages, anchor-answer synthesis, and final verification, while baselines retain different interaction pipelines. Matched inference-budget comparisons are required to determine whether accuracy gains reflect better routing or additional computation.
Questions for authors
See Weaknesses.
3 comments
Dear authors, could you clarify why the newly added GoA + Trajectory Verifier variant achieves exactly the same GPQA accuracy (35.35%) as the original GoA reported in Table 1?
Clarification on the GPQA accuracy for GoA + Trajectory Verifier
We sincerely thank the reviewer fYno for the careful reading and for raising this question. The identical accuracy is a coincidence at the aggregate level rather than a reuse of the original GoA predictions. Both variants correctly answer 70 out of 198 GPQA questions, resulting in the same accuracy of 35.35%. Their per-question predictions nevertheless differ substantially, and the changes introduced by the trajectory verifier happen to balance out in the aggregate accuracy.
The result indicates that the verifier substantially changes the decision outcomes but does not provide a net improvement when combined with the original GoA routing on GPQA. In contrast, its stronger performance under ARGOT's OT-based relational routing suggests that the benefit depends on the interaction between relational routing and final verification.
Many thanks to Reviewer fYno for appreciating the significance of our methodological innovations and key mechanisms. To address your insightful suggestions, we have conducted further empirical analyses to clarify the specific impacts of our underlying modules and assess the overall system efficiency.
We agree that, for multiple-choice tasks, the transport distributions represent the selected option and self-reported confidence without directly encoding the full supporting rationale. Their role is to provide a routing signal. The ground cost incorporates relationships between options, so transporting probability mass between semantically similar options incurs a different cost from transporting it between dissimilar options. The reasoning text is examined during subsequent communication and verification.
To assess OT’s contribution, we replace it with answer disagreement, JS divergence, or embedding similarity while retaining agent selection, verification, and the communication configuration.
As shown, OT consistently outperforms all alternative metrics. Compared with the strongest alternative on each benchmark, OT improves GPQA by 4.04 percentage points and HumanEval by 4.27 percentage points.
Although the transport distributions use selected answers and confidence scores, the task-adaptive ground cost captures semantic relationships between answer representations. These results suggest that OT provides a more effective basis for constructing support and challenge relations than simpler disagreement or similarity measures, even without explicitly encoding full reasoning trajectories into the transport distributions.
Thank you for highlighting this concern. We agree that comparing against majority voting and the original GoA does not fully isolate the contribution of evidence routing when the verification stage differs. To address this concern, we added two controlled comparisons: one applies the same trajectory verifier to GoA, and the other directly verifies independent initial responses without evidence routing. These experiments allow us to better distinguish the contributions of routing and verification.
With the same final verifier, ARGOT outperforms GoA routing by 4.55 percentage points, indicating that the improvement cannot be attributed solely to the verification stage.
Furthermore, applying the verifier directly to independent initial responses yields only 26.77%, compared with 39.90% for complete ARGOT. This 13.13-point gain demonstrates the importance of structured evidence routing before final verification.
These comparisons support the complementary contributions of relational communication and trajectory verification, rather than attributing the performance gains to either component alone.
Thank you for raising this concern. We conducted an additional end-to-end efficiency comparison on the GPQA. We report accuracy, total input/output tokens, inference time, and average LLM calls per question.
As shown, ARGOT achieves the highest accuracy while reducing output tokens by 41.7% and inference time by 31.5% compared with GoA. Although ARGOT requires more LLM calls, its lower generation volume and shorter runtime suggest more efficient use of individual interactions.
These results demonstrate a favorable accuracy-efficiency trade-off, indicating that ARGOT's performance improvements are not simply driven by increased token generation or longer inference time.