- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SEPAL, a multi-agent LLM collaboration framework designed to improve reasoning reliability while reducing error propagation among agents. SEPAL employs three independent Actor–Critic teams specializing in direct reasoning, evidence grounding, and verification. Each team undergoes role-specific supervised fine-tuning and preference optimization, followed by private iterative revisions. The final answers are combined through majority voting without sharing intermediate reasoning histories. Experiments across five LLM backbones and five question-answering benchmarks demonstrate consistent improvements over a single Actor–Critic baseline, with an average accuracy gain of 1.81 percentage points. Ablation studies further highlight the effectiveness of iterative revision and answer-level fusion.
Critical strengths
1. Clear Motivation and Practical Design: The paper addresses error propagation in multi-agent collaboration through independent Actor–Critic teams and a simple answer-level voting mechanism.
2. Consistent Experimental Results: Experiments across five LLM backbones and five benchmarks show consistent improvements over the single Actor–Critic baseline, supported by detailed ablation and revision analyses.
Critical weaknesses
-
Insufficient Validation of the Core Isolation Mechanism: The paper argues that private Actor–Critic interactions prevent error propagation and preserve reasoning diversity. However, no direct ablation compares private and shared communication under identical settings (same roles, training, revision rounds, and computational budget). Therefore, it remains unclear whether the improvements actually come from the proposed isolation mechanism or simply from using multiple independently generated answers.
-
Unfair Computational Budget and Limited Efficiency Analysis: SEPAL requires 30 model generation calls per question (15 Actor and 15 Critic calls), compared with only 10 calls for the single-pair ACC-Collab baseline, while achieving an average accuracy improvement of just 1.81 percentage points. The paper lacks comparisons under matched inference budgets, as well as detailed latency and token consumption analyses. Moreover, the revision analysis shows that the first revision already contributes approximately 89.2% of the total improvement, raising questions about the necessity of four revision rounds.
-
Limited Benefits from the Complex Training Pipeline: Table 3 shows that adding Critic DPO improves average accuracy from 75.74% to 76.30%, while further Actor DPO provides only a 0.01-point improvement. Furthermore, simpler configurations outperform the full method on some backbones, such as Qwen2.5-3B and Phi-4-mini. These results raise concerns about whether the additional preference optimization stages are necessary.
-
Insufficient Evidence for Role Specialization: Although SEPAL introduces Direct, Evidence, and Verification roles, all three teams share the same backbone and training questions. The paper does not compare the proposed role-specialized ensemble with three independently trained teams using identical roles. Consequently, it is difficult to determine whether the performance gains arise from meaningful role specialization or simply from independent sampling and majority voting.
-
Limited Statistical Reliability and Generalization: All results are based on a single evaluation run without standard deviations, confidence intervals, or statistical significance tests, making the relatively modest improvements difficult to assess. Additionally, experiments are restricted to short-answer question-answering benchmarks, leaving the effectiveness of SEPAL on open-ended generation and more complex reasoning tasks insufficiently explored.
Questions for authors
-
Effectiveness of the Isolation Mechanism: The central claim of SEPAL is that private Actor–Critic interactions prevent error propagation and preserve reasoning diversity. However, the paper lacks a direct comparison between private and shared communication. Could the authors provide an ablation that changes only the communication strategy while keeping the roles, training configurations, revision rounds, and computational budget identical? This would help determine whether isolation itself is responsible for the observed improvements.
-
Computational Fairness and Necessity of the Full Pipeline: SEPAL requires 30 model generation calls per question, compared with 10 for ACC-Collab, but achieves only a 1.81-point average accuracy improvement. Moreover, Table 3 shows that Actor DPO contributes only 0.01 points, while the first revision accounts for approximately 89.2% of the total revision gain. Could the authors clarify whether SEPAL still outperforms simpler alternatives under matched inference budgets, particularly configurations with fewer revision rounds or without Actor DPO? These comparisons would substantially affect the assessment of the method's efficiency and practical value.