- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CIPI (Conditional Integration of Peer Information), a multi-agent LLM reasoning framework that addresses the limitations of integrating peer feedback at fixed interaction rounds. Instead of immediately incorporating received feedback, CIPI maintains a feedback buffer and employs a state-dependent gate to determine when feedback should be inspected. An integrator proposes revisions, while a verifier evaluates accepted corrections before updating the reasoning state. The framework also supports deferred feedback inspection and rollback of rejected updates. Experiments across six benchmarks covering multi-hop question answering, mathematical reasoning, and code generation show that CIPI achieves higher task performance than the evaluated communication baselines, generally at the cost of increased token consumption. Additional ablation studies investigate the effects of feedback integration timing and comment selection.
Critical strengths
1. Clear Motivation and Well-Structured Design: The paper addresses the underexplored problem of when to integrate peer feedback in multi-agent LLM reasoning. The proposed framework combines state-dependent feedback inspection, verification, and rollback mechanisms in a structured and interpretable manner.
2. Broad Experimental Evaluation: The paper evaluates CIPI across six benchmarks covering multi-hop question answering, mathematical reasoning, and code generation. Additional ablation studies distinguish feedback integration timing from comment selection, while token usage and confidence intervals provide further insights into performance and efficiency.
Critical weaknesses
-
Insufficient Evidence for the Core Feedback Integration Mechanism: The main contribution of CIPI is its state-dependent decision on when to integrate peer feedback. However, the ablation results in Table 2 do not convincingly demonstrate the effectiveness of this mechanism. For example, CIPI-All performs slightly worse than Random-All on MuSiQue (68.34 vs. 68.40 F1) and achieves identical accuracy on OlymMATH (88.00%). Furthermore, the confidence intervals in Appendix Table 7 generally include zero. The paper also lacks direct evaluations of the gate's error-detection accuracy, false negatives, and the verifier's ability to reject harmful corrections.
-
Confounded Comparisons with Existing Methods: The main experiments do not isolate the contribution of CIPI's integration strategy. Different methods use different thinking settings, model-call structures, and output constraints. For example, EoT-Report disables thinking for all agents, whereas CIPI enables it for the Peer, Integrator, and Verifier. Additionally, CIPI employs a structured QA answer selector with source quotations and validation checks that are not applied to the baselines. Consequently, it is unclear how much of the reported improvement comes from conditional feedback integration rather than these additional design differences.
-
Limited Evaluation Scale and Generalizability: Despite covering six benchmarks, the experiments use only 100 questions per dataset, except for HumanEval (164 questions). Such small evaluation sets limit the statistical reliability of the reported improvements, particularly when some gains are relatively modest. For instance, the 95% confidence interval for CIPI's 5-point improvement over MP+SR on MATH-500 is [−1.00, 11.00], which includes zero. Moreover, all experiments rely on a single hosted model family, leaving the generalizability of CIPI across different LLM backbones insufficiently validated.
-
High Computational Cost and Lack of Budget-Matched Comparisons: CIPI introduces multiple additional model calls for feedback inspection, integration, and verification, resulting in substantially higher token consumption than the baselines. According to Table 1, CIPI consumes approximately 44%, 29%, and 35% more tokens than MP+SR on MATH-500, MuSiQue, and HumanEval, respectively. However, the corresponding performance improvements are not consistently substantial. The paper lacks comparisons under matched token budgets, as well as detailed latency analysis, making it difficult to assess the practical value of the proposed framework.
-
Poor Figure Quality and Unclear Method Presentation: Figure 1, the only methodological figure in the paper, suffers from unclear directional connections, insufficiently distinguished decision branches, and text-heavy boxes. In particular, the inspection, verification, commitment, and rollback paths are not clearly illustrated. Important components such as the feedback buffer, correction record, and hypothesis history are also not explicitly visualized. Given the complexity of the proposed controller, the figure does not effectively communicate the overall framework, making the method unnecessarily difficult to understand.
Questions for authors
-
Effectiveness of State-Dependent Feedback Integration: The central contribution of CIPI is determining when peer feedback should be integrated based on the agent's current reasoning state. However, Table 2 shows that CIPI-All performs slightly worse than Random-All on MuSiQue (68.34 vs. 68.40 F1) and achieves identical accuracy on OlymMATH (88.00%). Moreover, the confidence intervals in Appendix Table 7 include zero for these comparisons. Could the authors provide stronger evidence that state-dependent integration is more effective than random or fixed-timing strategies? In particular, how reliably does the gate identify situations requiring immediate correction, and how frequently does it miss critical feedback? These results would directly affect the assessment of the paper's central claim.
-
Fairness of Comparisons and Sources of Performance Gains: The main comparisons use different thinking settings, output constraints, and computational budgets across methods. For example, CIPI enables thinking for its Peer, Integrator, and Verifier, while EoT-Report disables thinking for all agents. CIPI also employs additional answer-validation mechanisms and generally consumes more tokens than the baselines. Could the authors clarify how much of CIPI's performance improvement can be attributed specifically to conditional feedback integration rather than these differences? Would the improvements remain under matched thinking configurations and inference token budgets? Such evidence is important for determining the effectiveness and practical value of the proposed method.
Response to Reviewer NmvY
Thank you for the detailed review. In short: (1) the log analysis shows that the gate fires on wrong states and avoids harmful early revisions, while the timing contrast in final scores is underpowered. (2) Table 1 is a system-level comparison, and the matched ablation arms are close in token use. (3) We will state the evaluation scope and redraw Figure 1.
Weaknesses
1. Evidence for the core integration mechanism.
In final scores CIPI-All trails Random-All by 0.07 F1 on MuSiQue and ties it on OlymMATH, and the intervals in Table 7 include zero (general response, item 2). The consistent effect in the ablation is selective inspection: full CIPI exceeds CIPI-All on all three datasets. To evaluate the gate directly, we analyzed the MuSiQue ablation logs, labeling each state as correct or wrong against the reference answer.
All arms use the same verifier, so the protection against harmful early changes comes from the gate.
2. Comparisons with existing methods.
The thinking settings, the QA selector and the APPS audit differ between CIPI and the baselines, and EoT-Report runs without thinking. Table 1 is therefore a system-level comparison, and we will describe it as such (general response, item 1).
3. Evaluation scale and generalizability.
The main comparison uses 100 tasks per benchmark and all 164 HumanEval tasks, and all runs use one hosted model (deepseek-flash). The MATH-500 interval ([−1.00, 11.00]) includes zero, as you note. We will state in the abstract that the results come from one model family and report the paired intervals alongside the point estimates.
4. Computational cost and budget matching.
On OlymMATH CIPI gains 9.00 points at a token ratio of 1.04 to MP+SR, and on APPS it passes 4 more tasks at a ratio of 1.01. On the other benchmarks the ratio is 1.29 to 1.44, and we make no efficiency claim there. The four ablation arms are close in token use (Table 2), so the ablation contrasts are approximately budget-matched. The cost report covers tokens, and we will state that wall-clock latency is not measured.
5. Figure 1.
We will redraw Figure 1 with the feedback buffer, correction record and hypothesis history as explicit components, and with separate paths for inspection, verification, commitment and rollback.
Questions
Q1. See item 1 above and item 2 of the general response.
Q2. See item 2 above and item 1 of the general response.