- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper first identifies correlated failures in multi-agent code generation and introduces REALMS-CG, an approach that combines model diversity with structure-aware candidate selection. REALMS-CG encourages different models to generate diverse solutions, groups semantically similar candidates, and retains only a small number of representatives from each group for the next stage. Experiments demonstrate improvements in performance while reducing resource consumption.
Critical strengths
- Clear and easy-to-follow: The paper identifies correlated failures in multi-agent code generation, and the method description is clear.
- The authors analyze error propagation across requirement decomposition, test generation, and code implementation, providing concrete examples from MBPP+/239 and LiveCodeBench abc400a. This explains how REALMS-CG mitigates correlated failures.
- Experiments demonstrate REALMS-CG can significantly improves performance.
Critical weaknesses
-
This paper uses Spectural Clustering to mitigates the influence of error propagation and information redundancy. However, semantic similarity does not imply correctness. Correct and slightly incorrect candidates may belong to the same cluster. Then medoid-based selection may discard a correct minority candidate while retaining representative but erroneous outputs. Although the paper presents end-to-end recovery examples, these do not demonstrate whether clustering itself preserves correct information or reduces error propagation. Controlled ablations against no clustering and budget-matched random sampling or simple semantic deduplication are needed. ( I'm not saying that clustering is a bad method, I just want to see the ablation study to demonstrate that the superior overall performance does not actually come from other factors such as LLM reranker / heterogeneous candidates).
-
The affinity construction and minimum cluster count require further justification. Equation (1) assigns positive affinity to almost every candidate pair, yielding a dense weighted graph without sparsification. While this is not inherently invalid, its effect on spectral structure and selection stability is not evaluated. Moreover, enforcing $ k_{\min}=2 $ excludes the possibility of a single coherent semantic group. The stated goal of preserving alternatives does not establish that such alternatives always exist. Comparisons with thresholded or kNN graphs and a cluster-selection procedure allowing would help assess the robustness of these choices.
Questions for authors
- What are the oracle Pass@1 scores before clustering, after clustering, and after final aggregation, respectively?
- See Weakness 1 above: How many tasks originally had a correct candidate that was removed during clustering? For how many tasks did a correct candidate remain after clustering, but the aggregator selected an incorrect one?
- Why does REALMS-CG (SC + LLM) run faster than the variant that uses only SC in the final stage?
I would raise my score if my concerns are addressed.
3 comments
I appreciate the author's efforts, which address some of my concerns. I will raise my score.
Thank you for your contribution.
Dear Reviewer mQBJ,
We sincerely thank the reviewer for their time and effort in carefully reviewing our paper and for providing constructive comments and valuable feedback, which have helped us improve the clarity, rigour, and overall quality of our work.
We thank the reviewer for recognising the failure analysis and for highlighting the distinction between semantic grouping and correctness preservation.
1. Random sampling and simple sematic deduplication.
Responding to Weakness #1 and the ablation request referenced in Question #2.
We have also added simple deduplication with uniform random forwarding at 10%, 30%, and 50%, together with Baseline + LLM selector (§4.7; Tables 3–4 and 7–10). These controls strengthen the evaluation beyond MoA: simpler approaches are competitive on MBPP+, while REALMS-CG achieves higher observed accuracy on LiveCodeBench.
Updated in Table 3 and Table 4, At 10% random sampling, it scored 74.60% for MBPP+ and 38.01% for LiveCodeBench. REALMS-CG (LLM last layer) outperformed by 12.15%. At the same time, random sampling does not group correlated failures, unlike REALMS-CG.
2. Dense affinity and minimum cluster count.
Responding to Weakness #2.
Setting Kmin=2 expresses the design objective of retaining multiple intermediate groups before selection (§3.1). This is to explore the feasibility of clustering correlated failure into distinct cluster while retaining correct output into another distinct cluster. As illustrated in Figure 11, We are able to use spectral clustering to concentrate correlated failures into distinct cluster.
Our evaluation establishes a reference configuration for future work comparing thresholded or nearest-neighbour graphs and cluster-selection rules that allow Kmin=1. Such comparisons would clarify when alternative graph structures improve candidate preservation and selection stability.
3. Oracle accuracy and discarded correct candidates.
Responding to Questions #1 and #2, and the correctness-preservation concern in Weakness #1.
Based on our logs, before clustering, its passed 207/378 MBPP+ tasks (54.76%) and 58/342 LiveCodeBench tasks (16.96%).
After REALMS-CG clustering and before final aggregation, it passed 289 MBPP+ tasks (76.46%) and 149 LiveCodeBench tasks (43.57%).
After final aggregation, it passed 282 tasks (74.60%) and 148 tasks (43.27%), respectively.
4. Why SC + LLM has lower total duration.
Responding to Question #3.
The SC variant uses matrix multiplication which requires more computation time while SC + LLM final selector variant is already optimized in term of computation time.
Our logs also show that additional runtime waiting or scheduling overhead is observed for the SC variant. This breakdown locates the observed efficiency difference and distinguishes it from the additional cost of final LLM selection.