acceptodds
Under review as a conference paper at ICLR 2027

REALMS-CG: Resilient Ensemble of Autonomous LLM Multi-agent with Spectral clustering for Code Generation

Abstract

Large language model (LLM) multi-agent systems are often built on the assumption that decomposing a task across multiple agents improves reliability. However, when one of the intermediate agents produces an error, it can propagate across stages rather than being corrected. We study this as correlated failure: the propagation of error across successive agent stages. We introduce Resilient Ensemble Of Autonomous LLM Multi-agent with Spectral clustering for Code Generation (REALMS-CG), a heterogeneous multi-LLM code-generation framework that combines model diversity with structure-aware candidate selection. REALMS-CG is implemented as a fully local pipeline using five compact LLMs ranging from 8B to 20B parameters, requiring less than 16GB of memory each. REALMS-CG embeds each LLM output into a semantic similarity graph and applies spectral clustering to reduce redundant output before downstream processing. We evaluated five configurations; single agent, homogeneous multi-agent generation, a Mixture-of-Agents (MoA)-style approach, and two REALMS-CG variants on 378 MBPP+ and 342 LiveCodeBench tasks. Beyond Pass@1, we traced failures across multiple agent stages and analysed the scalability using peak stage-average token input and runtime duration. REALMS-CG with an LLM final aggregator achieved the highest Pass@1 scores of 91.50% on MBPP+ and 43.27% on LiveCodeBench.

Primary area:
generative models
Keywords:
multi-agent systems, large language models, heterogeneous ensembles, spectral clustering, code generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mQBJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper first identifies correlated failures in multi-agent code generation and introduces REALMS-CG, an approach that combines model diversity with structure-aware candidate selection. REALMS-CG encourages different models to generate diverse solutions, groups semantically similar candidates, and retains only a small number of representatives from each group for the next stage. Experiments demonstrate improvements in performance while reducing resource consumption.

Critical strengths
  1. Clear and easy-to-follow: The paper identifies correlated failures in multi-agent code generation, and the method description is clear.
  2. The authors analyze error propagation across requirement decomposition, test generation, and code implementation, providing concrete examples from MBPP+/239 and LiveCodeBench abc400a. This explains how REALMS-CG mitigates correlated failures.
  3. Experiments demonstrate REALMS-CG can significantly improves performance.
Critical weaknesses
  1. This paper uses Spectural Clustering to mitigates the influence of error propagation and information redundancy. However, semantic similarity does not imply correctness. Correct and slightly incorrect candidates may belong to the same cluster. Then medoid-based selection may discard a correct minority candidate while retaining representative but erroneous outputs. Although the paper presents end-to-end recovery examples, these do not demonstrate whether clustering itself preserves correct information or reduces error propagation. Controlled ablations against no clustering and budget-matched random sampling or simple semantic deduplication are needed. ( I'm not saying that clustering is a bad method, I just want to see the ablation study to demonstrate that the superior overall performance does not actually come from other factors such as LLM reranker / heterogeneous candidates).

  2. The affinity construction and minimum cluster count require further justification. Equation (1) assigns positive affinity to almost every candidate pair, yielding a dense weighted graph without sparsification. While this is not inherently invalid, its effect on spectral structure and selection stability is not evaluated. Moreover, enforcing $ k_{\min}=2 $ excludes the possibility of a single coherent semantic group. The stated goal of preserving alternatives does not establish that such alternatives always exist. Comparisons with thresholded or kNN graphs and a cluster-selection procedure allowing k=1k=1 would help assess the robustness of these choices.

Questions for authors
  1. What are the oracle Pass@1 scores before clustering, after clustering, and after final aggregation, respectively?
  2. See Weakness 1 above: How many tasks originally had a correct candidate that was removed during clustering? For how many tasks did a correct candidate remain after clustering, but the aggregator selected an incorrect one?
  3. Why does REALMS-CG (SC + LLM) run faster than the variant that uses only SC in the final stage?

I would raise my score if my concerns are addressed.

3 comments

We thank the reviewer for recognising the failure analysis and for highlighting the distinction between semantic grouping and correctness preservation.

1. Random sampling and simple sematic deduplication.

Responding to Weakness #1 and the ablation request referenced in Question #2.

We have also added simple deduplication with uniform random forwarding at 10%, 30%, and 50%, together with Baseline + LLM selector (§4.7; Tables 3–4 and 7–10). These controls strengthen the evaluation beyond MoA: simpler approaches are competitive on MBPP+, while REALMS-CG achieves higher observed accuracy on LiveCodeBench.

Updated in Table 3 and Table 4, At 10% random sampling, it scored 74.60%  for MBPP+ and 38.01% for LiveCodeBench. REALMS-CG (LLM last layer) outperformed by 12.15%. At the same time, random sampling does not group correlated failures, unlike REALMS-CG.

2. Dense affinity and minimum cluster count.

Responding to Weakness #2.

Setting Kmin⁡=2K_{\min} = 2 expresses the design objective of retaining multiple intermediate groups before selection (§3.1). This is to explore the feasibility of clustering correlated failure into distinct cluster while retaining correct output into another distinct cluster. As illustrated in Figure 11, We are able to use spectral clustering to concentrate correlated failures into distinct cluster. 

Our evaluation establishes a reference configuration for future work comparing thresholded or nearest-neighbour graphs and cluster-selection rules that allow Kmin⁡=1K_{\min} = 1. Such comparisons would clarify when alternative graph structures improve candidate preservation and selection stability.

3. Oracle accuracy and discarded correct candidates.

Responding to Questions #1 and #2, and the correctness-preservation concern in Weakness #1.

Based on our logs, before clustering, its passed 207/378 MBPP+ tasks (54.76%) and 58/342 LiveCodeBench tasks (16.96%).

After REALMS-CG clustering and before final aggregation, it passed 289 MBPP+ tasks (76.46%) and 149 LiveCodeBench tasks (43.57%).

After final aggregation, it passed 282 tasks (74.60%) and 148 tasks (43.27%), respectively.

4. Why SC + LLM has lower total duration.

Responding to Question #3.

The SC variant uses matrix multiplication which requires more computation time while SC + LLM final selector variant is already optimized in term of computation time.

Our logs also show that additional runtime waiting or scheduling overhead is observed for the SC variant. This breakdown locates the observed efficiency difference and distinguishes it from the additional cost of final LLM selection.

I appreciate the author's efforts, which address some of my concerns. I will raise my score.

AuthorsOpenReview ↗

Thank you for your contribution.

Dear Reviewer mQBJ,

We sincerely thank the reviewer for their time and effort in carefully reviewing our paper and for providing constructive comments and valuable feedback, which have helped us improve the clarity, rigour, and overall quality of our work.

Reviewer_FoXDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

REALMS-CG combines five heterogeneous LLMs in a task-decomposition, test-generation, and code-generation pipeline. Spectral clustering selects representative intermediate outputs, followed by an LLM selecting the final program. Experiments report improved accuracy and reduced peak stage-average resource usage over the evaluated MoA-style baseline, alongside qualitative analyses of error propagation and recovery.

Critical strengths

1.The pipeline substantially improves accuracy over the evaluated MoA-style baseline on both benchmarks while reducing intermediate context usage, demonstrating practical value for local multi-model inference.

2.The trace analysis connects planning and testing errors to incorrect implementations. The authors also acknowledge statistically inconclusive gains over the strongest baseline and show that semantic clusters can contain incorrect outputs.

Critical weaknesses

1.Appendix A.6.2 determines MBPP+ success using base-test outcomes, whereas MBPP+ requires passing both base and additional tests. This conflicts with the benchmark description and leaves the reported MBPP+ accuracy insufficiently supported. GitHub

2.Compared with GPT-OSS, the reported gains are only 0.50 percentage points on MBPP+ and 4.38 on LiveCodeBench, with paired p-values of 0.8506 and 0.1006. As acknowledged, these results do not establish reliable superiority over the strongest single model.

3.The experiments lack token-matched random selection and simple semantic-deduplication controls. Consequently, they do not establish whether spectral structure provides benefits beyond reducing the amount of forwarded context.

4.Under the same top-10%-closest strategy, replacing final medoid selection with an LLM raises LiveCodeBench accuracy from 15.78% to 43.27%. Without independent generation followed by the same selector, the necessity of the preceding multi-stage pipeline remains unclear.

5.The failure analysis samples only mixed-outcome tasks, and recovery is illustrated through two cases. It does not quantify how often REALMS-CG prevents error propagation, discards correct alternatives, or recovers across a representative task sample.

6.Table 6 reports peak stage-average tokens and duration rather than total computation or end-to-end latency. The accounting for embedding, clustering, and final aggregation is unclear, limiting the resource-efficiency conclusions.

7.Six forwarding strategies are evaluated, but no separate selection set or repeated-seed evaluation is described. The highlighted configuration’s reliability is therefore uncertain, especially given the substantial performance variation across forwarding strategies.

Questions for authors

1.Do Tables 3 and 9 use base-only outcomes? Please provide accuracy and paired comparisons using the complete MBPP+ test suite.

2.Which model performs final selection, and how does it perform when selecting among independently generated solutions without intermediate agent stages?

3.Does spectral clustering outperform random forwarding and semantic deduplication under equal token budgets? How does the complete pipeline compare with repeated GPT-OSS sampling at comparable computational cost?

4.Can you report aggregate error-propagation and recovery rates, including the frequency of correct candidates being removed during forwarding?

1 comment

We thank the reviewer for identifying issues that have strengthened the benchmark evaluation, controls, and resource accounting.

1. Complete MBPP+ evaluation.

Responding to Weakness #1 and Question #1.

Yes, it was using the MBPP+ base test. We corrected success to require passing both base and additional tests. Tables 3, 7, and 9, Appendix A.6.2, the abstract, and conclusion reflect this correction.

Using the base test, REALMS-CG score 91.50% and outperformed other models that scored 78.72% on average, an improvement of over 13%. Using the additional test, it scored 74.60% and outperform other models that scored 66.63% on average, an improvement of over 10%. 

2. Final judging and the strongest baseline.

Responding to Weaknesses #2 and #4, and Question #2.

The final selector is GPT-OSS 20B, the strongest standalone model in our evaluation. 

The added Baseline + LLM final selector (independent selection w/o intermediate agent stage) comparison strengthens the evidence for intermediate processing on LiveCodeBench. REALMS-CG achieves 43.27% compared with 38.30%, an improvement of over 10%. This provides exploratory evidence that the complete pipeline can improve upon independent generation followed by the same final selector. 

3. Additional candidate-reduction controls.

Responding to Weakness #3, #7 and Question #3.

We have also added simple deduplication with uniform random forwarding at 10%, 30%, and 50%, together with Baseline + LLM selector (§4.7; Tables 3–4 and 7–10). These controls strengthen the evaluation beyond MoA: simpler approaches are competitive on MBPP+, while REALMS-CG achieves higher observed accuracy on LiveCodeBench.

Updated in Table 3 and Table 4, At 10% random sampling, it scored 74.60%  for MBPP+ and 38.01% for LiveCodeBench. REALMS-CG (LLM last layer) outperformed by 12.15%. At the same time, random sampling does not group correlated failures, unlike REALMS-CG.

4. Recovery and discarded candidates.

Responding to Weakness #5 and Question #4.

We analysed using only mixed-outcome tasks as this would provide meaningful information and contrast on why certain models are able to successfully solve a task while other models are unable to. The two cases help to illustrates an example from MBPP+ and LiveCodeBench on how errors are introduced in the first place. Our sampled analysis identifies recurring failure patterns: wrong logic and incomplete or missing implementations account for 52.63% of recorded MBPP+ failure occurrences (20/38) and 44.44% on LiveCodeBench (20/45) as shown in Table 12.

Appendix A.11 combines eigenvalue-based cluster selection, spectral visualisation, and  expose groups containing recurring errors. To the best of knowledge, REALMS-CG is the first to study correlated failures as illustrated in Figure 11. We are able to use spectral clustering to concentrate correlated failures into distinct cluster.

We intend to work on future work to specifically target to remove such correlated failure cluster.

5. Expanded resource accounting.

Responding to Weakness #6.

Appendix A.5 and Table 6 now include final selection and end-to-end duration. Mean MoA/REALMS-CG(SC-final)/REALMS-CG(LLM-final) durations are 331.07/27.70/17.45 minutes on MBPP+ and 389.45/112.13/34.06 minutes on LiveCodeBench. These include inference and embedding/retrieval, with clustering and medoid selection included in retrieval.

Reviewer_RoSPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

REALMS-CG is a multi-agent framework for code generation that uses spectral clustering and medoid-based selection to reduce redundant signal among task decomposition, test generation, and code generation stages. Evaluations are conducted on MBPP+ and LiveCodeBench.

Critical strengths

The filtering step substantially reduces intermediate input size and runtime compared with the MoA-style baseline. Keeping a smaller set of different ideas is a sensible approach.

Addresses a practical problem. Adding more agents can spread mistakes and fill the context with repetitive information. The paper tackles both issues and shows why more agents do not always mean better results.

Critical weaknesses

It is unclear how much spectral clustering actually helps. The gains could come from using different models, filtering outputs, or adding a final LLM aggregator. Ablation/Comparisons with random pruning, simple deduplication, and multiple samples would help separate these effects.

Filtering similar outputs does not necessarily remove errors. A representative output can still be wrong, and a correct but unusual answer might be discarded. The examples show that recovery is possible, but do not establish that clustering consistently reduces error propagation.

The main baseline makes the gains less convincing. The MoA-style baseline hits the 16,000-token context limit, so much of the improvement may come from avoiding context overload. Against the strongest single model, the gains are statistically inconclusive, leaving it unclear whether the extra complexity is worthwhile.

I think the work is a reasonable engineering experiment, but insufficient novelty for the conference.

Questions for authors

see weaknesses

1 comment

We thank the reviewer for recognising the practical importance of controlling redundant context and error propagation.

1. Additional controls and contribution of the pipeline.

Responding to Weakness #1.

Spectral clustering is used to provide globally structure awareness clustering in order to cluster correlated failure. To the best of knowledge, this work is the first to provide clustering of correlated failures.

We have also added simple deduplication with uniform random forwarding at 10%, 30%, and 50%, together with Baseline + LLM selector (§4.7; Tables 3–4 and 7–10). These controls strengthen the evaluation beyond MoA: simpler approaches are competitive on MBPP+, while REALMS-CG achieves higher observed accuracy on LiveCodeBench.

Updated in Table 3 and Table 4, At 10% random sampling, it scored 74.60% for MBPP+ and 38.01% for LiveCodeBench. REALMS-CG (LLM last layer) outperformed by 12.15%. At the same time, random sampling does not group correlated failures, unlike REALMS-CG.

2. Understanding correlated failures.

Responding to Weakness #2.

Appendix A.11 combines eigenvalue-based cluster selection, spectral visualisation, and expose groups containing recurring errors. To the best of knowledge, REALMS-CG is the first to study correlated failures as illustrated in Figure 11. We are able to use spectral clustering to concentrate correlated failures into distinct cluster.

We intend to work on future work to specifically target to remove such correlated failure cluster. 

3. Context-limited MoA and the strongest baseline.

Responding to Weakness #3.

On LiveCodeBench, REALMS-CG shown a clear improvement compared against the main strongest baseline by scoring 43.27% versus 38.89%, an improvement of over 10%.

REALMS-CG also has a lower and constant resource requirements as compared to MoA. Tables 3–4, Table 6, and Table 7–10 therefore quantify the performance claims. By increasing the token context limit (e.g. 32,000/64,000), more (potentially double or even exponential) expected resources would be necessary in order for MoA to match REALMS-CG (16000-token context limit) performance. 

4. Novelty and research value.

Responding to Weakness #4.

To the best of knowledge, REALMS-CG is the first to utilised spectral clustering to group correlated failures in distinct cluster. REALMS-CG is also the first to conduct empirical research and through empirical data, to demonstrate that there are existing correlated failures. REALMS-CG also uses spectral clustering with medoid based forwarding to mitigate correlated failure that random sampling is unable to.

By publishing these findings and worked examples, we aim to support future research on error-aware grouping, preservation of correct minority candidates, and reliable multi-agent selection.

Comments on the paper

2 comments
Area_Chair_9GAcOpenReview ↗

Please engage in the discussion

Dear Reviewers,

As the ddl of author-reviewer discussion is approaching, could you please read the authors' responses and update your reviews if necessary?

As you know, we have only three reviewers in total. So each of your assessments may have a crucial impact on the final decision. Please be active. Thanks!

Regards,

Your AC.

AuthorsOpenReview ↗

Rebuttal feedbacks and clarifications.

Dear Reviewer RoSP and Reviewer FoXD,

As the rebuttal period draws to a close, we sincerely thank the reviewers for their valuable feedback.

We believe our responses have adequately addressed the questions and concerns raised, and we respectfully invite the reviewers to reconsider their assessments and increase their scores accordingly.

We would greatly appreciate any further feedback and remain happy to clarify any outstanding concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.