- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ReM-MoA, which stores the outputs of all previous layers in a shared memory, ranked by an LLM Reviewer. Each agent in the next layer gets a different set of high and low scoring traces from this memory, which aims to improve both the quality and the diversity of the MoA system. It demonstrates good performance comparing other MoA variants while scaling on depth, width and token usage across 5 reasoning benchmarks; in the same time, the diversity converges more slowly than other methods. It is a smart and lightweight idea that fit well on MoA system.
Critical strengths
- It provides a lightweight method to improve the diversity of MoA, and it is easy to integrate into existing MoA systems. It perform good on both quality and diversity. At L=9, it reaches 72.8% average accuracy, compared with 68.3% for the best baseline. It also preserves diversity better and works with 7B to 32B proposer pools. Such good performance does not depend much on the Reviewer’s ability; an off-the-shelf 7B model already works well. Therefore, I think this idea is smart and can be widely applied in similar settings.
- It has clear ablation to demonstrate that the three core components are important, which make it more convincing. At L=9, removing cross-layer access lowers the average accuracy from 72.8% to 68.7% (−4.12), removing routing diversification lowers it to 70.2% (−2.66), and removing the rationales lowers it to 71.2% (−1.60).
Critical weaknesses
- Limited novelty compared with VeriMoA. VeriMoA already keeps a global record for all previous outputs with quality scores and give the top-n to the next layer. So the main adds on in this paper is making agents in one layer have different input and include low score traces as negative examples, which is not validated well as shown in the next point.
- The mixed routing is not compared with the Top-n only design. VeriMoA gives all agents the same top-n traces, while ReM-MoA mixes top and bottom traces and gives different agents different routes. This mixed design is the main difference from VeriMoA, but the paper never compares it with simply using Top-n for all agents. The only routing ablation compares different routes against one shared route, and it does not say which shared route is used. Without a Top-n only baseline, we cannot tell whether the low-scoring traces actually help, or how much ReM-MoA gains over a general version of VeriMoA.
Questions for authors
- Can you compare the mixed routing with a Top-n only version, where all agents share the same top-n traces (a general version of VeriMoA)? Also, which shared route is used in the “routing diversification” ablation?
- How much does reviewer contribute to the performance. How does ReM-MoA perform with a Reviewer that gives random scores? Does the ranking quality really matter? The Reviewer scores traces within each layer, but Top/Bot are selected across all layers. Are scores from different layers comparable?
- Does the method still help with newer models, harder benchmarks, or compared with a single stronger model at the same cost?
5 comments
Response to Reviewer kDhz (Part 2/3)
Q2. Contribution of the Reviewer, random scores, ranking quality, cross-layer comparability.
Random scores. We replaced every score with a uniform random integer in 1–10, with no Reviewer call and no rationales.
Table K2. Random-Score ablation. Accuracy (%), mean over the five benchmarks and five seeds, N=3. Sources: ReM-MoA from Table 8, − rationale from Table 11, Random-Score from new runs.
| Variant | L=1 | L=3 | L=5 | L=7 | L=9 |
|---|---|---|---|---|---|
| ReM-MoA | 62.3 | 67.9 | 70.8 | 72.3 | 72.8 |
| − rationale | 61.9 | 67.2 | 69.6 | 70.8 | 71.2 |
| Random-Score | 61.1 | 64.8 | 66.7 | 67.8 | 68.1 |
The Reviewer contributes 4.7 points at L=9: 3.1 from the quality of the ranking (71.2 vs. 68.1) and 1.6 from the rationales (72.8 vs. 71.2). Ranking quality therefore matters, and it matters in both directions: the distilled Reviewer, whose scores separate correct from incorrect traces better (AUROC 0.84 vs. 0.78), raises accuracy further to 75.2. At the same time, the system still scales with depth under random scores (61.1 to 68.1), because agents keep reading different entries from all earlier layers. In particular, 12.6 of the 17.3-point gain over Standard MoA comes from cross-layer memory and diversified routing alone. This agrees with the reviewer's observation that an off-the-shelf 7B Reviewer is sufficient.
Cross-layer comparability. We re-scored the 27 traces of an L=9 run in random triples drawn from three different layers (main pool, seed 0, 926 problems, three groupings, both Reviewers). Per-layer scores correlate with cross-layer scores at Spearman 0.83, close to the 0.86 between two cross-layer groupings (0.88 and 0.91 for the distilled Reviewer). The Top-2 entries selected from the 27 traces are correct 71.5% of the time under per-layer scores and 70.8% under cross-layer scores. The AUROC for separating correct from incorrect traces is 0.782 for same-layer pairs and 0.775 for cross-layer pairs. Scores are thus approximately comparable across layers. We have reported this analysis and noted the small residual dependence on the candidate set in the limitations paragraph.
Response to Reviewer kDhz (Part 3/3)
Q3. Newer models, harder benchmarks, and a single stronger model at the same cost.
Newer models and a harder benchmark. We ran MMLU-Pro (1,400 questions, 100 per domain) with the main pool and with three 8B models released in the first half of 2025, each serving as proposer, Reviewer and aggregator: Qwen3-8B (thinking disabled), DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-0528-Qwen3-8B.
Table K3. MMLU-Pro with the main pool and three 8B models. Accuracy (%) on a fixed 1,400-question subset of MMLU-Pro (100 questions from each of the 14 domains), mean±std over five seeds, N=3. Methods: Standard MoA, AttentionMoA and ReM-MoA with the off-the-shelf Reviewer. All numbers are new.
| Pool | Method | L=1 | L=3 | L=5 | L=7 | L=9 |
|---|---|---|---|---|---|---|
| Main (heterogeneous) | Std MoA | 48.4±0.8 | 50.1±0.6 | 49.0±0.8 | 47.7±0.6 | 46.4±0.7 |
| AttMoA | 49.5±0.5 | 52.0±0.3 | 53.5±0.4 | 54.0±0.8 | 54.2±0.3 | |
| ReM-MoA | 49.9±0.4 | 54.0±0.4 | 56.8±0.4 | 57.9±0.5 | 58.1±0.4 | |
| 3×Qwen3-8B | Std MoA | 52.4±0.7 | 54.1±0.8 | 53.0±0.8 | 52.1±0.8 | 51.0±0.8 |
| AttMoA | 53.0±0.7 | 55.8±0.8 | 57.1±0.8 | 57.4±0.9 | 57.9±0.4 | |
| ReM-MoA | 53.4±0.6 | 57.5±0.8 | 60.6±0.5 | 62.1±0.6 | 63.0±0.6 | |
| 3×R1-Distill-Llama-8B | Std MoA | 51.0±0.6 | 52.5±0.9 | 51.4±0.4 | 50.7±0.5 | 49.6±0.9 |
| AttMoA | 51.4±0.6 | 53.4±0.6 | 55.1±0.9 | 55.4±0.9 | 55.9±0.7 | |
| ReM-MoA | 52.0±0.9 | 55.5±0.4 | 58.2±0.8 | 59.7±0.6 | 60.6±0.4 | |
| 3×R1-0528-Qwen3-8B | Std MoA | 54.2±0.4 | 55.4±0.9 | 54.7±0.4 | 53.4±0.4 | 52.3±0.5 |
| AttMoA | 54.6±0.5 | 57.1±0.5 | 58.4±0.7 | 59.1±0.7 | 59.3±0.8 | |
| ReM-MoA | 55.1±0.3 | 59.0±0.7 | 62.1±0.7 | 63.4±0.9 | 64.3±0.5 |
ReM-MoA leads AttentionMoA by 3.9–5.1 points at L=9 across all four pools. Standard MoA degrades on all four. The three single-model ReM-MoA curves sustain growth through L=9, while the main pool plateaus at L=7–9 (+0.2). The R1-0528-Qwen3-8B pool starts highest but the ReM-MoA gain over Standard MoA (+12.0) remains comparable to the main pool (+11.7).
A single stronger model. The numbers come from the L=1 proposer traces of our 32B pool (Appendix C.2: Qwen2.5-32B-Instruct, Qwen2.5-Coder-32B-Instruct and DeepSeek-R1-Distill-Qwen-32B), on the five benchmarks with five seeds. One call to a single 32B model reaches 70.2–72.6, depending on the model. A single 32B model thus already matches ReM-MoA over the 7–8B pool at L=9 (72.8) at a fraction of the token cost, so when a stronger model can be deployed, it is the more cost-effective choice. ReM-MoA is complementary to model scale: on the same 32B pool, ReM-MoA at L=1 (74.8, five LLM calls rather than one) already exceeds the single-model range, at L=3 it reaches 80.5, and at L=9 it reaches 87.5, still leading AttentionMoA by 4.5 points (87.5 vs. 83.0 in Table 10). Standard MoA degrades on the 32B pool as well (Table 10), confirming that the depth-scaling pattern holds across model scales.
Response to Reviewer kDhz (Part 1/3)
We thank the reviewer for the constructive comments. The new results use the setting of the paper: N=3, the five benchmarks MATH, MMLU-redux, Formal Logic, CRUX and HellaSwag, and the main pool, with Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct and Qwen2.5-Coder-7B-Instruct as proposers and Qwen2.5-7B-Instruct as aggregator and off-the-shelf Reviewer. Each table below states its own benchmarks, models, metric and source.
W1. Novelty relative to VeriMoA.
We agree with the reviewer's description of VeriMoA: it keeps a global cache of earlier outputs with quality scores and gives the top-n to the next layer. Our related-work sentence "No prior MoA variant maintains a structured cross-layer memory ..." (lines 095–096) is too strong, and we have revised it to credit VeriMoA’s global cache. ReM-MoA differs in two ways. First, the quality signal needs no domain verifier: VeriMoA scores candidates by simulating hardware designs against testbenches and generates diversity through multiple domain-specific paths (direct HDL generation, C++ and Python), while our Reviewer is a general-purpose LLM that scores traces comparatively and writes a rationale. The Reviewer can also be strengthened without any verifier: the optional distilled Reviewer raises accuracy from 72.8 to 75.2 at L=9, with gains on all five benchmarks, including the three outside its distillation domains (Table 1). Second, reads are differentiated: instead of the same top-n for every agent, each agent reads a different mix of high- and low-scoring entries.
Beyond the architecture, the paper contributes a controlled characterization of depth scaling in existing MoA variants (our first contribution): under one shared pool and up to L=9, Standard MoA degrades, RMoA plateaus, and AttentionMoA saturates or plateaus on four of five benchmarks (Table 1). ReM-MoA is designed against these failure modes jointly, and its components are ablated separately (Table 1 and the variants below). VeriMoA already reports error accumulation in Standard MoA for HDL generation. What our study adds is that, on general reasoning tasks where such domain-specific paths are unavailable, a quality-ranked cache alone does not preserve diversity, which the Top-n-only comparison below quantifies.
W2 and Q1. Comparison with a Top-n-only design and which route the ablation uses.
We added two variants. Each agent receives two references, as in ReM-MoA. In Shared-Top and Distinct-Top, ties in score are broken once per layer so that all agents see the same ranking. In ReM-MoA, ties are broken independently for each agent.
Table K1. Routing variants. Accuracy (%), mean over the five benchmarks and five seeds, N=3. Vendi is the Vendi Score of the three proposer outputs at L=9, averaged over the five benchmarks (1 for identical outputs, 3 for orthogonal outputs), as in Table 12. Sources: ReM-MoA from Tables 8 and 12, Shared-Top and Distinct-Top from new runs.
ReM-MoA gains 3.9 points over the Top-n-only design at L=9, and the gap widens with depth (it is 2.0 points at L=3). Shared-Top also loses output diversity fastest. Adding low-scoring references and further reducing overlap between agents' reference sets together contribute the remaining 2.0 points (Distinct-Top → ReM-MoA).
On the existing ablation: in "− routing diversification" every agent uses the Contrastive route, Top-1 ∪ Bot-1, with ties broken independently per agent. The description at lines 413–414 ("one shared reference set") was imprecise and has been corrected.