acceptodds
Under review as a conference paper at ICLR 2027

DGTE: Disagreement-Gated Test-Time Evolution of Roles and Topologies in Multi-Agent Systems

Abstract

Large language model-driven multi-agent systems (MAS) use role specialization and communication to solve complex tasks, yet their test-time evolution typically relies on validation performance, task rewards, or executable feedback. For new tasks where task labels or trusted correctness scores are unavailable during generation, existing methods lack a reliable internal signal for deciding whether to continue evolving and how to modify the system. We therefore propose Disagreement-Gated Test-Time Evolution of Roles and Topologies (DGTE). For each task, DGTE starts from a frozen base system, runs multiple independent solver groups, and separately measures semantic disagreement among complete answers and operational disagreement caused by incomplete outputs. Their maximum forms a disagreement gate that determines whether to stop early, allocate another evolution round, or select a historical system state when the budget is exhausted. Execution traces, tool outputs, self-tests, and failure diagnostics guide bounded modifications to prompts, roles, helper agents, and communication topology. We validate DGTE on four benchmarks, including Humanity's Last Exam-Math, DeepResearchBench, BrowseComp-Plus, and Plancraft, and achieve SOTA performance on multiple benchmarks. The results show that DGTE achieves competitive performance across diverse task types; further analysis indicates that disagreement improves process completion, allocates additional computation, and helps repair incomplete outputs. DGTE has a modular design that can be seamlessly integrated into existing multi-agent methods without retraining the base model, improving system performance and process reliability with controlled overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent systems, test-time evolution, disagreement gating, adaptive computation, dynamic topology, solver groups, label-free control

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_F5tzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

DGTE uses disagreement among multiple independent solver groups as a gold-free gate for test-time multi-agent system evolution. It combines semantic disagreement among complete answers with operational disagreement from incomplete outputs. If the gate exceeds a threshold, a meta-controller updates prompts, roles, helper agents, and communication edges under bounded edit constraints. Experiments cover HLE-Math, DeepResearchBench, BrowseComp-Plus, and Plancraft.

Critical strengths

1:The high-level motivation—using disagreement as a label-free signal to decide whether to continue test-time MAS evolution—is reasonable and practically relevant.

2:The ablation on HLE-Math suggests that combining semantic and operational disagreement is better than either alone or no evolution.

Critical weaknesses

The semantic disagreement score Dsem is produced by an LLM evaluator, but the prompt, clustering rule, scoring procedure, and stability analysis are not adequately specified. This makes the central control signal effectively a black box and severely limits reproducibility.  The paper has serious presentation and rigor problems: DGTF and DGTE are used inconsistently, Table 1 is incomplete, cost accounting is not reported in sufficient detail, and baseline comparisons appear to use different backbones, judges, or budgets. In addition, the core idea of disagreement as a control signal for stopping, routing, or evolution is already well explored, so the incremental contribution is limited.

Questions for authors

What is the exact prompt and procedure for the semantic disagreement evaluator, and how stable is Dsem across repeated runs?

What is the average token and API cost per task for DGTE, including solver groups, evaluator calls, and meta-controller updates?

Are all baselines evaluated with the same backbone, judge, token budget, and grading protocol?

Is the method called DGTF or DGTE? Please resolve the inconsistency throughout the paper.

Why is there no AI use statement, and does the work comply with ICLR’s AI disclosure policy?

Why are many baseline entries missing in Table 1?

1 comment

The authors did not provide any rebuttal or response to my review. All of my original concerns about the paper remain unresolved. I will keep my original score.

Reviewer_tvKkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DGTE, a framework for task-local evolution of multi-agent systems without access to gold answers during generation. Starting from a frozen base configuration, multiple solver groups produce candidate answers. An LLM evaluates semantic disagreement among complete outputs, while the fraction of incomplete outputs defines operational disagreement. Their maximum controls early stopping and allocation of further evolution rounds; a meta-controller uses task-local traces to modify prompts, roles, helpers, and communication edges under discrete edit budgets. At budget exhaustion, the method returns an answer from the historical state with the lowest disagreement.

Critical strengths
  • Relevant motivation.
  • Interpretable control mechanism.
  • The evaluation includes gate ablations and fixed-round comparisons, and acknowledges correlated errors and limitations on open-ended reports.
Critical weaknesses
  • The disagreement signal is insufficiently specified and not directly validated as a predictor of useful additional evolution.
  • The experiments do not isolate structural adaptation from resampling, prompt revision, and completion repair.
Questions for authors
  1. Section 3.3 states that an LLM jointly compares complete answers and returns Dsem∈[0,1]D_{\mathrm{sem}}\in[0,1], but does not provide the evaluator prompt, scoring rubric, or a precise mapping from answer clusters to this scalar.
  2. Disagreement may decrease because the evolving topology correlates errors, rather than because the system improves. Although the paper acknowledges correlated errors, it does not measure whether declining disagreement accompanies improved correctness, shared mistakes, or merely reduced diversity.
  3. The no-evolution baseline executes the frozen system only once, whereas DGTE can repeatedly revise and re-execute it. Consequently, this comparison confounds configuration adaptation with additional sampling and accumulated task evidence.
  4. Comparing adaptive stopping with fixed K∈0,3,10K\in{0,3,10} is useful, but an intermediate cost–accuracy point does not by itself demonstrate that disagreement allocates computation better than a task-independent stopping schedule. A matched-cost random stopping policy, or a policy preserving the same distribution of evolution depths while permuting assignments across tasks, would more directly test that claim.
  5. The related-work discussion identifies TacoMAS, EVOCHAMBER, and DySCo as closely related approaches involving test-time adaptation, disagreement, or communication updates, yet none appears in Table 1. Meanwhile, some reported baseline results are imported from Skill-MAS without a row-level account of backbone, task subset, tool access, or budget.
  6. Section 4.1 specifies 168 HLE-Math evaluation problems, yet the ablation accuracies of 45.31%, 44.81%, and 43.17% in Table 2 cannot be obtained from binary accuracy over this set. These values appear arbitrary rather than derived from a clearly defined evaluation procedure, raising concerns about the reliability and reproducibility of the reported results. Please clarify the evaluation denominators, aggregation procedure, and whether different subsets or scoring protocols were used.
  7. Figure 2 shows all three solvers producing the same answer and disagreement reaching 0.0 at B3B_3, yet the trace continues to B10B_{10}. Under DGTE’s stated threshold, this run should terminate at B3B_3. If the figure comes from the forced fixed-budget experiment, it should be explicitly labeled as such rather than presented without qualification as an illustration of disagreement-gated evolution.
  8. Section 4.1 states that strict executable success is reported separately, but the supplied manuscript presents only the rubric-based score, including the headline 85.60% result. A rubric score should not be interpreted as successful execution, especially when the benchmark includes impossible goals.
Reviewer_qhXoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies test-time evolution/scaling of multi-agent systems (MAS), assuming gold answers or external scores are unavailable. In this setup, the authors propose a new MAS framework where agents are organized into several groups, each producing a final answer. Disagreement between groups then serves as a signal to guide evolution. The paper evaluates this approach on diverse benchmarks and reports significant improvements over other MAS baselines.

Critical strengths
  • The paper is clear and easy to follow.
  • The proposed method (disagreement-based MAS evolution) is very simple and seems effective on diverse tasks.
Critical weaknesses
  • The methodology is not sufficiently justified or explored. It is unclear to me why DGTE organizes agents into solver groups as the basic units of execution and comparison. There are many alternative topologies, such as a centralized meta-agent coordinating subagents, discussions or debates among subagents, or dynamic MAS structures based on task decomposition (e.g., DAGs). Disagreement signals could also be obtained and used for evolution in these settings. The authors adopt a very simple MAS configuration, whose practicality and efficiency in real-world deployments are unclear. Does the proposed evolution approach only work for such simple topologies? If so, I would view this as a limitation. I would encourage the authors to evaluate disagreement-based evolution across different MAS topologies to better establish its applicability.

  • The experiments are limited. The authors report only final scores, leaving unclear how much time each method takes,which is particularly important for test-time scaling approaches. Are the methods compared under matched budgets? What is the total number of agents and subagents in baselines? How are budgets controlled across the baselines and DGTE? In the related work section, the authors mention that most baselines require ground-truth answers or a validation set. Are these provided to the baseline methods in the experiments?

  • The source of the improvements is unclear. DGTE appears to outperform most MAS baselines by a large margin, but the paper does not clearly explain what drives these gains. Is disagreement-based evolution the key, or does the solver-group structure account for much of the improvement? Could the results suggest that the more complex MAS designs used by prior methods are unnecessary in these settings? Ablations separating the effects of the group structure and the evolution mechanism would help answer these questions.

Questions for authors

see above

1 comment

As the authors did not respond during the rebuttal period, I would keep my initial score.

Comments on the paper

3 comments

Dear Reviewers,

Could you please review the authors’ rebuttals and follow up with any further comments or questions?

Many thanks for your time and efforts.

Best regards, AC

Dear Reviewers,

As the AC–reviewer discussion phase is about to begin, please highlight any disagreements or inconsistencies across the reviews, as well as any concerns that remain unresolved after the author rebuttal.

Please share your perspectives on these points and engage with one another to help us reach a well-supported assessment of the paper.

Thank you for your time and contributions.

Best regards, AC

Dear Area Chair, The authors did not provide any rebuttal or response to my review. All of my original concerns about the paper remain unresolved. I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.