- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
DGTE uses disagreement among multiple independent solver groups as a gold-free gate for test-time multi-agent system evolution. It combines semantic disagreement among complete answers with operational disagreement from incomplete outputs. If the gate exceeds a threshold, a meta-controller updates prompts, roles, helper agents, and communication edges under bounded edit constraints. Experiments cover HLE-Math, DeepResearchBench, BrowseComp-Plus, and Plancraft.
Critical strengths
1:The high-level motivation—using disagreement as a label-free signal to decide whether to continue test-time MAS evolution—is reasonable and practically relevant.
2:The ablation on HLE-Math suggests that combining semantic and operational disagreement is better than either alone or no evolution.
Critical weaknesses
The semantic disagreement score Dsem is produced by an LLM evaluator, but the prompt, clustering rule, scoring procedure, and stability analysis are not adequately specified. This makes the central control signal effectively a black box and severely limits reproducibility. The paper has serious presentation and rigor problems: DGTF and DGTE are used inconsistently, Table 1 is incomplete, cost accounting is not reported in sufficient detail, and baseline comparisons appear to use different backbones, judges, or budgets. In addition, the core idea of disagreement as a control signal for stopping, routing, or evolution is already well explored, so the incremental contribution is limited.
Questions for authors
What is the exact prompt and procedure for the semantic disagreement evaluator, and how stable is Dsem across repeated runs?
What is the average token and API cost per task for DGTE, including solver groups, evaluator calls, and meta-controller updates?
Are all baselines evaluated with the same backbone, judge, token budget, and grading protocol?
Is the method called DGTF or DGTE? Please resolve the inconsistency throughout the paper.
Why is there no AI use statement, and does the work comply with ICLR’s AI disclosure policy?
Why are many baseline entries missing in Table 1?
The authors did not provide any rebuttal or response to my review. All of my original concerns about the paper remain unresolved. I will keep my original score.