- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TIDE, a framework for automatically designing and evolving LLM-based multi-agent systems through a Typed Intermediate Representation. TIDE separates high-level architectural design from low-level execution: a meta-agent generates a Typed IR specification, which is validated and compiled into an executable system by a deterministic compiler. The framework also incorporates an offline, experience-guided evolution loop that analyzes execution traces on a development set and extracts reusable design insights to refine task-family-specific design policies, without updating model parameters. -【T=he paper further uses execution traces to categorize system failures and investigate how evolution changes multi-agent coordination. Experiments on four mathematical reasoning and code-generation benchmarks show that TIDE achieves an average accuracy of 78.0%, outperforming the strongest reported baseline by 9.3 percentage points. Evolution also improves performance over the non-evolved TIDE system by an average of 7.2 percentage points. Overall, the paper presents a promising framework that combines structured system representation, deterministic compilation, and trace-guided optimization. However, further clarification of the individual contributions of its components and the evaluation protocol would strengthen the evidence supporting its claims.
Critical strengths
(1) A structured intermediate representation for automated multi-agent system design. TIDE offers a compelling approach to separating high-level MAS architecture design from low-level execution. Rather than requiring a meta-agent to generate complete executable code or restricting design to a predefined search space, the framework allows the meta-agent to specify system structure through a typed intermediate representation, while a deterministic compiler handles execution. This separation can reduce the coupling between architectural design and implementation details, while making system specifications easier to validate and inspect. The approach represents a useful systems-level contribution to automated MAS design.
(2) Trace-guided analysis and experience-driven evolution.The paper goes beyond reporting task-level performance by analyzing execution traces to identify recurring failure patterns and using the resulting insights to refine subsequent designs. The reported reduction in post-solution degradation on AIME illustrates how trace analysis can reveal coordination problems that would be difficult to identify from final accuracy alone. This provides a promising direction for making the optimization of automatically designed multi-agent systems more interpretable and systematic.
Critical weaknesses
(1) The individual contributions of Typed IR and the evolution components are not fully isolated.The reported results suggest that the complete TIDE framework is effective, with evolution improving average accuracy by 7.2 percentage points over the non-evolved system. However, the framework combines several components, including the typed representation, static validation, deterministic compilation, trace-based failure analysis, and experience-guided evolution. The current evidence does not fully establish how much each component contributes to the observed improvements, or whether the gains depend primarily on the evolution procedure rather than the Typed IR representation itself. More controlled ablations under matched model settings and computational budgets would help clarify the source of the performance gains and strengthen the paper's central claims.
(2) The framework's limitations and evaluation protocol require further clarification.TIDE constructs architectures as acyclic graphs from a registry of interaction primitives. This design supports static validation and deterministic execution, but it may limit the framework's expressiveness for tasks that require dynamic control flow or open-ended interaction. In addition, extending the primitive registry may require coordinated changes to the representation specification, compiler, validator, and associated prompts or analysis procedures. The evolution mechanism also relies on development-set feedback, which may not be readily available in every deployment setting. These considerations do not invalidate the proposed framework, but the paper would benefit from a clearer discussion of its applicability boundaries, extension costs, and deployment assumptions.
Questions for authors
(1) Could the authors provide additional controlled ablations to distinguish the contributions of Typed IR, static validation, deterministic compilation, trace-based analysis, and experience-guided evolution? In particular, under matched base models, execution primitives, prompting strategies, and computational budgets, how does TIDE compare with an alternative representation that does not use Typed IR? Such experiments would help clarify whether the observed improvements arise primarily from the representation, the evolution procedure, or their interaction.
(2) Could the authors clarify the relationship between the development-set sizes reported in Appendix Table 6 and the accuracy values in the main experimental results? Please specify the exact subsets and number of samples used to compute each reported metric, including AIME and HumanEval+, and explain how the development data are used during evolution and tuning. It would also be helpful to clarify whether the reported evaluation results are computed on samples that are strictly separate from those used for evolution.
Response to Reviewer vJ8t
Thank you for your positive and thoughtful review. We are glad you see TIDE as a systems-level contribution to automated MAS design, one that makes designs easy to inspect and their optimization more interpretable. In short: following your question, we now report results on the held-out test splits, where all conclusions hold, and new ablations show that both the typed IR and evolution contribute.
Q2. Data splits and held-out evaluation. Thank you for this question, which we answer part by part.
Evolution still improves every benchmark, and TIDE keeps the best average (74.8 vs. 68.7 for the strongest baseline), though no longer the best on every benchmark.
W1 / Q1. Where the gains come from. Both the representation and evolution contribute, and the representation alone already beats all baselines. TIDE lets the meta-agent use its design knowledge to orchestrate a system for each query (roles, nodes and their interactions), while a compiler handles execution, so no code writing or fine-tuning is needed. All methods share the same executor. We go through the components you listed:
W2. Limitations. We agree and have added a discussion of these points:
The main revisions are marked in blue in the revised PDF. We sincerely appreciate the effort you have invested in reviewing our work. We hope these clarifications address your concerns. Thank you again for your time!