acceptodds
Under review as a conference paper at ICLR 2027

Designing and Evolving Multi-Agent Systems via a Typed Intermediate Representation

Abstract

Automatic design of LLM-based multi-agent systems (MAS) has largely taken one of two forms: a meta-agent writes each system as free-form code, which is expressive but hard to validate and entangles design quality with coding ability; or a search over prebuilt operators, which is reliable but usually settles for one design per task family. We take a middle route in which the meta-agent designs and an engine implements. For each query the meta-agent emits a typed intermediate representation (IR) that composes agents from interaction primitives and specifies each agent by role, objective, and instructions. A rule-based compiler deterministically lowers each valid IR into an executable MAS, making every design statically checkable, executor-independent, and fully traceable from architecture to outcome. An experience-guided evolutionary loop then mines successful strategies and failure lessons from development traces to refine the meta-agent's design policy, updating no model parameters and adding no test-time cost. With a lightweight open-source executor, our system attains the best average accuracy across four benchmarks spanning math and coding, exceeding the strongest of eight baselines by 9.3 points, and evolution improves every benchmark. Because execution is fully traced, gains and losses are attributable: evolution repairs coordination while residual errors localize to executor capability, making failures of automatically designed MAS inspectable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent System

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vJ8tOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIDE, a framework for automatically designing and evolving LLM-based multi-agent systems through a Typed Intermediate Representation. TIDE separates high-level architectural design from low-level execution: a meta-agent generates a Typed IR specification, which is validated and compiled into an executable system by a deterministic compiler. The framework also incorporates an offline, experience-guided evolution loop that analyzes execution traces on a development set and extracts reusable design insights to refine task-family-specific design policies, without updating model parameters. -【T=he paper further uses execution traces to categorize system failures and investigate how evolution changes multi-agent coordination. Experiments on four mathematical reasoning and code-generation benchmarks show that TIDE achieves an average accuracy of 78.0%, outperforming the strongest reported baseline by 9.3 percentage points. Evolution also improves performance over the non-evolved TIDE system by an average of 7.2 percentage points. Overall, the paper presents a promising framework that combines structured system representation, deterministic compilation, and trace-guided optimization. However, further clarification of the individual contributions of its components and the evaluation protocol would strengthen the evidence supporting its claims.

Critical strengths

(1) A structured intermediate representation for automated multi-agent system design. TIDE offers a compelling approach to separating high-level MAS architecture design from low-level execution. Rather than requiring a meta-agent to generate complete executable code or restricting design to a predefined search space, the framework allows the meta-agent to specify system structure through a typed intermediate representation, while a deterministic compiler handles execution. This separation can reduce the coupling between architectural design and implementation details, while making system specifications easier to validate and inspect. The approach represents a useful systems-level contribution to automated MAS design.

(2) Trace-guided analysis and experience-driven evolution.The paper goes beyond reporting task-level performance by analyzing execution traces to identify recurring failure patterns and using the resulting insights to refine subsequent designs. The reported reduction in post-solution degradation on AIME illustrates how trace analysis can reveal coordination problems that would be difficult to identify from final accuracy alone. This provides a promising direction for making the optimization of automatically designed multi-agent systems more interpretable and systematic.

Critical weaknesses

(1) The individual contributions of Typed IR and the evolution components are not fully isolated.The reported results suggest that the complete TIDE framework is effective, with evolution improving average accuracy by 7.2 percentage points over the non-evolved system. However, the framework combines several components, including the typed representation, static validation, deterministic compilation, trace-based failure analysis, and experience-guided evolution. The current evidence does not fully establish how much each component contributes to the observed improvements, or whether the gains depend primarily on the evolution procedure rather than the Typed IR representation itself. More controlled ablations under matched model settings and computational budgets would help clarify the source of the performance gains and strengthen the paper's central claims.

(2) The framework's limitations and evaluation protocol require further clarification.TIDE constructs architectures as acyclic graphs from a registry of interaction primitives. This design supports static validation and deterministic execution, but it may limit the framework's expressiveness for tasks that require dynamic control flow or open-ended interaction. In addition, extending the primitive registry may require coordinated changes to the representation specification, compiler, validator, and associated prompts or analysis procedures. The evolution mechanism also relies on development-set feedback, which may not be readily available in every deployment setting. These considerations do not invalidate the proposed framework, but the paper would benefit from a clearer discussion of its applicability boundaries, extension costs, and deployment assumptions.

Questions for authors

(1) Could the authors provide additional controlled ablations to distinguish the contributions of Typed IR, static validation, deterministic compilation, trace-based analysis, and experience-guided evolution? In particular, under matched base models, execution primitives, prompting strategies, and computational budgets, how does TIDE compare with an alternative representation that does not use Typed IR? Such experiments would help clarify whether the observed improvements arise primarily from the representation, the evolution procedure, or their interaction.

(2) Could the authors clarify the relationship between the development-set sizes reported in Appendix Table 6 and the accuracy values in the main experimental results? Please specify the exact subsets and number of samples used to compute each reported metric, including AIME and HumanEval+, and explain how the development data are used during evolution and tuning. It would also be helpful to clarify whether the reported evaluation results are computed on samples that are strictly separate from those used for evolution.

1 comment
AuthorsOpenReview ↗

Response to Reviewer vJ8t

Thank you for your positive and thoughtful review. We are glad you see TIDE as a systems-level contribution to automated MAS design, one that makes designs easy to inspect and their optimization more interpretable. In short: following your question, we now report results on the held-out test splits, where all conclusions hold, and new ablations show that both the typed IR and evolution contribute.

Q2. Data splits and held-out evaluation. Thank you for this question, which we answer part by part.

  • Dev-set sizes and Table 2 accuracies. Table 7 (Table 6 in the submission) lists each benchmark's dev/test split, the accuracies in Table 2 were computed on the full benchmarks (e.g., all 60 AIME and 164 HumanEval+ problems), dev problems included.
  • Subsets and sample counts. We now report TIDE on the test splits only: 30 AIME, 4500 MATH, 115 HumanEval+ and 350 MBPP problems (table below).
  • Use of dev data. Dev data are used only during evolution. Intuitively, they work like practice problems: the meta-agent designs systems for them, a critic model reviews which designs succeed or fail, and reusable lessons are distilled into the meta-agent's design policy (e.g., add an agent only when it brings a distinct contribution). Only such lessons carry over, never answers or problem-specific content. Nothing else in TIDE is tuned on data, and TIDE w/o Evolution uses none.
  • Strict separation. We re-ran TIDE on the test splits only, with 3 seeds.
Test split (3 seeds)AIMEMATHHumanEval+MBPPAvg
TIDE w/o Evolution37.8 ± 5.191.1 ± 0.282.3 ± 0.572.3 ± 2.470.9
TIDE w/ Evolution46.7 ± 3.392.8 ± 0.484.3 ± 0.075.2 ± 1.474.8

Evolution still improves every benchmark, and TIDE keeps the best average (74.8 vs. 68.7 for the strongest baseline), though no longer the best on every benchmark.

W1 / Q1. Where the gains come from. Both the representation and evolution contribute, and the representation alone already beats all baselines. TIDE lets the meta-agent use its design knowledge to orchestrate a system for each query (roles, nodes and their interactions), while a compiler handles execution, so no code writing or fine-tuning is needed. All methods share the same executor. We go through the components you listed:

  • Typed IR and deterministic compilation: the IR is the compiler's input, and the compiler is the only way an IR design runs, so neither can be removed alone. Removing both means writing the system in another form, as our baselines do: code (ADAS; Hu et al., 2025), graphs (GPTSwarm; Zhuge et al., 2024), searched prompts and topologies (MASS; Zhou et al., 2026), dynamic agent teams (DyLAN; Liu et al., 2024), or a fine-tuned designer (MAS-GPT; Ye et al., 2025). Without evolution or any dev data, TIDE beats all of them on average. ADAS is the closest comparison: the same executor and a code-specialized designer of similar size (Qwen3-Coder-30B), but code instead of typed IR; it averages 62.9 vs. 70.9.
  • Static validation: this is a fully matched comparison: the same models, primitives, prompts and IR, with only the checks and repairs before execution switched off (w/o Evolution). Accuracy drops on every benchmark: AIME −12.2, MATH −9.2, HumanEval+ −21.7 and MBPP −29.4 points. Only 66.4% of first designs pass validation (68.5% after evolution).
  • Trace analysis and evolution: trace analysis is the input to evolution, so the two cannot be separated: without evolution its findings change nothing, and without traces evolution has nothing to learn from. Overall, 62.9 (code search) < 70.9 (typed IR) < 74.8 (typed IR + evolution): most of the gain comes from the representation, and evolution adds to it.

W2. Limitations. We agree and have added a discussion of these points:

  • Dynamic control flow: acyclic graphs do limit open-ended interaction. This trades some freedom for determinism and safety. TIDE still supports bounded loops and conditional routing, so every design stays checkable before it runs.
  • Primitive set: like Lego bricks, a few general primitives can build rich systems, in line with MASS's finding that only a small fraction of topologies matter. A new primitive needs only one registry entry and one lowering rule, so specific application domains can easily adapt their own primitive sets; better primitive sets are future work.
  • Dev feedback: evolution works best with verifiable feedback, a general limit of self-improvement, though learning from experience has proved effective (e.g., ReasoningBank; Ouyang et al., 2026). Without a dev set, TIDE w/o Evolution still beats all baselines on average and transfers to a new domain and model (Table 9).

The main revisions are marked in blue in the revised PDF. We sincerely appreciate the effort you have invested in reviewing our work. We hope these clarifications address your concerns. Thank you again for your time!

Reviewer_eHLTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TIDE, a framework for automatically designing LLM-based multi-agent systems using a typed IR. A meta-agent generates task-specific agent roles, instructions, and interaction structures, which are validated and compiled into executable systems by a deterministic compiler. An experience-guided evolution mechanism further improves design prompts using execution traces without updating model parameters.

Experiments on four math and coding benchmarks show that TIDE achieves 78.0% average accuracy, outperforming the strongest baseline (68.7%). Additional analyses examine system reliability, efficiency, architectural evolution, failure modes, and model scaling.

Critical strengths

1. Well-designed IR and deterministic compiler for MAS.

The paper presents a well-structured typed IR and deterministic compiler that effectively decouple MAS design from implementation, enabling flexible agent specifications while ensuring structural validity and reliable execution. I particularly appreciate the thoughtful design details, such as slot-based communication contracts, which automatically determine agents' read/write permissions and help prevent common coordination errors.

2. Effective experience-guided design evolution.

The paper introduces a simple yet effective mechanism for improving MAS designs through execution feedback. By extracting reusable lessons from successful and failed traces, the framework progressively refines the meta-agent's design policy without updating model parameters or introducing additional test-time search.

Critical weaknesses

Limited component-wise ablation studies. While the paper demonstrates the effectiveness of experience-guided evolution, it lacks ablation studies that isolate the contributions of other key design components, such as the typed IR, deterministic compiler, and slot-based communication constraints.

Questions for authors

Could the authors provide a concrete example showing how an interaction primitive, such as fan-out merge or iterative refinement, is compiled into typed communication channels and slot-level read/write sets?

3 comments
AuthorsOpenReview ↗

Response to Reviewer eHLT

Thank you for your constructive feedback. We are glad that you found the typed IR and compiler well designed, and that you appreciated the slot-based contracts. Letting the meta-agent design freely while keeping every design verifiable before it runs is exactly what TIDE aims for. In short: following your suggestion, we added component ablations, which show that validation, made possible by the typed IR, is the largest single contributor; we also added the compilation example you requested. Below, we address your points one by one.

W1. Component-wise ablations. Thank you for this suggestion. We should have explained one point more clearly in the paper: the typed IR is the only link between design and execution, so the IR and the compiler cannot be removed on their own. The IR is the compiler's input, and without them TIDE becomes a code-writing designer like ADAS (Hu et al., 2025; average 62.9 with the same executor, vs. 70.9 for TIDE w/o Evolution). We therefore disabled, one at a time, each mechanism that the IR and compiler provide. All runs use TIDE without evolution; each cell shows the accuracy change from the full system, averaged over 3 runs:

Mechanism switched offAIMEMATHHumanEval+MBPP
Validation & repair (typed IR)−12.2−9.2−21.7−29.4
Role safeguards (slot write rules)+2.2−1.0−0.9−2.0
Per-query design (one fixed structure)−1.1−0.20.0−0.3
  • Validation and repair matter most, and they exist because the IR is typed: every design is checked before it runs, and errors are fixed from structural feedback alone. Only 66.4% of first designs pass validation (68.5% after evolution). Repair takes just 0.445 extra rounds per design on average (0.425 after evolution).
  • Role safeguards (slot write rules): slot-based contracts are how the compiler derives every node's read and write sets, since node specifications name no channels, so the slot structure itself cannot be removed. We therefore ablate its write rules, which prevent a failure we observe in traces: a correct answer is found, and then a reviewer or verifier replaces it with a wrong one. With the safeguards, judging roles can write only verdicts and feedback, and only the final step writes the answer. Removing them lowers accuracy on MATH, HumanEval+ and MBPP; the AIME change is less than one problem per run, within run-to-run noise.
  • A fixed structure: across all benchmarks, we took the structure TIDE generated most often before evolution and used it for every problem. It loses little here because the problems within each benchmark are similar, and each agent's instructions are still written per problem. This matches a finding of MASS (Zhou et al., 2026), one of our baselines, that agent prompts are often the decisive factor, and only a small fraction of topologies make a real difference. Per-query design is most useful when tasks vary, as on the software-development benchmark rSDE-Bench (Table 9), where TIDE w/o Evolution beats hand-designed software-development systems with the same model (56.6% vs. 51.8% for ChatDev (Qian et al., 2024) and 8.3% for MetaGPT (Hong et al., 2024)).

Q1. How an interaction primitive compiles. Thank you for this question. A concrete example indeed makes the design much clearer. For fan_out_merge(dispatcher: d, workers: [w1, w2], merger: m), the compiler creates two block-local channels, plan and branch, and derives each node's access from its slot:

Node (slot)ReadsWrites
d (dispatcher)task, upstream outputplan
w1, w2 (worker i)planbranch[i]
m (merger)branchanswer

Because each worker writes to its own entry, the merge order is fixed no matter which worker finishes first. In iterative_refinement, the evaluator reads the draft but can write only its verdict and feedback, so a critic cannot overwrite a correct answer. We have added this example to Appendix H (Table 12).

The main revisions are marked in blue in the revised PDF. We sincerely appreciate the time and expertise you have devoted to our paper, and we hope these additions address your concerns. We would be grateful if you would take them into account in your final assessment.

Thank you for the detailed response and the additional experiments. I appreciate the authors' efforts in addressing my concerns. However, I remain uncertain about the distinct contributions of the key components. Therefore, I will maintain my original rating of 2 (Weak Reject).

AuthorsOpenReview ↗

Response to Reviewer eHLT

Thank you for your reply and for acknowledging the additional experiments. We would like to make sure we address exactly what remains uncertain, so here is a short summary of what each key component contributes.

ComponentMeasured contribution
Typed IR + compiler (vs. writing code)ADAS, a designer that writes each system as full code: 62.9 average vs. 70.9 for TIDE w/o Evolution, which uses no dev data
Static validation and repairSwitching it off: −12.2 / −9.2 / −21.7 / −29.4 on AIME / MATH / HumanEval+ / MBPP. Only 66.4% of first designs are valid
Slot write rulesRemoving them lowers MATH, HumanEval+ and MBPP (−1.0 / −0.9 / −2.0)
Experience-guided evolution+3.9 on average on the held-out test splits, improving all four benchmarks

The IR and the compiler cannot be switched off separately: the IR is the compiler's input, and the compiler is the only way a design runs. Likewise, the slot structure is how the compiler derives each node's read and write sets, since node specifications name no channels. We therefore isolated each mechanism they provide, and used the code-writing designer as the closest alternative representation.

If this summary resolves your uncertainty about the components, we would be grateful if you would consider raising your score. If not, could you let us know which component you find least clearly separated, or which comparison would resolve it? We will do our best to provide it within the remaining discussion period.

Reviewer_7AKbOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors use a typed intermediate representation to let an LLM design multi-agent systems without directly writing their executable code. A deterministic compiler turns these designs into runnable workflows, while a feedback-based evolution process uses previous successes and failures to improve future designs without updating model weights

Critical strengths

This paper focuses on separating MAS design from implementation, which makes automatically generated systems easier to validate, execute, and debug. The reported results are also promising compared to the listed baseline.

Critical weaknesses

1.In table 6, the authors claim that the benchmarks are divided into development and test sets. The development data are then used to evolve the task-family prompts through scored execution traces. However, the reported results appear to use at least some of the development examples again during evaluation. Table 2 reports 35.0% before evolution and 51.7% after evolution.Table 7 explicitly audits the 60 AIME tasks, reporting 39 incorrect answers before evolution and 29 afterward.These correspond exactly to the Table 2 results.

2.Table 3 evaluates executor scaling across five benchmarks, including AIME 2025 and AIME 2026. However, its reported "Average" is calculated using only GPQA, MBPP, and LiveCodeBench. The two AIME results are excluded without an explanation in the table or caption, while the similar scaling accuracy table 4 is fine.

3.The paper argues that typed IRs improve system reliability and design quality by separating architecture generation from code implementation. The main comparison evaluates TIDE against different MAS optimization methods, each with its own architecture, prompts, and search strategy. These differences make it difficult to attribute the performance gains specifically to the proposed representation. Further ablations are required

4.The authors report that only one of 3,448 executions produces an invalid IR. However, the implementation allows multiple regeneration attempts when validation fails, so this number does not tell us how frequently the meta-agent produces invalid designs initially. Authors should run a clearer analysis for this (average repair attempts, compilation failures, runtime failures, etc.)

5.The paper describes the evolution loop as learning reusable design strategies from execution traces. But the actual mechanism is relatively straightforward: an LLM reads successful and failed traces, summarizes useful lessons, rewrites the task-family prompt, and keeps the update if development accuracy improves. This is closely related to existing reflection-based prompt optimization.

6.Baseline fairness: TIDE uses a Qwen3-32B meta-agent to generate each design and Qwen3-4B executors to solve the task. Although Table 2 reports the executor model for every baseline, it is not sufficiently clear whether the automatic-design baselines receive comparable meta-agent capability, optimization budgets, tool access, and development feedback. Also, main results lack confidence intervals and multiple-seed evidence

7.The overall structure of the paper is understandable, but the visual presentation is inconsistent and some figures and tables look insufficiently checked ( inconsistent figure formatting for3, 4 figure 5-8, figure4 is oversized, table 9 is also in bad positioning). The paper acknowledges AI-assisted language editing. I cannot tell whether these formatting problems were caused by LLM assistance, but the final manuscript gives the impression of insufficient manual review after AI-assisted polishing

Questions for authors

See weaknesses listed above

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7AKb

We sincerely thank you for your careful review and constructive feedback. We are glad you value separating MAS design from implementation for easier validation and debugging. Following your close reading, we now report test-split results and fix the Table 3 average. In short: all main conclusions hold on the held-out test splits, and new ablations show where the gains come from. We first address W1, W3 and W6.

W1. Evaluation on held-out test splits. You are right: evolution used only the dev split, but our main results (Table 2) were computed on the full benchmarks. We re-ran TIDE on the test splits only, with 3 seeds:

Test split onlyAIMEMATHHumanEval+MBPPAvg
TIDE w/o Evolution37.8 ± 5.191.1 ± 0.282.3 ± 0.572.3 ± 2.470.9
TIDE w/ Evolution46.7 ± 3.392.8 ± 0.484.3 ± 0.075.2 ± 1.474.8

The conclusions hold: evolution improves all four benchmarks (+3.9 on average), TIDE has the best average (74.8 vs. 68.7 for the strongest baseline), and even w/o Evolution, which uses no dev data, beats every baseline on average. We withdraw the claim of being best on every benchmark.

W3. Where the gains come from. Instead of fixing one architecture by hand or by search, TIDE lets an LLM meta-agent, which already has rich design knowledge (Hu et al., 2025), design each query's system (roles, nodes and their interactions), while a compiler handles execution. The typed IR is the only link between the two, so it cannot be removed on its own. Without it, TIDE becomes a code-writing designer like ADAS, which averages 62.9 with the same executor, vs. 70.9 for TIDE w/o Evolution. We therefore disabled, one at a time, each mechanism it provides (TIDE w/o Evolution; change vs. the full system, mean of 3 runs):

Mechanism disabledAIMEMATHHumanEval+MBPP
Validation & repair−12.2−9.2−21.7−29.4
Role safeguards+2.2−1.0−0.9−2.0
Per-query design−1.1−0.20.0−0.3
  • Validation and repair, which the typed IR makes possible, matter most: only 66.4% of first designs are valid (see also W4).
  • Role safeguards let judging roles write only verdicts and feedback, so they cannot overwrite a correct answer. They help on three benchmarks; on AIME, +2.2 is less than one problem per run.
  • Per-query design: one fixed structure (TIDE's most frequent pre-evolution structure across all tasks) loses little: problems within each benchmark are similar in form, and each agent's instructions are still written per problem. When tasks vary, as on the software benchmark rSDE-Bench (Table 9), TIDE w/o Evolution beats hand-optimized MAS with the same model (56.6% vs. 51.8% for ChatDev and 8.3% for MetaGPT).

W6. Baseline fairness. As is standard, all methods share the executor, but designers may differ. Designer: ADAS uses Qwen3-Coder-30B as its meta-agent and MAS-GPT its official 32B designer fine-tuned from Qwen2.5-Coder-32B, whereas ours is off the shelf; even with an 8B designer and no evolution, TIDE beats every baseline on MBPP (Table 4). Budget and dev feedback: TIDE w/o Evolution uses no search or dev data, while search-based baselines use the dev split. Tools: in a new experiment without any tools, TIDE w/o Evolution averages 70.1, above every baseline. Seeds: Table 2 now reports TIDE's 3-seed results (W1).

Other points.

  • W2 (Table 3 average). Thank you for catching this error. The corrected average covers all five benchmarks: 30.0 → 36.9 → 41.0 from 8B to 32B executors, and the conclusion is unchanged.
  • W4 (validity before repair). The 1 invalid design in 3,448 is counted after repair. On the first attempt, 66.4% of designs are valid (68.5% after evolution). Repair is part of the engine and uses only structural feedback. Beyond the first generation, a design needs 0.445 repair rounds on average (0.425 after evolution), at most 5.
  • W5 (relation to prompt optimization). We agree the loop is deliberately simple and do not claim a new prompt optimizer. What is new is what it updates: how each query's system is designed, instead of one fixed pipeline's prompt.
  • W7 (presentation). The issues you note are all in the appendix, where Figures 4–8 and Table 9 did not match the style of the main text. All figures and tables were made and placed by hand. We have now unified them: Figure 4 is redrawn smaller in Figure 3's style, Figures 5–8 share one scale, and Table 9 (now 10) uses normal font size.

The main revisions are marked in blue in the revised PDF. We sincerely appreciate the effort you have put into reviewing our work. Your comments have made our evaluation more rigorous. We hope the revised results and new ablations address your main concerns, and we would be grateful if you would reconsider your score.

Reviewer_7AKbOpenReview ↗

Response to Authors

Dear Authors,

Thank you for your responses, which address most of my concerns about the presentation and experiments. However, the scope remains narrow, and the self-evolution mechanism offers only an incremental extension of existing feedback-driven prompt and workflow optimization methods. Thus, I will raise my score to 2.

AuthorsOpenReview ↗

Response to Reviewer 7AKb

Thank you for raising your score. We would like to briefly respond to the two remaining points.

  1. Evolution. We agree it builds on feedback-driven optimization, and we do not claim it as a new optimizer. Our core contribution is the typed IR and compiler, and the IR is also what makes evolution work. Evolution is only as effective as the representation it acts on. In raw code, a small edit can break the program or change many things at once. In an untyped graph, an edit says little about what each agent does. In TIDE, every design is built from a small set of typed primitives with explicit roles and data flow, so each change is a meaningful local change to the system, and it is checked before it runs. This is why a simple loop is enough. In Table 2, ADAS, which searches over code, averages 62.9, and GPTSwarm, which optimizes a graph, averages 63.8. Without evolution or any dev data, TIDE already reaches 70.9 and beats every baseline, and the evolution then adds 3.9 points and improves all four benchmarks. We kept the loop simple on purpose, to show that the IR is worth evolving on before adding a complex optimizer whose gains would be hard to attribute. Stronger evolution strategies can build on the same IR, and we see this as future work.

  2. Scope. Besides the four main benchmarks, the paper covers GPQA and LiveCodeBench (Tables 3 and 4) and multi-file software development on rSDE-Bench with a different model (Table 9), where TIDE outperforms ChatDev, AutoGen and MetaGPT. Broader agent settings are valuable future work.

If these clarifications address your remaining concerns, we would be grateful if you would consider raising your score further. Thank you again for the constructive discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.