acceptodds
Under review as a conference paper at ICLR 2027

MARS: Mechanism-Aware Redesign Search for Agentic Systems

Abstract

Automated agent design reduces manual engineering by searching for effective agentic systems. Representing agents as executable code enables a broad range of designs, but also creates a vast search space that is difficult to navigate. In this expansive space, coarse-grained evaluation feedback makes it difficult to attribute performance bottlenecks to specific parts of an agent’s design, resulting in a lack of targeted guidance during search. We introduce Mechanism-Aware Redesign Search (MARS), which establishes an explicit correspondence between executable agent programs and a mechanism-level semantic representation of their composition, instantiation, and interaction. This representation aligns agent design, failure attribution, and redesign at a common mechanism-level granularity. Grounded in executable code and observed execution, MARS attributes failures to individual mechanisms and their interactions, using the resulting diagnoses to guide targeted redesign. Across five tasks, MARS discovers agents that outperform the strongest automated-design baseline on each task while consuming fewer inference-time LLM tokens. Notably, MARS surpasses the best results achieved by all compared search baselines within the first two generations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LLM Agent, Automated Design, Agent Search, Workflow Generation, Agentic Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yy62OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MARS, which guides agentic program search through mechanism-level representations and failure diagnosis. Each candidate contains both executable code and a mechanism IR describing functional units, their implementations, and input–output relationships. Using execution records, the method attributes problems to individual mechanisms or to information flow and control relationships between them, then modifies the program through operations such as addition, deletion, replacement, and rewiring. The search can jointly adjust model calls, prompts, and deterministic computation. Results across five tasks show that the discovered programs outperform those produced by the compared search methods while using fewer inference tokens than the highest-scoring baseline on each task.

Critical strengths
  1. Using a common functional description to connect design, diagnosis, and modification is intuitive. Distinguishing the generation of an incorrect answer from the loss of a correct result downstream can give the search more specific directions for improvement. Mechanism implementations also allow freely generated code, enabling the division of work between model reasoning and programmatic computation to adapt to the task.
  2. The paper reports improvements on all five tasks and provides evolution histories that readers can inspect. On Logic-Q, the final program reaches 99.8% accuracy without invoking a language model at runtime, and unsuccessful intermediate parser implementations are also documented. The analysis of test sources and early stopping in coding tasks further demonstrates that the search can adjust workflows in response to specific execution problems.
Critical weaknesses
  1. AFlow already uses execution feedback to modify workflows, while Trace provides computational dependencies, intermediate results, and operator descriptions to an optimizer. MARS adds mechanism categories and layered diagnosis, but information flow and control relationships remain expressed in code, and concrete modifications are still generated through model-based analysis. I therefore see the main contribution as a more detailed organization of information for reflection. This is useful, but the current comparisons do not separate the contribution of the mechanism representation from that of additional diagnostic feedback. It remains unclear whether the representation constitutes a sufficiently substantial methodological advance.
  2. The paper emphasizes surpassing baselines within the first two generations, but its search curves report the highest test score observed up to each generation, whereas the algorithm selects candidates by validation score. This shows that good programs were generated early, but does not necessarily mean that the search would return them if stopped after two generations. Each generation also includes failure analysis and diagnostic aggregation, so generation count is not a direct measure of search cost. The reduction in inference tokens used by the final programs is clear; the advantage of the search process itself needs to be characterized more precisely.
Questions for authors
  1. With the same code search space and failure records, how does direct reflection on code compare with diagnosis using the mechanism IR? Are there existing controlled comparisons that isolate the contribution of the mechanism representation itself?
  2. If each generation reports only the candidate selected by validation performance, does the claim of surpassing baselines within the first two generations still hold? By that point, how much resource has been consumed by candidate generation, diagnosis, and validation, respectively?
5 comments
AuthorsOpenReview ↗

Response to Reviewer yy62 (Part 1/3)

We sincerely thank the reviewer for the careful reading of our paper and the positive assessment of MARS, particularly its methodological design and experimental results. We also greatly appreciate the insightful comments and constructive suggestions, which have helped us further clarify our contributions and improve the paper.

We address the concerns as follows.

Weakness 1 & Question 1: Methodological Contribution and Controlled Ablations

(1) Clarifying the methodological contribution of MARS.

We thank the reviewer for recognizing the value of mechanism-level information organization and for pointing out that the current experiments do not sufficiently distinguish the contributions of mechanism representation and diagnostic feedback. We would like to further clarify the core distinction between MARS and related work.

AFlow uses Workflow code, discrepancies between predictions and ground-truth answers, performance scores, and historical modification experience to guide workflow optimization. However, its feedback mainly concerns the overall predictions, expected outputs, and performance changes of the Workflow. It does not systematically record and use the actual execution inputs, intermediate outputs, and runtime states of individual Operators to localize component-level problems. Consequently, the optimizer primarily infers potential issues from the code structure and overall outcome feedback, without explicitly attributing execution failures to specific mechanisms and their intended responsibilities.

Trace represents workflow execution as a computational graph, recording operator dependencies, intermediate computational results, and how these results are used. It combines operator descriptions with output feedback to guide parameter updates. However, Trace does not explicitly establish a correspondence between the functional semantics of Agent mechanisms and actual execution failures. It therefore lacks a process for diagnosing actual execution behavior against design expectations at the mechanism level. In contrast, MARS uses Mechanism IR to explicitly describe the intended responsibilities, design-time inputs and outputs, and functional relationships of individual mechanisms. It aligns Agent design and failure diagnosis at the same functional-mechanism granularity, allowing actual execution evidence to reveal whether a mechanism fulfills its intended responsibility and whether a problem originates from interactions between mechanisms. These diagnoses then guide targeted redesign.

(2) Independent contributions of Mechanism IR and Structured Diagnosis.

We thank the reviewer for this important suggestion. To address the request for controlled comparisons, we conducted two additional ablation studies. We kept the code search space, failure records, search budget, and evaluation settings the same, and separately examined the contributions of mechanism representation and structured diagnosis:

  • w/o Mechanism IR: Remove Mechanism IR and perform reflection and search directly from Agent code and execution records.
  • w/o Structured Diagnosis: Retain Mechanism IR but remove the structured failure-attribution process.

The additional results are shown below. Values in parentheses denote the average inference tokens used by the final Agent per test example.

MethodE2H PassDROP F1GPQA Acc.HotpotQA F1Logic-Q Acc.
MARS (Full)87.0 (8932)91.9 (620)79.3 (2651)80.7 (3205)99.8 (0)
w/o Mechanism IR78.0 (19898)88.5 (1099)71.9 (1803)76.7 (1423)45.4 (6339)
w/o Structured Diagnosis72.8 (16727)88.7 (1190)75.7 (2484)77.4 (1456)68.0 (2283)

As shown, removing either Mechanism IR or Structured Diagnosis reduces performance on all five tasks. This indicates that mechanism-level representation and structured failure diagnosis both play important roles in translating execution feedback into targeted mechanism-level optimization, and that they jointly support the search effectiveness of MARS.

AuthorsOpenReview ↗

Response to Reviewer yy62 (Part 2/3)

Weakness 2 & Question 2: Candidate Selection, Reporting Protocol, and Search Cost

(1) Clarifying the candidate-selection rule and reporting protocol.

We thank the reviewer for carefully reading the paper and identifying that the candidate-selection and result-reporting logic was not sufficiently explained in the original version.

To clarify, MARS always selects the current incumbent according to the Validation Score during the actual search and uses it as the starting point for the next optimization round. Test Score does not participate in search decisions. Table 1 of the paper uses a Test-best reporting protocol primarily to ensure consistency with the reporting protocols of the other baselines, particularly ADAS, which uses the same protocol. Moreover, the performance advantage of MARS does not depend on Test-best reporting: even under a Validation-selected protocol, MARS outperforms all compared baselines on all five tasks. To further verify this, we compiled the Validation-selected candidates' performance from our existing experimental records.

(2) Validation-selected performance comparison.

We first recomputed the final results of all methods under a unified Validation-selected rule. The table reports the Test Set performance of the candidate selected by Validation Score, together with its average Test Inference Tokens (in parentheses).

MethodE2H PassDROP F1GPQA Acc.HotpotQA F1Logic-Q Acc.
ADAS79.4 (55750)79.7 (10119)70.2 (5637)77.3 (14208)47.8 (14622)
AFlow71.6 (19342)86.8 (1023)74.3 (1982)77.5 (13739)48.2 (6077)
MaAS57.6 (12439)53.6 (2964)40.4 (6826)64.9 (9063)32.4 (6929)
MARS (Ours)87.0 (8932)91.4 (463)78.9 (1996)79.6 (1647)99.8 (0)

The results show that, even when the Validation-selected candidate-selection rule is used consistently, MARS achieves the highest Test Performance on all five tasks.

(3) Actual search cost at Test-best First-Surpass.

To further address the concern about the actual cost of the search process, we follow the original paper's Test-best reporting protocol and calculate MARS's cumulative search cost when it first surpasses the strongest baseline. We compare this cost with the corresponding baseline costs.

The cost accounting covers candidate generation, case analysis, structured diagnosis, and candidate evaluation. Since DeepSeek API prices may vary with billing periods and pricing adjustments, we use token consumption as a unified measure of resource usage. We separately report cumulative token consumption for DeepSeek-V4-Pro (Preview, Thinking High; P) and DeepSeek-V4-Flash (Preview, Non-thinking; F), without mixing tokens from different models. This provides a more transparent and reproducible comparison that does not depend on a particular billing period or price.

1. Test-best First-Surpass vs. Baseline Full-search.

We first compare the cumulative cost when MARS first surpasses the strongest baseline in performance with the cumulative cost incurred by the corresponding baseline over its entire search.

TaskMARS First-SurpassMARS (P/F)Best Baseline (P/F)Ratio (P/F)
E2HGen 1 (82.0%)144119 / 690137776861 / 663782345.39× / 96.18×
DROPGen 1 (91.2%)77184 / 10668966284 / 21332480.86× / 20.00×
GPQAGen 2 (78.2%)119172 / 455592477995 / 236089504.01× / 51.82×
HotpotQAGen 2 (79.6%)130010 / 926056582090 / 367095004.48× / 39.64×
Logic-QGen 2 (80.0%)175428 / 970363566186 / 285857563.23× / 29.46×
AuthorsOpenReview ↗

Response to Reviewer yy62 (Part 3/3)

Weakness 2 & Question 2 (continued): First-Surpass Search Cost

2. Test-best First-Surpass vs. Baseline Test-best.

We further narrow the comparison to the point at which the baseline reaches its own best test performance, rather than the end of its entire search. Specifically, we compare the cumulative cost when MARS first reaches or surpasses the strongest baseline's Test-best performance with the cumulative cost incurred by that baseline to reach its own Test-best. This more directly reflects the search resources required by each method to obtain the corresponding performance.

TaskMARS First-SurpassMARS (P/F)Best Baseline (P/F)Ratio (P/F)
E2HGen 1 (82.0%)144119 / 690137776861 / 663782345.39× / 96.18×
DROPGen 1 (91.2%)77184 / 10668948691 / 14319830.63× / 13.42×
GPQAGen 2 (78.2%)119172 / 455592247897 / 171237932.08× / 37.59×
HotpotQAGen 2 (79.6%)130010 / 926056166145 / 186014681.28× / 20.09×
Logic-QGen 2 (80.0%)175428 / 970363209140 / 193947101.19× / 19.99×

Here, the P/F Ratio denotes the ratio of the baseline's cumulative Pro/Flash token consumption to that of MARS. A value greater than 1 indicates that MARS consumes fewer tokens for the corresponding model.

3. Test-best First-Surpass cost breakdown.

Finally, to further address the cost of candidate generation, failure analysis, diagnosis, and candidate evaluation, we break down MARS's cumulative cost at Test-best First-Surpass by stage. All costs are accumulated up to the corresponding First-Surpass Generation.

TaskFirst-SurpassSearchCase SummaryDiagnosisPro TotalEvaluation (Flash)
E2HGen 1170739559231454144119690137
DROPGen 120900417861449877184106689
GPQAGen 2280677064220463119172455592
HotpotQAGen 2278438250519662130010926056
Logic-QGen 24253910361429275175428970363

The cost components are defined as follows:

  • Search: Candidate generation and modification.
  • Case Summary: Reconstruction, analysis, and summarization of failure-case execution paths.
  • Diagnosis: Cross-case issue aggregation and structured mechanism-level failure attribution.
  • Pro Total: Cumulative Pro-model token consumption in Search, Case Summary, and Diagnosis.
  • Evaluation (Flash): Flash-model token consumption during candidate evaluation, including all input and output tokens.

These statistics account for cumulative token consumption in candidate generation, case analysis, structured diagnosis, and candidate evaluation, rather than measuring search efficiency only by the number of generations.

(4) Summary of search-cost comparisons.

Across the two cost-comparison protocols, on E2H, GPQA, HotpotQA, and Logic-Q, MARS consumes fewer cumulative Pro and Flash tokens at First-Surpass than the corresponding baseline.

For DROP, although MARS consumes more Pro tokens, it substantially reduces Flash token consumption. Based on the API prices of the two models, the overall search cost of MARS at First-Surpass is still lower.

Therefore, even when the costs of candidate generation, case analysis, structured diagnosis, and candidate evaluation are all included, MARS reaches performance above the strongest baseline with a lower overall search cost.

Reviewer_3KhzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MARS, a framework for automatically redesigning agent programs through execution-grounded failure diagnosis. Each candidate combines executable code with a semantic representation of its functional components, implementations, and interactions. The system diagnoses component and interaction failures, then uses these diagnoses to guide structural and implementation changes. Experiments on five benchmarks report higher test scores and lower inference token usage than the best-scoring automated-design baseline on each task. Case studies illustrate iterative code refinement and the replacement of model calls with deterministic computation.

Critical strengths
  • Overall, the paper is well written and easy to follow, with clear definitions and helpful diagrams.
  • The paper addresses a meaningful problem: translating observed execution failures into targeted changes to agent programs.
  • Experimental results show higher test scores than the compared baselines across all five benchmarks.
  • The discovered agents use fewer inference tokens than the best-scoring automated-design baseline on each task.
Critical weaknesses
  • The paper lacks ablations isolating the contributions of the semantic representation and mechanism-level diagnosis.
  • Comparisons with closely related optimizers are incomplete: Meta-Harness is absent, while GEPA is discussed but not evaluated.
  • Independent search repeats and uncertainty intervals for the main comparisons are not reported, making the smaller gains difficult to assess.
  • Evaluation uses only one search/execution model pair, leaving robustness to other model families untested.
  • The paper does not clearly explain which distinct failure modes motivate the selection of the five benchmarks.
Questions for authors
  • Under matched models and budgets, how much performance remains after removing the semantic representation or mechanism-level diagnosis?
  • How does MARS differ from Meta-Harness in feedback access and redesign guidance, and which differences account for its expected advantages?
  • Does mechanism-level diagnosis remain useful when coupled with tree-based or population-based search?
  • How consistent are the GPQA and HotpotQA gains across independent search runs?
  • What sanity checks validated the MaAS adaptation to the selected models and tasks, particularly given its results below CoT on four benchmarks?
  • Can the paper provide a unified cost breakdown covering candidate generation, diagnosis, and evaluation, together with end-to-end latency?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 3Khz (Part 1/4)

We sincerely thank the reviewer for the careful reading of our paper and for recognizing the importance of translating observed execution failures into targeted Agent-program improvements, as well as the clarity of our presentation and the performance and inference-token efficiency across five benchmarks. We also greatly appreciate the insightful comments and constructive suggestions, which have helped us further improve the paper.

We address each point below.

Weakness 1 & Question 1: Independent Contributions of Semantic Representation & Mechanism-level Diagnosis

To distinguish the contributions of Mechanism IR and Structured Diagnosis, we conducted two additional ablation studies.

For w/o Mechanism IR, we kept the same code search space, execution-failure records, and search budget, while removing only the mechanism-level semantic representation. Reflection and modification were instead performed directly from Agent code and execution feedback. For w/o Structured Diagnosis, we retained Mechanism IR but removed the structured failure-attribution process. Both ablations used the same model configuration and evaluation settings.

The results are shown below. Values in parentheses denote the final Agent's average inference-token consumption per test example.

MethodE2H PassDROP F1GPQA Acc.HotpotQA F1Logic-Q Acc.
MARS (Full)87.0 (8932)91.9 (620)79.3 (2651)80.7 (3205)99.8 (0)
w/o Mechanism IR78.0 (19898)88.5 (1099)71.9 (1803)76.7 (1423)45.4 (6339)
w/o Structured Diagnosis72.8 (16727)88.7 (1190)75.7 (2484)77.4 (1456)68.0 (2283)

These results show that both ablated variants achieve lower test performance than full MARS on all five benchmarks. This demonstrates that both components make practical contributions to MARS's performance.

Weakness 2 & Question 2: Differences from GEPA & Meta-Harness

(1) GEPA: Difference in optimization target and experimental scope

The main difference between GEPA and MARS lies in their optimization targets. GEPA iteratively optimizes the Prompts of existing modules in a task-specific, predefined Compound AI System by analyzing execution trajectories. Lines 13–14 of Algorithm 1 in the GEPA paper explicitly use UpdatePrompt to update the Prompt of a selected module and replace that module in the current candidate system.

Thus, GEPA studies Prompt Optimization within a given system architecture, whereas MARS studies open-ended Agent Design Search. Extending GEPA to search over complete Agent code and mechanism architectures would change its original optimization target and search setting. For this reason, we did not directly include the original GEPA as a baseline for Agent architecture search.

(2) Meta-Harness: Difference in feedback organization and redesign guidance

First, Meta-Harness starts optimization from a population of existing effective Harnesses, whereas MARS generates Agents from task requirements and does not rely on a pre-designed complete system.

More importantly, Meta-Harness primarily relies on a Coding Agent to autonomously infer modification directions from historical code, execution trajectories, and evaluation results. MARS, in contrast, uses Mechanism IR to associate execution evidence with the functional mechanisms of an Agent and their interactions. Structured Diagnosis then identifies specific design problems and provides targeted guidance for adjusting mechanism composition, instantiation, and interactions.

Therefore, the key distinction of MARS is not that it obtains more execution feedback, but that it converts execution evidence into mechanism-level design guidance, enabling Agent search to explore targeted solutions to identified design bottlenecks. We will add a citation to and discussion of Meta-Harness in the Related Work section of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 3Khz (Part 2/4)

Question 3: Combining Mechanism-level Diagnosis with Tree-based & Population-based Search

We believe that mechanism-level diagnosis is complementary to Tree-based Search and Population-based Search. A search strategy determines how candidate solutions are selected and explored, whereas mechanism-level diagnosis further guides which design problems a candidate should address and which alternative modifications can be explored.

For Tree-based Search, the tree structure and selection strategy primarily determine which existing node to expand next and how to allocate exploration resources; they do not directly determine how the current node should be modified. Mechanism-level diagnosis can identify a specific design bottleneck in the current node and propose multiple redesign alternatives based on the same diagnosis, which can then be explored as different child nodes. For example, as illustrated in Figure 1, when the output of Candidate Generation does not satisfy the required format, one can modify the Prompt of the existing mechanism, or add an LLM-based Normalization or Deterministic Normalization mechanism. In this way, tree search can explore multiple possible solutions to the same identified failure.

For Population-based Search, mechanism-level diagnosis can guide targeted mutations of different Agents and reveal the mechanism-level strengths, weaknesses, and potential complementarity of different individuals. For example, two Agents may perform well on different functional mechanisms. Mechanism-level representations and diagnoses can provide a basis for selecting and recombining mechanisms across individuals, allowing the search to combine complementary mechanisms and adjust their interactions to explore new Agent designs.

Therefore, mechanism-level diagnosis enables search to move beyond exploring candidate solutions solely at the level of overall performance and toward exploring the design of internal Agent mechanisms: tree search can explore different fixes for the same failure, while population-based search can use mechanism-level complementarity across individuals to discover new mechanism combinations.

Weakness 3 & Question 4: Independent-search Reproducibility & Performance Uncertainty

(1) Independent search experiments on GPQA and HotpotQA

A complete search and evaluation run requires substantial computational resources, particularly because some baselines generate Agents with high execution costs. Due to limited experimental resources, we did not perform multiple complete searches for every baseline.

For the two benchmarks of particular concern to the reviewer, GPQA and HotpotQA, we conducted two additional independent MARS searches on each benchmark, giving three independent search runs per task.

The results are shown below.

BenchmarkSearch RunMedian95% Bootstrap CI
GPQARun 1 (Original)79.3%[76.5%, 82.0%]
GPQARun 282.5%[79.9%, 85.1%]
GPQARun 381.6%[78.9%, 84.2%]
HotpotQARun 1 (Original)80.7%[78.4%, 82.9%]
HotpotQARun 281.2%[78.9%, 83.5%]
HotpotQARun 380.4%[78.1%, 82.6%]

The summary across the three independent searches is as follows:

BenchmarkMeanStdPerformance Range
GPQA81.1%1.7%79.3%–82.5%
HotpotQA80.8%0.4%80.4%–81.2%

The mean performance across the three independent searches is no lower than the corresponding result in the original main table, and the variation between search runs is small. These results indicate that MARS achieves consistent search performance on both benchmarks.

(2) Performance confidence intervals for the main methods

In addition, we provide 95% Bootstrap confidence intervals for the Test-best performance of all methods on the five benchmarks.

MethodE2H PassDROP F1GPQA Acc.HotpotQA F1Logic-Q Acc.
ADAS79.6 [76.0, 83.0]79.7 [77.0, 82.4]77.5 [74.6, 80.2]78.2 [75.8, 80.5]48.8 [44.4, 53.2]
AFlow71.6 [67.6, 75.4]86.8 [84.6, 88.9]74.3 [71.3, 77.2]77.5 [75.0, 80.0]48.2 [43.8, 52.6]
MaAS57.6 [53.2, 62.0]53.6 [50.7, 56.5]40.4 [37.0, 43.7]64.9 [62.1, 67.6]32.4 [28.4, 36.6]
MARS87.0 [84.0, 89.8]91.9 [90.1, 93.5]79.3 [76.5, 82.0]80.7 [78.4, 82.9]99.8 [99.4, 100.0]

The Bootstrap confidence intervals reflect performance uncertainty at the test-sample level.

AuthorsOpenReview ↗

Response to Reviewer 3Khz (Part 3/4)

Question 5: Sanity Checks for the MaAS Adaptation

For the MaAS adaptation, we followed the experimental configurations explicitly specified in the original paper, including the number of optimization rounds, the number of training repetitions, and the initial Operator set. For implementation details not specified in the paper, we referred to the official code repository for task adaptation.

To validate the adaptation, we performed the following checks. First, we confirmed that the input and output interfaces of each Operator were compatible with the current task and execution framework. Second, we ensured that MaAS and the other baselines used the same base-model configuration, answer parsing, and evaluation pipeline. Finally, we conducted end-to-end checks of the complete search process, candidate architecture execution, and final task evaluation.

Regarding MaAS's performance below CoT on four tasks, one possible explanation is that its predefined Operator space limits the flexibility of mechanism design. In our current task adaptation, most predefined Operators produce single-field outputs. Although MaAS can explore different combinations of Operators through architecture sampling and optimization, its ability to modify the predefined components themselves and explore their interaction patterns remains limited.

In contrast, MARS does not rely on a predefined library of executable components. Its search can freely adjust mechanism composition, instantiation, and interactions, preserving greater flexibility in exploring Agent implementations and their interaction structures.

Question 6: Unified Search Cost & End-to-end Latency

(1) Complete search-cost accounting

To provide a more comprehensive comparison of search overhead, we re-examined the complete experimental cost records and calculated the recordable token consumption of each method over its full search budget, rather than reporting only the average cost per generation or optimization round.

MethodE2H (P/F)DROP (P/F)GPQA (P/F)HotpotQA (P/F)Logic-Q (P/F)
MARS1138021 / 7431013532040 / 905391502162 / 2868887575296 / 4378725395493 / 970363
ADAS776861 / 66378234586300 / 10885412477995 / 23608950582090 / 36709500566186 / 28585756
AFlow145299 / 1694296866284 / 213324862014 / 152973551712 / 11197622114291 / 6610690
MaAS3744 / 56378573843 / 14397181308 / 30902861633 / 74149731325 / 4870123

Note: The unit is tokens. P/F denote:

  • P: DeepSeek-V4-Pro Preview (Thinking High), used for search, optimization, reflection, and diagnosis.
  • F: DeepSeek-V4-Flash Preview (Non-thinking), used for Agent execution and evaluation, or Workflow Sampling during training.

(2) Cost components and evaluation accounting

The cost components of the different methods are as follows:

MethodPro CostFlash Cost
MARSSearch + Case Summary + DiagnosisValidation Evaluation
ADASSearch + Reflection + DebugInitial + Generated Agent Evaluation
AFlowGraph OptimizationInitial Workflow + Subsequent Workflow Evaluation
MaASTextGrad Input + OutputFour Training-time Sampling Repetitions

(3) Cumulative First-Surpass cost and search efficiency

In addition to the complete search costs, we calculate the cumulative search cost when MARS first surpasses the strongest baseline's Test-best performance, to examine the resources required for MARS to discover a high-performing Agent early in the search.

First, we compare the cumulative cost of MARS at First-Surpass with the cost of the corresponding strongest baseline over its complete search.

Table 1. Test-best First-Surpass vs. Best Baseline Full-search

TaskMARS First-SurpassMARS (P/F)Best Baseline (P/F)Ratio (P/F)
E2HGen 1 (82.0%)144119 / 690137ADAS @ Gen 10: 776861 / 663782345.39× / 96.18×
DROPGen 1 (91.2%)77184 / 106689AFlow @ Round 11: 66284 / 21332480.86× / 20.00×
GPQAGen 2 (78.2%)119172 / 455592ADAS @ Gen 10: 477995 / 236089504.01× / 51.82×
HotpotQAGen 2 (79.6%)130010 / 926056ADAS @ Gen 10: 582090 / 367095004.48× / 39.64×
Logic-QGen 2 (80.0%)175428 / 970363ADAS @ Gen 10: 566186 / 285857563.23× / 29.46×
Reviewer_iuXBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MARS, which sets an agent's code next to a semantic description of its mechanisms. The description covers what the mechanisms do, how they are implemented, and how they interact. When validation fails, MARS reads the code, the mechanism description, and execution evidence. It proposes design edits or starts a new design. Five benchmarks test MARS against manually designed agents and against ADAS, AFlow, and MaAS. Table 1 reports the highest test performance for each task. On that task, token use remains below the baseline with the highest score. Case studies cover a workflow that revises code from execution results and a Logic-Q solver that replaces LLM calls with deterministic computation.

Critical strengths
  1. For code-based search, the representation is a workable unit: observed failures map to concrete edits. One sensible design choice separates a poor intermediate result from useful information that is discarded later. Appendix A.6 gives specific parsing failures and code-execution feedback as examples.

  2. MARS scores higher and uses fewer tokens. On E2H-Codeforces it scores 87.0, compared with 79.6 for ADAS, and uses 8932 inference tokens, compared with 87275. On DROP it scores 91.9, compared with 86.8 for AFlow, and uses 620 tokens, compared with 1023 (Table 1). The Logic-Q example shows the search can drop LLM calls entirely when a programmatic solution exists. These results make the approach worth studying beyond its conceptual formulation.

Critical weaknesses
  1. The experiments do not show what the mechanism-level representation contributes or what the diagnosis contributes. The paper argues that this correspondence improves redesign. Yet it never compares MARS with the same code-search procedure using ordinary execution feedback. It does not test a version without the mechanism-level representation, nor one without the structured diagnosis. The Logic-Q trajectory in Table 3 and Appendix A.6 is informative. Each successive candidate changes prompts, parsing, and computational structure at once. It cannot show which part of MARS produces the gains. That gap matters: the paper's main contribution is guidance for search, not the final agent programs.

  2. The paper does not say how candidates were chosen or how much the search changes between runs. Algorithm 1 returns the candidate with the highest validation score; Figure 3 reports the best test score so far. The paper should state whether validation results alone determined each agent in Table 1. It should also separate the test envelope from the trajectory of the validation-selected incumbent. No variation across independent search runs appears, and the main comparisons carry no uncertainty. This matters most for GPQA, where the gain over ADAS is 1.8 points on 166 test questions. Appendix A.3 provides five evaluations per question. These help assess execution variability, but they do not establish reproducibility across searches. Neither gap shows that test results guided the search.

Questions for authors
  1. Can the authors provide a controlled comparison between MARS and conventional feedback-driven code search, while keeping the underlying model, initialization, evaluation budget, and available search operations comparable? In particular, could the authors separately ablate the mechanism representation and structured diagnosis to determine whether these components provide measurable benefits beyond execution-trace feedback? Evaluating these variants on E2H-Codeforces and DROP would help substantiate the central claim of MARS.
  2. State how the Table 1 agents were chosen for each method and how many independent searches were run. Was test feedback kept out of every search and selection decision? Figure 3 should show the best-test envelope apart from the validation-selected incumbent's performance. Give any uncertainty that exists. Group repeated GPQA evaluations by question.
  3. With diagnosis and candidate evaluation included, what does one generation or optimization round cost for each method? Appendix A.4 states that Table 2 uses a different accounting method. Measured the same way, cumulative costs at the first-surpass points would show how the two-generation result relates to search efficiency in practice.
7 comments
AuthorsOpenReview ↗

Response to Reviewer iuXB (Part 1/4)

We sincerely thank the reviewer for the careful review and positive assessment of our work, especially the recognition of MARS's mechanism-level design and failure-attribution approach, as well as its experimental results in terms of performance and inference efficiency. We address the reviewer's comments point by point below.

Weakness 1 & Question 1: Independent Contributions of Mechanism IR and Structured Diagnosis

(1) Clarifying the methodological contribution.

We thank the reviewer for pointing out the need to clarify the respective roles of mechanism representation and structured diagnosis.

The central contribution of MARS is to establish an explicit correspondence between executable Agent programs and mechanism-level semantic representations, aligning Agent design, failure attribution, and targeted redesign at a common mechanism-level granularity.

Specifically, Mechanism IR describes Agent design along three dimensions: composition, instantiation, and interaction, and is explicitly aligned with the executable code. Based on this representation, MARS combines executable code with actual execution evidence to attribute failures to individual mechanisms and their interactions, and uses the resulting diagnoses to guide targeted mechanism-level redesign.

(2) Additional controlled ablation studies.

To address the reviewer's request for controlled comparisons, we conducted two additional ablation studies while keeping the underlying model, initialization, code search space, search budget, available edit operations, and execution feedback conditions the same:

  1. w/o Mechanism IR: Remove the mechanism-level representation and perform reflection and modification directly from Agent code and execution feedback.
  2. w/o Structured Diagnosis: Retain Mechanism IR but remove the structured failure-attribution process.

The additional results are shown below. Each cell reports test performance, with the final Agent's average inference tokens in parentheses.

MethodE2H PassDROP F1GPQA Acc.HotpotQA F1Logic-Q Acc.
MARS (Full)87.0 (8932)91.9 (620)79.3 (2651)80.7 (3205)99.8 (0)
w/o Mechanism IR78.0 (19898)88.5 (1099)71.9 (1803)76.7 (1423)45.4 (6339)
w/o Structured Diagnosis72.8 (16727)88.7 (1190)75.7 (2484)77.4 (1456)68.0 (2283)

Both ablated variants perform below the full MARS system on all five benchmarks. In particular, on the two benchmarks emphasized by the reviewer:

  • E2H-Codeforces: Removing Mechanism IR and Structured Diagnosis reduces the Pass Rate by 9.0 and 14.2 percentage points, respectively.
  • DROP: The corresponding F1 scores decrease by 3.4 and 3.2 percentage points, respectively.

These results show that both Mechanism IR and Structured Diagnosis make measurable contributions to MARS's search effectiveness.

Weakness 2 & Question 2: Candidate Selection, Search Trajectory, and Statistical Reliability

(1) Clarifying candidate selection and the use of Test Scores.

We thank the reviewer for carefully reading the paper and identifying a potential ambiguity between Algorithm 1 and Figure 3. We apologize that the original paper did not sufficiently explain the distinction between the candidate-selection rule and the final result-reporting protocol.

To clarify, MARS always selects the current best candidate according to Validation Score during the actual search and uses it as the basis for the next round of Edit Operations. When candidates have the same Validation Score, the candidate with fewer average inference tokens is preferred.

Table 1 of the original paper uses a Test-best reporting protocol: after the search is completed, it reports the candidate Agent with the highest test performance among all generated candidates. This choice is intended to maintain consistency with the reporting protocols used by existing baselines and to show the best Agent discovered by MARS during the search.

AuthorsOpenReview ↗

Response to Reviewer iuXB (Part 2/4)

Weakness 2 & Question 2 (continued): Candidate Selection, Search Trajectory, and Statistical Reliability

(2) Validation-selected performance comparison.

Although Test-best is used as the main result-reporting protocol for consistency with the baselines, we further examine whether MARS remains competitive when reporting only the best candidate actually selected by the algorithm according to Validation Score.

We therefore provide a performance comparison using a unified Validation-selected rule. For MARS and ADAS, we select the candidate with the highest Validation Score in each method's search process and report its corresponding Test Score and average inference tokens. For AFlow and MaAS, we retain their corresponding best Workflow results.

The additional results are shown below. Each cell reports test performance, with average inference tokens in parentheses.

MethodE2H PassDROP F1GPQA Acc.HotpotQA F1Logic-Q Acc.
ADAS79.4 (55750)79.7 (10119)70.2 (5637)77.3 (14208)47.8 (14622)
AFlow71.6 (19342)86.8 (1023)74.3 (1982)77.5 (13739)48.2 (6077)
MaAS57.6 (12439)53.6 (2964)40.4 (6826)64.9 (9063)32.4 (6929)
MARS (Ours)87.0 (8932)91.4 (463)78.9 (1996)79.6 (1647)99.8 (0)

As shown, even under a unified Validation-selected rule, MARS achieves the highest test performance on all five tasks.

Compared with the strongest automated-search baseline on each task, MARS improves performance by 7.6, 4.6, 4.6, 2.1, and 51.6 percentage points, respectively, for an average improvement of 14.10 percentage points.

(3) Additional Validation-selected Current-best Trajectory.

The original Figure 3 presents the Test-best Envelope, i.e., the highest Test Score among all candidate Agents generated up to each generation. This is consistent with the Test-best reporting protocol used in Table 1 and illustrates the best Agent discovered by MARS during the search.

In the revised paper, we will additionally provide the Validation-selected Current-best Trajectory for all five tasks. This trajectory will show the Test Score of the current best candidate selected according to Validation Score at each generation, separately from the original Test-best Envelope.

We will also revise the experimental setup and result analysis to clarify the distinction between the rule used to select the current best candidate according to Validation Score during the search and the Test-best protocol used to report the final best candidate.

(4) Independent searches and bootstrap confidence intervals.

Because a complete Agent search and evaluation process is computationally expensive, particularly when some baselines generate Agents with high execution costs, our experimental resources did not allow us to run multiple complete searches for every baseline. However, the original Figure 4 shows that multiple candidates generated by MARS lie on the global Pareto frontier, suggesting that the search discovers multiple competitive Agent designs rather than relying solely on a single accidentally strong candidate.

In addition, in response to the reviewer's particular concern about GPQA, we conducted two additional independent MARS searches, giving three independent search runs in total for this benchmark. Each additional experiment reran the complete search process from scratch, rather than repeatedly evaluating the same search result.

The results are shown below. The Median is the test accuracy of each search run, and the values in brackets are the 95% Bootstrap CIs.

BenchmarkSearch RunMedian95% Bootstrap CI
GPQARun 1 (Original)79.3%[76.5%, 82.0%]
GPQARun 282.5%[79.9%, 85.1%]
GPQARun 381.6%[78.9%, 84.2%]

The three independent searches achieve median performance of 79.3%, 82.5%, and 81.6%, respectively. Their results remain at a similar level, indicating that MARS's performance on GPQA is not an accidental outcome of a single search run.

Meanwhile, we provide 95% Bootstrap Confidence Intervals for ADAS, AFlow, MaAS, and MARS on all five benchmarks, under both the Test-best and Validation-selected reporting protocols.

AuthorsOpenReview ↗

Response to Reviewer iuXB (Part 3/4)

Weakness 2 & Question 2 (continued): Statistical Reliability

Test-best: 95% Bootstrap CIs.

Values in parentheses denote average inference tokens, and values in square brackets denote 95% CIs.

MethodE2H PassDROP F1GPQA Acc.HotpotQA F1Logic-Q Acc.
ADAS79.6 (87275) [76.0, 83.0]79.7 (10119) [77.0, 82.4]77.5 (28653) [74.6, 80.2]78.2 (14353) [75.8, 80.5]48.8 (15204) [44.4, 53.2]
AFlow71.6 (19342) [67.6, 75.4]86.8 (1023) [84.6, 88.9]74.3 (1982) [71.3, 77.2]77.5 (13739) [75.0, 80.0]48.2 (6077) [43.8, 52.6]
MaAS57.6 (12439) [53.2, 62.0]53.6 (2964) [50.7, 56.5]40.4 (6826) [37.0, 43.7]64.9 (9063) [62.1, 67.6]32.4 (6929) [28.4, 36.6]
MARS87.0 (8932) [84.0, 89.8]91.9 (620) [90.1, 93.5]79.3 (2651) [76.5, 82.0]80.7 (3205) [78.4, 82.9]99.8 (0) [99.4, 100.0]

Validation-selected: 95% Bootstrap CIs.

MethodE2H PassDROP F1GPQA Acc.HotpotQA F1Logic-Q Acc.
ADAS79.4 (55750) [75.8, 82.8]79.7 (10119) [77.0, 82.4]70.2 (5637) [67.1, 73.3]77.3 (14208) [74.9, 79.6]47.8 (14622) [43.4, 52.2]
AFlow71.6 (19342) [67.6, 75.4]86.8 (1023) [84.6, 88.9]74.3 (1982) [71.3, 77.2]77.5 (13739) [75.0, 80.0]48.2 (6077) [43.8, 52.6]
MaAS57.6 (12439) [53.2, 62.0]53.6 (2964) [50.7, 56.5]40.4 (6826) [37.0, 43.7]64.9 (9063) [62.1, 67.6]32.4 (6929) [28.4, 36.6]
MARS87.0 (8932) [84.0, 89.8]91.4 (463) [89.6, 93.1]78.9 (1996) [76.1, 81.7]79.6 (1647) [77.2, 81.9]99.8 (0) [99.4, 100.0]

GPQA: Question-level Bootstrap.

For GPQA, the test set contains 166 questions, and each question was evaluated independently five times, resulting in 830 evaluations.

Because the original experiments did not retain complete per-question execution records for all methods and all splits, we cannot consistently reconstruct Question-level Bootstrap intervals for every method. Rerunning the experiments would produce new Agent execution results that could not be matched one-to-one with the results reported in the original paper. However, for the per-question data that remain recoverable and correspond to the results reported in the original paper, we computed the following question-level 95% confidence intervals:

MethodSplit95% CI
AFlowTest[68.8%, 79.6%]
AFlowValidation[67.5%, 88.8%]
MARSValidation[68.8%, 91.9%]

Question 3: Unified Cost Accounting and Cumulative First-Surpass Cost

We thank the reviewer for pointing out that the number of generations alone cannot adequately reflect the actual search cost, particularly because each MARS generation also involves execution-case analysis, structured diagnosis, and candidate evaluation.

To address this concern, we re-examined the complete experimental cost records and recalculated the recordable token consumption for search and optimization across the four methods. We separately report token consumption for the two models:

  • P: DeepSeek-V4-Pro Preview Thinking High, used for search, optimization, reflection, and diagnosis.
  • F: DeepSeek-V4-Flash Preview Non-thinking, used for Agent execution and evaluation, or Workflow Sampling during training.

The cost accounting covers candidate generation, case analysis, structured diagnosis, and candidate evaluation. Since DeepSeek API prices may vary with billing periods and pricing adjustments, we use token consumption as a unified resource-usage measure and separately report cumulative token consumption for DeepSeek-V4-Pro (Preview, Thinking High; P) and DeepSeek-V4-Flash (Preview, Non-thinking; F).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.