- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MARS, which guides agentic program search through mechanism-level representations and failure diagnosis. Each candidate contains both executable code and a mechanism IR describing functional units, their implementations, and input–output relationships. Using execution records, the method attributes problems to individual mechanisms or to information flow and control relationships between them, then modifies the program through operations such as addition, deletion, replacement, and rewiring. The search can jointly adjust model calls, prompts, and deterministic computation. Results across five tasks show that the discovered programs outperform those produced by the compared search methods while using fewer inference tokens than the highest-scoring baseline on each task.
Critical strengths
- Using a common functional description to connect design, diagnosis, and modification is intuitive. Distinguishing the generation of an incorrect answer from the loss of a correct result downstream can give the search more specific directions for improvement. Mechanism implementations also allow freely generated code, enabling the division of work between model reasoning and programmatic computation to adapt to the task.
- The paper reports improvements on all five tasks and provides evolution histories that readers can inspect. On Logic-Q, the final program reaches 99.8% accuracy without invoking a language model at runtime, and unsuccessful intermediate parser implementations are also documented. The analysis of test sources and early stopping in coding tasks further demonstrates that the search can adjust workflows in response to specific execution problems.
Critical weaknesses
- AFlow already uses execution feedback to modify workflows, while Trace provides computational dependencies, intermediate results, and operator descriptions to an optimizer. MARS adds mechanism categories and layered diagnosis, but information flow and control relationships remain expressed in code, and concrete modifications are still generated through model-based analysis. I therefore see the main contribution as a more detailed organization of information for reflection. This is useful, but the current comparisons do not separate the contribution of the mechanism representation from that of additional diagnostic feedback. It remains unclear whether the representation constitutes a sufficiently substantial methodological advance.
- The paper emphasizes surpassing baselines within the first two generations, but its search curves report the highest test score observed up to each generation, whereas the algorithm selects candidates by validation score. This shows that good programs were generated early, but does not necessarily mean that the search would return them if stopped after two generations. Each generation also includes failure analysis and diagnostic aggregation, so generation count is not a direct measure of search cost. The reduction in inference tokens used by the final programs is clear; the advantage of the search process itself needs to be characterized more precisely.
Questions for authors
- With the same code search space and failure records, how does direct reflection on code compare with diagnosis using the mechanism IR? Are there existing controlled comparisons that isolate the contribution of the mechanism representation itself?
- If each generation reports only the candidate selected by validation performance, does the claim of surpassing baselines within the first two generations still hold? By that point, how much resource has been consumed by candidate generation, diagnosis, and validation, respectively?
5 comments
Response to Reviewer yy62 (Part 2/3)
Weakness 2 & Question 2: Candidate Selection, Reporting Protocol, and Search Cost
(1) Clarifying the candidate-selection rule and reporting protocol.
We thank the reviewer for carefully reading the paper and identifying that the candidate-selection and result-reporting logic was not sufficiently explained in the original version.
To clarify, MARS always selects the current incumbent according to the Validation Score during the actual search and uses it as the starting point for the next optimization round. Test Score does not participate in search decisions. Table 1 of the paper uses a Test-best reporting protocol primarily to ensure consistency with the reporting protocols of the other baselines, particularly ADAS, which uses the same protocol. Moreover, the performance advantage of MARS does not depend on Test-best reporting: even under a Validation-selected protocol, MARS outperforms all compared baselines on all five tasks. To further verify this, we compiled the Validation-selected candidates' performance from our existing experimental records.
(2) Validation-selected performance comparison.
We first recomputed the final results of all methods under a unified Validation-selected rule. The table reports the Test Set performance of the candidate selected by Validation Score, together with its average Test Inference Tokens (in parentheses).
| Method | E2H Pass | DROP F1 | GPQA Acc. | HotpotQA F1 | Logic-Q Acc. |
|---|---|---|---|---|---|
| ADAS | 79.4 (55750) | 79.7 (10119) | 70.2 (5637) | 77.3 (14208) | 47.8 (14622) |
| AFlow | 71.6 (19342) | 86.8 (1023) | 74.3 (1982) | 77.5 (13739) | 48.2 (6077) |
| MaAS | 57.6 (12439) | 53.6 (2964) | 40.4 (6826) | 64.9 (9063) | 32.4 (6929) |
| MARS (Ours) | 87.0 (8932) | 91.4 (463) | 78.9 (1996) | 79.6 (1647) | 99.8 (0) |
The results show that, even when the Validation-selected candidate-selection rule is used consistently, MARS achieves the highest Test Performance on all five tasks.
(3) Actual search cost at Test-best First-Surpass.
To further address the concern about the actual cost of the search process, we follow the original paper's Test-best reporting protocol and calculate MARS's cumulative search cost when it first surpasses the strongest baseline. We compare this cost with the corresponding baseline costs.
The cost accounting covers candidate generation, case analysis, structured diagnosis, and candidate evaluation. Since DeepSeek API prices may vary with billing periods and pricing adjustments, we use token consumption as a unified measure of resource usage. We separately report cumulative token consumption for DeepSeek-V4-Pro (Preview, Thinking High; P) and DeepSeek-V4-Flash (Preview, Non-thinking; F), without mixing tokens from different models. This provides a more transparent and reproducible comparison that does not depend on a particular billing period or price.
1. Test-best First-Surpass vs. Baseline Full-search.
We first compare the cumulative cost when MARS first surpasses the strongest baseline in performance with the cumulative cost incurred by the corresponding baseline over its entire search.
| Task | MARS First-Surpass | MARS (P/F) | Best Baseline (P/F) | Ratio (P/F) |
|---|---|---|---|---|
| E2H | Gen 1 (82.0%) | 144119 / 690137 | 776861 / 66378234 | 5.39× / 96.18× |
| DROP | Gen 1 (91.2%) | 77184 / 106689 | 66284 / 2133248 | 0.86× / 20.00× |
| GPQA | Gen 2 (78.2%) | 119172 / 455592 | 477995 / 23608950 | 4.01× / 51.82× |
| HotpotQA | Gen 2 (79.6%) | 130010 / 926056 | 582090 / 36709500 | 4.48× / 39.64× |
| Logic-Q | Gen 2 (80.0%) | 175428 / 970363 | 566186 / 28585756 | 3.23× / 29.46× |
Response to Reviewer yy62 (Part 3/3)
Weakness 2 & Question 2 (continued): First-Surpass Search Cost
2. Test-best First-Surpass vs. Baseline Test-best.
We further narrow the comparison to the point at which the baseline reaches its own best test performance, rather than the end of its entire search. Specifically, we compare the cumulative cost when MARS first reaches or surpasses the strongest baseline's Test-best performance with the cumulative cost incurred by that baseline to reach its own Test-best. This more directly reflects the search resources required by each method to obtain the corresponding performance.
| Task | MARS First-Surpass | MARS (P/F) | Best Baseline (P/F) | Ratio (P/F) |
|---|---|---|---|---|
| E2H | Gen 1 (82.0%) | 144119 / 690137 | 776861 / 66378234 | 5.39× / 96.18× |
| DROP | Gen 1 (91.2%) | 77184 / 106689 | 48691 / 1431983 | 0.63× / 13.42× |
| GPQA | Gen 2 (78.2%) | 119172 / 455592 | 247897 / 17123793 | 2.08× / 37.59× |
| HotpotQA | Gen 2 (79.6%) | 130010 / 926056 | 166145 / 18601468 | 1.28× / 20.09× |
| Logic-Q | Gen 2 (80.0%) | 175428 / 970363 | 209140 / 19394710 | 1.19× / 19.99× |
Here, the P/F Ratio denotes the ratio of the baseline's cumulative Pro/Flash token consumption to that of MARS. A value greater than 1 indicates that MARS consumes fewer tokens for the corresponding model.
3. Test-best First-Surpass cost breakdown.
Finally, to further address the cost of candidate generation, failure analysis, diagnosis, and candidate evaluation, we break down MARS's cumulative cost at Test-best First-Surpass by stage. All costs are accumulated up to the corresponding First-Surpass Generation.
| Task | First-Surpass | Search | Case Summary | Diagnosis | Pro Total | Evaluation (Flash) |
|---|---|---|---|---|---|---|
| E2H | Gen 1 | 17073 | 95592 | 31454 | 144119 | 690137 |
| DROP | Gen 1 | 20900 | 41786 | 14498 | 77184 | 106689 |
| GPQA | Gen 2 | 28067 | 70642 | 20463 | 119172 | 455592 |
| HotpotQA | Gen 2 | 27843 | 82505 | 19662 | 130010 | 926056 |
| Logic-Q | Gen 2 | 42539 | 103614 | 29275 | 175428 | 970363 |
The cost components are defined as follows:
- Search: Candidate generation and modification.
- Case Summary: Reconstruction, analysis, and summarization of failure-case execution paths.
- Diagnosis: Cross-case issue aggregation and structured mechanism-level failure attribution.
- Pro Total: Cumulative Pro-model token consumption in Search, Case Summary, and Diagnosis.
- Evaluation (Flash): Flash-model token consumption during candidate evaluation, including all input and output tokens.
These statistics account for cumulative token consumption in candidate generation, case analysis, structured diagnosis, and candidate evaluation, rather than measuring search efficiency only by the number of generations.
(4) Summary of search-cost comparisons.
Across the two cost-comparison protocols, on E2H, GPQA, HotpotQA, and Logic-Q, MARS consumes fewer cumulative Pro and Flash tokens at First-Surpass than the corresponding baseline.
For DROP, although MARS consumes more Pro tokens, it substantially reduces Flash token consumption. Based on the API prices of the two models, the overall search cost of MARS at First-Surpass is still lower.
Therefore, even when the costs of candidate generation, case analysis, structured diagnosis, and candidate evaluation are all included, MARS reaches performance above the strongest baseline with a lower overall search cost.
Response to Reviewer yy62 (Part 1/3)
We sincerely thank the reviewer for the careful reading of our paper and the positive assessment of MARS, particularly its methodological design and experimental results. We also greatly appreciate the insightful comments and constructive suggestions, which have helped us further clarify our contributions and improve the paper.
We address the concerns as follows.
(1) Clarifying the methodological contribution of MARS.
We thank the reviewer for recognizing the value of mechanism-level information organization and for pointing out that the current experiments do not sufficiently distinguish the contributions of mechanism representation and diagnostic feedback. We would like to further clarify the core distinction between MARS and related work.
AFlow uses Workflow code, discrepancies between predictions and ground-truth answers, performance scores, and historical modification experience to guide workflow optimization. However, its feedback mainly concerns the overall predictions, expected outputs, and performance changes of the Workflow. It does not systematically record and use the actual execution inputs, intermediate outputs, and runtime states of individual Operators to localize component-level problems. Consequently, the optimizer primarily infers potential issues from the code structure and overall outcome feedback, without explicitly attributing execution failures to specific mechanisms and their intended responsibilities.
Trace represents workflow execution as a computational graph, recording operator dependencies, intermediate computational results, and how these results are used. It combines operator descriptions with output feedback to guide parameter updates. However, Trace does not explicitly establish a correspondence between the functional semantics of Agent mechanisms and actual execution failures. It therefore lacks a process for diagnosing actual execution behavior against design expectations at the mechanism level. In contrast, MARS uses Mechanism IR to explicitly describe the intended responsibilities, design-time inputs and outputs, and functional relationships of individual mechanisms. It aligns Agent design and failure diagnosis at the same functional-mechanism granularity, allowing actual execution evidence to reveal whether a mechanism fulfills its intended responsibility and whether a problem originates from interactions between mechanisms. These diagnoses then guide targeted redesign.
(2) Independent contributions of Mechanism IR and Structured Diagnosis.
We thank the reviewer for this important suggestion. To address the request for controlled comparisons, we conducted two additional ablation studies. We kept the code search space, failure records, search budget, and evaluation settings the same, and separately examined the contributions of mechanism representation and structured diagnosis:
The additional results are shown below. Values in parentheses denote the average inference tokens used by the final Agent per test example.
As shown, removing either Mechanism IR or Structured Diagnosis reduces performance on all five tasks. This indicates that mechanism-level representation and structured failure diagnosis both play important roles in translating execution feedback into targeted mechanism-level optimization, and that they jointly support the search effectiveness of MARS.