acceptodds
Under review as a conference paper at ICLR 2027

Stabilizing Multi-Agent System Evolution Through Evidence-Guided Revision

Abstract

Persistent multi-agent systems should turn experience across recurring tasks into reusable problem-solving procedures that improve over time. Yet every accepted revision is inherited by later tasks and future revisions, so harmful updates can compound alongside useful ones. The central challenge is determining what evidence justifies allowing a revision to shape the system's future. We introduce EMAS, which turns cross-task experience into evidence-gated MAS revisions while keeping the underlying LLM fixed. Its Step graph aligns diagnoses and edits; recurring diagnoses across tasks trigger bounded proposals, and paired validation on separate instances determines retention. After two epochs, validation-selected checkpoints improve task-weighted held-out test accuracy over the initial MAS across four benchmarks by 4.10 and 12.09 percentage points with Kimi K2.6 and Qwen3.6-27B, respectively. Across matched V0–V20 Game24 trajectories, Full EMAS yields the lowest cumulative regression for both backbones. On Qwen, allowing one task to trigger a proposal and removing paired Validation raise this burden to and the Full-EMAS level, respectively. These results show that evidence-controlled inheritance supports continued improvement while limiting damage along the evolution trajectory.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
large language model agents, multi-agent systems, self-evolving agentic systems, automated agent design, agent workflow optimization, evidence-gated revision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_mhEzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce an MAS evolution method, which treats the overall system as a DAG and tries to modify each edge based on the feedback during test time. Testing on two models and 4 benchmarks, they demonstrate the effectiveness of their method. Besides, the authors also include some analysis to better explain their method.

Critical strengths
  1. Good formulation, which treat the overall process as a modification of a DAG.
  2. Solid experiments, including several benchmarks and models.
  3. I think the method is well designed.
Critical weaknesses
  1. The huge cost for the evolution process.
  2. We may need to design a specific framework for each dataset, which limits its usage significantly.
  3. I think the proposed method need a powerful critic and it cannot generalize to open-ended scenarios.
Questions for authors
  1. I think the first two figures look like AI-generated product, especially for figure 2. Would it possible to draw them on your own?
  2. I still worry about the effectiveness on real world problems. But at least, it is a decent research product. Could the authors explain more about how to use it in real world scenarios?
2 comments

Thank you for your positive comments on our problem formulation, experiments, and method design. We respond to each point below.

Q1: Figures 1 and 2

I think the first two figures look like AI-generated product, especially for figure 2. Would it possible to draw them on your own?

Thank you for the suggestion. We drew Figures 1 and 2 ourselves, with assistance from AI tools as disclosed in our AI Use Statement. We used a simple schematic style so that readers can quickly understand the workflow of EMAS. In the revised paper, we have revised Figure 2 to make it more concise and easier to read.

Q2: Use in real-world scenarios

I still worry about the effectiveness on real world problems. But at least, it is a decent research product. Could the authors explain more about how to use it in real world scenarios?

[…] it cannot generalize to open-ended scenarios.

We may need to design a specific framework for each dataset, which limits its usage significantly.

Large-scale studies find that real user requests concentrate in a small number of categories: nearly 80% of ChatGPT conversations fall into practical guidance, seeking information, and writing (Chatterji et al., 2025, NBER Working Paper 34255), and LMSYS-Chat-1M clusters the user prompts of 100K randomly sampled English conversations into 20 topic clusters (Zheng et al., ICLR 2024).

Deployment therefore works by category. Tasks are grouped by category, one EMAS system is maintained for each category, and new tasks are routed to the corresponding category. No manual design is needed for a new category: the Initial MAS for each category is designed fully automatically by an LLM, and the thresholds and acceptance rules of EMAS are identical across the 4 benchmarks and 2 backbones. As tasks accumulate, the system can continue to evolve, and each revision must pass paired Validation before it is deployed. This approach is suitable for real-world scenarios:

  • Inference cost can be reduced. A cost-oriented candidate is accepted only if its correct count on Validation is not lower than that of the current Version and its total token cost is strictly lower (Eq. (1)). For example, in one of our new trajectories, two revisions reduced tokens per task from 7,591 to 4,696 (−38%), while accuracy rose from 94.49% to 95.96%.
  • Continued updates limit damage to existing abilities. Without paired Validation, the V0–V20 cumulative regression rises from 5.02 pp to 157.97 pp. With it, the system can be updated continuously with much less regression.
  • Each update can be reviewed. Each change has an explicit basis (matching diagnoses across tasks) and an explicit acceptance condition, which makes it easy to inspect and trace in a real system. Reviewer 8qMA also noted that this makes the basis and the acceptance condition of each update clear and easy to inspect, and Reviewer Je5r noted that these design choices provide "a clear account of what may change and why a particular change is attempted."

We have added this motivation to the introduction (Section 1) of the revised paper.

Summary

  • Open-ended and real-world scenarios (Q2). Real user requests concentrate in a small number of categories, so one EMAS system can be maintained for each category. Every update is validated and can be reviewed.

Thank you again for your constructive feedback. We hope this response answers your questions.

Thanks for your response, I will keep my positive score.

Reviewer_8qMAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EMAS to control the performance degradation during the continuous update of multi-agent systems. The method first collects matching diagnoses from multiple tasks, then generates range-limited modifications. Subsequently, the system compares candidate versions with the current version on another set of validation samples. Modifications that improve the validation accuracy can be accepted. Modifications that reduce token consumption need to maintain the validation accuracy from not decreasing. The parameters of the underlying language model are always frozen. Experiments use two backbone models and four benchmarks. The checkpoints selected for validation have achieved an overall test accuracy improvement compared to the initial system. The ablation study on Game24 further analyzed the impact of cross-task gated diagnosis and paired validation on performance degradation.

Critical strengths
  1. The occurrence of the same diagnosis for multiple tasks does not guarantee the validity of the generated modifications. EMAS clearly distinguishes between these two issues. It proposes modifications based on the repeatedly occurring diagnoses, then checks the effect of the modifications using another set of samples, while also limiting the editing scope. This makes the basis and acceptance conditions for each update more clear, facilitating the inspection and analysis of the update process.

  2. Compared to the initial system, the version selected from the validation set achieved an overall test accuracy increase of 4.10 percentage points on Kimi and 12.09 percentage points on Qwen. The gating ablation study of Game24 further examined the performance degradation during continuous updates. Within each backbone, each setting used the same initial system and compared the first 20 accepted updates. The complete method achieved the lowest cumulative degradation on both backbones. These results demonstrate the performance improvement relative to the initial system and support the role of the two types of gating in reducing the cumulative degradation in the reported trajectories.

Critical weaknesses
  1. This paper organizes cross-task diagnosis matching, constrained modification, and independent validation into a continuously updated process. Semantic Backpropagation has integrated feedback aggregation and candidate validation, and TPGO has modified the system diagram structure through repeated error analysis. The main difference of EMAS lies in stricter diagnosis matching, support thresholds based on operation settings, and the combination method of constrained modification and independent validation. These designs have practical value, but the overall contribution is closer to the integration and refinement of existing feedback revision processes, with relatively limited incremental innovation at the method level.

  2. The paper did not conduct a direct experimental comparison between EMAS and the mechanism-based approaches. The authors also clearly stated that the existing external comparisons have different search and selection opportunities. κ = 1 and No Gate ablation support the role of gating within EMAS, but it is not yet certain how much additional benefit EMAS can bring compared to the existing revised methods, given comparable data access and search budgets.

Questions for authors
  1. Compared to the general aggregation of diverse sample feedback, what additional benefits does strict diagnostic matching bring? Are there any case studies or comparative results to support this?

  2. Regarding the modifications aimed at reducing token costs, can the verification mechanism effectively filter out the candidates that would compromise the accuracy? What evidence supports this?

  3. Under different task sequences and comparable task processing volumes, does the cumulative rollback advantage of Full EMAS still hold true compared to κ = 1?

4 comments

Thank you for your careful review and for your positive comments on how EMAS makes the basis and the acceptance condition of each update clear and easy to inspect. We respond to each point below.

Q1: Benefits of strict diagnostic matching

Compared to the general aggregation of diverse sample feedback, what additional benefits does strict diagnostic matching bring? Are there any case studies or comparative results to support this?

Semantic Backpropagation has integrated feedback aggregation and candidate validation, and TPGO has modified the system diagram structure through repeated error analysis.

Thank you for pointing this out. We agree with this summary: no single component is unique to EMAS. The difference lies in how evidence is assigned. An accepted change is inherited by later tasks and later revisions, so each change must answer two questions: should it be proposed, and should it be kept? EMAS assigns a different kind of evidence to each question. Table A compares Semantic Backpropagation (SB), TPGO, and EMAS.

Table A: How the three methods differ

SBTPGOEMAS
Existing componentsFeedback aggregation, candidate validationRepeated error analysis, graph-structure modificationHas the same components
Proposal authorization (Gate 1): what evidence is needed to propose a changeFeedback from a batch of failed samples, passed directly to the optimizer; agreement is not requiredFailures with similar descriptions are grouped togetherDiagnoses from multiple distinct tasks point to the same operation at the same location
Retention authorization (Gate 2): where the change is validatedA validation set (in its experiments, the training samples)Tasks that contributed to the error cluster (all originally failed)The Validation set, which is not used to construct proposals; paired comparison
Validation data vs. proposal dataOverlappingIdenticalSeparate; includes both originally correct and originally incorrect tasks

Similar failures do not necessarily need the same fix. Recurrence is evidence for a specific change only when the repair hypotheses match. Validating only on originally failed tasks shows whether the target is fixed, but not whether already-solved tasks are broken. Figure 1(c) shows exactly this case.

Table B: What each difference contributes. Each group replaces one part of EMAS with the corresponding mechanism of SB or TPGO and keeps everything else unchanged. The first group answers Q1.

Part of EMAS replacedMetricFull EMASAfter replacement
Matching evidence → random evidence (as in SB, Q1)Validation gain in correct answers (relative to Full EMAS; out of 218 Validation tasks)Reference−5.79, 95% CI [−13.69, −0.74]
Accepted proposals105
Improved on Test after acceptance5 / 10 (50%)1 / 5 (20%)
Paired Validation → count only how many originally failed tasks are fixed (as in TPGO)Accepted among 102 accuracy-oriented candidates11100
The 89 additionally accepted: fixed / broken / net change (mean, tasks)—6.61 / 51.81 / −45.20
Remove paired Validation (Table 2)V0–V20 cumulative regression (pp; Qwen / Kimi)5.02 / 15.56157.97 / 129.04
  • Group 1: evidence not required to match, as in SB. At the trigger events of Full EMAS, we replaced the evidence with the same number of diagnoses drawn at random, which are not required to match, and kept everything else unchanged (38 accuracy trigger events). Proposals become worse on all three metrics in Table B. Since the amount of evidence is the same, the difference comes from the diagnoses matching.
  • Group 2: validation only on originally failed tasks, as in TPGO. Using the records of the original Qwen trajectory, a candidate is accepted if it fixes at least 1 task that the current Version gets wrong on Validation; broken tasks are not counted. This is already stricter than TPGO, which only requires that performance does not get worse. This rule would accept 100 of the 102 candidates, and the 89 additional ones break far more tasks than they fix (51.81 vs. 6.61 on average). Checking only failed tasks cannot detect damage to tasks that were already solved; paired Validation can.

We have added the first control to Appendix C.3.3 of the revised paper (Table 12).

Q2: Can Validation filter out cost-oriented candidates that hurt accuracy?

Regarding the modifications aimed at reducing token costs, can the verification mechanism effectively filter out the candidates that would compromise the accuracy? What evidence supports this?

Validation filters out candidates that cause large accuracy losses. A cost-oriented candidate is accepted only if its correct count on Validation is not lower than that of the current Version and its total token cost is strictly lower (Eq. (1)). We analyzed the Full EMAS trajectory of Qwen on Game24 and found the following.

What is blocked. 144 cost-oriented candidates were evaluated on Validation. Of these, 132 saved tokens but lowered the correct count, and all of them were rejected. Their correct count dropped by 63.08 on average (median 55, out of 218), and 127 of them dropped by more than 10. Cost-oriented revisions are inherently high-risk, which is why validation is needed. In the single-step control of Q1, the 14 REMOVE-NODE proposals made with random evidence were also all rejected, with drops in correct count of 38 to 165.

What happens without validation. In the No Gate trajectory, where paired Validation is removed, cost-oriented candidates are committed directly after they are generated. Over V0–V20, cost-oriented revisions account for only 6 Versions, 5 of which lowered Test accuracy. These 5 Versions contributed 137.13 of the 157.97 pp of cumulative regression (86.8%). The largest single-step drop was 47.67 pp.

What passes. On the Full EMAS trajectory, of the 9 accepted cost-oriented revisions, 4 increased Test accuracy, 2 left it unchanged, and 3 decreased it, with a mean change of +0.01 pp. All of them reduced Test tokens per task, by 1.0% to 29.6%. We first use recurrence to locate precisely what to change, and then use paired Validation to confirm that the change does not lower Validation accuracy.

We have added this analysis to Appendix C.3.6 of the revised paper.

Q3: Cumulative regression under different task orders and matched task volume

Under different task sequences and comparable task processing volumes, does the cumulative rollback advantage of Full EMAS still hold true compared to κ = 1?

Setup. On Game24 with Qwen, we ran one Full EMAS trajectory and one κ=1 trajectory under each of 2 task orders. Diagnoses, proposals, and acceptance decisions were regenerated in every trajectory, and all trajectories start from the same Initial MAS. The table below cuts Full EMAS at the same number of Evolve samples as κ=1, and cuts the original trajectories at 1 and 2 epochs.

Cutoff (Evolve samples)Full EMAS accuracy (%)Full EMAS cumulative regression (pp)κ=1 accuracy (%)κ=1 cumulative regression (pp)
Order 1 (1,320)50.00 (V3)2.2168.63 (V10)5.27
Order 2 (1,544)48.53 (V2)6.2549.63 (V7)5.15
Original, 1 epoch (872)79.78 (V5)3.0678.92 (V9)5.76
Original, 2 epochs (1,744)79.90 (V6)3.0688.60 (V23)15.07

Results. At the same number of processed Evolve samples, Full EMAS has lower cumulative regression than κ=1 at 3 of the 4 cutoffs. The only exception is Order 2.

Conclusion. The advantage largely holds. In Order 2, Full EMAS regresses in only one early step (V1→V2) and then improves without further regression to 95.96% at V8, whereas κ=1 regresses repeatedly and ends below the Initial MAS (49.63% vs. 51.72%). The evolution of Full EMAS is therefore more stable. We have added these results to Appendix C.3.2 of the revised paper (Tables 10 and 11).

Summary

  • Benefit of matching diagnoses (Q1). With the same evidence volume, edit scope, and Validation budget, proposals based on matching diagnoses are better. With random evidence, the difference in Validation gain is −5.79 correct answers (95% CI [−13.69, −0.74]), half as many proposals are accepted (5 vs. 10), and a lower fraction of the accepted ones improve on Test (20% vs. 50%).
  • Cost-oriented revisions (Q2). All 132 candidates that saved tokens but lowered the correct count were rejected. The 9 accepted ones have a mean Test change of +0.01 pp, and all of them reduced tokens per task.
  • Cumulative regression (Q3). At 3 of the 4 matched-volume cutoffs, Full EMAS has lower cumulative regression than κ=1. The only exception is Order 2, where Full EMAS regresses in only one early step (V1→V2) and then improves without further regression, whereas κ=1 regresses repeatedly and ends below the Initial MAS.

Thank you again for your constructive feedback. We hope these additions answer your questions.

Dear Reviewer 8qMA,

Thank you again for your careful review. As the author–reviewer discussion closes on Nov 18 (AoE), we would like to kindly check whether our responses have addressed your concerns. Since our replies are long, we summarize them briefly here:

  1. Contribution relative to SB and TPGO. We agree that no single component is unique to EMAS. Our claim concerns how evidence is assigned: recurrence across distinct tasks authorizes a proposal, and paired validation on a separate set authorizes its retention. To test this, we replaced each part of EMAS with the corresponding mechanism of SB or TPGO while keeping everything else fixed (Table B in our response):
  • Non-matching, randomly drawn evidence (as in SB): lower validation gain (−5.79 correct answers, 95% CI [−13.69, −0.74]), half as many accepted proposals (5 vs. 10), and fewer Test improvements among them (20% vs. 50%).
  • Validating only on originally failed tasks (as in TPGO): 100 of 102 accuracy-oriented candidates would be accepted, and the 89 additional ones break far more tasks than they fix (51.81 vs. 6.61 on average). Because everything else is held fixed, these controls compare the mechanisms under the same data access and search budget, which is the condition you raised; they are not full re-implementations of SB and TPGO.
  1. Filtering cost-oriented candidates. All 132 candidates that saved tokens but lowered validation accuracy were rejected. The 9 accepted ones reduced Test tokens per task by 1.0–29.6%, with a mean Test accuracy change of +0.01 pp. Without paired validation, 5 cost-oriented versions account for 86.8% of cumulative regression (Appendix C.3.6).

  2. Task order and matched volume. With new independent trajectories under two task orders, cut at the same number of Evolve samples, Full EMAS has lower cumulative regression than κ = 1 at 3 of 4 cutoffs. In the exception (Order 2), Full EMAS regresses in a single early step and then reaches 95.96% without further regression (Appendix C.3.2, Tables 10–11).

If any concern remains, we would be glad to clarify before the deadline. If these results address your concerns, we would be grateful if you would consider revisiting your assessment.

Best regards, The Authors

Thanks for rebuttal! I will increase my score.

Comments on the paper

4 comments

Thank you for your careful review and for your positive comments on how EMAS separates the evidence for proposing a revision from the evidence for keeping it. We respond to each point below.

Q1: Independent evolution trajectories

How many independently generated evolution trajectories were run for each condition in Table 2? Do any of the reported repetitions regenerate diagnoses, proposals, and acceptance decisions? Please distinguish these from repeated execution of saved versions. Results across independent evolution runs or task orders, reporting both cumulative regression and V20 accuracy, would materially affect my assessment.

Thank you for the question. The submission has only one evolution trajectory per condition.

The repetitions reported in the paper are repeated executions of the evolved MAS on the Test set. They are used to reduce evaluation error. None of them regenerates diagnoses, proposals, or acceptance decisions.

To address this, we ran new independent evolution trajectories. Diagnoses, proposals, and acceptance decisions were all regenerated. The results are given below.

Setup. On Game24 with Qwen3.6-27B, we ran one Full EMAS trajectory and one κ=1 trajectory under each of two task orders (Order 1 and Order 2). We shuffled the order of the Evolve samples and used different random seeds, so every trajectory is different and these runs can better show the stability of our method. All trajectories start from the same Initial MAS.

Scope. Full EMAS was run until V10 or 4 epochs, and κ=1 was run until V10 or 2 epochs, whichever came first in each case. κ=1 was given fewer epochs because it moves through Versions much faster: its trigger condition is looser (a single diagnosis is enough to propose a revision), so it needs far fewer Evolve samples per accepted revision.

Results. Accuracies in the table are means over 3 Test executions.

TrajectoryVersionAccuracy (%)Change vs. V0 (pp; V0 = 51.72%)Cumulative regression (pp)Accepted / candidates evaluated on ValidationOptimization tokens
Full EMAS, Order 1V1075.37+23.654.0410 / 99227.1M
Full EMAS, Order 2V895.96+44.246.258 / 66107.0M
κ=1, Order 1V1068.63+16.915.2710 / 36127.7M
κ=1, Order 2V749.63−2.085.157 / 4579.3M

Endpoint accuracy. Full EMAS shows large gains in both orders: 75.37% at V10 in Order 1, and 95.96% at V8 in Order 2. In comparison, κ=1 reaches 68.63% and 49.63% at the Versions in the table. However, Full EMAS uses 1.8× and 1.3× as many optimization tokens as κ=1, respectively.

Cumulative regression. The two orders give different results. In Order 1, both settings reach V10, with cumulative regression of 4.04 pp for Full EMAS and 5.27 pp for κ=1. In Order 2, cumulative regression is 6.25 pp for Full EMAS and 5.15 pp for κ=1. In this order, all of the regression of Full EMAS comes from one step (V1→V2), and there is no regression afterward. The regression of κ=1 is spread over three steps. After 7 accepted revisions, κ=1 has a net change of −2.08 pp and shows no improvement.

Inference cost. In Order 2, Full EMAS reaches 94.49% at V6. The next two revisions (V6→V8) reduce inference tokens per task from 7,591 to 4,696 (−38%), while accuracy rises to 95.96%.

The original trajectory through V20. In the original trajectory, the two settings have almost the same accuracy at V10 (Full EMAS 85.05%, κ=1 84.68%), and their cumulative regression over V0–V10 differs by only 1.84 pp. The gap appears after V10, in the high-accuracy range. From V10 to V20:

  • Full EMAS has only 1.10 pp of cumulative regression. Its accuracy continues to rise to 94.12%, and its inference tokens per task drop from 6,766 to 3,447.
  • κ=1 has 7.60 pp of cumulative regression, with regressions in five of the ten steps. Its accuracy rises by only 2.94 pp net, and its inference tokens per task rise from 8,423 to 17,032.

The new trajectories end at V7–V10, before this range. We have added the new trajectories above to Appendix C.3.2 of the revised paper (Tables 10 and 11).

Q2: Cost of reaching V20

How do Full EMAS and κ = 1 compare in the number of task executions, candidate evaluations, and total optimization tokens required to reach V20?

Matching twenty accepted revisions does not match the amount of experience consumed, the number of candidates evaluated, or optimization cost.

Thank you for pointing this out. We agree: reaching V20 in both settings does not mean the two settings consumed the same resources. The table below gives the full cost breakdown up to V20 for the original trajectory (Qwen3.6-27B, Game24).

ItemFull EMASκ=1Full EMAS / κ=1
Evolve samples processed8,1041,5765.14
Triggers (accuracy / cost)260 (102 / 158)50 (43 / 7)5.20
Candidates evaluated on Validation (accuracy / cost)246 (102 / 144)47 (41 / 6)5.23
Accepted revisions (accuracy / cost)20 (11 / 9)20 (18 / 2)1.00
Acceptance rate (accepted / candidates evaluated on Validation)8.1%42.6%
Total optimization tokens639.5M326.1M1.96
Of which: paired Validation527.9M (82.5%)185.1M (56.8%)
V20 accuracy94.12%87.62%
Inference tokens per task of the V20 system (V0: 3,238)3,44717,0320.20

Where the cost goes. Up to V20, Full EMAS processes 5.14× as many Evolve samples as κ=1 and uses about 2× as many optimization tokens. Most of the difference comes from paired Validation. Only 8.1% of the candidates of Full EMAS are accepted, so it has to evaluate more candidates. Paired Validation accounts for 82.5% of the optimization tokens of Full EMAS, compared with 56.8% for κ=1.

What the cost buys. Of the triggers of Full EMAS, 158 (61%) are cost-oriented hypotheses, and 9 of its 20 accepted revisions are cost-oriented. κ=1 has only 2 cost-oriented revisions. The reason is that accuracy-oriented revisions take priority over cost-oriented revisions. Under Full EMAS, most executions are correct, and an accuracy-oriented revision is triggered only when the same diagnosis recurs across multiple failures. Cost-oriented revisions therefore also get a chance to be triggered. Under κ=1, a single failure triggers a revision, and these accuracy-oriented revisions crowd out cost-oriented ones. As a result:

  • The V20 system of Full EMAS uses only 3,447 tokens per task, with 94.12% accuracy;
  • The V20 system of κ=1 uses 17,032 tokens per task, about 4.9× that of Full EMAS, with 87.62% accuracy.

In other words, on this trajectory, the extra optimization cost of Full EMAS buys a final system that is both more accurate and much cheaper at inference time.

Comparison at the same number of processed Evolve samples:

Full EMAS (1 epoch)κ=1 (1 epoch)Full EMAS (2 epochs)κ=1 (2 epochs)
Version reachedV5V9V6V23
Accuracy (%)79.78 ± 2.2478.92 ± 0.9379.90 ± 1.7488.60 ± 1.10
Candidates evaluated on Validation25265353
Total optimization tokens48.9M112.1M128.0M381.3M
Inference tokens per task6,9398,7986,53916,768

At the same number of processed Evolve samples, κ=1 uses 2.3× (1 epoch) and 3.0× (2 epochs) as many optimization tokens as Full EMAS, and the resulting systems also use more inference tokens per task. After 1 epoch, the two settings have similar accuracy. After 2 epochs, κ=1 is higher. This supports your assessment: Full EMAS consumes more experience to reach V20, so its higher V20 accuracy is an advantage at the same number of accepted revisions, not at the same cost. We have added the cost breakdown and the comparison at the same number of processed Evolve samples to Appendix C.4.1 of the revised paper (Tables 16 and 17), where we also state this point.

Q2: Control with matched evidence volume

Does matching recurring diagnoses improve on a control that collects a comparable batch of traces before proposing a bounded edit, with the same paired validation rule? This would help determine whether diagnostic agreement itself accounts for the reported benefit.

A simple batch-based revision control with comparable evidence volume, edit scope, and validation budget would be substantially more diagnostic.

Thank you for the suggestion. We agree, and we have run this control.

Method. At the trigger events of Full EMAS, we kept the number of evidence samples fixed and replaced the evidence with diagnoses drawn at random from the same pool. These diagnoses are not required to point to the same revision. All other conditions are the same, including the edit scope (only an operation that appears in the evidence may be applied, and its location must also come from the evidence). We sampled 57 trigger events from the two new Full EMAS trajectories. The events were sampled before the control was run, regardless of whether the proposal of Full EMAS was accepted.

Full EMASRandom evidence
EvidenceMatching diagnoses that point to the same revisionDrawn at random; not required to match
Accuracy trigger events (38)
Validation gain in correct answers (relative to Full EMAS; out of 218)Reference−5.79, 95% CI [−13.69, −0.74]
By order (Order 1 / Order 2)Reference−4.68 / −7.31
Accepted proposals105
Improved on Test after acceptance5 / 10 (50%)1 / 5 (20%)
Median Test change after acceptance (pp)+0.19−1.23
Cost trigger events (19)
Validation gain in correct answers (relative to Full EMAS; out of 218)Reference−23.05, 95% CI [−40.76, −5.88]

Results. On accuracy trigger events, proposals based on matching diagnoses are better (a negative gain means that random evidence is worse than Full EMAS), and the direction is the same in both orders. On cost trigger events, random evidence also has a lower gain.

We have added this control to Appendix C.3.3 of the revised paper (Table 12).

Summary

My recommendation is primarily determined by the limited evidence for the distinctive recurrence mechanism and the reproducibility of its stability benefit. Clarification or additional evidence addressing these two points could change my assessment.

We have added new evidence on these two points. Our conclusions are as follows:

  • Recurrence mechanism (Q2). In a single-step control with matched evidence volume, edit scope, and Validation budget, proposals that require matching diagnoses are better (the difference in Validation gain, random evidence minus Full EMAS, is −5.79 correct answers, 95% CI [−13.69, −0.74], with the same direction in both orders). More of them are accepted (10 vs. 5), and a higher fraction of the accepted ones improve on Test (50% vs. 20%).
  • Reproducibility (Q1). We ran new independent trajectories under two random task orders, up to V10. Full EMAS shows large gains in both orders (+23.65 pp and +44.24 pp), compared with +16.91 pp and −2.08 pp for κ=1. Its cumulative regression is lower in Order 1 (4.04 vs. 5.27 pp). In Order 2, it is slightly higher (6.25 vs. 5.15 pp), but Full EMAS regresses in only one early step (V1→V2) and then improves without further regression to 95.96%, whereas κ=1 regresses repeatedly and ends below V0. The evolution of Full EMAS is therefore more stable.
  • Cost (Q2). Up to V20, Full EMAS processes 5.14× as many Evolve samples as κ=1 and uses about 2× as many optimization tokens, mostly for paired Validation. In return, its V20 system is more accurate (94.12% vs. 87.62%) and uses far fewer inference tokens (3,447 vs. 17,032 per task). At the same number of processed Evolve samples, κ=1 uses 2.3× (1 epoch) and 3.0× (2 epochs) as many optimization tokens as Full EMAS.

Thank you again for your constructive feedback. We hope this response addresses your concerns.

Dear Reviewer Je5r,

Thank you again for your detailed and constructive review. As the author–reviewer discussion closes on Nov 18 (AoE), we would like to kindly check whether our responses have addressed the two points you identified as determining your recommendation. Since our replies are long, we summarize them briefly here:

  1. Reproducibility of the stability benefit. We ran new independent evolution trajectories (Game24, Qwen3.6-27B) under two shuffled task orders, with diagnoses, proposals, and acceptance decisions all regenerated. Full EMAS improved over V0 by +23.65 pp and +44.24 pp, compared with +16.91 pp and −2.08 pp for κ = 1. Cumulative regression was lower for Full EMAS in Order 1 (4.04 vs. 5.27 pp) and slightly higher in Order 2 (6.25 vs. 5.15 pp), where all of it came from a single early step, followed by no further regression (Appendix C.3.2, Tables 10–11).

  2. Whether diagnostic agreement itself accounts for the benefit. Following your suggestion, we ran a control at 57 pre-sampled trigger events that keeps the evidence volume, edit scope, and paired-validation rule fixed but replaces matching diagnoses with randomly drawn ones. On accuracy triggers, random evidence yields lower validation gain (−5.79 correct answers, 95% CI [−13.69, −0.74], same direction in both orders), fewer accepted proposals (5 vs. 10), and fewer Test improvements among accepted ones (20% vs. 50%) (Appendix C.3.3, Table 12).

  3. Cost. We now report the full cost breakdown up to V20 and a comparison at a matched number of Evolve samples (Appendix C.4.1, Tables 16–17), and state explicitly that the V20 accuracy advantage holds at an equal number of accepted revisions, not at equal optimization cost.

If any concern remains, we would be glad to clarify before the deadline. If these results address your concerns, we would be grateful if you would consider revisiting your assessment.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.