- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies task arrival order in memory-based self-evolving coding agents. Holding the training tasks fixed, the authors find that different permutations lead to substantial differences in subsequent evaluation performance. They investigate the memory pathway using no-memory controls and comparisons across write policies, and propose a pilot protocol that evaluates candidate orders and selects one for subsequent runs. The experiments also show that a favorable order for one writer can be unfavorable for another. The protocol targets fixed, repeatedly graded workloads, with selection and confirmation performed on separate seeds but the same evaluation tasks.
Critical strengths
-
The paper provides a systematic empirical investigation of an important experimental variable. The fixed-content permutation design, no-memory controls, and evaluations across benchmarks, model scales, and write policies provide useful evidence that arrival order can materially affect memory-based adaptation.
-
The interaction between task order and the write policy is particularly informative. The finding that an order can help one writer and hurt another suggests that order should be evaluated jointly with the memory mechanism, rather than treated as a universally transferable curriculum.
-
The reporting is unusually careful about experimental scope and uncertainty. I appreciate the separation of pilot and confirmation seeds, the reported pre-registration and multiple-comparison procedures, and the explicit disclosure of the fixed-workload setting, pilot cost, and shared-harness baseline implementations.
Critical weaknesses
-
The comparison between order selection and write-time governance does not fully separate the value of the intervention from the value of searching over candidates. Order selection receives a 12-candidate, three-seed pilot, whereas the paper does not demonstrate an equivalent selection protocol over governance configurations. Although additional governance interventions are explored, a matched search would provide stronger support for the comparative conclusion.
-
The contribution is primarily empirical, with limited additional mechanistic explanation beyond the previously reported order sensitivity. The bank analyses document changes in stored experience, but do not clearly establish which properties determine the resulting gains. The practical benefit of the 36-trial pilot also remains uncertain outside workloads where its cost can be sufficiently amortized.
Questions for authors
-
Under the same pilot budget and evaluation access, how would selecting among governance configurations perform? A matched comparison would help distinguish the benefit of searching over alternatives from the specific benefit of optimizing task order.
-
Can the authors provide a concrete deployment example and an estimate of the reuse needed to justify the pilot cost? Since the protocol assumes a fixed, repeatedly graded workload, it would be helpful to explain its advantages relative to retaining successful solutions or reusing an already constructed memory bank.
-
Please reconcile the reference-free candidate count: Section 5.3 states four orders, whereas Appendices B and H state ten. Table 8 and Appendix J also report different permutation-test counts, 20,000 and 2,000.
-
The title does not mention the self-evolving agent setting, making the scope unclear at first glance. I suggest explicitly including “memory-based self-evolving agents” to distinguish this study from task-order effects in conventional model training or scheduling.
6 comments
2 · Analysis of Memory Bank Properties Driving Gains and Practical Value of Piloting (W2) In response to your concern that "the current analysis only shows that notes change, without explaining which properties drive the gains, and the practical value of piloting remains unclear," we added paired causal control experiments with and without memory at temperature 0.7, systematically tested correlations between macroscopic memory bank properties and final scores, and evaluated practical value in low-budget and unamortized scenarios. The results demonstrate that external memory is the physical carrier of the order effect, and the key determinant of gains is not a single static scalar, but the quality of early admitted experiences and their low-interference match with subsequent retrieval demands:
- Causal pathway isolation with/without memory under true stochastic sampling (temperature 0.7, Qwen2.5-14B-Instruct, HumanEval+, 12 orders × 2 control arms × 8 seeds = 192 units):
| Control Condition | 12-Order Max-Min Spread | 12-Order Confirmation Mean | Pilot-Selected Order Confirmation Pass Rate | Net Gain of Order Selection (95% CI) |
|---|---|---|---|---|
With Memory (memory) | 4.83pp | 71.40% | 73.60% | +2.20pp [+1.18, +3.38] |
Without Storing Notes (no-store) | 0.50pp | 70.96% | 70.67% | -0.29pp [-0.80, +0.27] |
This comparison directly confirms that the external memory bank is the physical carrier of the order effect.
- Testing memory bank properties that drive gains: Across the three benchmarks, we systematically examined the correlation () between macroscopic scalar properties—such as memory entry count, token volume, first activation step, and practice-phase pass rate—and confirmation scores (E5a):
| Macroscopic Memory Bank Property (Correlation ) | MBPP (Memory Prone to Saturation & Competition) | HumanEval+ (Fewer Practice Problems) | LiveCodeBench (Relatively Scarce Experience) |
|---|---|---|---|
| Effective Entry Coverage / Entry Count | -0.208 (mere volume increase shows weak negative correlation) | +0.598 | +0.388 |
| Intrinsic Solution Quality of Early Written Experiences | +0.521 | - | - |
| First Activation Step | -0.228 | +0.068 | -0.341 |
Combined with the finding that transferring orders across different writers hurts performance in 7/9 combinations, this reveals a clear mechanistic conclusion: what determines the gain is not a task-agnostic scalar like "more notes are better" or "writing earlier is better," but the alignment and low-interference balance between early admitted experiences and subsequent retrieval demands—which also mechanistically explains why static feature regression (-0.34 percentage points) cannot replace actual execution feedback.
- Practical value when a 36-trial pilot cannot be amortized: On one hand, for budget-constrained scenarios, piloting solely over 4 reference-free orders (only 12 runs, cutting cost to 1/3) still yields substantial gains of +1.23, +2.08, and +1.56 percentage points across the three benchmarks, and our pilot stability criterion can identify low-return problem pools before committing resources; on the other hand, even in one-off task streams with zero upfront order selection, our paper establishes a critical methodological evaluation standard—given that the exact same memory mechanism fluctuates by 4–11 percentage points across different arrival orders and relative rankings across methods can flip with order, future evaluations of any memory-based self-evolving agent must report means and variances across multiple shared task permutations to avoid biased conclusions drawn from a single default problem-ID order.
3 · Solution Caching, Reusing Existing Note Banks, and Deployment Payback Analysis (Q2) Regarding your question, "when fixed tasks repeat, is it more cost-effective to directly store successful solutions or reuse an existing note bank, and can you provide deployment examples and payback estimates?", we empirically compared "verbatim caching of successful solutions," "arbitrarily reusing an unselected existing note bank," and "freezing the optimal note bank via pilot order selection," and calculated the payback period in industrial scenarios. The results show that verbatim repeated problems can directly use caching, whereas homologous workflows with tweaked parameters must rely on textual note banks; in the latter case, frozen banks built from good versus bad orders differ by 3.40 to 4.00 percentage points, and locking in the optimal bank via a one-time pilot breaks even in roughly 3 to 8 rounds of reuse:
- Comparison with verbatim solution caching and deployment examples: When the problem text and unit tests in every evaluation are verbatim identical and replaying historical source code is permitted, caching is indeed more efficient—under the default OrderGuard configuration on MBPP, we measured that directly replaying the first passing code from the pilot phase hits 73/141 problems (51.77%, higher than the 46.52% of regenerative inference) with zero generation overhead on hit problems. However, in typical real-world enterprise deployments such as continuous integration (CI) automated bug fixing or daily homologous ticket pools, while workflows repeatedly operate around the same codebase and fixed task patterns, specific parameters, input fields, or contextual states shift slightly on each trigger (e.g., adapting similar APIs or varying initial object positions in interactive environments): here, copying historical solution code verbatim fails outright, requiring a transferable textual experience bank to guide generation; similarly, in algorithmic benchmark evaluations of self-evolving agents, evaluation protocols prohibit memorizing and replaying cached test-problem source code.
- Comparison with "reusing an arbitrary existing bank": The reason one cannot simply run the agent once to build a note bank and reuse it indefinitely is that an unselected note bank permanently carries the experience deficits caused by a single poor arrival order into every subsequent online run. Our frozen note bank experiments (E5b) demonstrate that, holding the evaluation process fixed, frozen note banks constructed from good versus bad orders differ by +3.40 percentage points on MBPP (46.24% vs. 42.84%) and +4.00 percentage points on LiveCodeBench (34.92% vs. 30.92%). Pilot order selection is designed precisely to lock in and reuse that optimal frozen textual note bank after a one-time offline calibration—subsequent deployments simply mount this frozen bank directly, requiring no repeated memory construction and adding zero online inference latency.
- Amortization payback estimate: A 36-trial pilot consumes 8,640, 5,760, and 8,640 model calls on MBPP, HumanEval+, and LiveCodeBench, respectively (wall-clock time on MBPP with 8-GPU concurrency is ~8.9 minutes, totaling ~0.87–1.23 GPU hours; using a 24-trial 2-seed pilot saves 1/3 of the cost while retaining 84%–99% of the gains). Under the break-even condition ,on MBPP () it improves by +1.65 percentage points over the mean of arbitrarily reusing an unselected note bank (solving ~2.33 more problems per round); in the aforementioned daily CI regression or ticket troubleshooting scenarios, each additionally resolved problem only needs to save a few minutes of manual engineering debugging cost (equivalent to 500–1,000 local 14B model calls), meaning all upfront pilot investment is recovered in about 4 to 8 rounds of reuse (or just 3 to 5 rounds for a 24-trial pilot).
Response on Numerical Verification, Equal-Budget and Joint Selection, Solution Caching Comparison, and Early-Sequence Mechanism Analysis
Thank you for taking the time to thoroughly review our paper and provide insightful comments.
Addressing your concerns regarding numerical discrepancies between the main text and appendix, distinguishing intrinsic method gains from candidate selection bias under an equal budget, the cost-effectiveness of directly caching solutions or reusing existing note banks on repeated tasks, the specific memory bank properties driving the gains and the practical value of piloting, as well as the scope of the title, we have verified the experimental grids and permutation test statistics one by one, added equal-budget single-dimension and joint search experiments, solution caching versus frozen bank comparisons, temperature-0.7 memory/no-memory controls, and memory property correlation analyses, and revised our title to Task Arrival Order Shapes Experience Accumulation in Memory-Based Self-Evolving Agents. Below, we report our detailed responses to each of your questions.
1 · Equal-Budget Comparison, Separation of Search Bias, and Joint Selection Performance Regarding your concern about "how the two methods perform under the exact same budget, and whether intrinsic method gains can be distinguished from the gains of selecting among candidates," we added two sets of experiments under a strictly equal budget (both using 36 target-pool pilot trials: seeds 0 to 2 for piloting, independent confirmation seeds 5 to 9 for evaluation): a single-dimension comparison and a joint
(order, write_rule)selection. The results show that the returns from memoryless search and admission-gating search are far lower than from order search, confirming that the gains do not simply stem from a "1-of-12" selection bias; meanwhile, equal-budget joint selection achieves synergistic improvements across all three benchmarks. First, the equal-budget single-dimension comparison and second, the equal-budget joint(order, write_rule)selection results are summarized below (all under a 36-trial pilot budget; single-dimension net gains are computed relative to the mean of their respective 12 candidates, with ties for first place resolved lexicographically for the primary rule while listing all tied items):(Order, Write Rule)Selection (12 Pre-sampled Pairs from 144 Combinations)k1_w0andk2_w0both at 46.52%)(k4_w3, hard_first): confirmation pass rate 46.24% (+1.54pp over 12-pair joint mean)k1_w0, 72.27%)(adaptive, interleaved): confirmation pass rate 73.33% (+1.29pp over 12-pair joint mean, +2.93pp over default referencek2_w3+random_0at 70.40%)k4_w0: +0.43pp (34.00%); periodic full-bank reset controlreset20: +3.50pp (37.38%)(k3_w3, random_7)at 36.00% and(k1_w3, random_4)at 33.69% (up to +2.49pp over 12-pair mean, +2.46pp over default reference)If the gains merely came from a generic "1-of-12" search bias, then memoryless search (only +0.06 to +0.26), 11-admission-gate search (3-benchmark mean +0.88), and task order search (mean +2.08) should yield similar returns; the measured gap proves that task order possesses a higher true signal space because it shapes substantially different memory banks. In joint selection, the winning configuration is never the "default order" or "default gate," but rather dynamically matches gating strengths to the early noise characteristics of the arrival order (e.g., pairing a strict gate
k4_w3whenhard_firstinduces high early exploration noise, and pairing anadaptivegate wheninterleavedprovides a balanced distribution). This proves that task order and write governance are not simple substitutes; jointly calibrating both under a fixed pilot budget effectively unlocks cross-dimensional synergy.