acceptodds
Under review as a conference paper at ICLR 2027

Choosing the Task Order Returns More Than Governing What Gets Written

Abstract

Self-evolving LLM agents improve by writing their own experience into a memory bank and reusing it on later tasks. Prior work has focused on what the bank should keep: which experiences to admit, distil or prune. We show that a variable one step earlier can matter more: the order in which the same tasks arrive. On execution-graded code benchmarks, permuting a training stream while holding its content fixed moves held-out success by 4-11 points, and the spread shrinks to evaluation noise once memory is disabled, locating the effect in what the agent writes. Order changes which experiences exist, and filtering them afterwards does not recover the gain: at identical cost per deployed run, choosing the order adds 1.71 points on average, while write-time governance and seven further admission and retrieval rules yield no significant gain. We turn this finding into the ordering lever, a pilot protocol that briefly tries each candidate order on the target task pool and commits to the winner. It raises all four write policies on HumanEval+ by 2.3-3.2 points, including Reflexion, ExpeL and Agent Workflow Memory, recovers over 90% of the best candidate's gain on 34 freshly drawn task pools, and comes with a reliability score, computed from the pilot alone, that tracks its payoff. The best order is specific to each writer: installing one writer's winning order on another usually hurts, while that writer's own pilot helps. For agents that learn from their own experience, arrival order is a design choice to be measured, not copied.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
elf-evolving agents, agentic memory, task ordering, path dependence, experience curation, lifelong learning, code generation benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wZCFOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies task arrival order in memory-based self-evolving coding agents. Holding the training tasks fixed, the authors find that different permutations lead to substantial differences in subsequent evaluation performance. They investigate the memory pathway using no-memory controls and comparisons across write policies, and propose a pilot protocol that evaluates candidate orders and selects one for subsequent runs. The experiments also show that a favorable order for one writer can be unfavorable for another. The protocol targets fixed, repeatedly graded workloads, with selection and confirmation performed on separate seeds but the same evaluation tasks.

Critical strengths
  1. The paper provides a systematic empirical investigation of an important experimental variable. The fixed-content permutation design, no-memory controls, and evaluations across benchmarks, model scales, and write policies provide useful evidence that arrival order can materially affect memory-based adaptation.

  2. The interaction between task order and the write policy is particularly informative. The finding that an order can help one writer and hurt another suggests that order should be evaluated jointly with the memory mechanism, rather than treated as a universally transferable curriculum.

  3. The reporting is unusually careful about experimental scope and uncertainty. I appreciate the separation of pilot and confirmation seeds, the reported pre-registration and multiple-comparison procedures, and the explicit disclosure of the fixed-workload setting, pilot cost, and shared-harness baseline implementations.

Critical weaknesses
  1. The comparison between order selection and write-time governance does not fully separate the value of the intervention from the value of searching over candidates. Order selection receives a 12-candidate, three-seed pilot, whereas the paper does not demonstrate an equivalent selection protocol over governance configurations. Although additional governance interventions are explored, a matched search would provide stronger support for the comparative conclusion.

  2. The contribution is primarily empirical, with limited additional mechanistic explanation beyond the previously reported order sensitivity. The bank analyses document changes in stored experience, but do not clearly establish which properties determine the resulting gains. The practical benefit of the 36-trial pilot also remains uncertain outside workloads where its cost can be sufficiently amortized.

Questions for authors
  1. Under the same pilot budget and evaluation access, how would selecting among governance configurations perform? A matched comparison would help distinguish the benefit of searching over alternatives from the specific benefit of optimizing task order.

  2. Can the authors provide a concrete deployment example and an estimate of the reuse needed to justify the pilot cost? Since the protocol assumes a fixed, repeatedly graded workload, it would be helpful to explain its advantages relative to retaining successful solutions or reusing an already constructed memory bank.

  3. Please reconcile the reference-free candidate count: Section 5.3 states four orders, whereas Appendices B and H state ten. Table 8 and Appendix J also report different permutation-test counts, 20,000 and 2,000.

  4. The title does not mention the self-evolving agent setting, making the scope unclear at first glance. I suggest explicitly including “memory-based self-evolving agents” to distinguish this study from task-order effects in conventional model training or scheduling.

6 comments
AuthorsOpenReview ↗

Response on Numerical Verification, Equal-Budget and Joint Selection, Solution Caching Comparison, and Early-Sequence Mechanism Analysis

Thank you for taking the time to thoroughly review our paper and provide insightful comments.

Addressing your concerns regarding numerical discrepancies between the main text and appendix, distinguishing intrinsic method gains from candidate selection bias under an equal budget, the cost-effectiveness of directly caching solutions or reusing existing note banks on repeated tasks, the specific memory bank properties driving the gains and the practical value of piloting, as well as the scope of the title, we have verified the experimental grids and permutation test statistics one by one, added equal-budget single-dimension and joint search experiments, solution caching versus frozen bank comparisons, temperature-0.7 memory/no-memory controls, and memory property correlation analyses, and revised our title to Task Arrival Order Shapes Experience Accumulation in Memory-Based Self-Evolving Agents. Below, we report our detailed responses to each of your questions.

1 · Equal-Budget Comparison, Separation of Search Bias, and Joint Selection Performance Regarding your concern about "how the two methods perform under the exact same budget, and whether intrinsic method gains can be distinguished from the gains of selecting among candidates," we added two sets of experiments under a strictly equal budget (both using 36 target-pool pilot trials: seeds 0 to 2 for piloting, independent confirmation seeds 5 to 9 for evaluation): a single-dimension comparison and a joint (order, write_rule) selection. The results show that the returns from memoryless search and admission-gating search are far lower than from order search, confirming that the gains do not simply stem from a "1-of-12" selection bias; meanwhile, equal-budget joint selection achieves synergistic improvements across all three benchmarks. First, the equal-budget single-dimension comparison and second, the equal-budget joint (order, write_rule) selection results are summarized below (all under a 36-trial pilot budget; single-dimension net gains are computed relative to the mean of their respective 12 candidates, with ties for first place resolved lexicographically for the primary rule while listing all tied items):

BenchmarkSingle-Dim Select Order (Ungated)Single-Dim Select Order (Default OrderGuard)Single-Dim Select Write Rule (12 Candidates)Equal-Budget Joint (Order, Write Rule) Selection (12 Pre-sampled Pairs from 144 Combinations)
MBPP+1.26pp [+0.86, +1.68] (45.53%)+1.65pp (46.52%)+0.74pp [+0.45, +1.00] (tied for 1st: k1_w0 and k2_w0 both at 46.52%)Selected (k4_w3, hard_first): confirmation pass rate 46.24% (+1.54pp over 12-pair joint mean)
HumanEval++2.69pp [+1.56, +3.82] (75.73%)72.00% (single-dim order selection within gating system)+1.47pp [+0.64, +2.11] (k1_w0, 72.27%)Selected (adaptive, interleaved): confirmation pass rate 73.33% (+1.29pp over 12-pair joint mean, +2.93pp over default reference k2_w3+random_0 at 70.40%)
LiveCodeBench+2.29pp [+1.54, +3.05] (35.23%)+2.86pp (36.77%)Winner among 11 admission & retrieval gates k4_w0: +0.43pp (34.00%); periodic full-bank reset control reset20: +3.50pp (37.38%)Pilot tied for 1st: (k3_w3, random_7) at 36.00% and (k1_w3, random_4) at 33.69% (up to +2.49pp over 12-pair mean, +2.46pp over default reference)

If the gains merely came from a generic "1-of-12" search bias, then memoryless search (only +0.06 to +0.26), 11-admission-gate search (3-benchmark mean +0.88), and task order search (mean +2.08) should yield similar returns; the measured gap proves that task order possesses a higher true signal space because it shapes substantially different memory banks. In joint selection, the winning configuration is never the "default order" or "default gate," but rather dynamically matches gating strengths to the early noise characteristics of the arrival order (e.g., pairing a strict gate k4_w3 when hard_first induces high early exploration noise, and pairing an adaptive gate when interleaved provides a balanced distribution). This proves that task order and write governance are not simple substitutes; jointly calibrating both under a fixed pilot budget effectively unlocks cross-dimensional synergy.

2 · Analysis of Memory Bank Properties Driving Gains and Practical Value of Piloting (W2) In response to your concern that "the current analysis only shows that notes change, without explaining which properties drive the gains, and the practical value of piloting remains unclear," we added paired causal control experiments with and without memory at temperature 0.7, systematically tested correlations between macroscopic memory bank properties and final scores, and evaluated practical value in low-budget and unamortized scenarios. The results demonstrate that external memory is the physical carrier of the order effect, and the key determinant of gains is not a single static scalar, but the quality of early admitted experiences and their low-interference match with subsequent retrieval demands:

  • Causal pathway isolation with/without memory under true stochastic sampling (temperature 0.7, Qwen2.5-14B-Instruct, HumanEval+, 12 orders × 2 control arms × 8 seeds = 192 units):
Control Condition12-Order Max-Min Spread12-Order Confirmation MeanPilot-Selected Order Confirmation Pass RateNet Gain of Order Selection (95% CI)
With Memory (memory)4.83pp71.40%73.60%+2.20pp [+1.18, +3.38]
Without Storing Notes (no-store)0.50pp70.96%70.67%-0.29pp [-0.80, +0.27]

This comparison directly confirms that the external memory bank is the physical carrier of the order effect.

  • Testing memory bank properties that drive gains: Across the three benchmarks, we systematically examined the correlation (rr) between macroscopic scalar properties—such as memory entry count, token volume, first activation step, and practice-phase pass rate—and confirmation scores (E5a):
Macroscopic Memory Bank Property (Correlation rr)MBPP (Memory Prone to Saturation & Competition)HumanEval+ (Fewer Practice Problems)LiveCodeBench (Relatively Scarce Experience)
Effective Entry Coverage / Entry Count-0.208 (mere volume increase shows weak negative correlation)+0.598+0.388
Intrinsic Solution Quality of Early Written Experiences+0.521--
First Activation Step-0.228+0.068-0.341

Combined with the finding that transferring orders across different writers hurts performance in 7/9 combinations, this reveals a clear mechanistic conclusion: what determines the gain is not a task-agnostic scalar like "more notes are better" or "writing earlier is better," but the alignment and low-interference balance between early admitted experiences and subsequent retrieval demands—which also mechanistically explains why static feature regression (-0.34 percentage points) cannot replace actual execution feedback.

  • Practical value when a 36-trial pilot cannot be amortized: On one hand, for budget-constrained scenarios, piloting solely over 4 reference-free orders (only 12 runs, cutting cost to 1/3) still yields substantial gains of +1.23, +2.08, and +1.56 percentage points across the three benchmarks, and our pilot stability criterion can identify low-return problem pools before committing resources; on the other hand, even in one-off task streams with zero upfront order selection, our paper establishes a critical methodological evaluation standard—given that the exact same memory mechanism fluctuates by 4–11 percentage points across different arrival orders and relative rankings across methods can flip with order, future evaluations of any memory-based self-evolving agent must report means and variances across multiple shared task permutations to avoid biased conclusions drawn from a single default problem-ID order.

3 · Solution Caching, Reusing Existing Note Banks, and Deployment Payback Analysis (Q2) Regarding your question, "when fixed tasks repeat, is it more cost-effective to directly store successful solutions or reuse an existing note bank, and can you provide deployment examples and payback estimates?", we empirically compared "verbatim caching of successful solutions," "arbitrarily reusing an unselected existing note bank," and "freezing the optimal note bank via pilot order selection," and calculated the payback period in industrial scenarios. The results show that verbatim repeated problems can directly use caching, whereas homologous workflows with tweaked parameters must rely on textual note banks; in the latter case, frozen banks built from good versus bad orders differ by 3.40 to 4.00 percentage points, and locking in the optimal bank via a one-time pilot breaks even in roughly 3 to 8 rounds of reuse:

  • Comparison with verbatim solution caching and deployment examples: When the problem text and unit tests in every evaluation are verbatim identical and replaying historical source code is permitted, caching is indeed more efficient—under the default OrderGuard configuration on MBPP, we measured that directly replaying the first passing code from the pilot phase hits 73/141 problems (51.77%, higher than the 46.52% of regenerative inference) with zero generation overhead on hit problems. However, in typical real-world enterprise deployments such as continuous integration (CI) automated bug fixing or daily homologous ticket pools, while workflows repeatedly operate around the same codebase and fixed task patterns, specific parameters, input fields, or contextual states shift slightly on each trigger (e.g., adapting similar APIs or varying initial object positions in interactive environments): here, copying historical solution code verbatim fails outright, requiring a transferable textual experience bank to guide generation; similarly, in algorithmic benchmark evaluations of self-evolving agents, evaluation protocols prohibit memorizing and replaying cached test-problem source code.
  • Comparison with "reusing an arbitrary existing bank": The reason one cannot simply run the agent once to build a note bank and reuse it indefinitely is that an unselected note bank permanently carries the experience deficits caused by a single poor arrival order into every subsequent online run. Our frozen note bank experiments (E5b) demonstrate that, holding the evaluation process fixed, frozen note banks constructed from good versus bad orders differ by +3.40 percentage points on MBPP (46.24% vs. 42.84%) and +4.00 percentage points on LiveCodeBench (34.92% vs. 30.92%). Pilot order selection is designed precisely to lock in and reuse that optimal frozen textual note bank after a one-time offline calibration—subsequent deployments simply mount this frozen bank directly, requiring no repeated memory construction and adding zero online inference latency.
  • Amortization payback estimate: A 36-trial pilot consumes 8,640, 5,760, and 8,640 model calls on MBPP, HumanEval+, and LiveCodeBench, respectively (wall-clock time on MBPP with 8-GPU concurrency is ~8.9 minutes, totaling ~0.87–1.23 GPU hours; using a 24-trial 2-seed pilot saves 1/3 of the cost while retaining 84%–99% of the gains). Under the break-even condition R⋅N⋅(gpp/100)⋅v≥Cp⋅cR \cdot N \cdot (g_{pp} / 100) \cdot v \ge C_{p} \cdot c ,on MBPP (N=141N = 141) it improves by +1.65 percentage points over the mean of arbitrarily reusing an unselected note bank (solving ~2.33 more problems per round); in the aforementioned daily CI regression or ticket troubleshooting scenarios, each additionally resolved problem only needs to save a few minutes of manual engineering debugging cost (equivalent to 500–1,000 local 14B model calls), meaning all upfront pilot investment is recovered in about 4 to 8 rounds of reuse (or just 3 to 5 rounds for a 24-trial pilot).
Reviewer_6iufOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Self-evolving LLM agents write their own experience into a memory bank, and prior work mostly asks what the bank should keep, i.e. write-time governance.

The authors permute the arrival order of a fixed task stream and propose the ordering lever, a pilot protocol that briefly tries candidate orders on the target pool and commits to the winner with a reliability score.

The innovation, imo, is showing that order acts through the bank and returns more than governance at equal per-run cost, while a copied order usually hurts.

Critical strengths

The pre-registration, the reserved confirmatory seeds and the no-memory control are careful work, tbh above the norm in this area. The sign flip in Section 3.2 and the bank content analysis in Section 4.2 are, imo, a genuinely useful observation.

Critical weaknesses

The setting is quite narrow. All evidence comes from Python code benchmarks in your Section 2.2 with one retrieve-then-generate agent on Qwen2.5 models. Imo the title promises more than that, e.g. no agent with planning loops, no task dependencies, no other feedback signal is tested. Either widen the experiments or narrow the claims.

I may not be fully sold on the evaluation structure. The pilot picks an order on the very held-out set the final number is scored, i.e. model selection on the test pool, and the authors admit it. Also all runs are at temperature 0, so the seed noise behind the intervals is just batched-inference nondeterminism, which feels like a narrow noise model to me. And the 34 fresh pools are resamples from the same benchmark universes, so transfer to a truly new task distribution seems not ti be tested?

Related work misses a few strands tbh. Task order selection in RL is a mature topic, e.g. Teacher-Student Curriculum Learning. The early lock-in of the bank is close to critical learning periods, e.g. Critical Learning Periods in Deep Networks, which would strengthen Section 3.2. Positioning against a recent memory survey, e.g. A Survey on the Memory Mechanism of Large Language Model based Agents, would help as well.

Questions for authors

see the weakness above

7 comments
AuthorsOpenReview ↗

Response on Experimental Coverage and Generalization Support, Evaluation Design, Related Work, and Readability

Thank you very much for taking the time to carefully review our paper and provide valuable, constructive feedback. We will promptly update the revised manuscript to further clarify the methodological pipeline and experimental descriptions so that the paper is easy to follow, addressing each of the issues you raised. In response to your core concerns regarding experimental coverage (model families, programming languages, planning loops, and feedback modes), same-pool order selection and temperature-0 noise settings, discussions of related literature on curriculum learning, critical learning periods, and agent memory, and the readability of our method workflow, we added experiments across 5 new model families, C++17, non-code reasoning, and 25-step interactive planning loops, supplemented temperature-0.7 stochastic sampling and disjoint unseen-problem transfer tests, articulated our academic positioning in relation to the literature you recommended, restructured our workflow explanation with a step-by-step summary and an intuitive example, and narrowed our title to Task Arrival Order Shapes Experience Accumulation in Memory-Based Self-Evolving Agents. Below, we report our specific improvements and results point by point.

Expanding Experimental Scope Across Models, Languages, Non-Code Domains, and Interactive Planning Loops In response to your concern that our experimental scope was too narrow—and your suggestion to either expand across different models, programming languages, planning loops, task dependencies, and feedback signals or narrow our title and claims—we have both precisely narrowed our title to Task Arrival Order Shapes Experience Accumulation in Memory-Based Self-Evolving Agents to focus on external memory experience accumulation, and added 2,600 complete experimental units (uniformly adopting 12 orders × 8 seeds + 8 no-write controls = 104 units per combination, with seeds 0–2 for pilot order selection and seeds 5–9 for independent confirmation). This expands our evaluation from single-model Python coding to 5 new non-Qwen model families, compiled C++17, competition mathematics, multi-hop QA reasoning, and interactive agents with 25-step planning loops and sparse environmental feedback. Across all 25/25 newly added combinations, point estimates of confirmation gains from order selection are positive:

First, expansion across 5 non-Qwen model backbones on the 3 Python benchmarks (1,560 units; 15/15 combinations are positive, ranging from +0.60 to +5.51 percentage points, with 14/15 having 95% confidence intervals entirely above 0) and second, cross-language expansion to official MultiPL-E C++ (g++ -std=c++17 compilation feedback, 85 practice problems and 75 evaluation problems, 6 models totaling 624 units; 6/6 are positive, with 5/6 models having confidence intervals strictly greater than 0) are summarized below:

Model BackbonePython: MBPPPython: HumanEval+Python: LCBCross-Language Expansion: MultiPL-E C++
Qwen2.5-14B (Primary Model)---+1.16pp [+0.47, +1.82]
Hermes-3-8B (Llama-3.1 Fine-tune)+0.92pp [+0.69, +1.09]+2.27pp [+1.60, +2.89]+0.60pp [+0.10, +1.08]+1.51pp [-0.04, +3.09]
Mistral-7B+0.93pp [+0.37, +1.50]+2.82pp [+1.20, +4.13]+0.76pp [+0.50, +0.91]+5.42pp [+3.91, +7.20]
DeepSeek-Coder-V2-Lite-Instruct+1.47pp [+1.23, +1.78]+1.02pp [-0.53, +2.58]+1.51pp [+1.17, +1.86]+2.69pp [+2.40, +2.96]
Mistral-Nemo-12B+1.65pp [+1.22, +2.10]+5.51pp [+4.93, +6.13]+1.32pp [+1.00, +1.58]+3.62pp [+2.33, +5.09]
Phi-4-14B+1.78pp [+1.51, +1.94]+1.71pp [+0.64, +2.44]+2.90pp [+2.24, +3.64]+1.91pp [+1.16, +2.56]

(Note: We also fully retain the observation of -0.22 percentage points on two seeds of HumanEval+ for the un-finetuned base Llama-3.1-8B for distinction.)

Third, expansion to multi-step planning loops and non-code feedback signals (4 new experimental suites totaling 416 units; 4/4 pilots yield a unique winner and all confirmation gains are positive):

Task & Environment SettingModel12-Order SpreadMemoryless Baseline / 12-Order MeanPilot-Selected Order Confirmation Rate & Gain Over Mean
ALFWorld Multi-Step Interactive Planning (up to 25-step Think/Act loop & sparse goal-completion feedback, 60 seen rooms practice / 72 unseen rooms valid_unseen evaluation)Qwen2.5-14B8.51pp61.94% / 65.07%68.61% (uniquely selected blocked_by_family, +3.54pp [+1.02, +6.16] over mean, +6.67pp over memoryless)
HotpotQA Multi-Hop Knowledge Reasoning QA (exact string match feedback, 87 practice / 120 evaluation problems)Qwen2.5-14B3.85pp65.33% / 66.44%68.17% (+1.72pp [+1.29, +2.08])
Big-Math Competition Mathematical Reasoning (exact numerical verification feedback, 96 practice / 120 evaluation problems)Qwen2.5-14B6.15pp56.00% / 56.60%58.83% (+2.24pp [+0.49, +3.99])
Big-Math Competition Mathematical Reasoning (same as above, non-Qwen model expansion)Phi-4-14B3.13pp- / 52.39%53.17% (+0.78pp [-0.68, +2.17])

Across 6 model families, two programming languages, non-code math/multi-hop QA, and 25-step interactive planning loops, these results demonstrate that whenever an agent accumulates and retrieves experience from a continuous task stream via an external memory bank, task arrival order substantially shapes its experience trajectory and final performance.

Evaluation Structure: Temperature-0.7 Stochastic Sampling and Disjoint Unseen-Task Transfer Addressing your questions regarding same-pool order selection and scoring, the narrow noise range reflected by seeds at temperature 0, and the fact that the 34 resampled pools came from the same benchmark universes, we supplemented 3-benchmark memory/no-memory control experiments under true stochastic sampling at temperature 0.7 (920 execution units) as well as cross-problem transfer tests where order-selection problems and scoring problems are strictly disjoint. The results show that under temperature-0.7 sampling noise, external memory continues to significantly carry the order effect (+2.20 to +3.62 percentage points on HumanEval+ vs. -0.29 percentage points without memory), while on completely isolated unseen problems, tasks with consistent structural and syntactic constraints exhibit stable positive transfer and open-ended tasks with larger topical shifts fluctuate around the mean:

First, 3-benchmark validation under true stochastic sampling noise (temperature = 0.7, totaling 920 execution units, seeds 0–2 for piloting, seeds 5–9 for independent confirmation):

Temp 0.7 Experimental DimensionSetting / Benchmark12-Order Spread or Control PerformanceSelect Task Order Confirmation PerformanceEqual-Budget Comparison / Causal Conclusion
HumanEval+ Memory/No-Memory Causal Control (192 units, 8 seeds)Memory Enabled (OrderGuard / Ungated)Spread 4.83pp / Ungated spread 5.33ppOrderGuard: 73.60% (+2.20pp [+1.18, +3.38] over mean 71.40%); Ungated: 75.20% (+3.62pp)Compared against memory disabled, confirms external memory remains a significant carrier of order effects under true sampling stochasticity
HumanEval+ Memory/No-Memory Causal Control (same as above)Memory Disabled (no-store)Spread drops sharply to 0.50pp70.67% vs. mean 70.96% (-0.29pp [-0.80, +0.27])Order effect vanishes without memory
Cross-3-Benchmark Equal-Budget ComparisonMBPPSelect rule confirmation rate: 44.96%Under OrderGuard: 45.67% (+1.08pp)Order selection robustly leads rule selection
Cross-3-Benchmark Equal-Budget ComparisonHumanEval+Select rule confirmation rate: 70.13%Ungated 75.20% / OrderGuard 73.60%Order selection significantly leads rule selection
Cross-3-Benchmark Equal-Budget ComparisonLiveCodeBenchLower pilot SNR on main pool, increased sampling variance3-seed pilot gain diluted near mean (-0.23pp)Single pool affected by sampling variance
Cross-3-Benchmark Mean Confirmation Pass Rate3-Benchmark Overall MeanSelect write rule mean: 49.03%Ungated order selection: 50.33%; OrderGuard order selection: 49.86%Task order selection leads equal-budget write rule selection under both memory settings
Reviewer_wC8MOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether the arrival order of a fixed training-task pool affects frozen-weight coding agents that write and retrieve textual memories. It reports substantial order-induced differences in held-out execution success, uses no-memory and writer-swap controls to locate the effect in the memory bank, and introduces an “ordering lever”: evaluate twelve candidate orders in a three-seed pilot, select the best, and evaluate that order on disjoint inference seeds. Across MBPP, HumanEval+, and LiveCodeBench, the authors compare this mechanism with write-admission governance, test in-h arness write-policy replicas of Reflexion, ExpeL, and AWM, and evaluate the protocol on 34 additional task pools. A pilot-only rank-stability statistic is proposed to indicate when selection is reliable.

Critical strengths
  • The core order effect is isolated with an unusually complete set of controls. The experiments hold task content fixed, freeze and hash the bank before held-out scoring, and pair evaluations on identical tasks (p. 3, Table 1; p. 17, Appendix B). Disabling memory reduces the spreads to 1.3–2.2 points, while the same selected order can reverse sign across writers (p. 5, Fig. 3). Robustness across Qwen2.5 7B–72B, a doubled MBPP pool, and a ReasoningBank port further reduces the risk that the phenomenon is specific to one backbone or store (pp. 18–20, Table 9). These results convincingly support the paper’s central scientific claim that arrival order can materially affect subsequent behavior through the memory pathway.
  • The selection protocol is evaluated prospectively and with strong reporting discipline. The paper separates pilot and scoring seeds, pre-registers falsification thresholds, reports writer-specific rather than only pooled effects, and releases every per-pool result in Table 14 (pp. 6–8, Tables 3–5; pp. 23–26, Tables 14–15). The copy-versus-own-pilot experiment is particularly informative: copied orders hurt in seven of nine cells, whereas writer-specific pilots help in six (p. 7, Table 4). Positive gains on all 34 additional pools, together with cheaper and reference-free variants, provide meaningful evidence that the protocol works within the paper’s stated fixed-workload regime (p. 8, Table 5; pp. 23–24, Appendix H).
Critical weaknesses
  • The headline ordering-versus-governance comparison is not matched in information or search budget. Order selection receives a 36-trial, target-pool-specific pilot that directly optimizes success on the same held-out tasks later used for scoring, albeit with disjoint inference seeds (pp. 6–7, Eq. 3; p. 17, Appendix B). In Table 2, the comparator is a fixed k=2k=2 gate rather than a governance method selected or tuned with an equivalent target-pool budget (p. 5). The wider governance panel does not close this gap: its rows use development seeds 0–4, different pools, controls, and numbers of paired cells, and the appendix explicitly states that the rows are not a ranking (pp. 19–21, Table 10). Thus, +1.71 versus −0.33 supports an advantage over the tested gate under the chosen accounting, but not the broader title-level conclusion that task ordering generally returns more than governing writes. A matched target-pool tuning comparison, or narrower comparative claims, is needed for this part of the contribution.
  • The deployment evidence concerns costly fixed-pool adaptation, not unseen-task generalization. Both Table 3 and the 34-pool study select an order using held-out-task scores and then report scores on those same tasks with new inference seeds; no task-disjoint validation/test split is used for order selection (p. 7, §5.1 “Information boundary”; pp. 23–24, Appendix H). The authors do acknowledge this fixed-workload scope, but “held-out” and “fresh pools” can nevertheless be read as stronger generalization than is demonstrated. Practical value also depends on a substantial upfront cost: 36 complete trials per pool and writer amount to 5,760–8,640 generator calls using the per-run counts in Table 7 (pp. 7 and 18). The paper reports per-run parity and amortization in principle, but not total-cost or break-even accounting. These issues limit the strength of the deployment claim rather than the existence of the order effect itself.
Questions for authors
  1. Can you provide a comparison in which write-governance selection or hyperparameter tuning receives the same target-pool information and a comparable pilot budget as order selection? If not, would you narrow the title and main comparative claims to the fixed k=2k=2 gate and the specific governance mechanisms tested?
  2. Is any existing experiment task-disjoint—selecting the order on validation tasks and scoring it on separate test tasks—or should all reported gains be understood strictly as fixed-pool model selection? Please make this distinction explicit in the abstract and Table 3/Table 5 captions.
  3. What are the total wall-clock and generator-call costs of the 24- and 36-trial pilots, and under what repetition rate or deployment horizon do they become worthwhile relative to a non-selected order?
4 comments
AuthorsOpenReview ↗

Response to Reviewer wC8M: Equal-Budget Fair Comparison, Fixed-Pool Scope Clarification, and Pilot Cost Accounting

Thank you very much for taking the time to carefully review our paper and provide constructive feedback. In response to your core concerns regarding the equal-budget fairness between task order selection and write governance, the distinction between fixed-pool evaluation and unseen-task transfer, and the total cost and amortization of the pilot phase, we conducted strictly matched 36-trial equal-budget comparisons across all three benchmarks, clarified the fixed-pool protocol while adding task-disjoint unseen-problem transfer experiments across 6 models, and compiled detailed call counts, wall-clock runtimes, and amortization break-even analyses, while narrowing our title to Task Arrival Order Shapes Experience Accumulation in Memory-Based Self-Evolving Agents. Below, we present our specific improvements and findings point by point.

Question 1: Equal-Budget Comparison Between Task Order Selection and Write Governance To address your question on whether write governance would match or outperform order selection when given the exact same target-pool pilot budget, we ran a strictly matched 36-trial target-pool pilot comparison with identical target-pool information, and narrowed our paper title to Task Arrival Order Shapes Experience Accumulation in Memory-Based Self-Evolving Agents while confining our comparative claims to the tested mechanisms and budgets. The results show that under the same 36-trial budget, task order selection achieves more than double the average net search gain (+2.08 percentage points) of tuning per-item write and retrieval governance rules (+0.88 percentage points):

In these matched-budget experiments, model parameters remain frozen throughout and the practice problem content is fixed. Selecting task order (12 candidate orders) and selecting note governance rules (12 candidate configurations: all 11 per-item write admission and retrieval gating hyperparameters, plus a windowed reset control reset20 that clears the entire memory bank every 20 problems, with the default order fixed to random_0) each receive 36 pilot trials (12 candidates × 3 seeds) on the same target evaluation pool using seeds 0 to 2, and are evaluated on independent confirmation seeds 5 to 9 (if a tie for first place occurs during piloting, the primary rule is chosen lexicographically while reporting all tied candidates, strictly never using confirmation seeds to pick winners). Applying a unified metric (confirmation score of the winner selected by the 36 pilot trials minus the mean confirmation score of its 12 candidates), the net search gains across the three benchmarks are as follows:

BenchmarkSelect Task Order (Ungated)Select Task Order (Default Gating OrderGuard)Select Write Rule (Pool of 11 Admission & Retrieval Gates)Broad Governance Pool (incl. Periodic Full-Bank Reset reset20)
MBPP+1.26pp [+0.86, +1.68] (45.53%)+1.65pp [+1.06, +2.19] (46.52%)+0.67pp (search within 11-admission pool)+0.74pp [+0.45, +1.00] (pilot tied for 1st: k1_w0 and k2_w0 both at 46.52%)
HumanEval++2.69pp [+1.56, +3.82] (75.73%, leading rule selection by +3.47pp)72.00% (locked within default k2_w3 gate, 95% CI of difference vs. rule selection covers 0; 36-trial joint order-governance search reaches 73.33%)+1.47pp [+0.64, +2.11] (k1_w0, 72.27%)+1.47pp [+0.64, +2.11] (k1_w0, 72.27%)
LiveCodeBench+2.29pp [+1.54, +3.05] (35.23%)+2.86pp [+2.32, +3.27] (36.77%)+0.43pp [+0.25, +0.59] (k4_w0, 34.00%)+3.50pp (reset20, forcibly clearing entire memory bank every 20 problems, 37.38%)
3-Benchmark Mean+2.08pp (52.16%)-+0.88pp (50.93%)+1.90pp (52.06%)

Across all three benchmarks under the exact same 36-trial pilot budget and unified accounting standard, the average net search gain of selecting task order (+2.08 percentage points, mean pass rate 52.16%) is more than 2.3 times that of per-item write admission gating tuning (mean +0.88 percentage points, mean pass rate 50.93%). Even when reset20—which periodically clears the entire memory bank—is included in the broad governance candidate pool, the average search gain (+2.08 vs. +1.90 percentage points) and average pass rate (52.16% vs. 52.06%) of order selection remain slightly higher or comparable. This directly addresses your concern about matched search budgets and corroborates our core finding: task arrival order shapes experience accumulation trajectories upstream, and tuning per-item admission thresholds alone cannot substitute for order selection (unless the memory bank is periodically wiped and rewritten).

Question 2: Fixed-Pool Scope Clarification and Task-Disjoint Unseen Problem Transfer Regarding your concern about whether pilot selection and final confirmation share the same evaluation problem pool and how selected orders transfer to strictly unseen tasks, we clarify that all original experiments (including the 34 resampled pools) evaluate fixed-pool selection—where the practice note-writing phase and evaluation phase are completely isolated, while order selection and confirmation occur on the same fixed evaluation pool using disjoint seeds. We will state this boundary explicitly in the abstract and the captions of Tables 3 and 5 in the revised manuscript, and we additionally conducted strictly task-disjoint unseen-problem transfer experiments across 6 models. The results show stable positive transfer when task structures and syntactic constraints align, and natural fluctuations around the mean under larger topical shifts:

In the original setup, "held-out problems" means held out relative to the practice problems, corresponding to fixed workflows such as regression test suites or routine maintenance problem pools (in practice, the total runtime for all 36 pilot trials on MBPP is only about 1.23 GPU hours, representing a low one-time calibration cost). Because task order operates by altering specific entries in the memory bank that are subsequently matched via retrieval by target problems, a single static order selected for problem pool A is not naturally expected to be optimal on problem pool B with a different distribution; this does not affect our conclusion that substantial order sensitivity (3.8–8.7 percentage points) exists in fixed-benchmark evaluations.

To directly evaluate task-disjoint generalization where order-selection problems and final scoring problems are completely separate, we added unseen-problem transfer experiments across our primary model and 5 cross-family models (6 models in total; including 100 unseen MBPP problems, 100 unseen functional LiveCodeBench problems, and a "38-problem order selection, 37-problem unseen test" split for HumanEval+ and MultiPL-E C++ since 75 evaluation problems remain after excluding the 85 practice problems). The confirmation performance of each setting relative to the 12-order mean is as follows:

Unseen Problem Evaluation SettingPositive Transfer Performance (vs. 12-order mean, in pp)Fluctuation Performance Under Larger Topical Shift / Without Re-selecting Order (in pp)
Cross-Lingual C++ Unseen Problems (MultiPL-E, 38 select / 37 test)5 out of 6 models achieve positive gains: Mistral-7B +2.25, Mistral-Nemo-12B +2.07, DeepSeek-Coder-V2 +1.04, Qwen2.5-14B +0.59, Phi-4-14B +0.41Tight structural and syntactic constraints yield stable overall positive transfer
Python HumanEval+ Unseen Problems (38 select / 37 test)Hermes-3-8B +4.95, Mistral-Nemo-12B +4.59, Qwen2.5-14B +1.53Mistral-7B -2.12, DeepSeek-Coder-V2 -1.71, Phi-4-14B -1.26 (minor pullback)
Python MBPP Unseen Problems (100 problems)Mistral-Nemo-12B +2.70Qwen2.5-14B -1.93; remaining models range between -1.57 and +0.22
Python LiveCodeBench Unseen Problems (100 problems)Mistral-Nemo-12B +1.32Qwen2.5-14B -2.10 (slightly below mean)

This pattern—"positive transfer when structurally similar, fluctuating under distribution shift"—clarifies the exact generalization boundary of our method: order selection adapts the memory bank composition to the retrieval structure of target tasks rather than producing a distribution-agnostic universal curriculum. For open-ended unseen tasks, this also points to a natural future direction: maintaining diverse memory banks constructed from several complementary orders in parallel and performing score-free online memory bank routing based on problem similarity at zero pilot cost.

Question 3: Pilot Call Counts, Wall-Clock Runtime, and Amortization Break-Even Accounting

In response to your request to quantify the total call overhead, wall-clock runtime, and practical amortization horizon of the pilot phase, we audited the exact generator call counts and concurrent wall-clock times across all three benchmarks and formulated an explicit break-even model. The accounting shows that a 24-trial pilot requires 3,840-5,760 generator calls (approximately 6.2 minutes wall-clock on MBPP with 8-GPU concurrency) and a 36-trial pilot requires 5,760-8,640 calls (approximately 8.9 minutes wall-clock), breaking even in approximately 3 to 5 rounds and 4 to 8 rounds of reuse, respectively, in routine repeated evaluation workflows:

Piloting is a one-time offline calibration: once the optimal order is selected and its corresponding frozen memory bank is saved directly, every subsequent deployment run requires no repeated memory construction and maintains zero additional inference latency for single-pass generation (unlike test-time multi-sampling scaling, it adds no compute overhead or response latency per online call). Based on the call count of a single complete trial (MBPP: 240 calls, HumanEval+: 160 calls, LiveCodeBench: 240 calls), the total upfront costs and measured runtimes are as follows:

Cost & Runtime MetricMBPP (240 calls/trial)HumanEval+ (160 calls/trial)LiveCodeBench (240 calls/trial)Notes & Comparative Explanation
Total Calls for 24-Trial Pilot (12 candidates x 2 seeds)5,760 calls3,840 calls5,760 callsSaves 1/3 of call cost while retaining 84%-99% of order selection gains
Total Calls for 36-Trial Pilot (12 candidates x 3 seeds)8,640 calls5,760 calls8,640 callsOne-time offline calibration, zero extra inference latency after deployment
Original 36-Trial Cumulative Worker-Hours1.23 hours1.72 hours13.81 hoursCumulative multi-GPU worker-hours rather than serial waiting time
New Equal-Budget 36-Trial Worker-Hours: Select Order / Select Write Rule0.87 / 0.87 hours0.97 / 1.02 hours3.75 / 3.84 hoursOverhead of selecting order and selecting write rules is on the exact same order of magnitude
8-GPU Concurrent Actual Wall-Clock Time (excl. model loading)24 trials: 369s (approximately 6.2 min); 36 trials: 535s (approximately 8.9 min)--Measured from shared concurrent run logs on MBPP

In scenarios involving repeated evaluation on fixed workflows or regression testing, let the number of target evaluation problems per round be NN, the improvement over an unselected order (12-order mean) be gppg_{pp}, the number of repeated rounds be RR, the cost per generation call be cc, and the human debugging and fixing value saved per additional automatically passed problem be vv; the pilot breaks even when R⋅N⋅(gpp/100)⋅v≥Cp⋅cR \cdot N \cdot (g_{pp} / 100) \cdot v \ge C_{p} \cdot c Taking MBPP (N=141N = 141, 36 pilot trials totaling 8,640 calls, approximately 0.87-1.23 GPU hours) as an example: under OrderGuard, it improves by +1.65 percentage points over the unselected 12-order mean (equivalent to stably solving approximately 2.33 more problems per round; or approximately 1.78 more problems per round under an ungated memory bank). In scenarios like daily CI regression suites or routine ticket pools, each additionally solved problem saves several minutes of manual troubleshooting time (equivalent to the cost of 500-1,000 local 14B model inference calls), meaning a 36-trial pilot requires only about 4 to 8 rounds of reuse (and a 24-trial pilot requires only about 3 to 5 rounds) to recover the entire one-time offline investment, whereas one-off, low-frequency tasks do not require pilot order selection.

We welcome any further comments on aspects that could be clarified or improved, and we will be glad to address them thoroughly. Thank you again for your time and constructive review.

Dear Reviewer, Please feel free to let us know if you have any further concerns or questions. We are ready to add additional experiments to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.