acceptodds
Under review as a conference paper at ICLR 2027

Harness Evolution as Learning: Approximation, Generalization, and Optimization Limits of Self-Improving Personal Agents

Abstract

As the capabilities of large language models (LLMs) continue to advance, increasing attention is turning to how to translate their abilities into useful behavior. Personal agents bring this question into everyday settings, where models are expected to serve individual users and continually adapt to their preferences. With the underlying model held fixed, such adaptation relies on harness engineering: designing and evolving the surrounding layer that manages context, memory, tools, and execution. Despite rapid progress, the factors governing effective harness evolution remain insufficiently understood. To narrow this gap, we investigate three central questions concerning harness architecture, harness scale, and self-evolution algorithms through complementary empirical and theoretical analyses. Empirically, we introduce a preference-oriented benchmark and systematically characterize the capabilities and limitations of personal agents associated with these three dimensions. Theoretically, we formulate harness evolution as a learning problem and explain these phenomena through approximation, generalization, and optimization errors. Analyses of reachable policies, capacity under finite interaction evidence, and biased update dynamics provide theoretical accounts of the observed phenomena. Together, these results offer a unified perspective on the limits of personalization through harness evolution and inform future harness design. We open-source our code through an anonymous GitHub repository at https://anonymous.4open.science/r/harness-as-learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, personal agents, harness engineering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s4hWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the capabilities and limitations of self-improving personal agents, which adapt to user preferences by evolving their control frameworks while keeping the underlying large language model (LLM) unchanged. The results indicate that: (1) context-based constraint frameworks can effectively handle simple preferences, while more complex preferences require external computation and execution control; (2) increasing memory size initially improves preference compliance but may ultimately lead to a decline in performance; (3) Existing self-evolutionary methods improve performance during early interactions but often reach a plateau, still showing a significant gap compared to the “Oracle” constraint framework. The authors further elucidated these observations through theoretical analyses.

Critical strengths
  1. Easy to follow.
  2. Important and timely study.
  3. Comprehensive experimental results and complementary analysis.
Critical weaknesses

The main weakness of this paper is that its key observations are largely intuitive and consistent with existing knowledge, offering limited novel insights. For example, the performance degradation observed with increasing memory size is not particularly surprising, as it could be attributed to various factors, such as long-context interference, hallucinations, or redundant information, rather than memory scaling itself. However, the paper does not sufficiently disentangle these factors or establish a clear connection between the empirical observations and the proposed theoretical explanations. Overall, the main findings are largely expected, which limits the novelty and significance of the paper's contributions.

Questions for authors

please see weakness

4 comments
AuthorsOpenReview ↗

rebuttal (part 1)

We thank the reviewer for recognizing the importance and timeliness of this study, its readability, and its comprehensive experiments and complementary analysis. Your concern about the specificity of the insights and the connection between theory and experiments has motivated additional controlled comparisons and trajectory audits. We address the weakness and the corresponding question together below.

1. Contribution: a learning formulation that guides diagnosis

Our contribution is to formulate personal-agent harness evolution as a learning problem and use this formulation to identify what to inspect and improve. With the underlying model frozen, the learned object includes the surrounding memory and execution mechanisms. This separates three questions: which behaviors the harness can realize, how reliably interaction evidence supports its contents, and how updates correct errors while preserving valid information.

AppWorld-P supplies a common executable setting for studying these questions through preference checks on actual actions. The resulting contribution combines this evaluation setting, theoretical mechanisms, and interventions that identify practical improvements. Section 3.3 organizes the three learning questions, Section 4 develops their empirical and theoretical analyses, and our newly enriched Appendix C reports the additional comparisons and diagnostics.

2. Additional experiments sharpen the actionable insights

The following summarizes what the new comparisons reveal beyond the aggregate performance trends:

QuestionNew diagnostic evidenceImplication for harness design
Q1: ArchitectureProcedures and examples narrow, and sometimes close, the gap from preference statements to runtime mechanisms.Separately assess preference specification, execution knowledge, and how that knowledge is acquired.
Q2: MemorySemantic repair improves long memory but worsens short memory; effects concentrate in particular rules and orderings.Evaluate content quality, memory organization, and task–rule configuration alongside memory length.
Q3: UpdatingClearer feedback improves ACE; local corrections can disturb other rules, and stored rules can still fail during execution.Diagnose correction, preservation of valid content, and execution separately.

Q1: Accurate preference specifications and execution knowledge play distinct roles. We completed a new 228-episode comparison of Base context, Procedural, Few-shot, and Runtime support. Base context supplies accurate preference descriptions and, for habitual-card inference, the shared 20-observation raw history. Procedural adds execution steps, Few-shot adds two worked examples to those steps, and Runtime support augments Base context with external execution mechanisms. For SMS checksums, violations decrease from 6/12 with Base context to 2/12 with Few-shot, matching Runtime support's 2/12 (using up to three attempts). For running totals, Procedural and Few-shot each reduce violations from 3/7 to 1/8. Runtime support achieves 0/6, across six of nine expected payment opportunities. Habitual-card results also favor Procedural (2/15) over Runtime support (3/14). All four habitual-card conditions share the same raw history, aligning Base context with the original From history reference's input.

These findings separate preference specification, execution knowledge, and the mechanism that supplies that knowledge. Procedures and examples enrich context with useful implementation steps. Q1's assertion-selection updater learns by selecting from a fixed statement bank; acquiring richer procedures is a distinct learning objective. Section 4.1 develops this distinction, and Appendix C.1 reports the behavioral gains from supplied execution guidance.

Q2: The effect of expanding learned memory depends on its content and use. Interaction-derived memories capture the errors, incomplete specifications, and repetition that arise during preference learning. All lengths draw from the same learned pool, holding upstream evidence fixed. We audited and semantically repaired the memories while preserving statement count, positions, orderings, and exact repetition relationships. Each condition contains 360 evaluation episodes:

Memory sizeOriginal memoryRepaired memory
10 statements123/620 = 19.84%165/667 = 24.74%
150 statements232/843 = 27.52%182/817 = 22.28%
AuthorsOpenReview ↗

rebuttal (part 2, finished)

Entries are violations divided by applicable episode–rule pairs. Repair improves long memory in all three orderings, particularly SMS greetings and sign-offs. Short memory worsens, largely through sign-off omissions in one ordering despite its unchanged sign-off sentence. The pooled length effect reverses after repair, with the repaired length comparison varying across orderings. These different responses suggest an interaction between content refinement and memory organization. Appendix C.2 reports higher violations for longer memory in the post-hoc non-SMS analysis under both content conditions. A separate analysis of expected task–rule pairs, excluding pre-identified task–preference conflicts, likewise gives 123/623 (19.74%) for repaired L=150L=150 versus 94/504 (18.65%) for repaired L=10L=10. These comparisons highlight content quality and task–rule configuration alongside memory length. The repair completes missing specifications while preserving statement count and revising sentence wording and token length.

Q3: Feedback quality improves learning, and execution diagnostics reveal further targets for improvement. Holding ACE and the interaction budget fixed, explicit corrective feedback reduces final violations from 43/102 (42.16%) to 35/113 (30.97%), with improvement in all three seeds; the contemporaneous oracle is 8/110 (7.27%). Each condition has 54 final evaluation episodes. Audits locate failures despite relevant stored instructions: 12 payment episodes lack an effective description, including one placing formatted text in an ignored argument. Eleven contain successful payments; one contains only failed attempts, scored under the original applicability definition. We also observe cross-rule update errors: correcting SMS behavior corrupts a payment-initials example; later feedback repairs it but temporarily corrupts the SMS signature, which is subsequently repaired. These observations distinguish update and execution failures and motivate different interventions.

3. Making the theory–experiment connection precise

Theory and interventions connect the learning formulation to specific design questions. The architecture analysis describes attainable behavior; the memory theorem relates feedback allocation to statement reliability; and the pool experiment examines expansion from accumulated experience. Section 4.2 and Appendix C connect these perspectives through the memory-repair results.

Proposition B.6 in Appendix B.5 gives a concrete repair–corruption mechanism, illustrated by the three-step ACE case in Appendix C.3. For two harness states with risk difference Δ\Delta, fixed corruption and repair probabilities 0<p,q<10<p,q<1 give stationary excess risk Δp/(p+q)\Delta p/(p+q). The proposition explains how repairable errors can coexist with a residual and identifies both correction and preservation as improvement objectives. The ACE trajectory shows the corresponding cross-rule editing failure and its subsequent repair. Together, these analyses guide information acquisition, preservation of valid content, and reliable execution of stored preferences.

We hope these additional results and clarifications make the paper's contribution and its implications for personal-agent design more concrete. We kindly refer you to our discussion with other reviewers, where we have largely enriched our work both empirically and theoretically.

We would also like to emphasize the motivation behind this work. The difficulties we study are familiar to many researchers building personal agents, and their recurrence makes understanding them especially important. Knowing that longer memories or repeated self-updates can fail still leaves open why they fail and which parts of the system should be improved. Our contribution is to introduce a learning perspective on harness evolution that connects these observations to approximation, generalization, and optimization. By combining controlled experiments with theoretical analysis, we provide a systematic interpretation of these phenomena and identify mechanisms that can produce them. This understanding translates recurring practical failures into concrete design questions about execution knowledge, reliable memory, and the preservation of correct content. We sincerely believe that explaining widely encountered phenomena is itself a meaningful scientific contribution and an essential step toward building more reliable personal agents.

Dear Reviewer s4hW,

With less than 6 hours remaining before the rebuttal deadline, we would like to know whether our response has addressed your concerns. Should you have any further questions, we would be happy to provide additional clarification.

Thank you once again for your time and effort.

Sincerely,

The Authors

We have posted a general response titled "Response After Rebuttal", in which we clarified our motivation. We hope this clarification can help the reviewers and ACs to understand our work better during the final assessment.

Reviewer_41DtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how personal agents adapt to user preferences by harness engineering, including changing the memory, tools, and execution around a frozen language model. The authors formulate harness evolution as a learning problem, relating harness architecture, memory scale, and update procedures to approximation, generalization, and optimization errors. The authors introduce a preference-oriented benchmark AppWorld-P which adds a preference-evaluation layer to AppWorld. In controled experiments, the authors show that external computation and execution checks improve compliance with preferences that explicit instructions do not reliably satisfy. Adding memory and applying several feedback-based update methods help initially, but further expansion or continued updates do not consistently improve performance. Under incomplete feedback, the tested self-evolution methods remain behind an oracle given complete preference descriptions. The theoretical analysis describe conditions under which limited harness capabilities, finite interaction evidence, and biased updates can produce these outcomes.

Critical strengths
  1. The paper proposed a useful perspective for understanding harness evolution. Treating harness evolution as a learning problem provides a way to reason about the roles of architecture, interaction evidence, and update procedures. The idea of using this framework to understand why adaptation stalls and where improvements may be needed is appealing.

  2. The emphasis on feedback quality and update reliability is valuable. Considering incomplete feedback and potentially mistaken interpretations raises an important question about how agents acquire reliable knowledge of user preferences. This perspective is useful for guiding the design of self-evolution methods, particularly how new rules are inferred, checked, and revised over time.

Critical weaknesses
  1. The link between the experimental findings and the proposed theoretical explanations is relatively weak. The experiments show that longer memories can reduce compliance and that repeated updates can leave a gap to the oracle. However, it is unclear whether the specific generalization and optimization mechanisms analyzed in the theory explain these findings. Memory redundancy and differences in available preference information provide alternative explanations that the current evaluation does not clearly distinguish.

  2. The theory assumes some of the limitations it aims to explain. For example, the self-evolution analysis assumes persistent update bias and then shows that an error gap remains. The paper does not sufficiently explain why the tested update methods would have this bias, leaving the underlying cause of stalled improvement unclear.

Questions for authors
  1. Does ACE still make mistakes when the feedback clearly states the correct preference? Your existing test with clearer feedback uses full-memory rewriting. Could you repeat this comparison using the same ACE updater for both feedback settings? If errors remain, please check whether the memory is wrong or the agent fails to follow correct memory. This would help identify where the remaining errors come from.

  2. Does longer memory hurt because it contains incorrectly learned preferences? Could you check the stored memories, then correct any mistaken rules and evaluate again without changing the memory size? This would help distinguish errors in learning the preferences from difficulty following a longer memory, making the connection to the generalization analysis clearer.

6 comments
AuthorsOpenReview ↗

rebuttal (part 1)

We thank the reviewer for recognizing the value of treating harness evolution as a learning problem and for the concrete suggestions on feedback quality and memory reliability. We have completed both requested experiments: a comparison using the same ACE updater under two feedback settings, and a memory-repair intervention at fixed statement counts. Below we report the results, trace the remaining errors, and give a constructive mechanism for the update-bias assumption.

W1: Connecting the theoretical explanations to the experimental findings

Our framework organizes distinct reliability questions and guides targeted interventions. Architecture determines the available behaviors, interaction evidence supports preference learning, and update procedures determine how that evidence changes the harness. Memory redundancy and incomplete preference information are therefore relevant aspects of the learning problem, and your suggestions help distinguish their empirical contributions.

In Q2, all memory lengths draw from the same previously learned pool, holding upstream learning evidence fixed as memory expands. This retains the errors, omissions, and repetition that arise in agent-generated memory. The theoretical analysis characterizes scaling under its statistical model, and the experiment examines the corresponding practical question with learned content. Section 4.2 and Appendix C.2 explain how these perspectives inform memory design. We kindly refer you to our enriched new experimental results and the following responses.

The new interventions make this connection more informative. The same-ACE comparison isolates the feedback setting while holding the updater fixed; the repair experiment changes the content at fixed statement counts. As detailed below, clearer feedback improves ACE, semantic repair improves long memory, and execution errors remain identifiable in the trajectories. Together, these findings connect the learning framework to concrete design choices: improving the information supplied to the updater, preserving valid memory during correction, and checking how stored preferences are realized in tool arguments.

W2: Why can feedback-driven updates produce persistent bias?

A concrete mechanism is that correcting one error can damage previously correct content. Our ACE implementation applies model-proposed additions and modifications directly to the stored memory. Consequently, local corrective feedback can improve its target rule while introducing a different error. This motivates distinguishing the ability to repair errors from the ability to preserve correct content.

A minimal repair–corruption proposition. ( Details in Appendix B.5) Write r(c)=R(πc∘f)r(c)=R(\pi_{c\circ f}). Consider two harness states, GG and BB, with risks r0r_0 and r0+Δr_0+\Delta, where 0≤r0<10\leq r_0<1 and 0<Δ≤1−r00<\Delta\leq1-r_0. Suppose each interaction moves G→BG\to B with fixed probability pp and B→GB\to G with fixed probability qq, with 0<p,q<10<p,q<1. Writing et=Pr⁡(ct=B)e_t=\Pr(c_t=B), we obtain

et+1=p(1−et)+(1−q)et=p+(1−p−q)et.e_{t+1}=p(1-e_t)+(1-q)e_t=p+(1-p-q)e_t.

With e∗=p/(p+q)e_*=p/(p+q),

\qquad \lim_{t\to\infty}\mathbb E[r(c_t)-r(G)] =\Delta\frac{p}{p+q}>0.$$ Thus, **every error can remain repairable while the process retains a positive stationary excess risk**. Holding the other parameters fixed, increasing $q$ reduces that risk. Eliminating corruption ($p=0$, $q>0$) permits convergence to $G$. This is also an explicit instance of Theorem 4.4's assumptions. With $\mathsf d_L(G,B)=1$, the Wasserstein contraction coefficient is $\rho=|1-p-q|<1$. For an idealized empirical objective equal to the population objective, the unique ERM is $G$, and $$\beta=W_L(\mathcal T_U\delta_G,\delta_G)=p,\qquad \mu=2\Delta,\qquad \epsilon_{n,L}=0.$$ These choices satisfy B.3–B.5. The proposition gives an exact stationary excess risk for this example; the theorem provides a more general lower bound. **What makes the corruption persistent?** A skipped update leaves the harness unchanged. Its corruption probability can therefore be decomposed as $$p=\Pr(\text{update triggered}\mid G) \Pr(G\to B\mid G,\text{update triggered}).$$ Execution errors despite correct memory, or feedback concerning another rule, can trigger further edits. Under stationary interaction and update mechanisms, positive update exposure and a positive conditional corruption probability yield $p>0$. ACE's accepted-feedback gate provides a concrete way to suppress updates after accepted behavior. This factorization identifies two design targets: controlling when edits are triggered and ensuring that corrective edits preserve existing preferences.
AuthorsOpenReview ↗

rebuttal (part 2)

The observed ACE trajectory demonstrates the mechanism's practical relevance. Under corrective feedback, one run exhibits the following sequence:

Training interactionFeedback concernsObserved memory change
6SMS greeting and sign-offA valid payment-initials example, (J.D.), becomes invalid (S.).
10Payment initialsInitials are repaired, but the SMS sign-off becomes the literal J.D..
11SMS sign-offThe sign-off is repaired to the user's first name.

This case demonstrates repair and collateral corruption in practice. The proposition explains their long-run balance under the specified update probabilities, linking a concrete editing mechanism to the optimization analysis. Appendix B.5 contains Proposition B.6 and its connection to Theorem 4.4; Appendix C.3 summarizes this three-step update case.

Q1: Does ACE still err with explicit corrective feedback, and where do the errors occur?

Yes. Clearer feedback improves the same ACE updater, and the remaining errors can be localized beyond missing preference information. We have run instance-feedback and corrective-feedback ACE with the same model, updater, 12-statement memory capacity, task sequences, execution interface, and 24 training interactions. Corrective feedback supplies the correct template for each currently violated rule. We also reran the oracle reference in the same experimental batch.

The final evaluation pools three training orders, six evaluation tasks, and three rollouts per task: 54 episodes per condition. The metric is violations divided by applicable episode–rule pairs; an episode can contribute several pairs.

Final evaluation after n=24n=24 training interactionsEpisodesViolations / applicable pairsViolation rate ↓
ACE: instance feedback5443/10242.16%
ACE: corrective feedback5435/11330.97%
Oracle preference descriptions, fixed reference548/1107.27%

Corrective feedback improves the aggregate rate in all three training orders. All five corrective templates have been disclosed in every run by interaction 6 at the latest. Thus, the comparison directly confirms that feedback informativeness explains part of the original shortfall, while revealing what remains after these templates become available.

We audited all 35 final corrective violations:

  • 24 pairs arise from 12 payment episodes lacking an effective description. Each violates both the category and initials rules. Eleven episodes contain successful payments with empty descriptions; one contains only failed payment attempts, retained under the original applicability definition. Eleven omit the required field; one places correctly formatted text in the ignored note argument instead of description.
  • Nine privacy violations occur on the task explicitly requesting public payments. We report these task–preference conflicts separately.
  • Two further privacy violations occur on non-conflicting tasks. The agent uses is_private, whereas the API requires private; the payments succeed with incorrect visibility.

These traces distinguish having relevant preference text from realizing it through effective tool arguments. The temporary update errors in W2 provide complementary evidence about preservation during learning. They were subsequently repaired: the three corrective memories stop changing after interactions 11, 2, and 5, respectively, and evaluation feedback is not returned to the updater. These later evaluations assess execution with fixed stored preferences, making the reliable use of memory a separate target for improvement.

The comparison also remains informative after removing the pre-identified public-payment conflict task: instance, corrective, and oracle rates become 28/87 = 32.18%, 14/86 = 16.28%, and 2/92 = 2.17%. Expected-rule coverage increases from 102/135 (75.56%) to 113/135 (83.70%) between the two ACE conditions, as reported in Table 6. Appendix C.3 reports the comparison, task and joint success, and applicability in Table 6, followed by the residual-error cases and concise metric definitions. Table 7 separately summarizes the original Q3 audit.

The resulting diagnosis identifies complementary improvements: supply explicit target specifications, preserve valid preferences during updates, and validate preference-bearing API arguments during execution. The matched-feedback comparison and trajectory audit connect these design choices to observed information, editing, and execution failures.

AuthorsOpenReview ↗

rebuttal (part 3, finished)

Q2: Does long memory underperform because some preferences were learned incorrectly?

Content quality contributes, and we have completed the requested intervention at fixed statement counts. We audited all 55 distinct assertions in the learned pool against the complete oracle preference specifications. Eleven were retained verbatim; 44 received semantic repairs addressing omissions, scope errors, unsupported requirements, or inaccurate formulations. The intervention preserves statement count, position, ordering, and the original exact-duplicate mapping. It also completes underspecified preferences, so we describe it as oracle-assisted semantic repair. Sentence wording and token length reflect the semantic repairs.

We evaluated original and repaired memories at L=10L=10 and L=150L=150, with three orderings and 360 episodes per cell:

Statement countOriginal memory: violations / applicable pairsRepaired memory: violations / applicable pairs
L=10L=10123/620 = 19.84%165/667 = 24.74%
L=150L=150232/843 = 27.52%182/817 = 22.28%

Long memory benefits from repair in all three orderings. Its overall violation rate decreases by 5.24 percentage points. The improvement is especially evident in SMS greetings and sign-offs.

Repair also changes the overall scaling pattern. The deterioration at L=10L=10 is largely driven by SMS sign-off omissions concentrated in one memory ordering. Although both the sign-off sentence and statement order are preserved, the affected ordering places the sign-off before the greeting, which becomes more explicit after repair. In the other orderings, the sign-off follows the greeting and the deterioration is much smaller. This suggests an interaction between repaired wording and relative instruction order: the more explicit later greeting may overshadow the earlier sign-off requirement during generation. Thus, even correctly stated preferences depend on their organization for reliable joint execution. Appendix C.2 discusses this interaction between content refinement and memory organization.

To isolate the abnormal influence of preference order, we additionally performed a post-hoc domain sensitivity analysis excluding the four SMS tasks, leaving 240 episodes per cell:

Statement countOriginal memoryRepaired memory
L=10L=1038/341 = 11.14%44/382 = 11.52%
L=150L=15068/492 = 13.82%62/463 = 13.39%

Longer memory remains modestly worse in this subset under both content conditions. A separate analysis of expected task–rule pairs, excluding pre-identified task–preference conflicts, gives 123/623 (19.74%) for repaired L=150L=150 and 94/504 (18.65%) for repaired L=10L=10. Together, these comparisons support the conclusion that expanding learned memory can reduce compliance even after content repair. They highlight content quality and task–rule configuration alongside memory length. Appendix C.2 describes the repair intervention, presents the overall and non-SMS results in Table 5, and reports the separate conflict-filtered comparison in the following discussion. Your proposed intervention has helped us make the relationship between learned content and executed behavior considerably more precise.

Reviewer_dfamOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies personalization through harness evolution around a frozen foundation model. It introduces AppWorld-P and examines three dimensions: the capability difference between context and control harnesses, the effect of memory scale, and the limits of feedback driven self evolution. The paper organizes these questions using approximation, generalization, and optimization errors, supported by conditional theoretical analyses and controlled experiments.

Critical strengths

The paper addresses a timely and practically relevant problem and provides a clear framework for analyzing different sources of failure in personal agent adaptation. AppWorld-P uses executable preference checks rather than subjective LLM based evaluation, and the experiments compare multiple harness mechanisms under shared models, tasks, and feedback protocols. The paper is also transparent about experimental settings, baseline adaptations, feedback limitations, and theoretical assumptions.

Critical weaknesses
  1. The proposed decomposition is not yet an operational error diagnosis. Although the decomposition is valid, the paper does not specify how to identify the empirical risk minimizer under weak, on-policy feedback, and the experiments do not separately measure approximation, generalization, and optimization errors.

  2. Q1 does not establish a general reachability limit for context harnesses. The “out-of-support” labels are based on empirical failure with one maximally explicit preference description, while Theorem 4.1 assumes a positive context class risk gap. The results demonstrate the benefit of control in the tested setting, but do not establish that context only approaches are fundamentally unable to satisfy these preferences.

  3. The Q2 experiment does not test the estimation mechanism in Theorem 4.2. The theory divides a fixed feedback budget across L statements, whereas the experiment draws statements from a previously learned pool and recycles them at larger lengths. The observed degradation could therefore reflect repetition, statement quality, or context interference rather than reduced evidence per statement. The increasing theoretical upper bound does not distinguish among these explanations.

  4. The Q3 oracle gap does not isolate optimization failure for the main methods. The authors acknowledge that standard feedback withholds two target literals and provide a fully corrective diagnostic, but that diagnostic uses a full memory rewrite updater rather than ACE, TEPA, or TRACE. Evaluating the main methods under both feedback conditions is needed to separate missing information from update failure.

  5. The primary metric leaves overall task performance unclear. Episodes without an applicable action are excluded from the preference-violation denominator, and applicability counts may differ across methods. Although the Q1 verifier partly mitigates this issue, task success rates, per rule applicability counts, and joint task and preference success are needed to assess whether lower violation rates correspond to better agent outcomes.

Questions for authors

My main concern has already been raised in the weakness section, please refer to that section for details.

7 comments
AuthorsOpenReview ↗

rebuttal (part 1)

We sincerely thank the reviewer for the careful and constructive assessment, and for recognizing the practical relevance of the problem, the value of executable preference checks, and the transparency of our experimental settings. Your comments have helped us clarify how the learning framework connects to the experiments. We have completed additional execution-guidance comparisons, a memory audit and repair study, a same-ACE feedback comparison, and an audit of applicability and task outcomes. We address each concern below.

W1: Operational meaning of the error decomposition

The decomposition provides theoretical reference points and guides targeted interventions into the harness-learning process. With the foundation model fixed, the harness class specifies which behaviors are available, the interaction evidence supports the preferences encoded in the harness, and the update algorithm determines the harness actually obtained. This formulation separates design questions that can otherwise be conflated when personalization fails.

The discussion in Section 3 supplies a theoretical reference for studying how architecture, interaction evidence, and updating affect the learned harness. Our experiments connect these questions to agent behavior through controlled interventions and failure localization:

QuestionIntervention and diagnosticWhat it helps examine
Q1: Architecture and implementationCorrect preference descriptions, supplied execution guidance, and runtime computation/controlThe distinction between specifying a preference and reliably implementing it
Q2: Scale under finite evidenceA shared learned-memory pool, length sweeps, and same-count semantic repairHow memory expansion interacts with the quality of learned content
Q3: Update procedureThe same ACE updater under different feedback, followed by memory and execution auditsMissing preference information, incorrect edits, and failures to follow stored rules

The additional studies below strengthen this diagnostic role. The same-ACE comparison identifies improvements from explicit corrections, and trajectory audits locate remaining errors at particular update or execution steps. Section 3.3 organizes the architecture, scale, and update questions, while Appendix C reports the additional comparisons and diagnostics. The learning formulation guides interventions and connects their outcomes to concrete harness-design decisions.

W2: Context reachability and the purpose of Q1

Q1 examines how preference specifications and execution knowledge contribute to reliable personalization. Golden preference descriptions provide the target requirements; procedures provide the computations, state tracking, and tool-use steps for implementing them.

Expressing an execution procedure and acquiring it through evolution are distinct learning questions. Q1's fixed candidate-bank updater assembles memory from existing preference statements. The added comparison supplies procedures and examples that enrich this content with implementation knowledge. Their behavioral gains identify execution knowledge as an additional learning objective alongside recovering the user's requirements.

Additional execution-guidance comparison. We evaluated a new batch of 228 episodes with Haiku under four conditions with extra procedure guidance: Base context supplies accurate preference descriptions and the shared raw history. Procedural adds hand-written execution steps to Base context. Few-shot adds two synthetic examples to Procedural. Runtime support augments Base context with external computation, feedback, or action rewriting during execution. Prompts were fixed before evaluation. The entries below are violations/applicable episode–rule pairs, with lower values indicating better compliance.

PreferenceBase contextProceduralFew-shotRuntime support
Private note format3/141/140/140/15
SMS checksum6/124/122/122/12
SMS sign-off0/91/91/90/9
Running spend total3/71/81/80/6
Habitual card4/142/152/133/14

For habitual-card inference, all four conditions receive the same 20-observation raw history. Base context therefore uses the preference-plus-history input of the original From history reference. The comparisons use three blocks per task family, with one rollout per task per block. Habitual-card blocks rotate bank labels over a shared observation pattern. Runtime support allows up to three attempts for format/checksum, while the other conditions use one. Its running-total mechanism has six applicable successful payments out of nine expected opportunities, compared with eight for each strengthened-context condition.

AuthorsOpenReview ↗

rebuttal (part 2)

These results show that supplied implementation knowledge substantially improves context-based execution and closes some observed gaps. Preference acquisition and execution-knowledge acquisition are therefore distinct design objectives. Procedures and examples enrich the context available to the actor, while runtime counters, computation, and action rewriting provide additional implementation mechanisms. Their effects identify useful targets for the evolution process.

Theorem 4.1 characterizes policy-class reachability under the latent-concept and risk-separation assumptions. Section 4.1 connects this architectural question with the acquisition of execution knowledge, and Appendix A.1 specifies the preference information supplied by each original harness condition. Appendix C.1 reports the strengthened-context comparison (Table 4). Personalization requires both learning what the user wants and obtaining mechanisms that reliably realize it. What we want to show is that, even a golden preference description is not sufficient for personal agent harness evolution. And for some preferences, a procedure guidance for realization is still essential.

W3: Fixed evidence, recycled memory, and Theorem 4.2

Sharing the learned pool across lengths holds the upstream interaction evidence fixed. This models a personal agent expanding or reorganizing its memory using accumulated experience. Longer memories select and recycle additional statements from that common body of evidence.

Theorem 4.2 analyzes this issue through a particular estimator, with m=⌊n/L⌋m=\lfloor n/L\rfloor independent feedback observations per statement. At fixed nn, its estimation bound is $ L\exp \left(-2\gamma^2\left\lfloor n/L\right\rfloor\right), $ and Corollary 4.3 adds the coverage term (d−L)+/d(d-L)_+/d. Together, the bounds describe the representational benefit of memory capacity and the evidence required for reliable statement estimation. The pool experiment studies expansion using agent-learned content under a fixed upstream evidence budget. Section 4.2 and Appendix C.2 connect these two perspectives, and the following intervention examines how the quality of that content affects the scaling results.

Additional memory audit and repair. To examine content quality directly, we corrected mistaken or incomplete statements while preserving statement count, positions, and the exact duplication pattern. We reran original and repaired memories at L=10L=10 and L=150L=150, with 360 evaluation episodes per condition:

Memory lengthOriginal memory: V/A (%)Repaired memory: V/A (%)
L=10L=10123/620 (19.84%)165/667 (24.74%)
L=150L=150232/843 (27.52%)182/817 (22.28%)

Repair improves long memory in all three orderings. Short memory worsens, with the increase concentrated in SMS sign-off failures in one ordering even though its sign-off statement is unchanged (see our discussion with reviewer 41Dt's Q2). This pattern suggests an interaction between content refinement and memory organization. The intervention preserves statement count while revising sentence wording and token length.

In a post-hoc analysis excluding SMS tasks, longer memory retains a modest disadvantage under both original content (13.82% versus 11.14%) and repaired content (13.39% versus 11.52%). A separate analysis of expected task–rule pairs, excluding pre-identified task–preference conflicts, likewise gives 123/623 (19.74%) for repaired L=150L=150 and 94/504 (18.65%) for repaired L=10L=10. These results support the conclusion that expanding learned memory can reduce compliance even after content repair.

We agree that an empirical phenomenon can be influenced by multiple factors. Memory length, content quality, and even the order of memory can affect the final violation rate. These are all realistic factors, and each can be analyzed from different perspectives. However, we would like to highlight that our theoretical analysis focuses on the memory length, because the primary aim of this study is to analyze harness evolution from a learning perspective. Our enriched experiment further decouples the influence of other factors and still validates our main claim.

Appendix C.2 describes the repair intervention, reports the overall and non-SMS results in Table 5, and presents the separate conflict-filtered comparison in the following discussion.

AuthorsOpenReview ↗

rebuttal (part 3)

W4: Separating missing information from update failure

Following your suggestion, we repeated the comparison using the same ACE updater under both feedback conditions. The model, task streams, memory capacity, training budget of 24 interactions, evaluation tasks, and step limits are held fixed. Instance feedback identifies the local problem, while corrective feedback additionally supplies the relevant target specification. The comparison uses three task-order seeds, with 54 final evaluation episodes per condition, and contemporaneous no-memory and oracle references.

ConditionViolations/applicable pairsConditional violation rate
No memory77/8887.50%
ACE + instance feedback43/10242.16%
ACE + corrective feedback35/11330.97%
Oracle context8/1107.27%

Corrective feedback improves all three seeds and reduces the pooled violation rate by 11.18 percentage points, however, a distinct remaining gap still existis. All five correction templates have been received by the sixth training interaction at the latest. The comparison demonstrates the value of explicit preference information, isolate the optimization failure, and enables a closer diagnosis of the remaining gap to the oracle.

The audit identifies concrete residual failures. Twelve final evaluation episodes have an empty effective payment description, producing 24 category/initials violations, despite relevant instructions in memory. Eleven contain successful payments, one contains only failed attempts, scored under the original applicability definition. Two further violations use is_private instead of the effective private parameter. Nine privacy violations occur on tasks explicitly requesting public payment and are reported separately as task–preference conflicts. Removing that pre-identified conflict task leaves rates of 32.18%, 16.28%, and 2.17% for instance, corrective, and oracle respectively.

The matched-updater comparison isolates feedback informativeness, while the execution audit identifies concrete targets for further improvement. Section 4.3 introduces this matched-ACE comparison alongside the theoretical analysis of update dynamics. Appendix C.3 reports the results and residual-error cases.

W5: Applicability, task success, and joint outcomes

Since we mainly discuss the harness evolution in personal agents, we would like to reclaim our aim first. Our primary research object is preference compliance when a relevant action is attempted or performed. Conditional preference violation measures this aspect of personalization, while task completion and joint success provide complementary information about the resulting agent outcomes. We agree that applicability is benificial to interpret the conditional rate, and have audited the saved results accordingly.

The scoring unit is an episode–rule pair. For each rule rr, ArA_r counts applicable episodes and VrV_r counts those that violate the rule; repeated calls within one episode still yield one judgment per rule. We report Vr/ArV_r/A_r, with N/A for a zero denominator. Separately, ErE_r counts task-expected opportunities, and coverage measures how many of those expected pairs become applicable. Official task success uses the original AppWorld evaluator. Joint success requires task success, applicability of every task-expected rule, and no scored preference violations. These aggregate success rates use the fixed episode denominator NN.

Audited original Q3 outcomes (Table 7(a)). Dynamic methods are evaluated at n=24n=24, oracle and no-memory are static references. The four main update methods pool three saved seeds, each with six tasks and three rollouts per task. Counts are pooled, whereas the original main-method curves average seed-level rates. The table below reproduces the original-cohort outcomes in Table 7(a) and adds no-applicable-episode counts from the same audit.

MethodNV/A (%)Expected coverageNo applicable rule/NTask success/NJoint success/N
ACE5462/129 (48.06%)129/1352/544/540/54
TEPA5464/135 (47.41%)135/1350/547/540/54
TRACE5473/129 (56.59%)129/1352/541/540/54
Reflexion54104/117 (88.89%)117/1356/544/540/54
Full rewrite1831/45 (68.89%)45/450/181/180/18
Corrective full rewrite187/9 (77.78%)9/4515/180/180/18
Oracle context183/42 (7.14%)42/451/181/181/18
No memory1837/42 (88.10%)42/451/181/180/18

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their constructive feedback. We have added comparisons of preference descriptions, execution guidance, and runtime support (Appendix C.1, Table 4), memory auditing and semantic repair at fixed statement counts with sensitivity analyses (Appendix C.2, Table 5), and a matched-ACE feedback comparison with update and execution case studies (Appendix C.3, Table 6). We also report per-rule applicability, task success, and joint task–preference success for the original Q3 experiments (Table 7). Beyond these, we have also enriched our paper theoretically. Proposition B.6 in Appendix B.5 provides a concrete repair–corruption mechanism showing how persistent harmful updates can sustain an error gap despite ongoing correction. These additions strengthen the connection between our learning framework and empirical findings, clarifying the roles of execution knowledge, memory reliability, and preservation of correct content in personal-agent evolution. Section 4 incorporates the corresponding clarifications and references to these additions. We have revised our submitted manuscript, with revisions marked in blue.

AuthorsOpenReview ↗

Response After Rebuttal

Dear Area Chair and Reviewers,

Thank you once again for the time and effort you have devoted to our paper, and for your constructive feedback.

Several reviewers have expressed limited confidence and suggested further discussion or expert judgment. We respect this careful approach and would like to briefly revisit our motivation and contribution to support the AC’s decision.

Allow us to explain this motivation from our own perspective. Personal agents are advancing rapidly, with new concepts and products continually emerging. Much of this progress is driven by engineering intuition and experience, across skills, memory, harness design, self-evolution, and recursive self-improvement. Once a technique proves useful, it is incorporated into a system. We value this progress, but as researchers, we also feel compelled to ask: why?

Moving from successful practice to scientific understanding requires us to investigate why a method works, when it works, and where its limits lie. History offers many examples of engineering achievements motivating deeper scientific explanations. Steam engines, for instance, were already in practical use when Carnot investigated the principles and limits of heat-engine efficiency. His work helped lay the foundations of thermodynamics and provided principles that would guide subsequent engineering advances.

Our study is motivated by this same impulse to ask “why.” We seek a new perspective on phenomena that have become familiar in personal-agent research and everyday use, while their underlying mechanisms remain insufficiently understood. Why can an agent still overlook a preference even after we repeatedly emphasize it in the prompt, as examined in Q1? Why can expanding its memory with more statements about our requirements make its behavior worse, as examined in Q2? And why, despite the considerable enthusiasm surrounding self-evolution, does sustained and reliable improvement remain difficult to achieve in practice, as examined in Q3? The familiarity of these problems makes understanding them especially important.

To investigate these questions, we formulate harness evolution as a learning problem and analyze it through the classical perspectives of approximation, generalization, and optimization. This formulation yields theoretical results that help explain when and why these difficulties can arise. For us, the exciting part was finding a systematic way to reason about problems that had long been difficult to formulate precisely. Beyond the individual results, we believe that viewing harness evolution as learning offers the community a useful analytical perspective, one that can inspire further questions, experiments, and approaches to designing personal agents.

We recognize that this framework has limitations and leaves open questions. Nevertheless, we believe it is a promising starting point. A new framework rarely answers every question when it first appears. Scientific understanding develops through continued scrutiny, refinement, and discussion, and we hope our work can contribute to that process.

To be candid, we have felt some disappointment during the review process because this central motivation has received less attention and recognition than we had hoped. We are grateful for the reviewers’ detailed comments on the experimental settings and theoretical proof. These comments have helped us deepen the study and make the paper more rigorous. At the same time, we had also hoped for a broader discussion of the theoretical perspective itself, the analytical approach it enables, and the possibilities it opens for future research.

We understand how challenging such a discussion can be in the current research environment. AI research is moving rapidly, the volume of submissions is enormous, and new techniques and concepts appear every few weeks. Each of us brings our own perspective and trusted ways of doing research. Communicating a new viewpoint clearly within a short review cycle is therefore difficult. The same rapid pace also creates pressure to move on to the next topic before an existing line of inquiry has had time to develop fully, particularly when its value is not immediately recognized. This can make it harder to pursue a question deeply and patiently. Perhaps this is one of the tensions of our time. Even so, we hope that our community continues to make room for those who pause to ask one more “why.”

These reflections revisit the motivation behind our paper and necessarily include some of our personal feelings about the work. We share them in the hope that they help communicate more clearly the perspective we are trying to contribute, as the reviewers and AC consider their final assessment.

Thank you again for your thoughtful engagement and the considerable effort you have invested in the review process. Whatever the outcome, we wish you continued success in your work and a happy life.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.