acceptodds
Withdrawn from ICLR 2027

EvoSpace: Co-Evolving the Harness and Edit Space for Memory-Grounded Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The performance of LLM-based agents depends not only on their underlying models but also on the harnesses that orchestrate their interactions with the environment. Recently, researchers have focused on harness self-evolution, in which agents iteratively refine their harnesses using operational experience within a harness edit space. However, dynamically aligning this edit space with evolving adaptation needs remains a central challenge. A fixed harness edit space may preclude necessary updates, whereas allowing all permissible edits initially increases the complexity of selecting and combining updates. We introduce EvoSpace to address this challenge by jointly evolving the agent harness and its corresponding edit space. As the harness evolves, feedback-driven diagnosis guides EvoSpace to progressively expand the permissible updates across components, scopes, and operations. This enables changes beyond the initial space without exposing all combinations from the outset. Across two established benchmarks, EvoSpace improves over the compared memory systems. On LoCoMo, EvoSpace achieves average relative gains across F1 and BLEU-1 of 8.62% with qwen3.6-flash and 13.71% with GPT-5.1 over EvolveMem. From the experimental analysis, our key insight is that the harness edit space governs which adaptations are possible, making its dynamic alignment with evolving adaptation needs essential for effective self-evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, Self-evolving, Harness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KF38OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoSpace, a method that evolves both an LLM agent harness and the set of edits available to modify it. The harness consists of Memory, Retrieval, and Prompt components. Based on failures from previous rounds, EvoSpace decides whether to expand the current edit space and then applies edits to the harness. The method is evaluated on LoCoMo and MemBench with two LLM backbones and improves over the compared memory systems.

Critical strengths

The paper studies an interesting problem in agent self-improvement: the set of allowed harness edits may itself need to change as new failure modes are identified. The experiments cover two LLM backbones, evolution trajectories, ablations, case studies, and transfer experiments.

Critical weaknesses
  1. I am concerned that the main evaluation does not separate adaptation from evaluation. The same query pool is used to provide feedback during harness evolution and to report the final performance. This is particularly concerning for Memory repair, since repaired memories can be query-specific. The transfer results in Appendix B.4 are helpful, but they do not show performance on unseen queries from the same task after evolution. It is therefore difficult to tell how much of the reported improvement would generalize to new queries.
  2. The edit-space ablation does not fully isolate the effect of dynamic edit-space adaptation. In Table 3, the Scheduled setting opens Memory repair at a fixed round without a priority hint, while Dynamic both chooses when to expand the space and passes the newly admitted capability to the editor as a priority hint. Therefore, the gap between Scheduled and Dynamic may come from either adaptive admission or the priority hint.
Questions for authors

1.Have you evaluated the evolved harness on held-out queries that were not used during the evolution process?

2.Could you separate the effect of dynamic capability admission from the priority hint given to EDIT, for example by evaluating Dynamic without the hint or Scheduled with the hint?

3.Which LLM is used for DIAGNOSE, META-EVOLVE, and EDIT? I could not find this clearly specified in the paper.

Reviewer_QDMfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EvoSpace, a framework for iterative harness optimization in memory-grounded question answering. EvoSpace maintains a predefined library of editing capabilities across Memory, Retrieval, and Prompt components. It progressively expands the subset of available operations based on failure diagnosis. This allows the harness to evolve within an adaptively expanding edit space over successive rounds. The authors evaluate EvoSpace on LoCoMo and MemBench using two LLM backbones and report improvements over existing memory systems. The evolution trajectories show substantial performance gains in later rounds, particularly after introducing Memory repair.

Critical strengths
  1. Clear formulation and detailed experimental setup. The paper presents a well-structured mathematical formulation that clearly distinguishes the agent harness from its available edit space. The appendix also provides relatively detailed descriptions of the experimental setup, implementation choices, and evolution protocol, which help readers understand the proposed framework and its evaluation.

  2. Useful ablation studies. The paper includes ablation studies comparing different edit-space configurations, including restricted, fully available, scheduled, and dynamically expanded spaces. These comparisons, along with component-level ablations, help assess which aspects of EvoSpace contribute to the reported performance improvements.

Critical weaknesses
  1. Potential evaluation leakage due to the harness evolution protocol. A major concern is that harness evolution appears to be performed on the same evaluation set used to report final performance. (Appendix D, line 1225: The same query pool supplies adaptation feedback and reported scores.) Unlike test-time scaling approaches that rely on independently generated feedback, such as agent discussions or reward models, EvoSpace uses correctness scores computed against gold answers to guide failure diagnosis and harness selection. Moreover, once Memory repair is enabled, the system uses previously failed questions and their original conversation histories to construct query-specific memory entries, and subsequently evaluates the updated harness on the same questions. Although the diagnosis agent does not directly receive gold answers, correctness feedback derived from the gold answer still influences the adaptation process. This raises substantial concerns about evaluation leakage and the validity of the reported performance improvements. The authors should clarify whether this understanding of the evaluation protocol is correct.

  2. Unclear methodological novelty and insufficient isolation of the proposed mechanism. The distinction between EvoSpace and existing self-evolving memory approaches, particularly EvolveMem, is not sufficiently clear. My understanding is that the primary contribution is progressively expanding an edit space based on failure diagnosis. However, all editing capabilities are predefined, and the proposed mechanism primarily controls when these capabilities become available. This raises questions about its methodological novelty beyond adaptive scheduling of existing editing operations.

    Furthermore, the experiments do not sufficiently disentangle the contribution of edit-space evolution from other factors. In particular, I would appreciate the following controlled comparisons:

    • Held-out evaluation: Assess the evolved harness on questions excluded from failure diagnosis, editing, and candidate selection to establish whether the improvements generalize beyond the adaptation queries.

    • Without Memory repair: Evaluate the performance improvements attributable to Retrieval and Prompt evolution alone, separating them from gains introduced by query-specific Memory repair.

Questions for authors

Please see the Weaknesses section for my questions and requests for clarification, particularly regarding the evaluation protocol and the contribution of the proposed edit-space evolution mechanism.

Reviewer_XQw3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EvoSpace, which treats the edit space in LLM agent harness self-evolution as an evolving state rather than a fixed design choice. Targeting long-term memory-grounded QA, it decomposes the harness into three components: Memory, Retrieval, and Prompt. In each round, a diagnosis agent analyzes failure traces to identify the responsible component. Based on this diagnosis, the system either opens a new capability from a preimplemented registry (expand) or retains the current space (keep). An edit agent then generates concrete modifications with the available capabilities, and the modified harness is retained or rolled back based on overall performance. On LoCoMo and MemBench with two backbones (qwen3.6-flash, GPT-5.1), the paper reports higher performance than memory systems including EvolveMem, along with edit-space and component ablations.

Critical strengths
  • The motivation of treating the edit space as an explicit state is clear, and the formulation cleanly separates opening a capability (admission) from executing a modification (operation).
  • Bundling modifications across components into a single candidate and evaluating it on whole-harness performance is a reasonable choice, since component-level improvements do not necessarily lead to better overall performance.
  • The paper reports consistent improvements across two benchmarks and two backbones.
  • The appendix details per-seed trajectories, round-by-round edits with adoption/rollback decisions, and case studies, making the evolution process easy to trace.
  • The component ablation shows that the contributions of Memory, Retrieval, and Prompt updates are not independent.
Critical weaknesses

W1. The problem formulation and the compared methods do not match the paper's claims.

The paper presents "a fixed, restricted edit space precludes necessary updates" as its central problem. However, EvoSpace's edit space is closed within a preimplemented capability registry O\mathcal{O}. "Expansion" does not create new modifications but unlocks one capability from O\mathcal{O}, and even Efull\mathcal{E}_{\text{full}} is merely the whole of O\mathcal{O}. In contrast, Meta-Harness, which the paper discusses in its related work, searches over harness implementations themselves spanning prompts, retrieval, and state updates, and thus covers a broader range of expressible modifications. This problem formulation therefore applies to config-search methods but not to code-level harness search. The remaining claim is that "an overly large space is hard to search under a limited budget," yet there is no comparison that tests this.

Moreover, while the paper claims a harness self-evolution methodology, its experiments compare only against memory systems (MemGPT, Mem0, A-Mem, MemBank, RecentMem, EvolveMem). There is no comparison with harness/prompt optimization methods such as Meta-Harness, RobustSGPO, and GEPA/TextGrad, or with self-evolving memory methods such as MemEvolve and MemSkill. The full list of capabilities in registry O\mathcal{O} is also not provided, making it impossible to judge what expansion actually means.

W2. The effect of the core contribution, diagnosis-guided adaptation, is not isolated.

The gain over EvolveMem conflates two factors, diagnosis-guided admission and the addition of the Memory repair capability, so it is unclear which one drives the performance. The paper's own results suggest that most of the gain comes from Memory repair.

  • The only observed admission is Memory repair, and according to Tables 5 and 6, it occurs at R5 in every seed on both datasets. As a result, it is indistinguishable from a fixed schedule.
  • The Scheduled setting, which opens Memory repair at R5 without diagnosis, accounts for a substantial portion of the gain over Limited. The remaining difference from Dynamic comes from the priority hint passed to EDIT rather than from edit-space adaptation, and since no variance is reported for the ablations, its significance is also unclear.

W3. The same questions are used for optimization and for reporting.

According to Appendix D, "the same query pool supplies adaptation feedback and reported scores." That is, the harness is modified over 7 rounds on the same question set, and the score of HbestH_{\text{best}}, the highest-scoring harness on those questions, is reported in Tables 1 and 2. Although the reference answer text is not given to diagnosis, the selection of questions to repair (prioritizing F1 = 0), the acceptance/rollback of candidates, and the selection of the final harness are all based on scores against gold answers. The reported numbers therefore cannot be interpreted as held-out performance.

Evolving on the same task is itself reasonable given the purpose of self-evolution, but performance should be measured on new questions from the same task. In addition, comparing against static memory systems that were run once without any answer signal is unfair because the two have different access to information.

W4. The premises of Memory repair are unrealistic, and there is no corresponding deployment scenario.

Memory repair (i) knows the evaluation questions in advance, (ii) knows from gold answers which questions were answered incorrectly, (iii) re-accesses the entire original conversation, and (iv) adds memories that are retrievable only for those questions. Since future questions and their answers are unknown in real deployment, it is unclear in what practical situation this mechanism would help. By construction, the gains from this stage do not transfer to new questions at all.

Yet a substantial portion of the reported improvement comes from this stage. Furthermore, because this process rereads the original conversation with knowledge of the question, it bypasses the very premise of memory systems,

Questions for authors
  1. Are there results comparing against harness optimization methods such as Meta-Harness under the same conditions?
  2. Can the authors show that the performance gain comes from diagnosis deciding when and what to open, rather than from simply having Memory repair available? For example, are there results comparing against opening capabilities randomly instead of by diagnosis, or opening them at different rounds?
  3. What is a realistic scenario in which Memory repair would be useful?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.