acceptodds
Under review as a conference paper at ICLR 2027

HoMe: Toward Practical Harness Optimization of Memory Systems for AI Assistants

Abstract

The behavior of memory systems for long-running AI assistants is governed by a harness, which consists of the code, prompts, and configurations for constructing, maintaining, and retrieving memory. However, optimizing the harness of a memory system remains largely manual, and existing automated methods restrict either the editable components or the search procedure. In this paper, we introduce HoMe, a pure evaluation-driven framework that iteratively evaluates candidate harnesses and derives each new candidate from any previously evaluated harness by editing any part of it. HoMe combines session-level memory versioning with execution traces to localize failures to the responsible sessions and harness components. It further reduces evaluation cost through smoke tests, candidate gating, adaptive sampling, and the reuse of constructed memory. We validate HoMe with the open-source memory system on two large-scale long-horizon memory benchmarks, LongMemEval-S and BEAM. With only six evaluated candidates per benchmark, HoMe discovers harnesses that improve end-task performance over the originally strong harness while consuming far fewer query-time tokens.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
personal memory, self-evolution, long-horizon evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_16R6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HoMe, an evaluation-driven framework for harness optimization. It combines session-level memory versioning and failure diagnosis with candidate gating, adaptive sampling, and memory reuse.

Critical strengths
  • The paper tries to address a relevant practical challenge: automating the optimization of long-term memory systems.

  • Experiments show that the optimized harnesses improve over the manually tuned ReMe baseline on both benchmarks.

Critical weaknesses
  • Comparisons with the most relevant optimization methods are missing. The introduction discusses Meta-Harness, EvolveMem, and MemPro as closely related work, but none is included in the experimental comparisons. The evaluated baselines (A-MEM, SimpleMem, and ReMe) are memory systems rather than harness optimization methods. These comparisons are insufficient to show HoMe’s effectiveness relative to existing optimization approaches.

  • The ablation study is incomplete. Table 3 shows evaluation-time reductions, but the contributions of key components to final performance remain unclear. In particular, session-level versioning, trajectory-guided attribution, and candidate gating are not individually evaluated.

  • The contribution seems incremental. HoMe largely follows the optimization paradigm of Meta-Harness, with additional mechanisms for memory versioning, diagnosis, and efficient evaluation. While potentially useful, these additions are not sufficiently justified as a substantial methodological advance.

Questions for authors
  • Can Figure 1 be improved? The image resolution is low, some text and labels are unclear, and the overall presentation is not clear enough. I recommend revising the figure to improve its clarity and readability.
4 comments
AuthorsOpenReview ↗

Comparison with MemPro: Experimental Alignment, Main Results, and Cross-Benchmark Transfer

W1

Comparisons with the most relevant optimization methods are missing. The introduction discusses Meta-Harness, EvolveMem, and MemPro as closely related work, but none is included in the experimental comparisons. The evaluated baselines (A-MEM, SimpleMem, and ReMe) are memory systems rather than harness optimization methods. These comparisons are insufficient to show HoMe’s effectiveness relative to existing optimization approaches.

Response.

We agree and have added MemPro, an automated memory optimizer, to the main and transfer experiments (Tables 1 and 2). Both methods start from the same tuned ReMe harness and share the runtime, editable plugin scope, coding-agent model, AgentScope CLI, construction and answering models, and scoring protocol. We optimize on training data only and measure final performance on independent test data.

MemPro uses fixed evaluation data, while HoMe dynamically samples using evaluation feedback. Both can access the complete BEAM-100K training set: MemPro evaluates it every round, whereas HoMe samples cases and queries after an initial full-training evaluation. On LongMemEval-S, full evaluation every round is prohibitively expensive. MemPro uses a subset sampled once, while HoMe dynamically samples after its initial full-training evaluation; cumulative formal case-evaluation counts are approximately matched.

Main experiments show higher answer quality and lower query cost on both benchmarks. On LongMemEval-S, HoMe improves the score from MemPro's 0.812 to 0.916 (+10.4 percentage points) and reduces average query tokens from 96.9K to 17.9K (−81.5%), using less than one-fifth as many tokens. On BEAM-1M, the score rises from 0.693 to 0.704 (+1.1 percentage points), while query tokens decrease from 65.4K to 60.4K (−7.6%). HoMe thus improves quality and efficiency even with million-token histories and sampled feedback, despite MemPro evaluating the complete BEAM training set every round. Its larger advantage over fixed-subset evaluation on LongMemEval-S supports the value of dynamic sampling in covering more difficult cases at lower cost.

Transfer experiments retain these advantages across benchmarks. BEAM-100K → LongMemEval-S yields scores of 0.908 for HoMe versus 0.800 for MemPro (+10.8 percentage points), with average query tokens of 26.6K versus 42.2K. LongMemEval-S → BEAM-1M yields scores of 0.639 versus 0.617 (+2.2 percentage points), with 39.0K versus 113.1K query tokens. Both directions preserve higher quality and lower query costs; in the latter, HoMe uses approximately one-third of MemPro's query tokens.

AuthorsOpenReview ↗

Methodological Contributions: Diagnosis, Candidate Gating, Adaptive Evaluation, and Search Efficiency

W3

The contribution seems incremental. HoMe largely follows the optimization paradigm of Meta-Harness, with additional mechanisms for memory versioning, diagnosis, and efficient evaluation. While potentially useful, these additions are not sufficiently justified as a substantial methodological advance.

Response.

Meta-Harness proposes a general iterative framework: read historical code, execution traces, and scores → diagnose and revise the harness → evaluate and record. Its retrieval experiment uses 40 iterations and produces 109 candidates. Applying this search scale directly to long-history memory optimization would additionally require replaying interaction histories, constructing memory, and answering many questions each round, making evaluation unaffordable under our budget (Meta-Harness §3 and §4.2). MemPro applies this type of evaluate-and-edit framework to memory optimization, allowing a single Evolving Agent to revise the entire harness and reducing costs through fewer iterations, static sampling, and construction reuse (MemPro §4). In practice, however, it still faces difficulties tracing errors to code, a preference for simple edits, and limited feedback coverage from fixed evaluation sets. HoMe redesigns diagnosis, candidate generation, and evaluation to address these problems.

  1. Tracing long-horizon errors to code mechanisms requires separating diagnosis from revision. Case investigation involves extensive sessions, memory, and retrieval records that can fill a coding agent's context, leaving it focused on superficial patches for individual errors. MemPro assigns investigation, attribution, and revision to the same agent. HoMe combines trajectory-guided attribution with parallel case diagnosis: Case Study Agents trace individual failures to harness defects, and the Diagnoser synthesizes shared causes into compact reports for the Optimizer, preserving its context for structural design. It also stores session-level checkpoints in Git and provides diff comparisons, reducing the cost of reconstructing memory changes from lengthy logs. Appendix A.5 shows how this design supports implementation of runtime evidence auditing and feedback-driven repair.

  2. Code-editing permissions alone can still lead to local prompt patching, requiring explicit guidance. In our MemPro reproductions on BEAM-100K and LongMemEval-S, all candidates modify only prompts or retrieval parameters, despite permission to edit the entire harness (Appendix A.6, Tables 10 and 11). The Gate Agent accepts only code-level mechanism changes in the first half of the budget. Prompt- or parameter-only tuning becomes eligible in the second half. Search first explores memory organization, retrieval, and answering through larger changes, then refines promising designs.

  3. Full evaluation every round is too expensive, while a small fixed sample repeatedly focuses search on the same cases, requiring dynamically refreshed feedback. MemPro evaluates either the full training set or a fixed subset sampled once. HoMe dynamically samples cases and queries using historical failures, low scores, uncertainty, and the number of rounds since the last evaluation, while retaining uniform exploration. The limited budget continually covers new difficult cases and tests repair hypotheses. For construction reuse, HoMe uses an independent equivalence review and retains a probability of reconstructing memory, reducing dependence on a single stochastic construction result.

These designs enable mechanism search at lower total cost. On BEAM-100K, HoMe uses 185.2M search tokens, 44.1% fewer than MemPro's 331.2M (Appendix A.8, Tables 12 and 13). Independent diagnosis and review increase developer-agent costs, but dynamic sampling and construction reuse reduce the dominant construction and query workloads by 45.4% and 59.0%, respectively.

Compared with MemPro, HoMe improves test scores by 10.4 percentage points on LongMemEval-S and 1.1 points on BEAM-1M while reducing query tokens by 81.5% and 7.6%, respectively (Table 1). Cross-benchmark transfer preserves these gains, improving scores by 10.8 and 2.2 points in the two directions while using fewer query tokens in both (Table 2).

AuthorsOpenReview ↗

Why the Three Components Work Together

W2

The ablation study is incomplete. Table 3 shows evaluation-time reductions, but the contributions of key components to final performance remain unclear. In particular, session-level versioning, trajectory-guided attribution, and candidate gating are not individually evaluated.

Response.

End-to-end leave-one-out ablations would require three additional full optimization runs. The reported BEAM-100K search used 185.2M tokens and took 10 hours 26 minutes. The three components form a sequential repair process, with each stage supplying evidence or guidance for the next.

Session-level versioning grounds fault localization in the history of the memory state. The same final-answer error can arise when information was never written, when a later session overwrote valid state, or when retrieval missed information that remained intact. Agents use checkpoints and session-level diffs to distinguish these cases by inspecting when information was written, what later sessions changed, and whether the relevant evidence was still available at retrieval time.

Trajectory-guided attribution traces failures to the responsible harness components. Structural repair also requires identifying defects shared across cases. HoMe supports this through parallel case diagnosis. Case Study Agents investigate individual failures, and the Diagnoser synthesizes their findings into a compact report for the Optimizer. This division keeps detailed case investigations out of the Optimizer's context while preserving the evidence it needs to reason about shared problems and design code-level repairs.

Candidate gating directs the search toward structural changes that address diagnosed causes. The Gate reserves early evaluations for code-level mechanism changes, so that proposed fixes to memory construction, indexing, retrieval, or answering receive evaluation budget. This makes it possible to test whether a structural revision resolves the shared failure identified during diagnosis.

Each stage addresses a different failure mode in the repair process. Removing any one changes what the remaining stages can accomplish.

  1. Without session-level versioning, reliable attribution becomes more expensive. Even a dedicated Diagnoser must reconstruct state transitions from raw multi-session construction logs. Deep inspection then becomes token-intensive, making agents more likely to rely on final answers alone. The same wrong answer can therefore send the agent to the wrong part of the harness even when diagnostic and revision roles are separate.

  2. Without trajectory-guided attribution and parallel case diagnosis, structural revisions lose a clear target. Checkpoints show how memory changed. Connecting those changes to shared code defects requires tracing failures and comparing cases. If the coding agent also performs detailed case investigations, case details can crowd out the context needed for cross-case reasoning. The Gate can still require a substantial code change, but the resulting edit may be weakly grounded in the observed failures or overfit to an isolated case.

  3. Without candidate gating, diagnosed causes may go untested. Even with an accurate cross-case diagnosis, a developer agent can spend early evaluations on low-risk, case-specific prompt patches. Those patches may address individual failures while leaving the underlying construction, indexing, retrieval, or answering mechanism unchanged.

The system-level comparison is with MemPro, which runs without HoMe's session-level versioning, trajectory-guided attribution and parallel case diagnosis, or candidate gating. HoMe scores 0.704 on BEAM-1M versus 0.693 for MemPro and uses 44.1% fewer optimization tokens. The contribution of each component to final performance remains unmeasured.

AuthorsOpenReview ↗

Revise Figure

Q1

Can Figure 1 be improved? The image resolution is low, some text and labels are unclear, and the overall presentation is not clear enough. I recommend revising the figure to improve its clarity and readability.

Response.

We redrew and replaced Figure 1 to make its text, labels, and workflow clearer. The figure shows HoMe's three components: Developer Agents diagnose failures and revise the harness; the Experiment Manager manages candidates and results, controls tool access and evaluation; and the Evaluation Environment evaluates candidates in isolated sandboxes. Each iteration follows diagnosis and revision → candidate review and evaluation planning → sandbox evaluation → feedback, allowing subsequent changes to build on previous evaluation and diagnostic results.

Reviewer_TBxXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

HoMe adapts agentic harness optimization to file-backed memory. It uses developer agents to inspect evaluation failures and revise a memory system's code, prompts, and configuration. The framework uses Git to checkpoint memory after each session to support diagnosis of missing, overwritten, or unretrieved information, and combines candidate gating, adaptive evaluation sampling, and construction-aware memory reuse to reduce search cost. Applied to ReMe with six revisions per benchmark, it reports improvements of 2-5% on 2 benchmarks, with approximately ~50% fewer query-stage tokens, also showing longer context transfer of optimizations done on smaller scale datasets.

Critical strengths
  1. A good piece of engineering work which also yields imporvements on a strong baseline. The cross transfer from 100k to 1M scale is particularly helpful than optimized in-domain performance.
  2. Maintaining memory snapshots in session history and git backed harness optimizations and evolution is a good principled way of optimizing harness for any domain.
Critical weaknesses
  1. Novelty distinction concern: Meta-Harness delegates diagnosis and code revision to an agent and explicitly states that it “imposes no parent-selection rule” in its section 3, contradicting HoMe’s characterization in lines 189–190. MemPro’s expansion prompt also specifies executable-code edits, debugging with memory states and traces, and reuse of memory when construction is unchanged in its figure 13. The filesystem-memory study cited in this paper also records per-task/per-chunk memory snapshots and reconstructs their evolution in its section 4.4. This establishes prior snapshot analysis, not the whole HoMe’s complete automated repair loop. So, the contribution to clarify is how session-level diffs guide harness edits, and what HoMe’s evaluation policy adds to these existing mechanisms.
  2. Table 1 compares HoMe’s resulting harness with A-MEM, SimpleMem, and manually tuned ReMe, but includes no competing automated harness-search method. It therefore establishes system-level gains without establishing an advantage over other optimizers. The case studies also provide no concrete overwrite episode linking a session diff to a harness repair. Table 3 measures evaluation speedups, but does not show whether the cheaper evaluation preserves the quality of the selected harness. So,these omissions leave the source of HoMe’s gains unclear...
  3. Across the six BEAM candidates, sampled and full-training scores have r = −0.083 in section 4.2.2 and the paper appropriately treats sampled scores as diagnostic. However, section 3.3 leaves final selection to the optimizer’s judgment under an unspecified token constraint. Table 7 selects B4 although B5 has a higher sampled score. Because the subsets differ, this choice is not inherently wrong, but its rationale and the evidence available to the selecting agent need to be specified..
Questions for authors
  1. Can you compare with an existing optimizer and ablate session-history access, using the same ReMe initialization, models, and total compute? Please include one concrete diff-guided repair.
  2. Please provide the final-selection prompt/rule and token constraint details.
  3. Can you report variability across repeated searches, paired test confidence intervals, and total optimization/construction costs including rejected candidates and smoke tests?
4 comments
AuthorsOpenReview ↗

Methodological Differences in Diagnosis, Candidate Gating, and Evaluation

W1

Novelty distinction concern: Meta-Harness delegates diagnosis and code revision to an agent and explicitly states that it “imposes no parent-selection rule” in its section 3, contradicting HoMe’s characterization in lines 189–190. MemPro’s expansion prompt also specifies executable-code edits, debugging with memory states and traces, and reuse of memory when construction is unchanged in its figure 13. The filesystem-memory study cited in this paper also records per-task/per-chunk memory snapshots and reconstructs their evolution in its section 4.4. This establishes prior snapshot analysis, not the whole HoMe’s complete automated repair loop. So, the contribution to clarify is how session-level diffs guide harness edits, and what HoMe’s evaluation policy adds to these existing mechanisms.

Response.

Meta-Harness proposes a general framework that iteratively diagnoses failures, generates harness candidates, and evaluates them, with a developer agent autonomously inspecting historical code and execution traces. Its retrieval experiment uses 40 iterations and produces 109 candidates. In long-horizon memory tasks, each evaluation must also process lengthy interaction histories and construct memory, making this search scale prohibitively expensive (Meta-Harness §3 and §4.2). MemPro applies this type of system-level iterative search to memory harness optimization, using fewer iterations, static sampling, and memory reuse when construction logic is unchanged to lower the cost. It permits edits to code, prompts, and parameters, with the same Evolving Agent investigating errors and revising the harness (MemPro §4).

HoMe is orthogonal to Meta-Harness's general search framework, introducing mechanisms tailored to diagnosis and search costs in long-horizon memory harness optimization. Furthermore, it differs from MemPro in three methodological aspects.

First, trajectory-guided attribution and parallel case diagnosis guide code revision. MemPro uses the same agent to investigate errors and implement changes. Long-history case studies generate substantial context, making it easy to remain at superficial patches for a few cases rather than tracing errors to code mechanisms. HoMe uses trajectory-guided attribution to trace failures to the responsible harness components. In parallel case diagnosis, Case Study Agents investigate individual failures, and the Diagnoser synthesizes shared causes into compact reports for the Optimizer. This leaves the Optimizer room to design structural changes, while parallel investigation also reduces elapsed time. We additionally manage session-level memory history with Git, supporting checkpoint inspection and session-level diff comparisons. Diagnostic agents can compare memory before and after relevant sessions to distinguish information never written, information subsequently changed, and information retained but not retrieved, then inspect the corresponding construction or retrieval logic. This integrates historical state comparison into diagnosis and code revision, reducing the cost of reconstructing memory evolution from lengthy construction logs.

Second, candidate generation uses a Gate Agent to reserve budget for mechanism exploration. Although MemPro permits unrestricted editing, the lack of constraints makes search susceptible to local optima, repeatedly adding prompt patches for failed cases. The Gate Agent accepts only code-level mechanism changes in the first half of the budget. Prompt- or parameter-only tuning becomes eligible in the second half. Search thus first takes larger steps across memory organization, retrieval, and answering mechanisms, then smaller steps to refine promising designs.

Third, evaluation uses dynamic sampling to continually refresh diagnostic feedback. MemPro evaluates either the full training set or the same static subset each round. The former is expensive; the latter repeatedly directs search toward a limited set of cases. HoMe dynamically adjusts case and query sampling priorities based on historical failures, low scores, uncertainty, and the number of rounds since the last evaluation, while retaining uniform exploration. This continually exposes new difficult cases and tests repair hypotheses at lower cost. For construction reuse, HoMe uses an independent equivalence review and retains opportunities to reconstruct memory, avoiding persistent reliance on a single stochastic construction result.

We have added MemPro as a new baseline and discussed the results in the following response.

AuthorsOpenReview ↗

Optimizer Comparison, Repair, and Sampling

W2 / Q1

Response.

Comparison with an automated optimizer and sources of improvement. The revision directly compares HoMe with MemPro. Both start from the same ReMe harness and use the same models, AgentScope CLI, runtime, editable scope, and scoring protocol. We approximately match cumulative case evaluations to align the main workload: construction, answering, and scoring account for about 80% of HoMe’s optimization tokens. On LongMemEval-S, MemPro performs eight formal evaluations on the same 50 training cases, while HoMe dynamically samples 28 cases per round after an initial full-training evaluation, yielding approximately equal cumulative case-evaluation counts. Both methods can access the complete BEAM-100K training set: MemPro evaluates all 20 cases and 400 queries each round, whereas HoMe samples 100 queries from 10 cases per round after its initial full-training evaluation, using substantially less computation.

In the main experiments, HoMe improves the LongMemEval-S test score from MemPro's 0.812 to 0.916 while reducing average query tokens from 96.9K to 17.9K, an 81.5% reduction. On BEAM-1M, the score increases from 0.693 to 0.704, while query tokens decrease from 65.4K to 60.4K, a 7.6% reduction (Table 1). Relative to the initial ReMe harness, HoMe also improves scores by 2.8 and 5.4 percentage points on the two benchmarks. Cross-benchmark transfer retains these advantages: BEAM-100K → LongMemEval-S yields scores of 0.908 versus 0.800 and query tokens of 26.6K versus 42.2K; LongMemEval-S → BEAM-1M yields scores of 0.639 versus 0.617 and query tokens of 39.0K versus 113.1K (Table 2).

The detailed appendix results explain differences in search behavior. All MemPro candidates across BEAM-100K and LongMemEval-S modify only prompts or retrieval parameters, without introducing new code-level construction or retrieval mechanisms (Appendix A.6, Tables 10 and 11). HoMe's execution trace instead shows trajectory-guided attribution and parallel case diagnosis, followed by the Optimizer implementing runtime evidence auditing and repairing code using validation feedback (Appendix A.5, Table 9). In this detailed trace, the repaired candidate improves from the initial 0.6120 to 0.6760 while reducing query tokens by 11.7%. Together with the action-indexing and source-aware retrieval examples, these records show how candidate gating, trajectory-guided attribution, and parallel case diagnosis move search toward mechanism changes that address shared problems across cases.

Mechanism attribution and repair localization using memory history and session-level diffs. We provide a case study showing how session-level memory history and diffs guide diagnosis and harness repair.

ExampleActual diff investigation and findingsAttribution and repair direction
1Inspection of the checkpoint where the update was written and subsequent checkpoints shows that the April 5 update was saved and retained in the final memory, with both old and new dates still present within the notes.Confirms that the update was preserved and directs investigation toward how retrieval and answering handle old and new information.
2Comparing 576a7c2f → 5543507 reveals a new note containing "$50 increased January budget." Comparing the next two checkpoints confirms that the note was neither changed nor deleted. Inspection of the query trace then shows that both top-5 search windows were occupied by transcripts repeating the old $35 value, excluding the $50 note.The update was written and remained available; attribution shifts to competition between notes and raw transcripts within the retrieval window.

Example 2 redirects investigation from memory writing and merging to retrieval competition. A returned passage also mentions $50 despite the $35 answer, prompting further investigation of temporal and scope handling during answering.

Although agents can inspect memory evolution through raw construction logs, doing so is costly. Git-managed session-level memory history and diff comparisons allow targeted inspection of memory changes, reducing the effort of reconstructing states step by step from lengthy construction logs and improving attribution efficiency.

Benefits of the sampling strategy. HoMe uses dynamic sampling on both benchmarks, while MemPro repeatedly evaluates the same static subset on LongMemEval-S and the full training set each round on BEAM-100K. HoMe's score improvement over MemPro is 10.4 percentage points on LongMemEval-S, substantially larger than the 1.1-point improvement on BEAM-1M; query token reductions are 81.5% and 7.6%, respectively. This difference is consistent with dynamic sampling continually introducing new cases and expanding diagnostic coverage, supporting its value when a fixed subset limits feedback coverage: a limited evaluation budget can expose more difficult problems and guide more generalizable changes.

AuthorsOpenReview ↗

Final Selection: Score-First Instructions, Comparison Tables, and the Choice of B4

W3 / Q2

W3

Across the six BEAM candidates, sampled and full-training scores have r = −0.083 in section 4.2.2 and the paper appropriately treats sampled scores as diagnostic. However, section 3.3 leaves final selection to the optimizer’s judgment under an unspecified token constraint. Table 7 selects B4 although B5 has a higher sampled score. Because the subsets differ, this choice is not inherently wrong, but its rationale and the evidence available to the selecting agent need to be specified..

Q2

Please provide the final-selection prompt/rule and token constraint details.

Response.

Final selection chooses the harness to deliver from candidates that have already been formally evaluated. The principle is score first, query tokens second: prefer higher answer quality and, when quality is similar, fewer tokens, without a weighted score–token objective or trade-off. Cases and questions differ across evaluation rounds, so aggregate scores cannot directly rank candidates. By comparison, average query token consumption on sampled subsets is strongly correlated with that on the complete training set (r = 0.870), making sampled tokens a useful reference for query-cost comparison. Hence, the main selection challenge is estimating overall answer quality. To control selection costs, HoMe does not re-evaluate all candidates on a common evaluation set or the complete training set. Instead, the Optimizer selects using evaluation results, diagnostic reports, and candidate changes recorded during the search. HoMe automatically generates paired comparison tables for shared cases and questions, summarizing score differences, the numbers of improvements and regressions, and token consumption. The Optimizer uses these statistics to compare performance on identical questions and the diagnoses to assess whether a change resolves the original problem. When necessary, it can directly compare any two historical candidates before selecting the version best supported by the evidence. The relevant system-prompt instructions are:

Score improvement is more important than query token reduction. Compare scores on common cases and queries, weigh coverage, difficulty, regressions and lineage, and do not rank sampled candidates by aggregate score alone. Prefer quality, stability, query cost and simplicity. When exhausted, select the best existing evaluated branch without another formal validation.

The complete prompts have been provided in Appendix A.9.

For example, B4 and B5 score 0.670 and 0.715 on different subsets, so B5’s higher aggregate score does not establish better overall answer quality. B4 restricts the topic index to coverage questions such as summarization and counting and consolidates duplicate notes, preserving fine-grained ordering evidence. B5 adds a chronological source-path map, but shows substantial score declines on questions shared with B4, raising concerns about overfitting. The Optimizer therefore selects B4 for its stronger answer-quality evidence on directly comparable questions.

AuthorsOpenReview ↗

Search Stability, Paired Test Confidence Intervals, and End-to-End Optimization Costs

Q3

Can you report variability across repeated searches, paired test confidence intervals, and total optimization/construction costs including rejected candidates and smoke tests?

Response.

Stability. Due to the high cost of a full end-to-end optimization trajectory (185.2M tokens for the reported BEAM run), we performed two independent BEAM searches starting from the same initial ReMe harness, hyperparameters, and formal evaluation budget, evaluating the harness selected by each search on the same BEAM-1M test set. The two scores are 0.704 and 0.720, a difference of 1.6 percentage points and a mean of 0.712; average query token consumption is 60.4K and 68.3K. Relative to the initial ReMe score of 0.650, the two runs improve by 5.4 and 7.0 percentage points, respectively, while both use fewer query tokens than ReMe's 101.8K. Both runs lie on the empirical Pareto frontier of the compared methods (Appendix A.4, Figure 4).

The stability of the performance gain comes from consistency in diagnoses and modification directions during optimization (Table 8). Both searches independently identify several shared defects in the initial harness: transcripts crowding notes out of retrieval, excessive search output, insufficient cross-session evidence coverage, and loss of speaker attribution or update relations during construction. They then propose corresponding mechanisms such as source-balanced retrieval, evidence compression, chronological indexes, and explicit preservation of attribution and update relations. The implementations address the same underlying problems through different means: query-guided line selection and session compression both reduce excessive search output, while source-path maps and monthly timelines both support reading evidence in chronological order.

Paired test confidence intervals. Paired bootstrap resampling over identical BEAM-1M test queries shows statistically significant improvements over the initial ReMe harness for both Run 1 (+0.054, 95% CI: [+0.028, +0.082], p=0.0002p = 0.0002) and Run 2 (+0.070, 95% CI: [+0.042, +0.090], p<0.0001p < 0.0001).

Optimization cost. Appendix A.8 (Tables 12 and 13) reports token consumption for the BEAM-100K optimization runs: 185.2M for HoMe versus 331.2M for MemPro, a 44.1% reduction. HoMe's total covers smoke tests, debugging, all developer-agent activity, and evaluation (memory construction, answering, and scoring), including work on rejected candidates. The MemPro total excludes its smoke tests. Memory construction decreases from 162.6M to 88.8M, and query evaluation from 144.0M to 59.1M. Sampling and construction reuse reduce the dominant evaluation workload, yielding a substantial overall reduction even as developer-agent tokens increase from 24.7M to 37.4M. Counts include input and output tokens without discounting cache hits. HoMe's end-to-end wall-clock times are 13 hours 5 minutes on LongMemEval-S and 10 hours 26 minutes on BEAM.

Reviewer_9tBUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HoMe uses coding agents to revise memory-system code, prompts, and configurations, with session-level versioning for diagnosis and cheaper candidate evaluation. On LongMemEval-S and BEAM-1M, the optimized ReMe harness improves answer scores while reducing query-time tokens.

Critical strengths
  1. The gains come on top of a tuned ReMe baseline: scores improve from 88.8% to 91.6% and from 65.0% to 70.4%, with 59% and 41% fewer query-time tokens (Table 1).
  2. Session-level memory diffs provide a useful way to distinguish information lost during construction from retrieval or answering failures (Section 3.2).
  3. The BEAM-100K harness transfers to BEAM-1M and also improves LongMemEval-S, providing evidence beyond the optimization setting (Table 2).
Critical weaknesses
  1. The baselines are memory systems, not harness optimizers. Without a budget-matched optimization baseline or component ablations of answer quality, the benefit of HoMe over a simpler coding-agent loop remains unclear.
  2. Final selection relies on optimizer judgment, but its reliability is unclear. Across six candidates in the reported BEAM run, sampled and full-training answer scores have r = -0.083; the samples are intended for diagnosis, not ranking. Only one search per benchmark is reported (§3.3; §4.2.2).
Questions for authors
  1. Under the same model and compute budget, how does HoMe compare with a basic evaluate-and-edit coding agent, particularly with session-level diagnosis removed?
  2. What selection instructions and score-token trade-off does the optimizer use, and do independent searches with that same procedure retain the reported gains?
2 comments
AuthorsOpenReview ↗

Comparison with MemPro: Search Design, Quality, and Efficiency

W1 / Q1

W1

The baselines are memory systems, not harness optimizers. Without a budget-matched optimization baseline or component ablations of answer quality, the benefit of HoMe over a simpler coding-agent loop remains unclear.

Q1

Under the same model and compute budget, how does HoMe compare with a basic evaluate-and-edit coding agent, particularly with session-level diagnosis removed?

Response.

We have added MemPro, as a memory harness optimization baseline. It follows an evaluate-and-edit loop in which a single developer agent analyzes evaluation feedback, attributes errors, and revises the entire harness, including code, prompts, and parameters, without a dedicated session-level diagnostic agent. Each round evaluates either the full training set or the same subset sampled at the start, without adapting the evaluation set to historical feedback.

Both HoMe and MemPro start from the same ReMe harness and use the same models, AgentScope CLI, runtime, editable scope, and scoring protocol. The complete BEAM-100K training set is available to both: MemPro evaluates all 20 cases and 400 queries in each round, while HoMe samples 100 queries from 10 cases per round after an initial full-training evaluation. On LongMemEval-S, evaluating all 250 training cases in each round is too expensive. Following its original paper, MemPro evaluates a fixed random subset of 50 cases; HoMe samples 28 cases per round after its initial full-training evaluation. On LongMemEval-S, MemPro runs eight formal evaluations to approximately match HoMe’s cumulative case evaluations and keep evaluation costs comparable.

On LongMemEval-S, HoMe achieves a test score of 0.916, outperforming MemPro’s 0.812 while reducing average query tokens from 96.9K to 17.9K. On BEAM-1M, the score increases from 0.693 to 0.704, while query tokens decrease from 65.4K to 60.4K. The advantage also holds in cross-benchmark transfer: BEAM-100K → LongMemEval-S yields scores of 0.908 versus 0.800 and query tokens of 26.6K versus 42.2K; LongMemEval-S → BEAM-1M yields scores of 0.639 versus 0.617 and query tokens of 39.0K versus 113.1K (Table 2; HoMe followed by MemPro throughout).

We further investigate the search behaviors behind these methods. Appendices A.5 and A.6 illustrate differences in search behavior: all MemPro candidates across BEAM-100K and LongMemEval-S modify only prompts or retrieval parameters. Despite permission to revise the entire harness, both searches remain focused on local prompt and parameter adjustments, without introducing new code-level construction or retrieval mechanisms. Accumulating such patches can overfit observed cases while leaving underlying construction and retrieval defects unresolved, limiting performance on unseen queries. HoMe, in contrast, implements runtime evidence auditing based on independent diagnosis, repairs errors using validation feedback, and proposes multiple harnesses with structural changes and new retrieval mechanisms.

Two design choices help explain this difference. First, HoMe combines trajectory-guided attribution with parallel case diagnosis: Case Study Agents trace individual failures to the responsible harness components, and the Diagnoser synthesizes recurring causes into a compact report, preserving the Optimizer’s context for cross-case reasoning and structural design. Second, candidate gating directs the search toward these changes. During the first half of the budget, the Gate Agent accepts only code-level mechanism changes and rejects prompt- or parameter-only tuning. This allocates early evaluations to testing structural hypotheses, preventing the search from spending its budget on successive patches for individual cases. Prompt and parameter tuning becomes eligible in the second half to refine promising designs.

Excluding test-set evaluation, HoMe uses 185.2M end-to-end tokens on BEAM-100K versus MemPro’s 331.2M, a 44.1% reduction (Appendix A.8, Tables 12 and 13). Memory construction and query evaluation dominate the cost. Dynamic sampling and construction reuse reduce memory-construction tokens by 45.4% and query-evaluation tokens by 59.0%, offsetting the additional diagnosis and review costs.

AuthorsOpenReview ↗

Final Harness Selection and Stability across Independent Searches

W2 / Q2

W2

Final selection relies on optimizer judgment, but its reliability is unclear. Across six candidates in the reported BEAM run, sampled and full-training answer scores have r = -0.083; the samples are intended for diagnosis, not ranking. Only one search per benchmark is reported (§3.3; §4.2.2).

Q2

What selection instructions and score-token trade-off does the optimizer use, and do independent searches with that same procedure retain the reported gains?

Response.

HoMe selects both a parent harness for the next revision and a final harness from formally evaluated candidates. We interpret the question as referring to final selection.

Final-selection reliability comes from comparing candidates on shared questions and checking whether their changes resolve the diagnosed problems. The Optimizer receives per-question results, diagnostic reports, code changes, and revision hypotheses. After each formal evaluation, HoMe automatically generates comparison tables against historical candidates on shared cases and questions, reporting mean scores, score differences, improvement and regression counts, and query tokens. Thus, although sampling makes aggregate scores across rounds incomparable, the Optimizer can judge candidates using direct comparisons and evidence that the proposed repairs work. Sampled and full-training average token consumption also correlate strongly (r = 0.870), supporting cost comparisons. Selection prioritizes answer quality, considering token savings when scores are similar, without an explicit weighted score–token trade-off.

The system prompt makes these principles explicit and requires selection from existing evaluated candidates when the budget is exhausted:

Optimize answer quality by supplying precise evidence at lower query-time token cost. Score improvement is more important than query token reduction. Every proposed change should aim for both.

Compare scores on common cases and queries, weigh coverage, difficulty, regressions and lineage, and do not rank sampled candidates by aggregate score alone. Prefer quality, stability, query cost and simplicity.

When exhausted, select the best existing evaluated branch without another formal validation. Finish with a clean checkout of the best evaluated branch and full commit. Choosing the baseline is valid if improvements are unsupported.

The full instructions are provided in Appendix A.9.

The choice of B4 illustrates this procedure (Table 7). B4, B5, and B6 score 0.670, 0.715, and 0.677 on their respective subsets. B4 restricts the topic index to coverage questions such as summarization and counting and consolidates duplicate notes, preventing note-level overviews from replacing fine-grained ordering evidence. B5 adds a chronological source-path map; B6 adds Conversation Flow line ranges for cross-session reading. Although their aggregate sampled scores are higher, B5 and B6 regress on questions shared with B4 and substantially increase token consumption, suggesting overfitting. The Optimizer therefore selects B4 based on the shared-question results and the improvements it already provides.

Independent case investigations also make it less likely that one agent’s misattribution steers the search. Git preserves evaluated alternatives so the Optimizer can return to promising branches, while the Gate encourages exploring different mechanisms before tuning, reducing premature convergence on case-specific patches.

Empirical stability. We conduct two independent BEAM searches with identical initial ReMe harnesses, hyperparameters, and evaluation budgets. Their selected harnesses achieve BEAM-1M scores of 0.704 and 0.720 with 60.4K and 68.3K average query tokens. Both improve on ReMe's 0.650 score and 101.8K tokens, confirming that the same search and selection procedure retains quality and efficiency gains across runs (Appendix A.4, Figure 4).

Consistency extends to diagnosis and proposed improvements. Both searches identify transcripts crowding notes out of retrieval, excessive search output, insufficient cross-session coverage, and lost attribution and update relations. They propose source-balanced retrieval, evidence compression, chronological indexes, and preservation of attribution and update relations (Table 8). These results show agreement in the problems identified and the resulting modification directions, alongside stable final performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.