- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes HoMe, an evaluation-driven framework for harness optimization. It combines session-level memory versioning and failure diagnosis with candidate gating, adaptive sampling, and memory reuse.
Critical strengths
-
The paper tries to address a relevant practical challenge: automating the optimization of long-term memory systems.
-
Experiments show that the optimized harnesses improve over the manually tuned ReMe baseline on both benchmarks.
Critical weaknesses
-
Comparisons with the most relevant optimization methods are missing. The introduction discusses Meta-Harness, EvolveMem, and MemPro as closely related work, but none is included in the experimental comparisons. The evaluated baselines (A-MEM, SimpleMem, and ReMe) are memory systems rather than harness optimization methods. These comparisons are insufficient to show HoMe’s effectiveness relative to existing optimization approaches.
-
The ablation study is incomplete. Table 3 shows evaluation-time reductions, but the contributions of key components to final performance remain unclear. In particular, session-level versioning, trajectory-guided attribution, and candidate gating are not individually evaluated.
-
The contribution seems incremental. HoMe largely follows the optimization paradigm of Meta-Harness, with additional mechanisms for memory versioning, diagnosis, and efficient evaluation. While potentially useful, these additions are not sufficiently justified as a substantial methodological advance.
Questions for authors
- Can Figure 1 be improved? The image resolution is low, some text and labels are unclear, and the overall presentation is not clear enough. I recommend revising the figure to improve its clarity and readability.
4 comments
Methodological Contributions: Diagnosis, Candidate Gating, Adaptive Evaluation, and Search Efficiency
W3
The contribution seems incremental. HoMe largely follows the optimization paradigm of Meta-Harness, with additional mechanisms for memory versioning, diagnosis, and efficient evaluation. While potentially useful, these additions are not sufficiently justified as a substantial methodological advance.
Response.
Meta-Harness proposes a general iterative framework: read historical code, execution traces, and scores → diagnose and revise the harness → evaluate and record. Its retrieval experiment uses 40 iterations and produces 109 candidates. Applying this search scale directly to long-history memory optimization would additionally require replaying interaction histories, constructing memory, and answering many questions each round, making evaluation unaffordable under our budget (Meta-Harness §3 and §4.2). MemPro applies this type of evaluate-and-edit framework to memory optimization, allowing a single Evolving Agent to revise the entire harness and reducing costs through fewer iterations, static sampling, and construction reuse (MemPro §4). In practice, however, it still faces difficulties tracing errors to code, a preference for simple edits, and limited feedback coverage from fixed evaluation sets. HoMe redesigns diagnosis, candidate generation, and evaluation to address these problems.
-
Tracing long-horizon errors to code mechanisms requires separating diagnosis from revision. Case investigation involves extensive sessions, memory, and retrieval records that can fill a coding agent's context, leaving it focused on superficial patches for individual errors. MemPro assigns investigation, attribution, and revision to the same agent. HoMe combines trajectory-guided attribution with parallel case diagnosis: Case Study Agents trace individual failures to harness defects, and the Diagnoser synthesizes shared causes into compact reports for the Optimizer, preserving its context for structural design. It also stores session-level checkpoints in Git and provides diff comparisons, reducing the cost of reconstructing memory changes from lengthy logs. Appendix A.5 shows how this design supports implementation of runtime evidence auditing and feedback-driven repair.
-
Code-editing permissions alone can still lead to local prompt patching, requiring explicit guidance. In our MemPro reproductions on BEAM-100K and LongMemEval-S, all candidates modify only prompts or retrieval parameters, despite permission to edit the entire harness (Appendix A.6, Tables 10 and 11). The Gate Agent accepts only code-level mechanism changes in the first half of the budget. Prompt- or parameter-only tuning becomes eligible in the second half. Search first explores memory organization, retrieval, and answering through larger changes, then refines promising designs.
-
Full evaluation every round is too expensive, while a small fixed sample repeatedly focuses search on the same cases, requiring dynamically refreshed feedback. MemPro evaluates either the full training set or a fixed subset sampled once. HoMe dynamically samples cases and queries using historical failures, low scores, uncertainty, and the number of rounds since the last evaluation, while retaining uniform exploration. The limited budget continually covers new difficult cases and tests repair hypotheses. For construction reuse, HoMe uses an independent equivalence review and retains a probability of reconstructing memory, reducing dependence on a single stochastic construction result.
These designs enable mechanism search at lower total cost. On BEAM-100K, HoMe uses 185.2M search tokens, 44.1% fewer than MemPro's 331.2M (Appendix A.8, Tables 12 and 13). Independent diagnosis and review increase developer-agent costs, but dynamic sampling and construction reuse reduce the dominant construction and query workloads by 45.4% and 59.0%, respectively.
Compared with MemPro, HoMe improves test scores by 10.4 percentage points on LongMemEval-S and 1.1 points on BEAM-1M while reducing query tokens by 81.5% and 7.6%, respectively (Table 1). Cross-benchmark transfer preserves these gains, improving scores by 10.8 and 2.2 points in the two directions while using fewer query tokens in both (Table 2).
Why the Three Components Work Together
W2
The ablation study is incomplete. Table 3 shows evaluation-time reductions, but the contributions of key components to final performance remain unclear. In particular, session-level versioning, trajectory-guided attribution, and candidate gating are not individually evaluated.
Response.
End-to-end leave-one-out ablations would require three additional full optimization runs. The reported BEAM-100K search used 185.2M tokens and took 10 hours 26 minutes. The three components form a sequential repair process, with each stage supplying evidence or guidance for the next.
Session-level versioning grounds fault localization in the history of the memory state. The same final-answer error can arise when information was never written, when a later session overwrote valid state, or when retrieval missed information that remained intact. Agents use checkpoints and session-level diffs to distinguish these cases by inspecting when information was written, what later sessions changed, and whether the relevant evidence was still available at retrieval time.
Trajectory-guided attribution traces failures to the responsible harness components. Structural repair also requires identifying defects shared across cases. HoMe supports this through parallel case diagnosis. Case Study Agents investigate individual failures, and the Diagnoser synthesizes their findings into a compact report for the Optimizer. This division keeps detailed case investigations out of the Optimizer's context while preserving the evidence it needs to reason about shared problems and design code-level repairs.
Candidate gating directs the search toward structural changes that address diagnosed causes. The Gate reserves early evaluations for code-level mechanism changes, so that proposed fixes to memory construction, indexing, retrieval, or answering receive evaluation budget. This makes it possible to test whether a structural revision resolves the shared failure identified during diagnosis.
Each stage addresses a different failure mode in the repair process. Removing any one changes what the remaining stages can accomplish.
-
Without session-level versioning, reliable attribution becomes more expensive. Even a dedicated Diagnoser must reconstruct state transitions from raw multi-session construction logs. Deep inspection then becomes token-intensive, making agents more likely to rely on final answers alone. The same wrong answer can therefore send the agent to the wrong part of the harness even when diagnostic and revision roles are separate.
-
Without trajectory-guided attribution and parallel case diagnosis, structural revisions lose a clear target. Checkpoints show how memory changed. Connecting those changes to shared code defects requires tracing failures and comparing cases. If the coding agent also performs detailed case investigations, case details can crowd out the context needed for cross-case reasoning. The Gate can still require a substantial code change, but the resulting edit may be weakly grounded in the observed failures or overfit to an isolated case.
-
Without candidate gating, diagnosed causes may go untested. Even with an accurate cross-case diagnosis, a developer agent can spend early evaluations on low-risk, case-specific prompt patches. Those patches may address individual failures while leaving the underlying construction, indexing, retrieval, or answering mechanism unchanged.
The system-level comparison is with MemPro, which runs without HoMe's session-level versioning, trajectory-guided attribution and parallel case diagnosis, or candidate gating. HoMe scores 0.704 on BEAM-1M versus 0.693 for MemPro and uses 44.1% fewer optimization tokens. The contribution of each component to final performance remains unmeasured.
Revise Figure
Q1
Can Figure 1 be improved? The image resolution is low, some text and labels are unclear, and the overall presentation is not clear enough. I recommend revising the figure to improve its clarity and readability.
Response.
We redrew and replaced Figure 1 to make its text, labels, and workflow clearer. The figure shows HoMe's three components: Developer Agents diagnose failures and revise the harness; the Experiment Manager manages candidates and results, controls tool access and evaluation; and the Evaluation Environment evaluates candidates in isolated sandboxes. Each iteration follows diagnosis and revision → candidate review and evaluation planning → sandbox evaluation → feedback, allowing subsequent changes to build on previous evaluation and diagnostic results.
Comparison with MemPro: Experimental Alignment, Main Results, and Cross-Benchmark Transfer
W1
Response.
We agree and have added MemPro, an automated memory optimizer, to the main and transfer experiments (Tables 1 and 2). Both methods start from the same tuned ReMe harness and share the runtime, editable plugin scope, coding-agent model, AgentScope CLI, construction and answering models, and scoring protocol. We optimize on training data only and measure final performance on independent test data.
MemPro uses fixed evaluation data, while HoMe dynamically samples using evaluation feedback. Both can access the complete BEAM-100K training set: MemPro evaluates it every round, whereas HoMe samples cases and queries after an initial full-training evaluation. On LongMemEval-S, full evaluation every round is prohibitively expensive. MemPro uses a subset sampled once, while HoMe dynamically samples after its initial full-training evaluation; cumulative formal case-evaluation counts are approximately matched.
Main experiments show higher answer quality and lower query cost on both benchmarks. On LongMemEval-S, HoMe improves the score from MemPro's 0.812 to 0.916 (+10.4 percentage points) and reduces average query tokens from 96.9K to 17.9K (−81.5%), using less than one-fifth as many tokens. On BEAM-1M, the score rises from 0.693 to 0.704 (+1.1 percentage points), while query tokens decrease from 65.4K to 60.4K (−7.6%). HoMe thus improves quality and efficiency even with million-token histories and sampled feedback, despite MemPro evaluating the complete BEAM training set every round. Its larger advantage over fixed-subset evaluation on LongMemEval-S supports the value of dynamic sampling in covering more difficult cases at lower cost.
Transfer experiments retain these advantages across benchmarks. BEAM-100K → LongMemEval-S yields scores of 0.908 for HoMe versus 0.800 for MemPro (+10.8 percentage points), with average query tokens of 26.6K versus 42.2K. LongMemEval-S → BEAM-1M yields scores of 0.639 versus 0.617 (+2.2 percentage points), with 39.0K versus 113.1K query tokens. Both directions preserve higher quality and lower query costs; in the latter, HoMe uses approximately one-third of MemPro's query tokens.