- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that skill evolution is less effective for long-horizon tasks, where useful evidence is buried in long contexts. It proposes ContextEvo, which keeps the model fixed and instead evolves the harness's context policy across input assembly, history maintenance and context orchestration. Each round reconstructs evidence from held-in trajectories, attributes failures through cross-case screening and decision-point counterfactual replay, and applies constrained edits to the policy. The authors start from the Pi-agent harness and run two iterations on LHTB, DeepSWE and a hard BrowseComp-Plus subset, with DeepSeek-V4-Flash and GPT-5.6-Luna as backbones. They report improvements of 3.1–4.4 pp on full-benchmark scores and performance competitive with OpenCode, Codex and OpenClaw.
Critical strengths
S1. The paper addresses an important problem with a useful abstraction. Treating context management as an evolvable policy over the whole information lifecycle is a meaningful step beyond evolving prompts or skills. The analysis of why skill evolution struggles in long-horizon settings, including invocation gaps and local goal drift (Sec. 6.1), is insightful.
S2. The evolution pipeline is concrete and well documented. Case-card extraction, paginated review, explicit screening thresholds, bounded counterfactual replay and the constrained editor are all specified in App. B. The accompanying prompts are included, which supports reproducibility.
S3. The experiments are broad, and the claims are carefully worded. The paper covers three benchmarks, two backbones and four harnesses, together with compression baselines, ablations and transfer experiments. It also distinguishes observed, local and causal evidence, and acknowledges that decision-point replay does not establish task-level causality.
Critical weaknesses
W1. The headline numbers include training tasks, and the held-out gains are mixed. The abstract and introduction emphasize the All column, which includes the held-in tasks used for evolution. On held-out tasks alone (Table 2), the picture is weaker. DeepSeek on BrowseComp-Plus drops by 1.0 pp at iteration 1 despite a 9.2 pp held-in gain, Luna on DeepSWE falls from 18.5 to 15.4 and then 13.8, and Luna on BrowseComp-Plus drops by 6.1 pp at iteration 1. Several of the positive changes amount to only one to three tasks. Held-out performance should be the primary metric throughout.
W2. All results come from single runs without variance estimates. Each task is attempted once (A.2), on held-out sets of 26, 65 and 98 tasks. The ablations suggest substantial noise. If "w/o Context-Policy Update" leaves the native policy unchanged, it should match Native, yet the two differ by about 10.7 pp on held-in tasks (32.57 vs. 43.30; Table 6). Please define each ablation precisely and report at least three runs with confidence intervals.
W3. The comparison with established harnesses is overstated. Under GPT-5.6-Luna, Codex clearly outperforms ContextEvo, scoring 44.3 vs. 14.2 on DeepSWE and 60.1 vs. 42.0 on BrowseComp-Plus. The large gaps between backbones suggest integration issues rather than context pressure: Pi-agent, for instance, scores 64.6 on DeepSWE with DeepSeek but only 13.3 with Luna. The 26.4 pp cross-harness gain on OpenClaw also needs care. It is computed on all 174 questions, including the held-in ones. It also uses Qwen3.8-27B as the judge, which conflicts with the same-model setup described in Sec. 4.
W4. The closest baselines and the cost of evolution are missing. Meta-Harness, Agentic Harness Engineering and ACE are cited but not compared under a matched budget. Without such a comparison, the specific value of restricting edits to context-policy surfaces remains unclear. The cost of evolution, in analysis and editing tokens, replay calls and wall-clock time, is also not reported.
W5. The presentation and layout need polishing. Figures 1 and 2 contain numerous typos (e.g., "Lifetcyle", "Decsion", "Couterfactual"). Some cross-references are broken ("Table 3.2", "Table 5.1"), and Tables 3 and 4 duplicate each other. In addition, the wrapped tables at the bottom of p. 7 leave noticeable white space. This carries over to the top of p. 8, where two half-width lines are followed by an empty column. I would kindly suggest revising the layout, for example by removing the duplicate table, to improve readability.
Questions for authors
Please see the Weaknesses section.
3 comments
Weakness 3: Comparison with existing harnesses
W3. The comparison with established harnesses is overstated. Under GPT-5.6-Luna, Codex clearly outperforms ContextEvo, scoring 44.3 vs. 14.2 on DeepSWE and 60.1 vs. 42.0 on BrowseComp-Plus.
We understand this concern. ContextEvo is designed as an external component applicable to different harnesses. Our main experiments use Pi-agent as the base harness, and Appendix C.8 and Table 18 report its effects on other harnesses. Our central claim is that ContextEvo can improve the adaptive context-management capabilities of an existing harness. Because the Codex harness used in our comparison was not open source, we could not integrate ContextEvo into it to test whether it could further improve its performance.
We interpret your concern about “integration issues” as a question about the effects of model capability and model–harness compatibility on evaluation. Our main controlled comparison isolates the contribution of ContextEvo by holding the harness and base model fixed and changing only whether ContextEvo is applied.
For OpenClaw and OpenCode, we additionally provide the held-in and held-out scores below. Qwen3.8-27B was used only for benchmark scoring and did not participate in policy evolution.
| Harness | Method | Held-in (76) | Held-out (98) | All (174) |
|---|---|---|---|---|
| OpenCode | Base | 35.53 (27/76) | 34.69 (34/98) | 35.06 (61/174) |
| OpenCode | ContextEvo | 42.11 (32/76) | 39.80 (39/98) | 40.80 (71/174) |
| OpenClaw | Base | 22.37 (17/76) | 16.33 (16/98) | 18.97 (33/174) |
| OpenClaw | ContextEvo | 46.05 (35/76) | 44.90 (44/98) | 45.40 (79/174) |
Weakness 4: Baseline comparisons, budgets, and evolution cost
W4. The closest baselines and the cost of evolution are missing. Meta-Harness, Agentic Harness Engineering and ACE are cited but not compared under a matched budget. Without such a comparison, the specific value of restricting edits to context-policy surfaces remains unclear.
Thank you for this suggestion. We have added the following comparisons and cost measurements.
1. ContextEvo achieves the strongest performance in our long-horizon comparison with Meta-Harness and Self-Harness. We reproduced both baselines on LHTB. ContextEvo achieves the best overall performance while requiring fewer tokens per task and lower evolution cost.
| Method | Held-in ↑ | Held-out ↑ | Overall ↑ | Execution Cost ↓ | Evolution Cost ↓ |
|---|---|---|---|---|---|
| Pi-agent (Base) | 43.30 | 40.40 | 41.70 | 8.57 | — |
| ContextEvo | 47.10 ± 2.85 | 44.62 ± 2.52 | 45.70 ± 0.64 | 7.52 | 114.74 |
| Self-Harness | 45.53 | 41.93 | 43.49 | 8.95 | 1,213.45 |
| Meta-Harness | 46.24 | 43.15 | 44.49 | 9.19 | 1,101.01 |
Note. Execution Cost is the average token consumption(M tokens/task), while Evolution Cost is the total token consumption required to obtain the evolved harness (M tokens).
2. ACE's evolution mechanism faces limitations in this long-horizon setting. We reproduced its complete Generator → Reflector → Curator → playbook update pipeline on LHTB. This setting places substantial pressure on the real-time handling of incoming information from the environment, including tool observations. ACE's approach to evolving history and context maintenance was less effective in this setting:
| Candidate | Train | dev | Choice | Held-out ↑ |
|---|---|---|---|---|
| Empty playbook | 55.56 | 58.64 | Selected | 40.48 |
| ACE playbook v1 | 55.56 | 38.33 | Rejected | 20.17 |
| ACE playbook v2 | 39.22 | 46.74 | Rejected | 28.58 |
3. We report the model-token consumption of each evolution stage. The table below reports costs in millions of tokens. The largest cost comes from validating the proposed changes through counterfactual replay.
| Benchmark | Review | Attribution | Reconstruction | Policy Update | Replay | Total |
|---|---|---|---|---|---|---|
| LHTB | 0.35 | 0.37 | 0.28 | 1.08 | 112.66 | 114.74 |
| DeepSWE | 0.30 | 0.21 | 1.29 | 0.96 | 460.21 | 462.96 |
| BCP | 0.20 | 0.18 | 0.93 | 0.79 | 138.34 | 140.44 |
Note. Review: paginated trajectory review; Attribution: failure attribution; Reconstruction: model-visible context reconstruction; Policy Update: policy editing; Replay: counterfactual replay. All values are in millions of tokens (M).
Weakness 5: Presentation and layout
Thank you for pointing out these issues. We have corrected them in the revision version.
We sincerely thank the reviewer for the careful review and constructive suggestions. Your feedback has helped us strengthen our work through additional experiments and analyses, which have been incorporated into the revised manuscript. We greatly appreciate your time and consideration.
Thank you for the authors’ response. However, I still do not believe that the paper meets the standard for acceptance at this conference, and I will therefore retain my current score.
Thank you for your careful review and constructive suggestions, which help us improve both the experimental setup and the presentation of our results.
We have made the following revisions to the paper:
Here are our detailed responses to your concerns.
Weaknesses 1 and 2: Held-out regressions and result stability
Thank you for these suggestions. We agree that held-out performance should be the primary metric.
Our case analysis identified two sources of variability: (1) long-horizon executions can enter loops, run for extended periods, and time out; and (2) binary per-task scoring in the benchmarks can produce substantial changes in aggregate scores when only a few task outcomes change. These factors contribute to variability in held-out performance. We therefore removed the execution time limit and repeated the ContextEvo procedure three times. The held-out means and standard deviations are reported below:
The mean score across three rollouts improves over the baseline in every benchmark setting, supporting the effectiveness of our method on held-out tasks.
Besides, thank you also for identifying the ambiguity in “w/o Context-Policy Update.” This was a labeling error: the intended variant is “w/o Constrained Policy Editing.” It removes the safety constraints described in Section 4.3, which validate and reject updates that could cause harness failures. The ablation therefore measures performance without this constraint mechanism.