- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PerfRace, a benchmark of 100 long-running performance optimization tasks on real software projects without reference solutions, and BoxBox, a method that lets the agent call a boxbox() tool to pause, review its trajectory, rewrite its own harness, and hot-reload the new harness while keeping the execution context. On 50 PerfRace test tasks with GPT-5.6-Luna, BoxBox reaches 1.70x Speedup@4h, compared with 1.54x for Harness Scaling and 1.44x for the frozen harness, at lower cost than the frozen harness.
Critical strengths
- PerfRace is a useful contribution. Most agent benchmarks still evaluate a final state against a reference, while PerfRace is open-ended, workload-first and long-horizon, and the construction pipeline looks careful. Given how much attention open-ended self-improvement is getting now, I think this benchmark is relevant to the community.
- The paper is generally clearly written, and the system design (git-versioned harness, kernel-managed hot reload, rollback) is easy to follow.
Critical weaknesses
-
Gains are shown only on the authors' own benchmark. The authors propose a benchmark and then show their own method is SOTA on it. There are already several efficiency optimization benchmarks (e.g. GSO, SWE-fficiency, FormulaCode, KernelBench), and BoxBox should be evaluated on more of them. All experiments also use a single model and a single initial harness.
-
Unclear why fewer optimizations work better. BoxBox reviews a median of 5 times per run, fewer than Harness Scaling (about 11) and Periodic BoxBox, yet performs best. I would like finer-grained ablations that separate context compaction, review timing, and review frequency (e.g. random timing with the same number of reviews).
-
Many key details are missing. What exactly is the "predefined scope" in Bounded BoxBox? Since most edits are review feedback (40.8%) and prompt rules (42.9%), scope should not matter much, yet Bounded drops to 1.349. How does the agent decide when to call
boxbox()? The task-side prompts are not given, and with a 30-minute cooldown the agent may simply trigger right after it ends, which is close to periodic. One may suspect that prompt engineering here is playing a heavy role. -
Limited novelty. (minor) Agents rewriting their own code during a task has been explored before, e.g. Gödel Agent (Yin et al., 2024), Darwin Gödel Machine (Zhang et al., 2025), SICA (Robeyns et al., 2025), Live-SWE-agent (Xia et al., 2025), and Continual Harness (Karten et al., 2026), which the paper cites. The new part is mainly agent-triggered timing plus hot reloading, and the paper should make this delta clear.
Questions for authors
- What is the editable scope in Bounded BoxBox, and what is the distribution of modification types for each ablation variant?
- Can you release the task-side
boxbox()tool description and prompts, and report when reviews are triggered (e.g., how often right after the cooldown ends)? - After a hot reload, if the system prompt or tool definitions change, the prompt cache will miss for the whole context. How much does this cost, and what is the cache hit rate for each method?
- How would Frozen and HS perform with the same context compaction, polling and fault recovery? (Just for curiosity)
3 comments
Response to Reviewer 1Qx8 (2/2)
W3/Q1–Q2: Editable scope and review triggering
Bounded BoxBox follows Self-Harness's bounded-edit principle by restricting each review to one of five predefined surfaces:
- System prompts: persistent instructions and task-facing feedback.
- Skills: reusable procedures available to the agent.
- Tools/observability: tools for measurement, profiling, and execution inspection.
- Context/memory: context selection, compaction, and persistent memory.
- Runtime control: execution flow, polling, and fault recovery.
The categories reported in Figure 5(c) are non-exclusive, post-hoc labels rather than predefined editable surfaces. Review feedback is the handoff accompanying an intervention, and prompt-rule updates can accompany changes to tools or execution logic. Their 40.8% and 42.9% frequencies therefore do not imply that most interventions are confined to those surfaces. Restricting each review to one predefined surface reduces speedup from 1.584× to 1.349×.
We add the complete task-side boxbox() tool description and triggering prompt to Appendix D.4 (Lines 2552–2570). They explain what invoking boxbox() does and direct the agent to request review only when accumulated trajectory evidence can improve subsequent optimization. The triggering prompt is short and fixed across all 50 tasks, with no task-specific heuristic.
To prevent repeated reviews before enough new evidence accumulates, we set a 30-minute cooldown between reviews. We group 515 non-initial review triggers by the elapsed time since the preceding review completed:
| Elapsed time | Triggers | Share |
|---|---|---|
| ≤31 min | 61 | 11.8% |
| (31, 35] min | 143 | 27.8% |
| (35, 40] min | 92 | 17.9% |
| (40, 50] min | 105 | 20.4% |
| >50 min | 114 | 22.1% |
Only 11.8% occur within one minute after the cooldown ends, while 60.4% occur more than five minutes later; the agent therefore does not simply trigger review as soon as it becomes available.
Together, these results show that BoxBox benefits from letting the agent decide both which harness surface to revise and when accumulated evidence is sufficient to request a review.
W4: Distinction from prior self-improving agents
Thank you for encouraging a clearer comparison with prior self-improving agents. BoxBox differs in two respects:
- Agent-triggered, on-demand review. DGM and SICA evolve agents through benchmark-evaluated outer loops; Live-SWE-agent considers tool creation after each step; and Continual Harness invokes a Refiner every F steps. Gödel Agent autonomously selects self-updates, but within a dedicated validation-driven optimization cycle. BoxBox instead lets the task-solving agent request review from accumulated evidence during ongoing task execution.
- Task-specific full-harness revision. Live-SWE-agent focuses on custom-tool synthesis, while Continual Harness refines predefined prompt, sub-agent, skill, and memory components. DGM, SICA, and Gödel Agent optimize reusable agent implementations or policies against benchmark or validation performance. BoxBox uses experience from the ongoing task to revise any part of the executable harness used for that task.
Q3: Prompt caching
We thank you for raising this concrete technical concern. A harness revision can cause a cache miss on the first API call after reload; subsequent calls reuse the revised prefix. Since harness revisions are infrequent, BoxBox retains a token-weighted cache hit rate of 98.76%, compared with 99.24% for Frozen. This small reduction is outweighed by fewer model calls: BoxBox reduces mean model cost from 23.22 to 16.43. As a future improvement, we can incorporate expected cache misses into the review strategy as an explicit cost constraint.
Q4: Shared initial harness
Frozen, HS, and BoxBox already start from the same Terminus-KIRA harness, including identical context compaction, polling, and fault-recovery mechanisms. The comparison therefore already controls these initial capabilities. The changes reported in our case studies—such as extending a polling window or adjusting compaction and recovery logic—are task-specific revisions produced by BoxBox from the accumulated trajectory.
In summary, we thank you for these constructive comments. We hope the additional experiments and clarifications address your concerns.
References
[1]: Karpathy. Autoresearch. 2026.
(https://github.com/karpathy/autoresearch)
A gentle follow-up as the discussion closes
Thank you for highlighting PerfRace's relevance to open-ended self-improvement. Recent autoresearch systems illustrate this direction. Karpathy's Autoresearch evaluates an autonomous experiment--measurement--revision loop through GPU-based nanochat training [1]. One study finds that accumulated context can narrow exploration and cause discovery to plateau, based on four optimization tasks [2]. Another finds that agents become trapped in idea basins, based on 13 long-horizon research and engineering tasks [3]. Together, these results motivate accessible benchmarks for measuring sustained autoresearch progress.
PerfRace provides this evaluation through 100 workload-first tasks constructed from real software projects. Each task makes autoresearch measurable as a repeated loop of proposing an optimization, running the workload, and using correctness and performance feedback to guide the next iteration. Its construction and Pair Verification pipeline provide a scalable way to build autoresearch tasks from public software projects.
As the discussion period is set to close in 24 hours, we would appreciate it if you could let us know whether our rebuttal has addressed your concerns. If you find these additions sufficient, would you consider updating the overall rating to 4?
Referencecs
[1]: Karpathy. Autoresearch. 2026. (https://github.com/karpathy/autoresearch)
[2]: Tan et al. Scaling Discovery Requires Forgetting. 2026. (https://arxiv.org/abs/2610.20563)
[3]: Cai et al. Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents. 2026. (https://arxiv.org/abs/2610.07447)
Response to Reviewer 1Qx8 (1/2)
Thank you for your constructive suggestions on broadening the empirical evaluation and clarifying review timing, editable scope, and BoxBox's distinction from prior self-improving agents.
W1: Benchmark, model, and initial-harness coverage
We appreciate your observation that PerfRace's workload-first, long-horizon design is relevant to growing interest in open-ended self-improvement. It evaluates whether agents can complete the full optimization workflow on real application workloads, from profiling and controlled measurement to proposing, implementing, and validating improvements. Karpathy's Autoresearch illustrates growing interest in this experiment–measurement–revision loop [1]. The cited code-optimization benchmarks differ in two respects (Lines 1574–1593):
einsum; Appendix B.1), while KernelBench evaluates GPU kernels for tensor computations. PerfRace starts from end-to-end project workflows on commodity CPUs, leaving the agent to profile the application and determine what to optimize.To evaluate BoxBox beyond PerfRace, we compare Frozen and BoxBox on EdgeBench's five Systems & Software Engineering tasks using GPT-5.6 Luna and Terminus-KIRA. We run each method three times per task under a four-hour budget and report arithmetic-mean scores from EdgeBench's native evaluator (Lines 514–527).
BoxBox improves performance on four of five tasks and reduces model cost on all five. Its mean cost per trial decreases by 39.2%, from $21.81 to $13.26.
For the main comparison, we control the initial harness across methods. Following Meta-Harness, all methods start from Terminus-KIRA while retaining their original optimization mechanisms (Lines 369–373). We further evaluate BoxBox with Grok 4.6 and from an OpenHands initial harness on the same 14 tasks (Lines 499–512).
BoxBox improves speedup by 9.0% with Grok 4.6 and 8.0% from OpenHands while reducing mean model cost by 27.5% and 37.6%, respectively. These results provide further evidence for BoxBox's effectiveness on an additional benchmark and with an additional model and initial harness.
W2: Review timing and frequency
Harness Scaling revises the harness after the task agent submits and completes an attempt, so its review count is determined by submission timing rather than an independently controlled schedule. To isolate review frequency, we add Periodic-45, which changes only the Periodic BoxBox interval from 30 to 45 minutes. It produces a median of five reviews per run, matching Full BoxBox's median.
Fewer reviews are therefore not the source of the gain: at the same typical review count, agent-triggered reviews achieve higher speedup at lower cost than scheduled reviews.