acceptodds
Under review as a conference paper at ICLR 2027

BoxBox: Recursive Harness Self-Improvement for Long-Running Performance Optimization

Abstract

Automated harness optimization improves agents by learning from trajectories and task feedback. Existing offline methods use reference answers to optimize harnesses for reuse on held-out tasks. However, many real-world tasks lack reference solutions, while long runs expose new difficulties that offline-optimized harnesses may not address. To study harness optimization in these settings, we introduce , a benchmark for long-running performance optimization of software projects without reference solutions, and further investigate whether agents can use task-specific experience to recursively improve their own harnesses during an ongoing task. To this end, we introduce , which enables agent-triggered harness revision and kernel-managed hot reloading while preserving execution context. We evaluate on under matched test-time budgets, comparing it with Frozen Harness, offline Harness Evolution, and Harness Scaling. achieves higher speedup with more efficient use of task time and model cost, reaching a 4h of \(1.70\times\), compared with \(1.54\times\) for the strongest baseline. Code is available at https://anonymous.4open.science/r/BoxBox-1C35/.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Agent Harnesses, Recursive Self-Improvement, Test-Time Adaptation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_1Qx8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PerfRace, a benchmark of 100 long-running performance optimization tasks on real software projects without reference solutions, and BoxBox, a method that lets the agent call a boxbox() tool to pause, review its trajectory, rewrite its own harness, and hot-reload the new harness while keeping the execution context. On 50 PerfRace test tasks with GPT-5.6-Luna, BoxBox reaches 1.70x Speedup@4h, compared with 1.54x for Harness Scaling and 1.44x for the frozen harness, at lower cost than the frozen harness.

Critical strengths
  1. PerfRace is a useful contribution. Most agent benchmarks still evaluate a final state against a reference, while PerfRace is open-ended, workload-first and long-horizon, and the construction pipeline looks careful. Given how much attention open-ended self-improvement is getting now, I think this benchmark is relevant to the community.
  2. The paper is generally clearly written, and the system design (git-versioned harness, kernel-managed hot reload, rollback) is easy to follow.
Critical weaknesses
  1. Gains are shown only on the authors' own benchmark. The authors propose a benchmark and then show their own method is SOTA on it. There are already several efficiency optimization benchmarks (e.g. GSO, SWE-fficiency, FormulaCode, KernelBench), and BoxBox should be evaluated on more of them. All experiments also use a single model and a single initial harness.

  2. Unclear why fewer optimizations work better. BoxBox reviews a median of 5 times per run, fewer than Harness Scaling (about 11) and Periodic BoxBox, yet performs best. I would like finer-grained ablations that separate context compaction, review timing, and review frequency (e.g. random timing with the same number of reviews).

  3. Many key details are missing. What exactly is the "predefined scope" in Bounded BoxBox? Since most edits are review feedback (40.8%) and prompt rules (42.9%), scope should not matter much, yet Bounded drops to 1.349. How does the agent decide when to call boxbox()? The task-side prompts are not given, and with a 30-minute cooldown the agent may simply trigger right after it ends, which is close to periodic. One may suspect that prompt engineering here is playing a heavy role.

  4. Limited novelty. (minor) Agents rewriting their own code during a task has been explored before, e.g. Gödel Agent (Yin et al., 2024), Darwin Gödel Machine (Zhang et al., 2025), SICA (Robeyns et al., 2025), Live-SWE-agent (Xia et al., 2025), and Continual Harness (Karten et al., 2026), which the paper cites. The new part is mainly agent-triggered timing plus hot reloading, and the paper should make this delta clear.

Questions for authors
  1. What is the editable scope in Bounded BoxBox, and what is the distribution of modification types for each ablation variant?
  2. Can you release the task-side boxbox() tool description and prompts, and report when reviews are triggered (e.g., how often right after the cooldown ends)?
  3. After a hot reload, if the system prompt or tool definitions change, the prompt cache will miss for the whole context. How much does this cost, and what is the cache hit rate for each method?
  4. How would Frozen and HS perform with the same context compaction, polling and fault recovery? (Just for curiosity)
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1Qx8 (1/2)

Thank you for your constructive suggestions on broadening the empirical evaluation and clarifying review timing, editable scope, and BoxBox's distinction from prior self-improving agents.

W1: Benchmark, model, and initial-harness coverage

We appreciate your observation that PerfRace's workload-first, long-horizon design is relevant to growing interest in open-ended self-improvement. It evaluates whether agents can complete the full optimization workflow on real application workloads, from profiling and controlled measurement to proposing, implementing, and validating improvements. Karpathy's Autoresearch illustrates growing interest in this experiment–measurement–revision loop [1]. The cited code-optimization benchmarks differ in two respects (Lines 1574–1593):

  • Workload. GSO, SWE-efficiency, and FormulaCode assess final artifacts on predefined microbenchmarks (e.g., a selected API call such as NumPy einsum; Appendix B.1), while KernelBench evaluates GPU kernels for tensor computations. PerfRace starts from end-to-end project workflows on commodity CPUs, leaving the agent to profile the application and determine what to optimize.
  • Evaluation framework. PerfRace supports iterative optimization by repeatedly evaluating submitted project snapshots, returning correctness and performance feedback, and tracking best-so-far progress throughout a shared budget, rather than evaluating only a final artifact.

To evaluate BoxBox beyond PerfRace, we compare Frozen and BoxBox on EdgeBench's five Systems & Software Engineering tasks using GPT-5.6 Luna and Terminus-KIRA. We run each method three times per task under a four-hour budget and report arithmetic-mean scores from EdgeBench's native evaluator (Lines 514–527).

TaskFrozen scoreBoxBox scoreFrozen costBoxBox costRelative to Frozen
ANN QPS ↑46075291$7.260$2.760+14.9%
Exchange ↑23092235$21.93$7.770−3.2%
FFmpeg ↑1.1811.285$14.53$11.67+8.9%
Integer ↑11.9722.47$30.26$28.12+87.7%
VLIW cycles ↓22591969$35.08$16.00+12.9%

BoxBox improves performance on four of five tasks and reduces model cost on all five. Its mean cost per trial decreases by 39.2%, from $21.81 to $13.26.

For the main comparison, we control the initial harness across methods. Following Meta-Harness, all methods start from Terminus-KIRA while retaining their original optimization mechanisms (Lines 369–373). We further evaluate BoxBox with Grok 4.6 and from an OpenHands initial harness on the same 14 tasks (Lines 499–512).

SettingMethodSpeedup@4hMean model costSpeedup/$
Grok 4.6 + Terminus-KIRAFrozen1.9297×$78.630.05580
BoxBox2.1036×$56.980.1544
GPT-5.6 Luna + OpenHandsFrozen1.7317×$17.480.2771
BoxBox1.8701×$10.900.6141

BoxBox improves speedup by 9.0% with Grok 4.6 and 8.0% from OpenHands while reducing mean model cost by 27.5% and 37.6%, respectively. These results provide further evidence for BoxBox's effectiveness on an additional benchmark and with an additional model and initial harness.

W2: Review timing and frequency

Harness Scaling revises the harness after the task agent submits and completes an attempt, so its review count is determined by submission timing rather than an independently controlled schedule. To isolate review frequency, we add Periodic-45, which changes only the Periodic BoxBox interval from 30 to 45 minutes. It produces a median of five reviews per run, matching Full BoxBox's median.

ConditionTriggerMedian reviews/runSpeedup@4hMean costSpeedup/$
Periodic-30Every 30 min71.508×$21.580.1338
Periodic-45Every 45 min51.478×$22.410.1206
Full BoxBoxAgent-triggered51.584×$15.070.3452

Fewer reviews are therefore not the source of the gain: at the same typical review count, agent-triggered reviews achieve higher speedup at lower cost than scheduled reviews.

AuthorsOpenReview ↗

Response to Reviewer 1Qx8 (2/2)

W3/Q1–Q2: Editable scope and review triggering

Bounded BoxBox follows Self-Harness's bounded-edit principle by restricting each review to one of five predefined surfaces:

  • System prompts: persistent instructions and task-facing feedback.
  • Skills: reusable procedures available to the agent.
  • Tools/observability: tools for measurement, profiling, and execution inspection.
  • Context/memory: context selection, compaction, and persistent memory.
  • Runtime control: execution flow, polling, and fault recovery.

The categories reported in Figure 5(c) are non-exclusive, post-hoc labels rather than predefined editable surfaces. Review feedback is the handoff accompanying an intervention, and prompt-rule updates can accompany changes to tools or execution logic. Their 40.8% and 42.9% frequencies therefore do not imply that most interventions are confined to those surfaces. Restricting each review to one predefined surface reduces speedup from 1.584× to 1.349×.

We add the complete task-side boxbox() tool description and triggering prompt to Appendix D.4 (Lines 2552–2570). They explain what invoking boxbox() does and direct the agent to request review only when accumulated trajectory evidence can improve subsequent optimization. The triggering prompt is short and fixed across all 50 tasks, with no task-specific heuristic.

To prevent repeated reviews before enough new evidence accumulates, we set a 30-minute cooldown between reviews. We group 515 non-initial review triggers by the elapsed time since the preceding review completed:

Elapsed timeTriggersShare
≤31 min6111.8%
(31, 35] min14327.8%
(35, 40] min9217.9%
(40, 50] min10520.4%
>50 min11422.1%

Only 11.8% occur within one minute after the cooldown ends, while 60.4% occur more than five minutes later; the agent therefore does not simply trigger review as soon as it becomes available.

Together, these results show that BoxBox benefits from letting the agent decide both which harness surface to revise and when accumulated evidence is sufficient to request a review.

W4: Distinction from prior self-improving agents

Thank you for encouraging a clearer comparison with prior self-improving agents. BoxBox differs in two respects:

  • Agent-triggered, on-demand review. DGM and SICA evolve agents through benchmark-evaluated outer loops; Live-SWE-agent considers tool creation after each step; and Continual Harness invokes a Refiner every F steps. Gödel Agent autonomously selects self-updates, but within a dedicated validation-driven optimization cycle. BoxBox instead lets the task-solving agent request review from accumulated evidence during ongoing task execution.
  • Task-specific full-harness revision. Live-SWE-agent focuses on custom-tool synthesis, while Continual Harness refines predefined prompt, sub-agent, skill, and memory components. DGM, SICA, and Gödel Agent optimize reusable agent implementations or policies against benchmark or validation performance. BoxBox uses experience from the ongoing task to revise any part of the executable harness used for that task.

Q3: Prompt caching

We thank you for raising this concrete technical concern. A harness revision can cause a cache miss on the first API call after reload; subsequent calls reuse the revised prefix. Since harness revisions are infrequent, BoxBox retains a token-weighted cache hit rate of 98.76%, compared with 99.24% for Frozen. This small reduction is outweighed by fewer model calls: BoxBox reduces mean model cost from 23.22 to 16.43. As a future improvement, we can incorporate expected cache misses into the review strategy as an explicit cost constraint.

Q4: Shared initial harness

Frozen, HS, and BoxBox already start from the same Terminus-KIRA harness, including identical context compaction, polling, and fault-recovery mechanisms. The comparison therefore already controls these initial capabilities. The changes reported in our case studies—such as extending a polling window or adjusting compaction and recovery logic—are task-specific revisions produced by BoxBox from the accumulated trajectory.

In summary, we thank you for these constructive comments. We hope the additional experiments and clarifications address your concerns.

References

[1]: Karpathy. Autoresearch. 2026.
(https://github.com/karpathy/autoresearch)

AuthorsOpenReview ↗

A gentle follow-up as the discussion closes

Thank you for highlighting PerfRace's relevance to open-ended self-improvement. Recent autoresearch systems illustrate this direction. Karpathy's Autoresearch evaluates an autonomous experiment--measurement--revision loop through GPU-based nanochat training [1]. One study finds that accumulated context can narrow exploration and cause discovery to plateau, based on four optimization tasks [2]. Another finds that agents become trapped in idea basins, based on 13 long-horizon research and engineering tasks [3]. Together, these results motivate accessible benchmarks for measuring sustained autoresearch progress.

PerfRace provides this evaluation through 100 workload-first tasks constructed from real software projects. Each task makes autoresearch measurable as a repeated loop of proposing an optimization, running the workload, and using correctness and performance feedback to guide the next iteration. Its construction and Pair Verification pipeline provide a scalable way to build autoresearch tasks from public software projects.

As the discussion period is set to close in 24 hours, we would appreciate it if you could let us know whether our rebuttal has addressed your concerns. If you find these additions sufficient, would you consider updating the overall rating to 4?

Referencecs

[1]: Karpathy. Autoresearch. 2026. (https://github.com/karpathy/autoresearch)

[2]: Tan et al. Scaling Discovery Requires Forgetting. 2026. (https://arxiv.org/abs/2610.20563)

[3]: Cai et al. Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents. 2026. (https://arxiv.org/abs/2610.07447)

Reviewer_PM6SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies online harness adaptation for long-running software performance optimization. It introduces PerfRace, a benchmark of 100 tasks from real software projects, and BoxBox, which allows the task-solving agent to request revisions to its executable harness. A separate kernel validates and activates revisions while preserving execution context and providing recovery. On 50 held-out tasks with three runs per task and a shared four-hour test-time budget, BoxBox achieves an aggregate speedup of 1.702x, compared with 1.539x for the strongest baseline. The experiments compare frozen, offline-optimized, and externally revised harnesses, with ablations on review timing and editable scope.

Critical strengths
  1. The paper addresses a useful practical problem: adapting an agent's executable harness to difficulties encountered during a long task. Context preservation, hot reloading, and recovery make this a concrete system contribution.

  2. PerfRace provides repository-based performance optimization tasks without prescribed target patches. The results across 50 held-out tasks and three runs per task provide encouraging evidence that the combined BoxBox system improves optimization performance under a matched test-time budget.

Critical weaknesses
  1. The ablations do not fully separate executable harness editing from review timing and context preservation, leaving the source of the gains unclear.

  2. The experiments use only Terminus-KIRA as the starting harness. Without stronger harness baselines, the gains may partly reflect fixes to this initial harness.

  3. Evaluation is limited to the authors' PerfRace benchmark; generalization to other benchmarks remains untested.

Questions for authors
  1. Can the authors add matched-budget controls for context-preserving Harness Scaling and a context-preserving agent that receives review feedback without editing executable harness code?

  2. Please compare a stronger fixed harness with BoxBox initialized from that harness on PerfRace, using the same model, budget, and evaluation protocol.

  3. Are there results on other benchmark?

2 comments
AuthorsOpenReview ↗

Response to Reviewer PM6S

Response to Reviewer PM6S

Thank you for your constructive suggestions on strengthening BoxBox's effectiveness and generalization analyses. We hope the following response addresses your concerns regarding the paper's soundness.

W1/Q1: Ablation study of context preservation and executable revision

To clarify the source of the gains, we add two ablations as follows.

For context preservation, the Harness Scaling implementation used for Terminal-Bench applies each post-attempt harness revision in a new trajectory, preserving project progress without carrying over the active conversation, tool state, or running processes (Lines 366–369). The BoxBox Kernel provides the mechanism to activate each revision within the ongoing task session while preserving its execution context. To isolate context preservation, we implement Context-refresh as a BoxBox variant that clears the task conversation after each review.

To isolate executable revision, we implement Feedback-only as a BoxBox variant that keeps the executable harness fixed while the review expresses its intended edits as equivalent feedback rules or steps (Lines 461–491).

ConditionSpeedup@4hMean model costSpeedup/$
Frozen1.389$18.700.2577
Context-refresh1.460$9.2050.2894
Feedback-only1.550$26.070.1673
Full BoxBox1.584$15.070.3452

Full BoxBox achieves the highest speedup and Speedup/$. Its gains over Context-refresh and Feedback-only show that task-context preservation and executable revision each contribute to performance and cost efficiency.

W2/Q2: Initial harness

For the main comparison, we control the initial harness across methods. Following Meta-Harness, all methods start from Terminus-KIRA while retaining their original optimization mechanisms (Lines 369–373).

To evaluate BoxBox from a stronger initial harness, we initialize both Frozen and BoxBox with OpenHands and compare them on the same 14 tasks using GPT-5.6 Luna under the same budget and evaluation protocol (Lines 499–512):

MethodSpeedup@4hMean model costSpeedup/$
Frozen + OpenHands1.7317×$17.480.2771
BoxBox + OpenHands1.8701×$10.900.6141

Starting from the stronger OpenHands harness, BoxBox improves speedup from 1.7317× to 1.8701× while reducing mean model cost by 37.6% and more than doubling Speedup/$. These results show that BoxBox performs task-specific harness adaptation from different initial harnesses.

W3/Q3: Cross-benchmark generalization

To evaluate whether BoxBox generalizes beyond PerfRace, we compare Frozen and BoxBox on EdgeBench's five Systems & Software Engineering tasks, which optimize executable software performance under correctness constraints. We use GPT-5.6 Luna with Terminus-KIRA and run each method three times per task under a four-hour budget. Task scores are arithmetic means across runs under EdgeBench's native evaluator, and costs are averaged per trial (Lines 514–527).

TaskFrozen scoreBoxBox scoreFrozen costBoxBox costRelative to Frozen
ANN QPS ↑46075291$7.260$2.760+14.9%
Exchange ↑23092235$21.93$7.770−3.2%
FFmpeg ↑1.1811.285$14.53$11.67+8.9%
Integer ↑11.9722.47$30.26$28.12+87.7%
VLIW cycles ↓22591969$35.08$16.00+12.9%

BoxBox improves performance on four of five tasks and reduces model cost on all five. Its mean model cost per trial decreases by 39.2%, from 21.81 to 13.26, showing that the gains extend beyond PerfRace.

We sincerely thank you for these constructive suggestions and hope the added ablations and generalization experiments address your concerns. We would appreciate your reconsideration of the overall assessment.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer PM6S,

Thank you again for your thoughtful comments on mechanism attribution, the initial harness, and evaluation beyond PerfRace. As the discussion period is coming to an end, we would greatly appreciate it if you could review our follow-up response. We added Context-refresh and Feedback-only ablations to isolate task-context preservation and executable harness revision. We also initialized both Frozen and BoxBox from OpenHands under a matched protocol and evaluated BoxBox on EdgeBench's five Systems and Software Engineering tasks. All results are incorporated into the revised manuscript.

We hope these additions are helpful in your assessment and would sincerely appreciate your reconsideration of the overall evaluation. If any questions or reservations remain, please let us know.

Best,
Authors

Reviewer_Mfs7OpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This article introduces a benchmark and a harness self-improvement framework to explore agents' abilities in long-term tasks. The benchmark includes 100 real repositories, aiming to improve code performance. The paper also proposes a harness update method, which modifies the harness and continues the task by kernel management while preserving context. Experiments show that the proposed method achieves acceleration, outperforming baseline methods.

Critical strengths
  1. The article uses a combination of automation and manual review to build a dataset for harness evaluation consisting of 100 projects, which involves a lot of work and is quite meaningful.
  2. The article proposes a new method, and the concept of a kernel is fairly novel for agent evolution.
  3. The proposed method in the article has been experimentally validated.
Critical weaknesses
  1. Readability. Some sentences are a bit unclear, especially in the induction section. For example, the authors mention HE and Frozen Harness without explaining what they mean. Also, Figures 1-3 have too much text, which makes them harder to read.
  2. In the benchmark, each task is a test generated by the agent, and humans haven't independently verified the reliability of the verifier. So, it seems like concerns such as agents learning to hack the reward haven't really been addressed. The paper does mention that a red team checked for cheating, but the details are unclear, like whether they found any vulnerabilities or what proportion of attacks occurred. In other words, the agent is evolving, generating, and verifying itself for cheating without human involvement, which is a pretty big problem on its own.
  3. The experiments in the article aren't thorough enough; they only used gpt5.6 Luna and terminus-kira, ignoring work like continual harness and self-harness.
  4. The author didn’t show that multiple rounds of revisions made the harness keep getting better, like they didn’t show how much improvement there was after the 1st, 2nd, or 3rd harness updates, and they also didn’t explain if later updates were better than the earlier ones.
Questions for authors

See Weaknesses 1-4.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Mfs7 (1/2)

W1: Presentation and baseline definitions

Thank you for the constructive presentation suggestions. We have defined both baselines at their first mention (Lines 67–69): Frozen Harness keeps the initial harness fixed throughout task execution, while Harness Evolution optimizes the harness offline on training tasks and selects it by validation performance before fixed-harness test evaluation.

We also revised Figures 1–3 (Lines 27–41, 164–184, and 234–249) by shortening captions, reducing in-figure text, and enlarging labels.

W2: Human verification and reward-hacking checks

We thank you for highlighting the need to clarify human verification of verifier reliability and reward-hacking risks. In PerfRace, this control is designed into the task-construction process: every task passes Pair Verification, whose Integrity gate requires a human reviewer to examine the red-team trial results, diagnose the verifier weakness exposed by each successful attack, and request a targeted fix before approving the task (Lines 214–219).

We now provide the requested details on the scale and attack categories of red-team testing. Appendix B.4.2 records at least 175 red-team trials across all 100 tasks; 40 tasks have at least two trials (Lines 2098–2116). We report the task coverage of five attack mechanisms; a task can cover multiple mechanisms:

Attack classDescriptionTasks covered
Faked successful outputReturn a successful-looking output without performing the required computation.59
Test manipulationAlter the workload, tests, or scoring path, as documented in prior benchmark reward-hacking studies [2,3].43
Hard-coded or shortcut outputProduce an accepted result by hard-coding the answer or bypassing required work.66
Hidden-test extractionObtain hidden inputs, tests, or expected outputs and use them to construct an accepted result.23
Cached-result reuseReuse precomputed results or stale artifacts instead of executing the measured workload.20

When a red-team trial exposes a verifier weakness, the human reviewer requests a targeted fix, and the construction agent hardens the verifier and replays the attack before Integrity approval. In the PDAL task, correctness and performance ran sequentially and shared /tmp. A red-team implementation saved the intermediate output produced during correctness checking and reused it during timed runs, obtaining a speedup above 13×. The hardened verifier clears candidate-writable runtime state after each operation, and replaying the attack no longer earns speedup.

Beyond this Trial-driven hardening, PerfRace also prevents reward hacking through two construction and evaluation choices:

  • Workload-driven construction. Ludwig et al. [1] show that agents can retrieve benchmark reference patches from local Git history or upstream repositories. PerfRace constructs tasks from real workloads rather than existing performance pull requests or commits, so project history contains no reference patch to retrieve.
  • Shadow verification. Verifier-owned tests, held-out inputs, and expected outputs remain outside the agent container. Each submission is evaluated in an isolated shadow, preventing test manipulation, hard-coding from hidden answers, and reuse of evaluation caches (Lines 2121–2130).

Finally, we manually audited the trajectories and project changes of five high-speedup BoxBox runs from the formal evaluation on the held-out test split (17.34×–156.64×). All five gains came from substantive project-code optimizations, and we found no evidence of reward hacking.

W3: Controlled comparisons and generalization

To compare harness-optimization methods, we control the initial harness across methods. Following Meta-Harness, each method starts from Terminus-KIRA while retaining its core design (Lines 355–364).

We also include ablations that adapt the core mechanisms of Continual Harness and Self-Harness to this controlled setup. Their original implementations start from a minimal Pokémon interface and a minimal DeepAgent harness, respectively. Periodic BoxBox adopts Continual Harness's scheduled refinement (Lines 462–467), while Bounded BoxBox adopts Self-Harness's bounded-edit principle (Lines 468–473). They reach 1.508× and 1.349× speedup, respectively, compared with 1.584× for Full BoxBox.

To evaluate generalization across models and initial harnesses, we add two matched Frozen–BoxBox comparisons (Lines 489–505):

SettingMethodSpeedup@4hMean costSpeedup/$
Grok 4.6 + Terminus-KIRAFrozen1.9297×$78.630.0558
BoxBox2.1036×$56.980.1544
GPT-5.6 Luna + OpenHandsFrozen1.7317×$17.480.2771
BoxBox1.8701×$10.900.6141

BoxBox improves speedup while reducing model cost in both settings, showing that its gains extend across backbone models and initial harnesses.

AuthorsOpenReview ↗

Response to Reviewer Mfs7 (2/2)

W4: Improvement across repeated reviews

To measure how performance changes as reviews accumulate, we pair each BoxBox run with the Frozen run for the same task and repeat index, then compare their best-so-far correct speedups at the same elapsed time at the start of each BoxBox review. For each review round, we aggregate runs geometrically within each task and tasks harmonically, following Speedup@4h, and report BoxBox's relative advantage over Frozen. Rounds 6+ pool all later reviews.

Review roundReviewsTasksAdvantage at review start
1st15050-0.12%
2nd14950+6.21%
3rd14450+10.00%
4th12549+15.33%
5th8043+13.69%
6+4428+22.66%

Performance is nearly matched at the first review (-0.12%). BoxBox's advantage grows after completed reviews and reaches 22.66% for rounds 6+, showing that later revisions continue to improve performance.

In conclusion, we sincerely thank you again for your constructive feedback. We hope these clarifications and additional analyses address your concerns, and we would appreciate your reconsideration of the overall assessment.

References

[1]: Ludwig et al. Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks. 2026.
(https://arxiv.org/abs/2609.06780)

[2]: Wang et al. Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack. 2026.
(https://arxiv.org/abs/2605.12673)

[3]: Von Arx et al. Recent Frontier Models Are Reward Hacking. 2025.
(https://metr.org/blog/2025-06-05-recent-reward-hacking/)

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer Mfs7,

Thank you again for your constructive comments on presentation, benchmark integrity, baseline coverage, and improvement across reviews. As the discussion period is coming to an end, we would greatly appreciate it if you could review our follow-up response. We revised the baseline definitions and figures, documented the human-reviewed Integrity gate and 175 red-team trials, and manually audited five high-speedup evaluation runs for reward hacking. We also clarified the controlled Continual Harness and Self-Harness adaptations, added Grok 4.6 and OpenHands evaluations, and reported BoxBox's advantage at successive review rounds.

We hope these additions are helpful in your assessment and would sincerely appreciate your reconsideration of the overall evaluation. If any questions or reservations remain, please let us know.

Best,
Authors

Reviewer_xZ2gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BOXBOX enables agents to revise their executable harness during ongoing tasks while preserving context. The paper also introduces PERFRACE, a 100-task software optimization benchmark. BOXBOX achieves 1.702× speedup versus 1.539× for the strongest baseline.

Critical strengths
  1. Practical benchmark with realistic workloads and careful correctness verification.
  2. Concrete system supporting harness updates, state preservation, and recovery.
  3. Substantial evaluation with shared time budgets and explicit model-cost accounting.
Critical weaknesses
  1. Gains are not clearly separated from additional reflection or context preservation. The recursive adaptation lacks a direct ablation.
  2. Ablations cover only 14 tasks, with unclear selection criteria and no uncertainty estimates.
  3. Best-so-far scoring may amplify measurement noise. The independent remeasurement of selected snapshots is unclear.
  4. Cost-efficiency metrics are sensitive to outliers, and evidence is limited to one model and initial harness.
Questions for authors
  1. Do gains persist with feedback-only review, independent remeasurement, and identical per-run dollar budgets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer xZ2g (1/2)

Thank you for recognizing the practical value of PerfRace and the concrete system design of BoxBox, and for suggesting additional ablations to clarify BoxBox's effectiveness. We respond to each point below.

W1/Q1: Ablation of reflection and context preservation

In BoxBox, feedback serves as the handoff that communicates the review's diagnosis and selected intervention to resumed task execution. The review instruction states: "State the diagnosis and selected intervention. When the intervention activates a new capability, explain how the Target Agent should use it."

To separate the effect of executable harness revision from reflection over the accumulated trajectory, we add a Feedback-only ablation. It retains all review steps and agent-triggered timing while fixing the harness. The review expresses its intended harness edits as equivalent feedback rules or steps for the resumed task session, preserving the trajectory-based reflection.

As for context preservation, we provide a Context-refresh experiment that clears the task-conversation history after each review while preserving the workspace, terminal sessions, and running processes. Section 5.3 and Table 2 report:

ConditionSpeedup@4hMean model costSpeedup/$
Frozen1.389$18.700.2577
Feedback-only1.550$26.070.1673
Context-refresh1.460$9.2050.2894
Full BoxBox1.584$15.070.3452

Full BoxBox reaches 1.584× speedup and 0.3452 Speedup/$, compared with 1.550×/0.1673 for Feedback-only and 1.460×/0.2894 for Context-refresh. The results show that reflecting on the accumulated trajectory guides subsequent task execution, while executable revision incorporates task-specific experience into reusable harness behavior and context preservation avoids reconstructing prior progress. Together, these mechanisms give Full BoxBox the highest overall cost efficiency. We have added both ablations and their results to Section 5.3 and Table 2 (Lines 461–491).

W2/Q1: Ablation task selection and uncertainty estimation

The 14 tasks were randomly sampled from the held-out test set before running the ablations. They span six PerfRace domains: five data systems, four developer tools, two machine-learning workloads, and one task each from scientific computing, media processing, and geometry. Following our Experimental Setup, all ablation settings use the same tasks with three independent four-hour runs per task. As defined in Metrics (Eq. 2), we take the geometric mean across runs to account for model stochasticity and further aggregate tasks with the harmonic mean, following GSO [1], to prevent a few extreme speedups from dominating the result.

To estimate uncertainty, we use the Delta method [2] to compute the standard error (SE) by propagating the within-task log-speedup variance through the same aggregation. Let GiG_i be task ii's geometric-mean speedup, HH the final harmonic mean, and si2s_i^2 the sample variance of its RR log-speedups:

\approx \frac{H^2}{N} \sqrt{\sum_{i=1}^{N}\frac{s_i^2}{R G_i^2}}.$$ This quantity estimates how much the final Speedup@4h would vary if we reran the same tasks, due only to model stochasticity. | | Frozen | Feedback-only | Context-refresh | Bounded | Periodic | Full BoxBox | |---|---:|---:|---:|---:|---:|---:| | SE | 0.0534 | 0.0427 | 0.0719 | 0.0209 | 0.0652 | 0.0401 | Across all methods, the SE ranges from 0.0209 to 0.0719, indicating limited variation in the aggregate results due to model stochasticity. ## W3/Q1: Measurement reliability and verification The Judge measures each submitted snapshot through multiple independent timed executions, which it aggregates into a verification score. Pair Verification determines and freezes each task's warmup count, repetition count, and aggregation rule before evaluation. Reliability approval requires a coefficient of variation below 10%; all 99 wall-clock tasks meet this threshold, while the remaining gas-measurement task is deterministic (Appendix B.4). This protocol keeps measurement noise within the reliability range established during Pair Verification.
AuthorsOpenReview ↗

Response to Reviewer xZ2g (2/2)

W4: Cost-efficiency metric

To test sensitivity to outliers, we report the median task-level Speedup/$. We also compare methods under identical per-run model-cost budgets, taking each run's best correct speedup within the budget and applying the same geometric-then-harmonic aggregation as Speedup@4h:

MethodMedian Speedup/$Speedup@$5Speedup@$10Speedup@$15
Frozen0.07021.421×1.432×1.438×
Harness Evolution0.05431.240×1.246×1.247×
Harness Scaling0.11761.411×1.473×1.532×
BoxBox0.15761.624×1.679×1.695×

To evaluate generalization across models and initial harnesses, we conduct two matched comparisons:

SettingMethodSpeedup@4hMean costSpeedup/$Median Speedup/$
Grok 4.6 + Terminus-KIRAFrozen1.9297×$78.630.05580.0267
BoxBox2.1036×$56.980.15440.0368
GPT-5.6 Luna + OpenHandsFrozen1.7317×$17.480.27710.1726
BoxBox1.8701×$10.900.61410.1773

Across the four methods, BoxBox achieves the highest median speedup per dollar and the highest speedup at each matched cost budget. It also improves speedup and reduces model cost with both Grok 4.6 and OpenHands.

We thank you for these constructive comments and hope the additions address your concerns.

References

[1]: GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents
(https://arxiv.org/abs/2505.23671)

[2]: Oehlert, G. W. A Note on the Delta Method. The American Statistician, 1992.
(https://doi.org/10.1080/00031305.1992.10475842)

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer xZ2g,

Thank you again for your thoughtful and constructive comments, which have helped us strengthen the effectiveness and reliability analyses. As the discussion period is coming to an end, we would greatly appreciate it if you could review our follow-up response. Following your suggestions, we added Feedback-only and Context-refresh ablations, documented the sampling and uncertainty protocol for the ablation set, clarified that each verification score aggregates multiple independent measurements, and reported median and matched-budget cost-efficiency results. We also evaluated BoxBox with Grok 4.6 and from an OpenHands initial harness. These results and clarifications are incorporated into the revised manuscript.

We hope these additions are helpful in your assessment and we kindly hope that you will take them into account in your final evaluation. If any questions or reservations remain, please let us know.

Best,
Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers and Area Chair

Dear Reviewers and Area Chair,

We thank the reviewers for their constructive feedback and for recognizing the following contributions:

  • PerfRace provides realistic, open-ended, and long-horizon software optimization tasks constructed from real application workloads (xZ2g, Mfs7, PM6S, 1Qx8).
  • BoxBox provides a concrete system supporting online harness optimization with context preservation and recovery (xZ2g, Mfs7, PM6S, 1Qx8).
  • BoxBox enables task-specific harness improvement, allowing the agent to revise its tools, prompts, memory, and execution logic according to difficulties encountered during the ongoing task (PM6S).

During the rebuttal, we added controlled ablations, generalization experiments, and further analyses of reliability, cost efficiency, and benchmark integrity. These results and clarifications have been incorporated into the revised manuscript, with revisions highlighted in blue.

  1. Effectiveness of executable revision, context preservation, and agent-triggered timing (W1 of xZ2g; W1 of PM6S; W2–W3 of 1Qx8): We added Feedback-only and Context-refresh ablations. On the 14-task ablation set, Full BoxBox reaches 1.584× speedup and 0.3452 Speedup per dollar, compared with 1.550×/0.1673 for Feedback-only and 1.460×/0.2894 for Context-refresh. We further added Periodic-45, which matches Full BoxBox’s median of five reviews per run but reaches 1.478× speedup and 0.1206 Speedup per dollar. Together, these ablations isolate the contributions of executable harness revision, task-context preservation, and agent-triggered review timing, supporting the effectiveness of the complete BoxBox design.

  2. Generalization across benchmarks, models, and initial harnesses (W4 of xZ2g; W3 of Mfs7; W2–W3 of PM6S; W1 of 1Qx8): We evaluated BoxBox on EdgeBench’s five Systems & Software Engineering tasks, where it improves performance on four tasks and reduces model cost on all five. With Grok 4.6, BoxBox improves speedup by 9.0% and reduces mean model cost by 27.5%. Starting from OpenHands, it improves speedup by 8.0% and reduces cost by 37.6%. These experiments extend the evidence beyond PerfRace, GPT-5.6 Luna, and Terminus-KIRA.

  3. Human verification and benchmark integrity (W2 of Mfs7): We reported at least 175 red-team trials covering five reward-hacking mechanisms. Every task passes a human-reviewed Integrity gate, where successful attacks lead to targeted verifier hardening and replay before approval. PerfRace further prevents reference-patch retrieval through workload-driven construction and isolates verifier-owned tests through shadow verification. We also manually audited five high-speedup BoxBox runs from the held-out evaluation and found that their gains resulted from substantive project-code optimizations.

  4. Measurement reliability and uncertainty estimation (W2–W3 of xZ2g): Each submitted snapshot is measured through multiple independent timed executions and aggregated using a task-specific protocol frozen during Pair Verification; all wall-clock tasks satisfy the required measurement-variance threshold. For each task, we geometrically aggregate three independent agent runs to summarize multiplicative speedup under model randomness, then aggregate tasks harmonically to limit the influence of extreme speedups. We propagate the within-task log-speedup variance through the same aggregation using the Delta method, obtaining SE values from 0.0209 to 0.0719 across methods.

  5. Improvement across successive reviews (W4 of Mfs7): We paired every BoxBox run with the corresponding Frozen run and compared their performance at the same task time before each review. Their performance is nearly matched at the first review (−0.12%), while BoxBox’s advantage increases after completed reviews and reaches 22.66% for rounds 6+, showing that later task-specific harness revisions continue to improve performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.