acceptodds
Under review as a conference paper at ICLR 2027

VibeTraining-Bench: Data–Trainer–Harness Co-Design for Iterative System Improvement

Abstract

Recursive self-improvement requires research agents that can iteratively improve a designated target agent system as a whole, rather than one component in isolation. A target agent system's performance depends jointly on its training data, trainer, and inference harness, and because these components interact, the most useful intervention can shift as the system evolves. Existing benchmarks either fix a single component in advance or open only training-side components. They therefore do not evaluate whether research agents can select, coordinate, and revise interventions across training and inference, a capability we call cross-component co-design. We introduce VibeTraining-Bench,, a benchmark of five tasks spanning terminal interaction, software engineering, multi-turn tool use, and mathematical reasoning. Within each task, research agents start from the same initial target agent system and training setup, then use validation scores and execution trajectories to guide iterative changes to the permitted components under a fixed budget. To limit reward hacking, an isolated grader evaluates the selected target agent system once on hidden test tasks that the research agent cannot access. Evaluating six research-agent configurations, we find large hidden-test gains (32.9% to 53.7% on SWE-bench Verified; 20.3% to 52.1% on -Bench), yet improvement is not guaranteed and component exploration remains uneven: one unrestricted SWE run does not edit the harness, although a separate harness-only run improves the initial target agent system. Validation search is also non-monotonic, and validation leaders do not always lead on the hidden test. These results expose a gap between finding effective system changes and reliably exploring and selecting them.

Primary area:
datasets and benchmarks
Keywords:
Recursive self-improvement, Harness, Vibe Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_gFfPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VibeTraining-Bench evaluates research agents that improve a designated target system under bounded compute and wall-clock resources. All five tasks permit changes to training data and the trainer; SWE-bench Verified and Terminal-Bench additionally permit changes to the target system's inference harness. Research agents autonomously request validation scores and execution trajectories, revise candidates, and select one system for isolated hidden-test evaluation. Six researcher-model/harness configurations improve on the initial Qwen3.5-9B-Base systems. The paper analyzes component choices, search trajectories, resource use, and engineering episodes, with the aim of evaluating coordination across training and inference.

Critical strengths
  1. Joint access to training and the deployed inference harness is a useful evaluation setting. On two tasks, researchers can respond to model changes by revising prompts, tool interfaces, and runtime behavior, while the grader retains control of task environments and scoring (§3). This extends training-side optimization to a broader system design problem. The hidden-test boundary is also a useful infrastructure contribution.

  2. The trajectory analysis provides concrete, informative evidence. An Astra SWE harness change at fixed weights improves 45 to 55 solves on 148 matched validation tasks (§5.2). Other cases show that fewer parser errors need not improve task success, and that apparent aggregate gains can disappear on common task identities (Appendices A–B). These examples provide useful lessons about evaluating autonomous experimentation.

Critical weaknesses
  1. The distinctive contribution needs more precise positioning, and the abstract combines results that do not establish the same claim. PostTrainBench already permits joint choices of training data, training methods, and hyperparameters, together with repeated experiments (§2; §5.2, “Iteration within SFT”). Accordingly, Table 1's claim that PostTrainBench does not support iterative revision appears incorrect under the table's own definition of repeated propose–evaluate–revise cycles. PostTrainBench fixes evaluation templates; the meaningful extension here is editable target inference harnesses, supported on two of five tasks, together with the protected evaluation infrastructure. I recognize this extension, but the broad co-design framing should be assessed against that narrower contribution.

    The abstract then places overall score gains, uneven component exploration, non-monotonic validation search, and validation/test ranking differences into one conclusion about reliable co-design. Overall gains and candidate-selection difficulties can also arise in training-only optimization; the highlighted τ³ result itself comes from a task with a fixed harness. Moreover, the separate H-only SWE run improves the initial system, whereas the unrestricted Kimi run ends with different weights. This does not establish that the latter missed a beneficial harness intervention at its selected checkpoint, as §4.5 appropriately acknowledges. The main scientific finding should distinguish demonstrated training–inference interaction from more general search and selection behavior.

  2. The evidence does not yet adequately calibrate what the proposed co-design benchmark measures. Table 4 covers one researcher using independent searches, rather than matched checkpoint/harness interventions, and lacks a separate SWE D/T-only run. On Terminal, unrestricted search reaches 22.4% versus 13.5% for D/T-only, but differences in the entire search trajectory prevent attributing that gap specifically to coordinated interventions. The fixed-weight Astra example is valuable local evidence; edit counts and final scores do not establish how reliably configurations identify component interactions or improve over a simple training-only recipe under equal resources. All configurations use SFT, and there is one research run per configuration/task, without uncertainty estimates. The evidence therefore supports observations about these traces more strongly than stable conclusions about researcher exploration tendencies. A benchmark need not introduce a new training algorithm, but its claimed capability requires clearer calibration.

    The result provenance also needs reconciliation. Table 3 reports final Terminal scores while Figures 9 and 11 describe the corresponding held-out receipts as pending or unavailable. These may be stale captions, but their relationship to the reported final scores is unclear. The introduction reports 13.0% for Terminal D/T-only, versus 13.5% in Table 4. Variable validation denominators also need to be reconciled with Appendix C.3's rule that timeouts and execution failures receive zero. These issues matter to a benchmark centered on candidate selection and reliable scoring.

Questions for authors
  1. Why is PostTrainBench marked as lacking iterative revision in Table 1? Can the contribution and abstract distinguish the new training–inference edit scope from existing iterative training-side optimization, and identify which evidence specifically supports coordination?
  2. What existing budget-matched training-only or fixed-recipe results calibrate the benchmark? Can matched checkpoint/harness comparisons separate the value of a harness intervention from different search trajectories? Please distinguish case observations from claims requiring repeated research runs.
  3. Can you reconcile the pending/unavailable receipts, the 13.0%/13.5% discrepancy, and changing validation denominators, and provide selected artifact identifiers with matching final receipts and per-task outcomes? What overlap checks were actually performed on the provisioned training resources, beyond relying on dataset-source reports?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

W1 & Q1: Positioning of the contribution, and evidence that separates co-design from general search behavior.

(1) Positioning relative to PostTrainBench. We agree that PostTrainBench supports iterative revision and have corrected Table 1 (iterative revision: ✗ → ✓). Our benchmark differs from PostTrainBench in two key aspects:

  • Training–inference edit scope. We open the target inference harness for modification in two of the five tasks (SWE-bench Verified and Terminal-Bench 2.1), extending the editable space beyond training data and recipes.

  • Protected evaluation infrastructure. We isolate hidden-test data and grading services from the research workspace to limit reward hacking.

(2) Mapping findings from the Abstract to our contributions and general search behavior.

  • Findings from cross-component co-design (the two {D, T, H} tasks): We ground all co-design claims in SWE-bench Verified and Terminal-Bench 2.1. The Terminal 2×2 experiment (see (3a) below) provides direct evidence of model–harness interaction, while Table 4 and the additional SWE experiment (see (3b) below) reveal uneven component exploration.
  • Findings from protected evaluation (across all five tasks): Our protected evaluation protocol keeps hidden-test data and grading services inaccessible during research, enabling trustworthy validation-to-test comparisons under broad component edit access. These comparisons reveal cases where validation leaders do not lead on the hidden test (Section 4.3, Fig. 3).
  • Findings from general search behavior (across all five tasks): The recorded validation trajectories reveal non-monotonic search behavior (Section 4.3, Fig. 3).

Accordingly, we have revised the Abstract (Lines 24–36) to distinguish findings specific to cross-component co-design from those enabled by protected evaluation and more general search observations.

(3) Controlled evidence for co-design. We add two controlled experiments:

(a) Terminal-Bench 2.1 (Kimi-K3). We evaluate the initial and final selected models with the initial and evolved harnesses from the same Kimi-K3 unrestricted run.

ModelInitial harnessEvolved harness
Initial model8.98.9
Final selected model15.722.4

With the initial model fixed, harness editing yields no improvement (8.9 vs. 8.9). With the initial harness fixed, training improves performance from 8.9 to 15.7. Combining the trained model with the evolved harness further improves performance to 22.4. This provides controlled evidence of checkpoint-dependent model–harness interaction: harness editing alone does not improve the initial model but becomes beneficial after training improves the model's capabilities (+6.7 percentage points). These results show that the benefit of harness interventions can depend on the current state of the target system, motivating research agents to adaptively coordinate training and inference improvements.

(b) SWE-bench Verified (Kimi-K3). Starting from Kimi-K3's final selected model from the unrestricted run, we conduct an additional 6-hour harness-only optimization with model weights fixed.

ModelInitial harnessEvolved harness
Final selected model51.753.4

With the same checkpoint, harness editing improves performance by 1.7 percentage points. This reveals a missed opportunity for harness exploration in Kimi-K3's original unrestricted run: despite having access to harness editing, the agent left unexplored a potentially beneficial direction that further improved its selected system. This illustrates how our benchmark can identify shortcomings in cross-component co-design. We include these results in Table 4 (b) and Lines 371–379.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

W2 & Q2: Baseline calibration, controlled comparisons, and multi-run stability

(1) Budget-matched fixed-recipe SFT baseline and SWE D/T-only reference. We add a fixed-recipe SFT baseline on SWE-bench Verified using the initial data pool, default trainer, and initial inference harness, under the same 24-hour research budget and GPU allocation:

MethodSWE-bench Verified Test
Fixed-recipe SFT (24h)46.0
GPT-6-Astra (full co-design)53.7

GPT-6-Astra's unrestricted research outperforms the fixed-recipe baseline by 7.7 percentage points. This baseline provides a budget-matched reference for evaluating the gains from autonomous system optimization. Regarding the missing SWE D/T-only reference in Table 4, Kimi-K3's unrestricted D/T/H run made zero harness edits, thus providing an observed D/T-only reference. We include these results in the appendix (Lines 1153–1173).

(2) Matched checkpoint/harness comparisons. As detailed in W1 & Q1 (3a)–(3b), our two new experiments measure the effect of harness changes while holding model weights fixed, avoiding confounding from comparisons between independent search trajectories. On Terminal-Bench 2.1, applying the evolved harness yields no gain on the initial model (8.9 vs. 8.9) but improves the trained model from 15.7 to 22.4 (+6.7), showing that harness optimization can further improve a trained model. On SWE-bench Verified, harness-only optimization on Kimi-K3's fixed final checkpoint improves hidden-test performance from 51.7 to 53.4.

(3) Repeated runs and distinguishing case observations from multi-run claims. Due to the short rebuttal period, we provide two independent runs for Kimi-K3 and GPT-6-Astra on SWE-bench Verified and Terminal-Bench 2.1 (the two tasks opening the full {D, T, H} space), as well as 3-Bench (mean ± std):

Research agentSWE ValSWE TestTerminal ValTerminal Test3-Bench Val3-Bench Test
Kimi-K341.4 ± 0.949.6 ± 3.025.0 ± 1.420.2 ± 3.158.6 ± 2.150.8 ± 1.9
GPT-6-Astra45.7 ± 0.553.3 ± 0.625.0 ± 1.413.0 ± 0.862.9 ± 4.049.3 ± 0.6

Across both runs, our core multi-run claims hold:

  • Relative performance is consistent across runs. GPT-6-Astra outperforms Kimi-K3 on SWE-bench Verified (53.3 vs. 49.6), while Kimi-K3 outperforms GPT-6-Astra on Terminal-Bench 2.1 (20.2 vs. 13.0). On 3-Bench, Kimi-K3 also achieves a higher mean hidden-test score than GPT-6-Astra (50.8 vs. 49.3).

  • Validation performance does not fully predict hidden-test performance. On Terminal-Bench 2.1, Kimi-K3 and GPT-6-Astra achieve the same mean validation score (25.0), yet GPT-6-Astra scores 13.0 on the hidden test, below Kimi-K3 (20.2) and Qwen-3.8-Max in Table 3 (17.9). Similarly, on 3-Bench, GPT-6-Astra achieves a higher mean validation score (62.9 vs. 58.6), yet trails Kimi-K3 in mean hidden-test performance (49.3 vs. 50.8).

  • Kimi-K3 consistently leaves the SWE harness unchanged. Kimi-K3 made 0 harness edits in both runs.

These repeated runs provide preliminary evidence for the consistency of these observations. We include these results in the appendix (Lines 1095–1127).

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

W2 & Q3: Reporting consistency, artifact provenance, and overlap checks.

  • Typo and stale captions. We fixed the Introduction typo for Terminal D/T-only (13.0% to 13.5%, matching Table 4) and updated the outdated captions in Figures 9 and 11.

  • Validation denominators. Case-study figures originally quoted raw runner logs that reported the completed-task count as the denominator, whereas all final hidden-test scores are computed over the full test set. To avoid confusion, we have updated these figures to use the full requested validation set (counting timeouts as zero), aligning reporting with Appendix C.3.

  • Artifact provenance and final receipts. We reconciled the selected candidates in Figures 9 and 11 with their final grading jobs over the full 89-task Terminal-Bench 2.1 hidden-test partition: GPT-5.6-Sol (c07_balanced_marker_resume, job ID fa00378fae78431f, 8/89 solved) and GPT-6-Astra (c05_recovery_mixed, job ID f592f5fddf4147ec, 12/89 solved). We additionally provide the binary outcomes for all 89 tasks in Appendix H (Table 12, Lines 1193–1237).

  • Overlap checks. Beyond dataset-source reports (Appendix C.2), we directly measured 13-gram overlap between all provisioned training pools and our validation/hidden-test sets. For each evaluation problem, we computed the fraction of its unique 13-grams that appear in the union of 13-grams across the corresponding training pool, and reported the mean ratio across all problems:

BenchmarkValTest
Terminal-Bench 2.10.0030.018
SWE-bench Verified0.0030.004
3-Bench00
Toolathlon00
AIME0.0120.002

The overlap ratios are consistently low, with the highest mean overlap being 0.018 and no detected 13-gram matches on 3-Bench and Toolathlon. These results suggest limited direct textual overlap between the provisioned training resources and evaluation problems. We have included this analysis in Appendix C.2 (Lines 1043–1049 and Table 7).

Reviewer_mh4qOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VibeTraining-Bench to evaluate agents that improve a target system by modifying its training data, trainer, and inference harness. It includes five tasks, two of which allow changes to all three components. Research agents work within fixed time and GPU budgets, use validation scores and execution traces to guide their experiments, and submit a final system for hidden-test evaluation. The authors evaluate six research-agent configurations and analyze both their final performance and the interventions made during the research process.

Critical strengths
  1. The benchmark studies a useful problem: whether an agent can identify which part of a system needs improvement and adjust its approach as experiments reveal new problems. Allowing changes to both training and inference in the same research run makes it possible to study decisions that are difficult to evaluate when only one component is editable. This is a worthwhile setting for research on automated agent development.

  2. The experiments cover several types of tasks, and the case studies help explain what the research agents actually do. I found the examples of environment selection and stopping stalled execution informative. The authors also report changes that fail, such as parser repairs that reduce errors but do not improve task success. These observations make the paper more useful than a comparison of final scores alone.

Critical weaknesses
  1. The experiments do not clearly establish what is gained from coordinating changes across components. An improved final system could result from several independently useful changes, without requiring the co-design capability emphasized in the paper. The component ablations use separate research runs, so differences between them do not isolate this effect. The case studies show agents switching between components, but provide limited evidence that one intervention changes the value of another. A controlled comparison using the original and evolved harnesses with both the initial and trained checkpoints would help establish whether such interactions matter.
  2. The paper gives limited attention to how well the evolved harness changes transfer. The hidden-test results, particularly the SWE harness-only experiment, do provide evidence of improvement on unseen instances. However, the detailed examples mostly report validation results, and the final results often combine changes to the model and harness. It is therefore difficult to tell which harness changes remain useful beyond the setting in which they were developed. Testing selected changes with another checkpoint or a compatible task environment would help distinguish reusable improvements from narrow adaptations. RRSI [1] studies this distinction explicitly and would be useful context for the discussion. My concern is the limited analysis of transfer, rather than the absence of a comparison with this concurrent work.

[1] Xia et al. RRSI: Regularized Recursive Self-Improvement of Agent Harnesses. 2026. https://arxiv.org/abs/2609.24972

Questions for authors
  1. Do you have controlled results showing that the benefit of a harness change depends on the accompanying model or training changes? For example, how do the original and evolved harnesses perform with the initial and trained checkpoints?
  2. Do the improvements from an evolved harness generalize to other compatible datasets without further optimization, when compared with the original harness using the same model weights and evaluation budget?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

W1 & Q1: Controlled evidence of model–harness interaction.

We conduct a controlled 2×2 evaluation on the Terminal-Bench 2.1 hidden test set. We combine the initial and final selected model checkpoints with the initial and evolved harnesses from the same Kimi-K3 unrestricted run.

ModelInitial harnessEvolved harness
Initial model8.98.9
Trained model15.722.4

These results provide direct evidence of interaction between model training and harness evolution. With the initial model, the evolved harness yields no improvement (8.9 vs. 8.9). However, after model training, the initial harness becomes a performance constraint: replacing it with the evolved harness improves performance from 15.7 to 22.4. These findings provide controlled evidence of model–harness complementarity, highlighting the benefits of coordinating model and harness evolution. We include these results in Table 4 (b) and Lines 375–379.


W2 & Q2: Generalization of evolved harnesses.

Thank you for this valuable suggestion. Following the cross-benchmark transfer evaluation setting in RRSI [1], we use SWE-bench Verified and Terminal-Bench 2.1 as the in-domain and out-of-domain environments, respectively. We evaluate the initial and evolved mini-SWE-agent harnesses from the Kimi-K3 SWE harness-only run using the same base model checkpoint and evaluation budget, without further optimization.

Evaluation benchmarkInitial SWE harnessEvolved SWE harness
SWE-bench Verified (in-domain)32.938.0
Terminal-Bench 2.1 (OOD, mini-SWE-agent)10.17.9

While the evolved harness improves performance on SWE-bench Verified from 32.9 to 38.0, its performance on Terminal-Bench 2.1 drops from 10.1 to 7.9. These results suggest that agent-discovered harness improvements do not necessarily transfer beyond the environment in which they were developed, which reflects task-specific adaptation. This observation is consistent with the generalization challenges discussed in RRSI [1]. We have added this cross-task analysis and cited RRSI [1] in the revised paper (Appendix E, Lines 1128-1151).

[1] Xia et al. RRSI: Regularized Recursive Self-Improvement of Agent Harnesses. 2026. https://arxiv.org/abs/2609.24972

AuthorsOpenReview ↗

A kind reminder as the deadline is approaching

Dear Reviewer,

We again express our deep gratitude to you for spending the time and efforts on our submission. Because it is quite close to the end of rebuttal, we deeply appreciate that the remaining questions / concerns, if any, can be posted earlier, such that we have enough time to address them. We truly hope that our response can clarify your concerns.

We fully understand that it may take time to read our rebuttal in-depth. If there is any additional explanation or experiments that can save your time to understand our paper and clarify the concerns, we will be more than happy to do so.

Respectfully,

Authors

Thank you for the additional experiments, which largely address both concerns. I maintain my original rating.

AuthorsOpenReview ↗

Reply Rebuttal Comment by Authors

Thank you very much for acknowledging our additional experiments and for maintaining your positive assessment. We are glad that our rebuttal largely addressed your concerns, and we truly appreciate your constructive feedback, which has helped strengthen the revised manuscript.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

W1 & Q1: Robustness across independent runs

Due to the short rebuttal period, we prioritized two independent runs for Kimi-K3 and GPT-6-Astra on SWE-bench Verified and Terminal-Bench 2.1 (the two tasks that open the full {D, T, H} editable space), as well as τ3\tau^3-Bench. Below we report the validation and hidden-test scores of the selected systems across the two runs (mean ± std).

Research agentSWE ValSWE TestTerminal ValTerminal Testτ3\tau^3-Bench Valτ3\tau^3-Bench Test
Kimi-K341.4 ± 0.949.6 ± 3.025.0 ± 1.420.2 ± 3.158.6 ± 2.150.8 ± 1.9
GPT-6-Astra45.7 ± 0.553.3 ± 0.625.0 ± 1.413.0 ± 0.862.9 ± 4.049.3 ± 0.6

The key findings remain consistent across runs:

(1) Cross-task leadership remains consistent across the evaluated runs. GPT-6-Astra leads on SWE-bench Verified (53.3 vs. 49.6), while Kimi-K3 leads on Terminal-Bench 2.1 (20.2 vs. 13.0). On τ3\tau^3-Bench, Kimi-K3 also achieves a slightly higher mean hidden-test score than GPT-6-Astra (50.8 vs. 49.3). Overall, these results support our finding that no single research-agent configuration leads across all tasks.

(2) Validation leaders do not always lead on the hidden test. On Terminal-Bench 2.1, Kimi-K3 and GPT-6-Astra tie as validation co-leaders at 25.0, the highest among all six configurations. However, GPT-6-Astra scores only 13.0 on the hidden test, below Kimi-K3 (20.2) and Qwen-3.8-Max in Table 3 (17.9). Similarly, on τ3\tau^3-Bench, GPT-6-Astra achieves a higher mean validation score (62.9 vs. 58.6), yet trails Kimi-K3 in mean hidden-test performance (49.3 vs. 50.8). These results further illustrate that strong validation performance does not necessarily translate to strong hidden-test performance.

(3) Kimi-K3 consistently leaves the SWE harness unchanged. Kimi-K3 made 0 harness edits in both runs.

We have included these results in the revised appendix (Lines 1095–1127).


Q2: Controlled analysis of harness contributions

We conduct two additional experiments to examine the contribution of harness evolution.

(a) Terminal-Bench 2.1 (Kimi-K3). We evaluate the initial and evolved harnesses from the same unrestricted run with the final selected model fixed.

ModelInitial harnessEvolved harness
Final selected model15.722.4

With model weights fixed, the evolved harness improves hidden-test performance from 15.7 to 22.4.

(b) SWE-bench Verified (Kimi-K3). Starting from the final selected model of the unrestricted run, we conduct an additional 6-hour harness-only optimization with model weights fixed.

ModelInitial harnessEvolved harness
Final selected model51.753.4

With model weights fixed, the additional harness-only optimization improves hidden-test performance from 51.7 to 53.4. Since Kimi-K3 did not edit the SWE harness during its original unrestricted run, this result provides evidence of a potentially beneficial improvement direction that Kimi-K3 failed to explore during that run. We have included these results in Table 4 (b) and Lines 371–379.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W2 & Q3: Specialized co-design harness, accessibility, and base-model diversity

(1) Specialized co-design harnesses. To our knowledge, there is currently no research-agent harness specifically designed for open-ended cross-component co-design. Existing works such as Harnessforge [1] and CoHARL [2] focus on co-evolving model policies and inference harnesses through predefined optimization procedures but do not serve as autonomous research-agent harnesses analogous to Codex or Claude Code. Our benchmark establishes strong coding-agent baselines for this open-ended co-design setting while providing a standardized platform for future purpose-built research agents.

(2) Smaller-budget accessibility. Co-design targets agentic tasks that require iterative model training and multi-turn environment rollouts, a process that naturally takes time. Importantly, the budget in our protocol is configurable. Researchers can run evaluations under a smaller uniform budget (e.g., 12 hours) or with a smaller base model like Qwen3.5-4B-Base. Comparisons remain fair as long as all compared agents share the same budget.

(3) Different base model. Given the limited rebuttal period, we additionally evaluated Kimi-K3 on SWE-bench Verified using Qwen3.5-4B-Base.

SystemSWE-bench Verified
Initial system19.4
Final system39.7

On Qwen3.5-4B-Base, Kimi-K3 improves the target system from 19.4 to 39.7, and exploration across {D, T, H} remains uneven at this smaller scale. This result provides preliminary evidence that substantial system gains and uneven component exploration can also occur with a smaller base model. We plan to conduct additional experiments with the 4B base model. We have included the result in Appendix G (Lines 1176-1191).

[1] Chen, Mingju, et al. "Harnessforge: Joint harness and policy evolution for adaptive agent systems." arXiv preprint arXiv:2606.01779 (2026).

[2] Chen, Jianwen, et al. "CoHARL: Co-Evolving Agent Policies and Executable Harnesses with Reinforcement Learning." arXiv preprint arXiv:2610.36469 (2026).

AuthorsOpenReview ↗

General Response

We thank all reviewers and the AC for their time and careful review. We appreciate the reviewers' recognition of the importance of the problem, the value of our evaluation setting, and the insights provided by our empirical findings (KRuV, mh4q, gFfP). Following their constructive suggestions, we have uploaded a revised paper (changes highlighted in blue) and carefully responded to every raised concern. Below, we summarize the key revisions:

  • Controlled model–harness interaction (KRuV, mh4q, gFfP): Added a controlled 2×2 evaluation crossing the initial and final models with the initial and evolved harnesses on Terminal-Bench 2.1 to demonstrate model–harness interaction, along with a fixed-weight harness-only optimization experiment on Kimi-K3's unrestricted SWE checkpoint showing further room for improvement (Section 4.5, Table 4(b)).
  • Multi-run stability (KRuV, gFfP): Reported results from two independent runs each for Kimi-K3 and GPT-6-Astra across SWE, Terminal, and τ3\tau^3-Bench, showing consistent per-task leaders, validation–test discrepancies, and zero SWE harness edits by Kimi-K3 across both runs (Appendix D, Table 8).
  • Cross-benchmark harness transfer (mh4q): Evaluated the transfer of Kimi-K3's harness evolved on SWE-bench Verified to Terminal-Bench 2.1 without further optimization, and added discussion of RRSI (Appendix E, Table 9).
  • Budget-matched SFT calibration (gFfP): Added a 24h fixed-recipe SFT baseline on SWE-bench Verified under the same compute budget (Appendix F, Table 10).
  • Smaller base-model scale (KRuV): Evaluated Kimi-K3 on SWE-bench Verified starting from Qwen3.5-4B-Base (Appendix G, Table 11).
  • Overlap audit and provenance (gFfP): Measured 13-gram overlap across all five benchmarks, fixed the Introduction typo, aligned validation denominators, and added 89-task Terminal-Bench 2.1 receipts and job IDs (Appendix C.2, Table 7; Figures 9 & 11; Appendix H, Table 12).
  • Positioning and Abstract Revision (gFfP): Corrected our characterization of PostTrainBench's iterative revision capability and revised the Abstract to clearly distinguish our contributions (Abstract, Table 1).
AuthorsOpenReview ↗

A kind reminder as the deadline is approaching

Dear Reviewer,

We again express our deep gratitude to you for spending the time and efforts on our submission. Because it is quite close to the end of rebuttal, we deeply appreciate that the remaining questions / concerns, if any, can be posted earlier, such that we have enough time to address them. We truly hope that our response can clarify your concerns.

We fully understand that it may take time to read our rebuttal in-depth. If there is any additional explanation or experiments that can save your time to understand our paper and clarify the concerns, we will be more than happy to do so.

Respectfully,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.