- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VibeTraining-Bench evaluates research agents that improve a designated target system under bounded compute and wall-clock resources. All five tasks permit changes to training data and the trainer; SWE-bench Verified and Terminal-Bench additionally permit changes to the target system's inference harness. Research agents autonomously request validation scores and execution trajectories, revise candidates, and select one system for isolated hidden-test evaluation. Six researcher-model/harness configurations improve on the initial Qwen3.5-9B-Base systems. The paper analyzes component choices, search trajectories, resource use, and engineering episodes, with the aim of evaluating coordination across training and inference.
Critical strengths
-
Joint access to training and the deployed inference harness is a useful evaluation setting. On two tasks, researchers can respond to model changes by revising prompts, tool interfaces, and runtime behavior, while the grader retains control of task environments and scoring (§3). This extends training-side optimization to a broader system design problem. The hidden-test boundary is also a useful infrastructure contribution.
-
The trajectory analysis provides concrete, informative evidence. An Astra SWE harness change at fixed weights improves 45 to 55 solves on 148 matched validation tasks (§5.2). Other cases show that fewer parser errors need not improve task success, and that apparent aggregate gains can disappear on common task identities (Appendices A–B). These examples provide useful lessons about evaluating autonomous experimentation.
Critical weaknesses
-
The distinctive contribution needs more precise positioning, and the abstract combines results that do not establish the same claim. PostTrainBench already permits joint choices of training data, training methods, and hyperparameters, together with repeated experiments (§2; §5.2, “Iteration within SFT”). Accordingly, Table 1's claim that PostTrainBench does not support iterative revision appears incorrect under the table's own definition of repeated propose–evaluate–revise cycles. PostTrainBench fixes evaluation templates; the meaningful extension here is editable target inference harnesses, supported on two of five tasks, together with the protected evaluation infrastructure. I recognize this extension, but the broad co-design framing should be assessed against that narrower contribution.
The abstract then places overall score gains, uneven component exploration, non-monotonic validation search, and validation/test ranking differences into one conclusion about reliable co-design. Overall gains and candidate-selection difficulties can also arise in training-only optimization; the highlighted τ³ result itself comes from a task with a fixed harness. Moreover, the separate H-only SWE run improves the initial system, whereas the unrestricted Kimi run ends with different weights. This does not establish that the latter missed a beneficial harness intervention at its selected checkpoint, as §4.5 appropriately acknowledges. The main scientific finding should distinguish demonstrated training–inference interaction from more general search and selection behavior.
-
The evidence does not yet adequately calibrate what the proposed co-design benchmark measures. Table 4 covers one researcher using independent searches, rather than matched checkpoint/harness interventions, and lacks a separate SWE D/T-only run. On Terminal, unrestricted search reaches 22.4% versus 13.5% for D/T-only, but differences in the entire search trajectory prevent attributing that gap specifically to coordinated interventions. The fixed-weight Astra example is valuable local evidence; edit counts and final scores do not establish how reliably configurations identify component interactions or improve over a simple training-only recipe under equal resources. All configurations use SFT, and there is one research run per configuration/task, without uncertainty estimates. The evidence therefore supports observations about these traces more strongly than stable conclusions about researcher exploration tendencies. A benchmark need not introduce a new training algorithm, but its claimed capability requires clearer calibration.
The result provenance also needs reconciliation. Table 3 reports final Terminal scores while Figures 9 and 11 describe the corresponding held-out receipts as pending or unavailable. These may be stale captions, but their relationship to the reported final scores is unclear. The introduction reports 13.0% for Terminal D/T-only, versus 13.5% in Table 4. Variable validation denominators also need to be reconciled with Appendix C.3's rule that timeouts and execution failures receive zero. These issues matter to a benchmark centered on candidate selection and reliable scoring.
Questions for authors
- Why is PostTrainBench marked as lacking iterative revision in Table 1? Can the contribution and abstract distinguish the new training–inference edit scope from existing iterative training-side optimization, and identify which evidence specifically supports coordination?
- What existing budget-matched training-only or fixed-recipe results calibrate the benchmark? Can matched checkpoint/harness comparisons separate the value of a harness intervention from different search trajectories? Please distinguish case observations from claims requiring repeated research runs.
- Can you reconcile the pending/unavailable receipts, the 13.0%/13.5% discrepancy, and changing validation denominators, and provide selected artifact identifiers with matching final receipts and per-task outcomes? What overlap checks were actually performed on the provisioned training resources, beyond relying on dataset-source reports?
6 comments
Rebuttal by Authors (2/3)
W2 & Q2: Baseline calibration, controlled comparisons, and multi-run stability
(1) Budget-matched fixed-recipe SFT baseline and SWE D/T-only reference. We add a fixed-recipe SFT baseline on SWE-bench Verified using the initial data pool, default trainer, and initial inference harness, under the same 24-hour research budget and GPU allocation:
| Method | SWE-bench Verified Test |
|---|---|
| Fixed-recipe SFT (24h) | 46.0 |
| GPT-6-Astra (full co-design) | 53.7 |
GPT-6-Astra's unrestricted research outperforms the fixed-recipe baseline by 7.7 percentage points. This baseline provides a budget-matched reference for evaluating the gains from autonomous system optimization. Regarding the missing SWE D/T-only reference in Table 4, Kimi-K3's unrestricted D/T/H run made zero harness edits, thus providing an observed D/T-only reference. We include these results in the appendix (Lines 1153–1173).
(2) Matched checkpoint/harness comparisons. As detailed in W1 & Q1 (3a)–(3b), our two new experiments measure the effect of harness changes while holding model weights fixed, avoiding confounding from comparisons between independent search trajectories. On Terminal-Bench 2.1, applying the evolved harness yields no gain on the initial model (8.9 vs. 8.9) but improves the trained model from 15.7 to 22.4 (+6.7), showing that harness optimization can further improve a trained model. On SWE-bench Verified, harness-only optimization on Kimi-K3's fixed final checkpoint improves hidden-test performance from 51.7 to 53.4.
(3) Repeated runs and distinguishing case observations from multi-run claims. Due to the short rebuttal period, we provide two independent runs for Kimi-K3 and GPT-6-Astra on SWE-bench Verified and Terminal-Bench 2.1 (the two tasks opening the full {D, T, H} space), as well as 3-Bench (mean ± std):
| Research agent | SWE Val | SWE Test | Terminal Val | Terminal Test | 3-Bench Val | 3-Bench Test |
|---|---|---|---|---|---|---|
| Kimi-K3 | 41.4 ± 0.9 | 49.6 ± 3.0 | 25.0 ± 1.4 | 20.2 ± 3.1 | 58.6 ± 2.1 | 50.8 ± 1.9 |
| GPT-6-Astra | 45.7 ± 0.5 | 53.3 ± 0.6 | 25.0 ± 1.4 | 13.0 ± 0.8 | 62.9 ± 4.0 | 49.3 ± 0.6 |
Across both runs, our core multi-run claims hold:
-
Relative performance is consistent across runs. GPT-6-Astra outperforms Kimi-K3 on SWE-bench Verified (53.3 vs. 49.6), while Kimi-K3 outperforms GPT-6-Astra on Terminal-Bench 2.1 (20.2 vs. 13.0). On 3-Bench, Kimi-K3 also achieves a higher mean hidden-test score than GPT-6-Astra (50.8 vs. 49.3).
-
Validation performance does not fully predict hidden-test performance. On Terminal-Bench 2.1, Kimi-K3 and GPT-6-Astra achieve the same mean validation score (25.0), yet GPT-6-Astra scores 13.0 on the hidden test, below Kimi-K3 (20.2) and Qwen-3.8-Max in Table 3 (17.9). Similarly, on 3-Bench, GPT-6-Astra achieves a higher mean validation score (62.9 vs. 58.6), yet trails Kimi-K3 in mean hidden-test performance (49.3 vs. 50.8).
-
Kimi-K3 consistently leaves the SWE harness unchanged. Kimi-K3 made 0 harness edits in both runs.
These repeated runs provide preliminary evidence for the consistency of these observations. We include these results in the appendix (Lines 1095–1127).
Rebuttal by Authors (3/3)
W2 & Q3: Reporting consistency, artifact provenance, and overlap checks.
-
Typo and stale captions. We fixed the Introduction typo for Terminal D/T-only (13.0% to 13.5%, matching Table 4) and updated the outdated captions in Figures 9 and 11.
-
Validation denominators. Case-study figures originally quoted raw runner logs that reported the completed-task count as the denominator, whereas all final hidden-test scores are computed over the full test set. To avoid confusion, we have updated these figures to use the full requested validation set (counting timeouts as zero), aligning reporting with Appendix C.3.
-
Artifact provenance and final receipts. We reconciled the selected candidates in Figures 9 and 11 with their final grading jobs over the full 89-task Terminal-Bench 2.1 hidden-test partition: GPT-5.6-Sol (c07_balanced_marker_resume, job ID fa00378fae78431f, 8/89 solved) and GPT-6-Astra (c05_recovery_mixed, job ID f592f5fddf4147ec, 12/89 solved). We additionally provide the binary outcomes for all 89 tasks in Appendix H (Table 12, Lines 1193–1237).
-
Overlap checks. Beyond dataset-source reports (Appendix C.2), we directly measured 13-gram overlap between all provisioned training pools and our validation/hidden-test sets. For each evaluation problem, we computed the fraction of its unique 13-grams that appear in the union of 13-grams across the corresponding training pool, and reported the mean ratio across all problems:
| Benchmark | Val | Test |
|---|---|---|
| Terminal-Bench 2.1 | 0.003 | 0.018 |
| SWE-bench Verified | 0.003 | 0.004 |
| 3-Bench | 0 | 0 |
| Toolathlon | 0 | 0 |
| AIME | 0.012 | 0.002 |
The overlap ratios are consistently low, with the highest mean overlap being 0.018 and no detected 13-gram matches on 3-Bench and Toolathlon. These results suggest limited direct textual overlap between the provisioned training resources and evaluation problems. We have included this analysis in Appendix C.2 (Lines 1043–1049 and Table 7).
Rebuttal by Authors (1/3)
W1 & Q1: Positioning of the contribution, and evidence that separates co-design from general search behavior.
(1) Positioning relative to PostTrainBench. We agree that PostTrainBench supports iterative revision and have corrected Table 1 (iterative revision: ✗ → ✓). Our benchmark differs from PostTrainBench in two key aspects:
Training–inference edit scope. We open the target inference harness for modification in two of the five tasks (SWE-bench Verified and Terminal-Bench 2.1), extending the editable space beyond training data and recipes.
Protected evaluation infrastructure. We isolate hidden-test data and grading services from the research workspace to limit reward hacking.
(2) Mapping findings from the Abstract to our contributions and general search behavior.
{D, T, H}tasks): We ground all co-design claims in SWE-bench Verified and Terminal-Bench 2.1. The Terminal 2×2 experiment (see (3a) below) provides direct evidence of model–harness interaction, while Table 4 and the additional SWE experiment (see (3b) below) reveal uneven component exploration.Accordingly, we have revised the Abstract (Lines 24–36) to distinguish findings specific to cross-component co-design from those enabled by protected evaluation and more general search observations.
(3) Controlled evidence for co-design. We add two controlled experiments:
(a) Terminal-Bench 2.1 (Kimi-K3). We evaluate the initial and final selected models with the initial and evolved harnesses from the same Kimi-K3 unrestricted run.
With the initial model fixed, harness editing yields no improvement (8.9 vs. 8.9). With the initial harness fixed, training improves performance from 8.9 to 15.7. Combining the trained model with the evolved harness further improves performance to 22.4. This provides controlled evidence of checkpoint-dependent model–harness interaction: harness editing alone does not improve the initial model but becomes beneficial after training improves the model's capabilities (+6.7 percentage points). These results show that the benefit of harness interventions can depend on the current state of the target system, motivating research agents to adaptively coordinate training and inference improvements.
(b) SWE-bench Verified (Kimi-K3). Starting from Kimi-K3's final selected model from the unrestricted run, we conduct an additional 6-hour harness-only optimization with model weights fixed.
With the same checkpoint, harness editing improves performance by 1.7 percentage points. This reveals a missed opportunity for harness exploration in Kimi-K3's original unrestricted run: despite having access to harness editing, the agent left unexplored a potentially beneficial direction that further improved its selected system. This illustrates how our benchmark can identify shortcomings in cross-component co-design. We include these results in Table 4 (b) and Lines 371–379.