acceptodds
Under review as a conference paper at ICLR 2027

HarnessTTT: Learn to Optimize Harness at Test-Time for Scientific Discovery

Abstract

Scientific discovery with language models requires finding the best valid solution within a finite computational budget. We study how feedback from this search can improve the procedure used to conduct it. We introduce HarnessTTT, a test-time learning framework that trains a harness proposer while keeping the executor frozen. The proposer generates executable harnesses comprising system prompts, reusable skills, tools, middleware, and execution controls. Each harness guides a budgeted discovery rollout, whose outcome provides feedback for updating the proposer. A discovery-oriented policy-gradient objective combines gap-normalized rewards, leave-one-out advantages (RLOO), and max-sharpening to favor high-performing proposals. Generated components are validated before execution, and the search retains the best valid solution and selected harnesses across rounds. Across thirteen discovery tasks in mathematics, systems, heuristic programming, astrodynamics, and biology, HarnessTTT with a frozen 9B executor improves on OpenEvolve with the same executor on all eleven benchmark tasks (e.g., from 1.17 to 2.635 on circle packing), improves on the discovery-fine-tuned Finch-9B by up to 92%, holds the best open-weight sub-10B result on nine, and reaches 0.380919 on Erdős minimum overlap, better than the best human reference of 0.380927. A frozen 27B executor improves on the 9B executor on eleven of thirteen tasks, and transferring the learned harnesses to larger executors matches or exceeds the strongest reported system of any backbone on eight of thirteen tasks and comes within 1% on four more. Together, these results show that harness search can itself be learned at test time: a trained proposer lets a small frozen executor search effectively, and the harnesses it finds carry over to stronger executors.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test time training, harness, scientific discover

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_VgzAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces HarnessTTT, a test-time RL framework that trains a harness proposer while keeping the executor frozen. The proposer generates executable harnesses and learns from discovery rollouts. Experiments on 13 scientific discovery tasks show strong performance and promising cross-executor transferability.

Critical strengths
  • Compelling motivation. Learning to improve the search procedure rather than the executor itself is an interesting direction, especially given the potential for reusable and transferable harnesses.

  • Strong empirical results. The method achieves competitive results across diverse scientific discovery tasks, with promising improvements when transferring learned harnesses to stronger executors.

Critical weaknesses
  • Incomplete compute comparison. HarnessTTT incurs additional costs from proposer inference, RL training, harness validation, and tool execution. Comparisons under matched GPU-hours, token budgets, and total evaluation costs would make the results more convincing.

  • Insufficient RL ablations. The paper lacks component-wise ablations of its RL design, such as near-tie filtering and max-sharpening, making it difficult to identify which components contribute most to the improvements.

  • Limited evidence of generalization. While cross-executor replay demonstrates harness transferability, it remains unclear whether the trained proposer can generalize its harness optimization ability to unseen tasks.

Questions for authors

see Weaknesses

3 comments

We thank the reviewer for the constructive feedback. In response, we report a full GPU-hour accounting with same-executor baselines at comparable compute, add a three-seed component-wise ablation of the RL objective, and clarify the intended scope of generalization. All new results will be included in the revised version.

1. Compute Comparison (W1)

W1: Incomplete compute comparison. HarnessTTT incurs additional costs from proposer inference, RL training, harness validation, and tool execution. Comparisons under matched GPU-hours, token budgets, and total evaluation costs would make the results more convincing.

All model costs are counted in GPU-hours. We report H100 GPU-hours from our job logs. They cover proposer inference (including harness validation and tool repair), executor rollouts (including generated-tool execution and paired-control rollouts), and LoRA training. All models in these campaigns are served on our own GPUs, so token usage is reflected in GPU time. We did not aggregate campaign-level token totals and therefore use GPU-hours as the common budget across methods. Evaluation is bounded per rollout by B = 20 full evaluator calls for the 9B campaign and B = 5 for the 27B campaign; subsampled probe() calls are capped separately (Appendix D). Ranges below are across tasks, so components need not sum exactly to the totals. Appendix E describes the paired-control rollouts incorrectly; we will correct it in the revised version.

Component per roundCount (9B / 27B)GPU-hours (9B / 27B)
Proposer episodes (incl. validation and repair)32 / 84.5–7.0 / 1.3–2.4
Executor rollouts (candidate + paired control)64 / 167.7–12.6 / 2.0–6.6
LoRA update (3 epochs, 8 GPUs)1 / 1≈ 5.9 / 1.3–3.1
Total per round96 + 1 / 24 + 118.7–25.5 / 7.5–10.0
Total per task (× 20 rounds)1,920 + 20 / 480 + 20370–510 / 150–200

We ran ShinkaEvolve and GEPA with the same frozen Qwen3.5-9B executor and evaluator at about 400 GPU-hours each. Both evolve programs rather than harnesses, so we match total GPU-hours rather than evaluator calls per trajectory. We list OpenEvolve from Table 1 with its measured cost.

TaskHarnessTTT-9BOpenEvolve (Table 1)ShinkaEvolveGEPA
CP26 (sum of radii ↑)2.635983 (≈ 390 GPU-h)1.172702 (≈ 360 GPU-h)2.3379 (≈ 400 GPU-h)2.2296 (≈ 400 GPU-h)
Hadamard (det ratio ↑)0.921053 (≈ 370 GPU-h)0.397184 (≈ 360 GPU-h)0.5742 (≈ 400 GPU-h)0.4642 (≈ 400 GPU-h)
AC2 (c2 ↑)0.933900 (≈ 430 GPU-h)0.880100 (≈ 360 GPU-h)0.9245 (≈ 400 GPU-h)0.8558 (≈ 400 GPU-h)

At comparable compute, HarnessTTT finds the best solution on all three tasks, and on CP26 and Hadamard it uses less compute than ShinkaEvolve and GEPA. GEPA did not improve on the initial program on AC2. Across Table 1, the OpenEvolve runs used about 360 GPU-hours per task, somewhat below HarnessTTT-9B's 370–510, and HarnessTTT improves on them on all eleven tasks (nine of nine excluding the quarantined LLM-SQL and Txn). Our new 15-round campaigns (Section 2) use about three quarters of a full campaign (≈ 280–380 GPU-hours), similar to or below OpenEvolve's ≈ 360, and the full method's three-seed means still exceed OpenEvolve's Table 1 results on all four tasks tested there.

2. Component-wise RL Ablations (W2)

W2: Insufficient RL ablations. The paper lacks component-wise ablations of its RL design, such as near-tie filtering and max-sharpening, making it difficult to identify which components contribute most to the improvements.

We ablated the main components of the objective, including standard RLOO. We used three seeds on four tasks, with the frozen Qwen3.5-9B executor, K = 32 proposals per round, B = 20, and 15 rounds instead of the 20 rounds of Table 1. The last row is a frozen proposer that receives no weight or context update (the control of Figure 3c).

ObjectiveAC1 ↓AC2 ↑CP26 ↑Hadamard ↑
Full method1.5049 ± 0.00220.9268 ± 0.01412.4043 ± 0.12380.7656 ± 0.1474
RLOO only (α = 0)1.5137 ± 0.00240.9002 ± 0.00452.1851 ± 0.33340.4956 ± 0.0266
RLOO + std normalization1.5160 ± 0.00040.8961 ± 0.01102.0047 ± 0.01130.5428 ± 0.0175
Without near-tie filtering (δ = 0)1.5141 ± 0.00190.9086 ± 0.00361.9804 ± 0.05200.5292 ± 0.0337
Without uτ (normalize by |bτ|)1.5052 ± 0.00120.9362 ± 0.00602.4142 ± 0.07350.5271 ± 0.0159
Frozen proposer (no update)1.5131 ± 0.00070.9066 ± 0.00911.9988 ± 0.15580.5406 ± 0.0201

Max-sharpening and near-tie filtering are each necessary. Removing either one worsens performance on all four tasks, in most cases to roughly the level of the frozen proposer. Removing the reference uτ leaves performance mostyly unchanged within the seed spread on three tasks, but lowers it on Hadamard. The gains therefore come from the combination of components designed for best-of-search with indirect, noisy rewards (§3.1), rather than from RL training alone.

3. Generalization to Unseen Tasks (W3)

W3: Limited evidence of generalization. While cross-executor replay demonstrates harness transferability, it remains unclear whether the trained proposer can generalize its harness optimization ability to unseen tasks.

HarnessTTT is instance-wise test-time learning for scientific discovery. Each campaign trains the proposer on a single open problem, and the goal is the best solution to that problem, as in TTT-Discover, which likewise trains per problem. Specializing the proposer to the instance at hand is therefore the intended behavior rather than a limitation. We do not claim that the trained proposer transfers to unseen tasks; the phrase "across tasks and rounds" in §2 is inaccurate, since each campaign trains its own proposer, and we will correct it. What we do claim transfers is the harness. Because the harness is explicit and separate from the executor's weights, a harness learned with the 9B executor improves the mean score of all five unseen executors in our replay study (Figure 4). Transferring learned harnesses to GPT-5.5 also matches or exceeds the strongest reported result on six of the eleven non-quarantined tasks (Table 3). Training a proposer that generalizes across problems is a natural extension, which we leave to future work.

Dear Reviewer,

Thank you again for your time and thoughtful comments on our paper. We understand this is a busy period, and we appreciate your time and effort.

As the discussion period closes in less than two days, we wanted to check whether our response has addressed your concerns. If there are any remaining questions or points that would benefit from further clarification, we would be very happy to discuss them before the deadline.

Thank you very much for your consideration.

Best regards, The Authors

Reviewer_g7gJOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed HarnessTTT, a framework that can improve performance on discovery tasks. Unlike prior approaches which solve problems either by fine-tuning the model or by prompt engineering, HarnessTTT contains 2 parts: the first part is a LM with frozen parameters (executor), which solves the problem; the second part is another LM, the proposer, which writes executable harnesses to guide the executor and is trained with RL at test time using the executor's scores. The harness includes prompts, skills, tools, middleware and execution control, etc. The authors report better results on 13 discovery tasks in mathematics, systems, etc, and on the 11 tasks shared with OpenEvolve and Finch-9B they beat both with the same frozen 9B executor.

Critical strengths

The idea is genuinely new: instead of fine-tuning the model to solve the problem, the authors chose to train a second model to write the solver's harness. This direction is worth pursuing, but it does not support acceptance in the current form. The key experiment, the trained-vs-frozen proposer comparison, is only run on 2 problems with 1 seed, and the training recipe itself is assembled from existing components.

The evaluation is broad and covers 5 domains such as math, systems, biology, etc. And the authors are candid about its gaps: 13 tasks in 5 domains, three ablations, a cross-executor replay, and the authors acknowledge that runs are single-seed, compute is not matched, and two tasks are quarantined. I note this as a strength of the writing, not of the evidence: the admissions do not make the single-seed results any more reliable.

Critical weaknesses
  1. The novelty is limited. The framework proposed by this paper is very similar to RLPrompt (Deng et al., 2022) and TEMPERA (Zhang et al., 2023), where they train a policy with RL to write prompts for a frozen task model. Neither of these two papers was cited or discussed in this paper. In addition, the idea of whole-harness editing already exists, as the authors themselves acknowledge by citing MetaHarness (Lee et al., 2026a) and Harness-R1 (Shao et al., 2026). Also, the 5-part executable harness is built on the NexAU agent framework (Nex-AGI Team, 2026) rather than introduced here. In summary, the novelty of this paper is very limited.
  2. The central claim, that a trained proposer is needed, is not established. The paper hangs on "training the proposer beats not training it". The 13-task comparison in Table 2 (HarnessTTT-9B vs. "Context, 9B") is a single run per task, and on several tasks the margin is well within run-to-run noise (AC1 +0.34%, AC2 +0.82%, V2 +0.34%, EPLB −0.47%, AHC039 tie). The "Context, 9B" condition is defined only in passing in §4.3 ("no gradient update and two frozen-model analyst calls after the cold round"), which is a weak in-context baseline. The cleaner control, a frozen proposer sampling the same K = 32 harnesses per round, appears only in Fig. 3c on CP26 and Hadamard, again a single campaign. Mature in-context evolution methods (GEPA, ShinkaEvolve) are only quoted from their own papers in Table 3 with different backbones; none is run with the same executor and budget. A human-written harness baseline is absent entirely. To establish the claim, the authors should: (1) rerun Table 2 with at least three seeds and report mean and spread for the Context and HarnessTTT rows; (2) run the frozen-proposer control of Fig. 3c (same K = 32, no update) on all 13 tasks; (3) run GEPA and ShinkaEvolve with the same frozen Qwen3.5-9B executor and the same 20-call budget; (4) add a human-written harness baseline on at least the five math tasks; (5) define the "Context, 9B" condition precisely in the main text.
  3. The statement about 9B → 27B comparison is confounded. Table 2 attributes the improvement to executor size, but Appendix E and Table 6 show the two campaigns differ in 4 different settings the same time: number of proposals per round (32 vs. 8), number of evaluator calls per rollout (20 vs. 5), the reward baseline (shared incumbent vs. a paired control rollout at the same seed), and the handling of all-tied groups (frontier rescue vs. exclusion). The main text mentions only the first setting. This confounded design cannot answer whether the larger executor helps. In addition, the abstract and conclusion state that the 27B executor improves on "eleven of thirteen" tasks, while Table 2's caption and its own numbers give ten (two ties, one regression of −12.4% on Hadamard). The authors should rerun the 27B executor under the 9B protocol (K = 32, B = 20, shared-incumbent baseline, same tie handling) and correct the count.
  4. The reward uses the published best score as a normalizer, which is unfair because the other baseline model this paper is benchmarking against doesn't have this information (Eq. 1; Appendix F, Table 7). For the 9B campaigns uτ is the published ≤10B SOTA; for the 27B campaign it is the maximum of the seed score, the best human value, the ≤10B SOTA and the Finch-9B score. While the incumbent is below uτ, every proposer gradient is scaled by the remaining gap to it. None of the compared methods has access to this information, and in a genuine discovery setting the ceiling is unknown. The authors' statement that uτ is "never shown to the executor" does not address the issue: the proposer's parameters are shaped by it. The authors should rerun the 9B campaign with uτ removed (normalizing by |bτ| only, as in the fallback branch of Eq. 1) and report whether the Table 1 results hold.
  5. The statement that HarnessTTT "improves on OpenEvolve with the same executor on all eleven benchmark tasks" is not accurate as a like-for-like comparison, because the cost of the proposer is left out of the bill. The comparison counts only executor rollouts. Appendix E shows that each round launches 96 agent trajectories, but only the 32 executor rollouts are counted in the comparison; the other 64 are proposer episodes. On top of that, every round fine-tunes the proposer's LoRA adapter for 3 epochs on 8 GPUs. Over 20 rounds this is 1,920 trajectories plus 20 training passes per task, against the 640 executor rollouts that appear in the cost accounting.; OpenEvolve and Finch have no such layer. The authors themselves concede that "ranks are not compute-matched" (§4.2). The scaling result in Fig. 3b has the same problem: K = 32 beats K = 2 at 16 times the per-round cost. This indicates that that more compute helps, not that the method does. The authors should report total GPU-hours per method and provide a compute-matched comparison against OpenEvolve on the same executor.
Questions for authors
  1. How does HarnessTTT differ from RLPrompt and TEMPERA beyond the size of the generated object? Please discuss these in related work.
  2. What exactly is the "Context, 9B" condition in Table 2, and is it the same as the context arm of §4.3? Why is the frozen-proposer control of Fig. 3c (same K = 32, no update) not run on all 13 tasks?
  3. Can you report means and spread over at least three seeds for Tables 1 and 2 and Fig. 3? §4.1 states every entry is a single campaign. On Erdős, AC1, AC2, EPLB and V2 the margins over the next system or over the Context row are below 1%, which a single run of a best-of-32 search cannot distinguish from noise.
7 comments

We thank the reviewer for the detailed and constructive review. In response, we ran three-seed comparisons of trained and frozen proposers on eight tasks, ran ShinkaEvolve and GEPA with the same executor at comparable GPU-hours, added task-specific human-written harnesses on the five math tasks, reran our method without the reference score uτ, and report a full GPU-hour accounting. We also acknowledge the errors the reviewer identified and will correct them in the revised version, which will also include the new results below.

1. Novelty Relative to Prompt and Harness Optimization (W1, Q1)

W1: The framework proposed by this paper is very similar to RLPrompt (Deng et al., 2022) and TEMPERA (Zhang et al., 2023)... Neither of these two papers was cited or discussed... the idea of whole-harness editing already exists... MetaHarness (Lee et al., 2026a) and Harness-R1 (Shao et al., 2026). Also, the 5-part executable harness is built on the NexAU agent framework...

Q1: How does HarnessTTT differ from RLPrompt and TEMPERA beyond the size of the generated object? Please discuss these in related work.

We thank the reviewer for these references. Since the differences follow directly from the problem we study, we first restate our motivation and contribution, and then compare.

Motivation and contribution: HarnessTTT targets scientific discovery on a single open problem. There is no ground truth, and success is the best valid solution found within an evaluation budget. Solving such a problem requires long agentic trajectories in which the executor repeatedly writes, evaluates, and repairs programs. What matters is therefore the procedure that governs the whole trajectory, i.e., the harness, rather than a single prompt. The only learning signal is a black-box evaluator score, observed through stochastic rollouts. Our contribution is to turn this signal into self-improvement of the search procedure itself. A proposer, initialized from the same 9B/27B base model as the executor, is trained at test time to write complete executable harnesses. It uses a reward function designed for this setting: gap-normalized rewards, parent-grouped leave-one-out advantages without variance normalization, and max-sharpening toward the best-of-group outcome (§3.1). A validation pipeline guards generated code against evaluator leakage (§3.2). No stronger teacher, labeled data, or offline training set is involved.

Each prior work differs from this setting in at least one essential respect:

RLPromptTEMPERAMeta-HarnessHarness-R1HarnessTTT
Proposer learningRL, before deploymentRL, before deploymentNone (frozen)SFT + RL, before deploymentRL, at test time
Relies on a frontier model or teacher✗✗✓ (Opus-4.6 proposer)✓ (GPT-5.5 teacher)✗
Edit spacePrompt tokensInstruction, exemplars, verbalizersHarness programCode hooks at 4 lifecycle pointsFull harness, incl. new tools
ObjectiveAverage scoreAverage accuracyAverage rewardAverage success changeBest solution found

Comparison with prior work (Q1): Beyond the size of the generated object, RLPrompt and TEMPERA train a prompt policy before deployment and optimize average performance on NLP tasks, where each prompt is scored directly. TEMPERA's test-time edits come from a pretrained policy that is not updated on the test instance. Neither addresses long trajectories, best-of-search objectives, or learning on the problem being solved. Meta-Harness edits whole harnesses, but with a frozen frontier proposer (Claude Code with Opus-4.6), so experience never accumulates in the proposer's weights. Harness-R1 trains an editor, but it initializes the editor by SFT on GPT-5.5 patches, restricts edits to four lifecycle hooks without new tools, and trains offline for average success. HarnessTTT, in contrast, writes complete harnesses including new tools, such as the block-family sweeper whose update coincided with surpassing the best human construction on Erdős (Appendix G). It learns only from the evaluator scores of its own rollouts, without a teacher or demonstrations, and its objective outperforms the standard GRPO used by Harness-R1 in our setting (Figure 3c). We will add this discussion, including RLPrompt and TEMPERA, to Related Work in the revised version.

Contemporaneous work: Harness-R1 appeared on arXiv on August 3, 2026, within the contemporaneous window defined by the ICLR 2027 reviewer guidelines (on or after July 17, 2026), and is available only on arXiv. We cite it and include it here for completeness.

On NexAU: We claim neither the NexAU runtime nor the five-part harness format as a contribution. Decomposing a harness into prompts, skills, tools, and middleware is standard practice in harness evolution, for example, in AHE (arXiv:2604.25850, also built on NexAU), Co-Harness (arXiv:2607.22688), Harness² (arXiv:2610.14224), VERSE (arXiv:2610.02616), and WHALE (arXiv:2609.00196). These systems edit the harness with a frozen model and do not train the proposer; the one method that trains a separate harness editor around a frozen executor, Harness-R1, patches four fixed lifecycle hooks. We adopt this decomposition because a learned proposer needs a controllable edit space, in which each proposal is a typed partial spec that can be validated, sandboxed, and compared within its group (Appendices C–D). **What is new in HarnessTTT is a proposer trained at test time, without a stronger teacher, to write the full spec. **** We will revise the corresponding statements in §1, §3, and Figure 6 and its caption, which contrast HarnessTTT only with partial-harness methods, and cite these systems.

2. Evidence that Training the Proposer Matters (W2, Q2, Q3)

W2: The central claim, that a trained proposer is needed, is not established... The 13-task comparison in Table 2 (HarnessTTT-9B vs. "Context, 9B") is a single run per task, and on several tasks the margin is well within run-to-run noise... The cleaner control, a frozen proposer sampling the same K = 32 harnesses per round, appears only in Fig. 3c on CP26 and Hadamard... Mature in-context evolution methods (GEPA, ShinkaEvolve)... none is run with the same executor and budget. A human-written harness baseline is absent entirely.

Q2: What exactly is the "Context, 9B" condition in Table 2, and is it the same as the context arm of §4.3? Why is the frozen-proposer control of Fig. 3c (same K = 32, no update) not run on all 13 tasks?

Q3: Can you report means and spread over at least three seeds for Tables 1 and 2 and Fig. 3?... On Erdős, AC1, AC2, EPLB and V2 the margins over the next system or over the Context row are below 1%...

We agree that multiple seeds help characterize how reliably a method finds good solutions. In scientific discovery, the goal is the best valid solution to a single problem, and prior discovery work such as TTT-Discover and SimpleTES accordingly reports results from single search campaigns. We nevertheless ran the requested controls, at a scale that fits the rebuttal period.

Multi-seed comparison of trained and frozen proposers (Q2, Q3): Rather than rerunning the Context row, we compared the trained proposer against the control the reviewer describes as cleaner: a frozen proposer that samples the same K = 32 harnesses per round and receives no update of any kind. Figure 3c was designed as an ablation of the training objective, which is why the paper ran this control on two tasks only; we now run it with three seeds on eight tasks. These include the five tasks named in Q3 as having margins below 1% (Erdős, AC1, AC2, EPLB, V2) and both tasks of Figure 3c. Together they span four of the paper's five domains: mathematics (Erdős, AC1, AC2, CP26, Hadamard), systems (EPLB), algorithm engineering (AHC058), and astrodynamics (V2). The two arms share the code, edit space, seed program, frozen Qwen3.5-9B executor, evaluator, K = 32, and B = 20, and each seed uses the same decode seeds in both arms. The only difference is whether the proposer is updated. To fit 48 campaigns (2 arms × 3 seeds × 8 tasks) into the rebuttal period, we ran 15 rounds instead of the 20 rounds of Table 2.

Erdős ↓AC1 ↓AC2 ↑EPLB ↑V2 ↓CP26 ↑Hadamard ↑AHC058 ↑
Trained proposer0.380917 ± 0.0000231.5049 ± 0.00220.9268 ± 0.01410.12722 ± 0.000543.4653 ± 0.00452.4043 ± 0.12380.7656 ± 0.1474625.4 ± 60.8
Frozen proposer (no update)0.381065 ± 0.0001451.5131 ± 0.00070.9066 ± 0.00910.12734 ± 0.000353.5913 ± 0.12671.9988 ± 0.15580.5406 ± 0.0201550.8 ± 76.5
Improvement (positive = trained better)+0.000148+0.0082+0.0202−0.00012+0.1260+0.4055+0.2250+74.6

The trained proposer achieves the better mean on seven of eight tasks. The improvement exceeds one standard deviation of both arms on Erdős, AC1, AC2, CP26, and Hadamard, which include three of the tasks named in Q3. On AHC058 and V2 it exceeds the trained arm's standard deviation but not the frozen arm's. On EPLB the two arms are within noise, and we do not claim an improvement there.

Definition of "Context, 9B" (Q2): "Context, 9B" in Table 2 is the same condition as the context arm of §4.3. The proposer is the same frozen Qwen3.5-9B checkpoint as in HarnessTTT-9B and receives no gradient update. After the cold first round, two calls to the same frozen model act as analysts: they read the previous round's harness proposals together with their executor outcomes and scores, and their analysis is placed in the proposer's context for the next round. Everything else is identical to HarnessTTT-9B: the executor, evaluator, initial harness, seed program, number of proposals per round, 20 evaluator calls per executor trajectory, and number of rounds. Table 2 uses K = 32, while §4.3 runs the same arm at K = 8 to match the proposer arm in Figure 3a. We will state this definition in the main text.

Same-executor baselines (ShinkaEvolve and GEPA): We ran ShinkaEvolve and GEPA with the same frozen Qwen3.5-9B executor and evaluator. Both evolve programs rather than harnesses, so a per-trajectory budget of 20 evaluator calls has no direct counterpart; we instead compare at similar total compute in H100 GPU-hours. On these three tasks, a HarnessTTT-9B campaign costs 370–430 GPU-hours, including proposer episodes, candidate and paired-control executor rollouts, and LoRA updates (see our response to W5). Each baseline received about 400 GPU-hours, and we list OpenEvolve from Table 1 with its measured cost.

TaskHarnessTTT-9BOpenEvolve (Table 1)ShinkaEvolveGEPA
CP26 (sum of radii ↑)2.635983 (≈ 390 GPU-h)1.172702 (≈ 360 GPU-h)2.3379 (≈ 400 GPU-h)2.2296 (≈ 400 GPU-h)
Hadamard (det ratio ↑)0.921053 (≈ 370 GPU-h)0.397184 (≈ 360 GPU-h)0.5742 (≈ 400 GPU-h)0.4642 (≈ 400 GPU-h)
AC2 (c2 ↑)0.933900 (≈ 430 GPU-h)0.880100 (≈ 360 GPU-h)0.9245 (≈ 400 GPU-h)0.8558 (≈ 400 GPU-h)

With comparable compute, HarnessTTT finds the best solution on all three tasks. On CP26 and Hadamard, it does so with less compute than ShinkaEvolve and GEPA. GEPA did not improve on the initial program on AC2. The three-seed means of our shorter 15-round campaigns (2.404, 0.766, and 0.927 at ≈ 290, 280, and 320 GPU-hours; three-seed table above) also exceed both baselines on all three tasks.

Human-written harness: We manually wrote a task-specific harness for each of the five math tasks, assisted by a coding agent, and ran each with the same frozen Qwen3.5-9B executor and budget as HarnessTTT-9B.

Erdős ↓AC1 ↓AC2 ↑CP26 ↑Hadamard ↑
Human-written harness, 9B0.4231181.5111980.9233731.9846540.514036
HarnessTTT-9B0.3809191.5046800.9339002.6359830.921053

HarnessTTT outperforms the human-written harnesses on all five tasks. The gap is largest on CP26 (1.985 → 2.636) and Hadamard (0.514 → 0.921), and smaller on AC1 and AC2.

Reviewer_YoYLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HarnessTTT for per-problem test-time learning of agent harnesses in computational discovery. It trains a proposer to generate executable harnesses while keeping the task-solving executor frozen. Downstream search outcomes update the proposer through a policy-gradient objective combining gap-normalized rewards, RLOO, and max-sharpening. The evaluation covers fourteen discovery tasks, comparisons across adaptation strategies, and cross-executor reuse of learned harnesses.

Critical strengths

HarnessTTT adapts a 9B proposer using LoRA while keeping the executor frozen, limiting the trainable parameters and supporting reuse with executors whose weights are inaccessible. The proposer generates prompts, tools, skills, and middleware, producing executable adaptations that can be inspected, modified, and reused across executors.

Critical weaknesses
  1. TTT-Discover [1] already trains per problem, and Harness-R1 [2] trains a harness editor around a frozen executor. HarnessTTT expands the edit space, but does not clearly isolate this contribution from proposer training. Cross-executor reuse shows harness portability, not proposer generalization.

  2. Single-rollout evaluations and single-campaign results leave the gains uncertain. Figure 5a matches rollout counts, not total cost. Stronger cost-controlled comparisons with executor training and training-free adaptation, such as GEPA [3], are needed.

References:

[1] Yuksekgonul et al. Learning to Discover at Test Time. arXiv:2601.16175, 2026.

[2] Shao et al. Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories. arXiv:2608.02276, 2026.

[3] Agrawal et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv:2507.19457, 2025.

Questions for authors
  1. Does proposer training consistently help across seeds when the edit space, starting programs, and budgets are held fixed?
  2. How much improvement comes from generated numerical tools rather than better LLM-guided search?
  3. What are the total training, inference, and evaluation costs? Does HarnessTTT reach comparable solution quality faster or more cheaply than the alternatives?
4 comments

We thank the reviewer for the constructive feedback. In response, we added a three-seed edit-space ablation, three-seed results for the trained proposer on eight tasks, a full GPU-hour accounting, and same-executor baselines at comparable compute. All new results will be included in the revised version.

1. Relation to TTT-Discover and Harness-R1 (W1)

W1: TTT-Discover [1] already trains per problem, and Harness-R1 [2] trains a harness editor around a frozen executor. HarnessTTT expands the edit space, but does not clearly isolate this contribution from proposer training. Cross-executor reuse shows harness portability, not proposer generalization.

HarnessTTT is the only one of the three that self-evolves its search procedure without touching the executor or relying on a stronger model. TTT-Discover also learns per problem at test time, but it only use one model as the role of executor and updates the executor's own weights while HarnessTTT uses one model as two roles (executor and proposer) and updaets proposer's weight.For TTT-discover, what it learns stays inside those weights and cannot be inspected or moved to another executor (to enable weak to strong transfer). Harness-R1 keeps the executor frozen, but it is not self-improvement setting (where use the same model as proposer and exectuor): its 9B editor is first initialized by SFT on patches written by a GPT-5.5 teacher, then trained offline across task splits for average success. Its edits are also restricted to code hooks at four fixed lifecycle points, with no new tools. HarnessTTT starts its proposer from the same base model as the executor, learns only from its own discovery outcomes on the problem being solved, and writes complete harnesses, including new tools. Harness-R1 appeared on arXiv on August 3, 2026, within the ICLR 2027 contemporaneous window; we cite it and will add this comparison to Related Work.

TTT-DiscoverHarness-R1HarnessTTT
What is trainedExecutor weightsHarness editorHarness proposer
ExecutorUpdated (needs weights)FrozenFrozen (no weight access needed)
Stronger model or teacher✗✓ (SFT on GPT-5.5 patches)✗
When it learnsTest time, per problemBefore deployment, on task splitsTest time, per problem
What it producesWeights, tied to one executorCode hooks at 4 lifecycle pointsFull harness incl. new tools, reusable across executors

Separating the edit space from proposer training. We trained the proposer with one component of the edit space disabled at a time (no new skills, no new tools, or no new middleware). We used three seeds, K = 32, B = 20, and 15 rounds. The frozen proposer uses the full edit space and receives no update. Both training and the edit space contribute, and their interaction is task-dependent. On CP26 and AHC058, the trained proposer's mean stays above the frozen full-space proposer under every restriction, so the gain from training does not hinge on any single component. On CP26, every restriction lowers the score, most of all for skills and middleware. On Hadamard, training with a restricted edit space (0.526–0.544) stays at the level of the untrained full-space proposer (0.541), while their combination reaches 0.766, so there the two are complementary. The full space is not uniformly best: on AHC058, disabling new tools scores higher (736.6 vs. 625.4).

CP26 ↑Hadamard ↑AHC058 ↑
Trained, full edit space2.4043 ± 0.12380.7656 ± 0.1474625.4 ± 60.8
Trained, without new skills2.1274 ± 0.22080.5256 ± 0.0381575.4 ± 110.5
Trained, without new tools2.3375 ± 0.13440.5317 ± 0.0429736.6 ± 9.2
Trained, without new middleware2.0844 ± 0.16070.5436 ± 0.0215630.8 ± 56.5
Frozen (no update), full edit space1.9988 ± 0.15580.5406 ± 0.0201550.8 ± 76.5

Harness portability is the intended form of transfer. Like TTT-Discover, HarnessTTT is instance-wise test-time learning: the proposer specializes to the problem at hand, and we do not claim that it transfers to unseen tasks. The phrase "across tasks and rounds" in §2 is inaccurate, since each campaign trains its own proposer, and we will correct it. What transfers is the harness. This is precisely what weight-level adaptation such as TTT-Discover cannot provide, and it lets the learned procedure be reused with five other executors (Figure 4) and with the closed GPT-5.5 (Table 3).

2. Multiple Seeds and Cost-Controlled Comparisons (W2, Q1, Q3)

W2: Single-rollout evaluations and single-campaign results leave the gains uncertain. Figure 5a matches rollout counts, not total cost. Stronger cost-controlled comparisons with executor training and training-free adaptation, such as GEPA [3], are needed.

Q1: Does proposer training consistently help across seeds when the edit space, starting programs, and budgets are held fixed?

Q3: What are the total training, inference, and evaluation costs? Does HarnessTTT reach comparable solution quality faster or more cheaply than the alternatives?

Multiple seeds (Q1). We repeated the HarnessTTT-9B campaign with three seeds on eight tasks spanning four of the paper's five domains (K = 32, B = 20, 15 rounds), and did the same for a frozen proposer that receives no update. The two arms share the edit space, starting program, initial harness, executor, evaluator, seeds, and budget, and differ only in whether the proposer is trained. As in the paper, each harness is scored by a single rollout during search; the three-seed campaigns measure the combined effect of rollout-level and campaign-level noise on the final result.

Erdős ↓AC1 ↓AC2 ↑EPLB ↑V2 ↓CP26 ↑Hadamard ↑AHC058 ↑
Trained proposer0.380917 ± 0.0000231.5049 ± 0.00220.9268 ± 0.01410.12722 ± 0.000543.4653 ± 0.00452.4043 ± 0.12380.7656 ± 0.1474625.4 ± 60.8
Frozen proposer (no update)0.381065 ± 0.0001451.5131 ± 0.00070.9066 ± 0.00910.12734 ± 0.000353.5913 ± 0.12671.9988 ± 0.15580.5406 ± 0.0201550.8 ± 76.5
Improvement (positive = trained better)+0.000148+0.0082+0.0202−0.00012+0.1260+0.4055+0.2250+74.6

With the edit space, starting programs, and budgets held fixed, the trained proposer has the better mean on seven of eight tasks. The improvement exceeds one standard deviation of both arms on Erdős, AC1, AC2, CP26, and Hadamard. On AHC058 and V2 it exceeds the trained arm's standard deviation but not the frozen arm's, and on EPLB the two arms are within noise.

Total cost (Q3). We report H100 GPU-hours from our job logs, covering proposer inference, executor rollouts with their evaluator calls, and LoRA training. Ranges are across tasks, so components need not sum exactly to the totals. Appendix E describes the paired-control rollouts incorrectly; we will correct it in the revised version.

Component per roundCount (9B / 27B)GPU-hours (9B / 27B)
Proposer episodes32 / 84.5–7.0 / 1.3–2.4
Executor rollouts (candidate + paired control)64 / 167.7–12.6 / 2.0–6.6
LoRA update (3 epochs, 8 GPUs)1 / 1≈ 5.9 / 1.3–3.1
Total per round96 + 1 / 24 + 118.7–25.5 / 7.5–10.0
Total per task (× 20 rounds)1,920 + 20 / 480 + 20370–510 / 150–200

Cost-controlled comparisons (W2, Q3). We ran GEPA and ShinkaEvolve with the same frozen Qwen3.5-9B executor and evaluator. Both evolve programs rather than harnesses, so we match total GPU-hours rather than evaluator calls per trajectory. We list OpenEvolve from Table 1 with its measured cost.

TaskHarnessTTT-9BOpenEvolve (Table 1)ShinkaEvolveGEPA
CP26 (sum of radii ↑)2.635983 (≈ 390 GPU-h)1.172702 (≈ 360 GPU-h)2.3379 (≈ 400 GPU-h)2.2296 (≈ 400 GPU-h)
Hadamard (det ratio ↑)0.921053 (≈ 370 GPU-h)0.397184 (≈ 360 GPU-h)0.5742 (≈ 400 GPU-h)0.4642 (≈ 400 GPU-h)
AC2 (c2 ↑)0.933900 (≈ 430 GPU-h)0.880100 (≈ 360 GPU-h)0.9245 (≈ 400 GPU-h)0.8558 (≈ 400 GPU-h)

With comparable compute, HarnessTTT finds the best solution on all three tasks, and on CP26 and Hadamard it does so with less compute than ShinkaEvolve and GEPA. GEPA did not improve on the initial program on AC2. Across Table 1, HarnessTTT also improves on OpenEvolve with the same executor on all eleven tasks (nine of nine excluding the quarantined LLM-SQL and Txn). On cost, our 15-round campaigns above use about three-quarters of a full campaign (≈ 280–380 GPU-hours), similar to or below OpenEvolve's ≈ 360, and their three-seed means still exceed OpenEvolve's Table 1 results on all seven shared tasks (within the seed spread on EPLB only). For executor training, Figure 3a, which we believe the reviewer refers to, compares the three routes at matched rollouts only, and we do not have a GPU-hour-matched executor-training run. For reference only, since it uses a different pipeline and base model, TTT-Discover's reported Qwen3-8B results in Table 1 are matched or exceeded by HarnessTTT-9B on Erdős, AC1, and CP26, while TTT-Discover is ahead on AC2.

3. Contribution of Generated Tools (Q2)

Q2: How much improvement comes from generated numerical tools rather than better LLM-guided search?

On the tasks we ablated, the gains are not explained by numerical tools alone; most of the improvement comes from better-guided search. The edit-space ablation in our response to W1 isolates the tools. With the proposer trained and everything else fixed, disabling new tools changes the score by only −0.067 on CP26 (within the seed spread), lowers it substantially on Hadamard (0.766 → 0.532), and raises it on AHC058 (625 → 737). Within these 9B campaigns, how often the executor calls the tools does not predict their value across tasks either. With the full edit space, generated tools are called in 60%, 26%, and 22% of executor rollouts on CP26, Hadamard, and AHC058, yet removing the most heavily used tool, on CP26, changes the score least. Inspecting the generated code explains this. In the 9B harnesses on the six tasks we inspected (CP26, Hadamard, AC2, AHC058, EPLB, AC1), only the CP26 tool performs the core numerical work: it constructs a staggered packing, solves a linear program for the radii, and refines the coordinates. The other tools mainly inspect the candidate program or return fixed heuristics. On Hadamard, for example, the tool reads the program and returns a rewrite plan (multi-start search, log-determinant screening of sign flips, row and column normalization) without computing any matrix. The large Hadamard drop therefore reflects the loss of procedural guidance delivered through a tool, not of numerical computation. Tools remain a useful part of the edit space on some tasks (Hadamard here; Figure 4), but their value is task-dependent. On these tasks, the gains thus come mainly from how the harness as a whole, including skills, middleware, and tool-delivered guidance, steers the executor's search, which is what the proposer learns to improve.

Dear Reviewer,

Thank you again for your time and thoughtful comments on our paper. We understand this is a busy period, and we appreciate your time and effort.

As the discussion period closes in less than two days, we wanted to check whether our response has addressed your concerns. If there are any remaining questions or points that would benefit from further clarification, we would be very happy to discuss them before the deadline.

Thank you very much for your consideration.

Best regards, The Authors

Reviewer_UxNtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HarnessTTT trains a proposer to write executable harnesses for a frozen executor during scientific search. The harness can change prompts, skills, tools, middleware, and execution settings. Rewards come from the best valid solution found during each rollout. The proposer is updated with gap-normalized rewards, leave-one-out advantages, and a sharpening term that favors stronger proposals. The paper evaluates discovery tasks across several domains and tests whether learned harnesses remain useful with other executors.

Critical strengths

This paper is written pretty competently. I like the separation between learning and execution. Updating the proposer lets the executor remain frozen, while the learned behavior is expressed through inspectable code and instructions. This is useful when executor weights are unavailable or the executor may change later. The paper compares updating proposer weights, executor weights, and context, and also includes a frozen proposer and GRPO. These are useful checks that go beyond comparing only final benchmark scores. Improvements span several mathematical and algorithmic tasks. The fixed-harness replay also provides a more direct test of reuse than simply switching to a stronger model and continuing the search.

Critical weaknesses

The main weakness is that it has a large overlap with Meta-Harness (https://arxiv.org/abs/2603.28052) but never directly compares to it and doesn't seem to have any overlap with their paper. Their baseline contextual agent is like a weaker version of the Meta-Harness harness because it doesn't have access to all of the context. Novelty can be seen as low, as it is just applying RL techniques on top of a Meta-Harness type problem. The full recipe beats GRPO, but there is no RLOO-only comparison. Sharpening itself rescales by reward range, so removing variance normalization does not by itself establish robustness to executor noise. Repeated rollouts of the same harness would help. LLM-SQL and Txn are described as quarantined, but remain in headline frontier counts and the cross-executor aggregate. I appreciate the disclosure, but the main claims should consistently exclude these results and explain why each task was set aside.

Questions for authors

See Weaknesses section

4 comments

We thank the reviewer for the positive assessment and the constructive suggestions. In response, we added a three-seed, component-wise ablation of the objective that includes RLOO-only, report three-seed results for the trained proposer, and explain why LLM-SQL and Txn were set aside, with all headline counts recomputed without them. All new results will be included in the revised version.

1. Novelty: Relation to Meta-Harness and the RL Objective (W1, W2)

W1: The main weakness is that it has a large overlap with Meta-Harness... but never directly compares to it and doesn't seem to have any overlap with their paper. Their baseline contextual agent is like a weaker version of the Meta-Harness harness because it doesn't have access to all of the context. Novelty can be seen as low, as it is just applying RL techniques on top of a Meta-Harness type problem.

W2: The full recipe beats GRPO, but there is no RLOO-only comparison.

The key difference is self-improvement. Meta-Harness (cited in Figure 1(b), §1, and §2) proposes harnesses with Claude Code running Opus-4.6, which reads the code, scores, and traces of all prior candidates; its improvement comes from a frontier model and since it use the closed source frontier model, the weights, however, never change. On the other hand, the learned information is not guided into the model weight but context. HarnessTTT introduces no stronger model and focuses on the self-improvement settings. The proposer starts from the same 9B/27B model as the executor and improves only from its own search outcomes, by updating its weights at test time. The closest comparison under this constraint is a Self-Harness-style setting (Zhang et al., 2026b; cited in §2 alongside Meta-Harness), in which the same frozen model proposes harnesses in context instead of updaing model weights from its own earlier outcomes, with no stronger model involved. This is the Context setting in Table 2. (named Context,9B) There, the frozen Qwen3.5-9B proposer receives an analysis of the previous round's proposals and scores from the same model, while the executor, edit space, budget, and number of rounds are identical to HarnessTTT-9B. Against this baseline, HarnessTTT improves on 11 of 13 tasks (9 of 11 without the quarantined LLM-SQL and Txn), by up to +78.1% on Hadamard; it ties on AHC039 and is 0.47% lower on EPLB. Both proposers read the same fixed-size digest (§3.2), and the Context proposer additionally receives this analysis, so it has at least as much in-context information as the trained proposer, and the gain is not explained by missing context; the ablation below, which includes a never-updated proposer under identical seeds, isolates the weight update. The reviewer is right that this baseline sees less than Meta-Harness's proposer. Meta-Harness itself, however, is not a like-for-like baseline in this setting. Its proposer is a fixed frontier model that never improves, and running it would bring that model into the loop: its access to the full history of all prior candidates relies on a frontier coding agent to navigate it, while models use such long contexts poorly (§1, §3.2). We will add a dedicated discussion of Meta-Harness to Related Work in the revised version.

Not simply RL on a Meta-Harness-type problem. To show that the gains come from our objective rather than from applying RL as such, we ablated its main components, including the RLOO-only comparison the reviewer requested. We used three seeds on four tasks, with K = 32 proposals per round, B = 20, and 15 rounds. The last row is a frozen proposer that receives no weight or context update (the control of Figure 3c).

ObjectiveAC1 ↓AC2 ↑CP26 ↑Hadamard ↑
Full method1.5049 ± 0.00220.9268 ± 0.01412.4043 ± 0.12380.7656 ± 0.1474
RLOO only (α = 0)1.5137 ± 0.00240.9002 ± 0.00452.1851 ± 0.33340.4956 ± 0.0266
RLOO + std normalization1.5160 ± 0.00040.8961 ± 0.01102.0047 ± 0.01130.5428 ± 0.0175
Without near-tie filtering (δ = 0)1.5141 ± 0.00190.9086 ± 0.00361.9804 ± 0.05200.5292 ± 0.0337
Without uτ (normalize by |bτ|)1.5052 ± 0.00120.9362 ± 0.00602.4142 ± 0.07350.5271 ± 0.0159
Frozen proposer (no update)1.5131 ± 0.00070.9066 ± 0.00911.9988 ± 0.15580.5406 ± 0.0201

Standard RL recipes help much less in this setting. RLOO with variance normalization stays within noise of, or below, the frozen proposer on all four tasks, and standard GRPO plateaus early (Figure 3c). Removing max-sharpening (RLOO only) or near-tie filtering worsens performance on all four tasks, in most cases to roughly the level of the frozen proposer. Removing the reference uτ leaves performance unchanged within the seed spread on three tasks, but lowers it on Hadamard.

2. Robustness to Executor Noise (W3)

W3: Sharpening itself rescales by reward range, so removing variance normalization does not by itself establish robustness to executor noise. Repeated rollouts of the same harness would help.

Sharpening and noise. We agree with the reviewer's analysis: because its temperature scales with the reward range, the sharpening term is scale-invariant, so dropping variance normalization does not by itself establish robustness. Two other design choices limit how much small differences can drive the update. Near-tie filtering collapses reward differences below δ before the range-scaled sharpening is applied, so near-tied groups carry no sharpening signal. The sharpening term also enters with weight α = 0.3, so its contribution to any advantage is bounded by 0.3 at every reward scale (Eq. 3), whereas variance normalization rescales the entire advantage. Whether the remaining sharpening amplifies noise in practice is an empirical question, which the ablation of Section 1 addresses: adding sharpening to RLOO improves the mean on all four tasks (full method vs. RLOO only), removing near-tie filtering worsens performance on all four, and RLOO with variance normalization is within noise of or below the frozen proposer. We will correct the explanation in §4.3 accordingly.

Repeated campaigns under executor noise. Rather than repeating rollouts of one fixed harness, we repeated the whole HarnessTTT-9B campaign from scratch with three seeds on eight tasks (K = 32, B = 20), and did the same for a frozen proposer that receives no update. This tests the update end to end under executor noise: executor sampling varies at every rollout of every round, and an update that mainly amplified this noise would fail to improve on a proposer that is never updated. The two arms share the same seeds and budget and differ only in whether the proposer is updated.

Erdős ↓AC1 ↓AC2 ↑EPLB ↑V2 ↓CP26 ↑Hadamard ↑AHC058 ↑
Trained proposer0.380917 ± 0.0000231.5049 ± 0.00220.9268 ± 0.01410.12722 ± 0.000543.4653 ± 0.00452.4043 ± 0.12380.7656 ± 0.1474625.4 ± 60.8
Frozen proposer (no update)0.381065 ± 0.0001451.5131 ± 0.00070.9066 ± 0.00910.12734 ± 0.000353.5913 ± 0.12671.9988 ± 0.15580.5406 ± 0.0201550.8 ± 76.5
Improvement (positive = trained better)+0.000148+0.0082+0.0202−0.00012+0.1260+0.4055+0.2250+74.6

Under executor noise, the trained proposer has the better mean on seven of eight tasks. The improvement exceeds one standard deviation of both arms on Erdős, AC1, AC2, CP26, and Hadamard. On AHC058 and V2 it exceeds the trained arm's standard deviation but not the frozen arm's, and on EPLB the two arms are within noise. Across seeds, the trained proposer's standard deviation is at most about 1.5% of the mean on Erdős, AC1, AC2, EPLB, and V2, and larger on CP26 (5%), AHC058 (10%), and Hadamard (19%).

3. Quarantined Tasks (W4)

W4: LLM-SQL and Txn are described as quarantined, but remain in headline frontier counts and the cross-executor aggregate. I appreciate the disclosure, but the main claims should consistently exclude these results and explain why each task was set aside.

Both tasks were set aside because of evaluator flaws. Our benchmark tasks use the evaluators distributed with SkyDiscover (https://github.com/skydiscover-ai/skydiscover), following Finch (Lee et al., 2026b); they are also used by EvoX and other prior work. While manually inspecting programs during evolution, we found loopholes in two of them. On Txn, the validity check only verifies that the returned schedule is a permutation of the indices it contains, not that every transaction is scheduled. A program can therefore drop transactions to lower its makespan and still pass. This flaw has since been documented independently by Oved et al. ("Evolution or Illusion? Rethinking Evaluation in LLM Evolutionary Search," arXiv:2609.19799, September 17, 2026), who report programs scoring about four times higher than any prior result in this way. On LLM-SQL, the evaluator does not check whether the reordered table preserves all rows and values. To our knowledge, none of the baseline systems reported in Table 3 discuss these issues. To keep the comparison with them fair, we report results under the original evaluators. We manually checked the final programs we report on both tasks and found no use of either loophole. Because the same cannot be guaranteed for every intermediate program during evolution, we marked both tasks as quarantined. The reviewer is right that they nevertheless remained in several headline counts and in the Figure 4 aggregate. Excluding both tasks leaves every count-based conclusion intact:

ClaimIncluding LLM-SQL and TxnExcluding them
Improves on OpenEvolve with the same 9B executor (Table 1)11 of 119 of 9
Best open-weight ≤10B result (Table 1)9 of 118 of 9
HarnessTTT-9B improves on Context, 9B (Table 2)11 of 139 of 11
27B improves on 9B (Table 2)*10 of 138 of 11
Matches or exceeds the reported frontier with transfer (Table 3)8 of 136 of 11
Within 1% of the reported frontier (Table 3)4 more4 more

In the revised version, we will state all headline counts (including Figure 5c) without these tasks, recompute the cross-executor aggregate of Figure 4 without them, discuss the flaws in a dedicated appendix, and, following Oved et al., report evolution results under repaired evaluators.

Dear Reviewer,

Thank you again for your time and thoughtful comments on our paper. We understand this is a busy period, and we appreciate your time and effort.

As the discussion period closes in less than two days, we wanted to check whether our response has addressed your concerns. If there are any remaining questions or points that would benefit from further clarification, we would be very happy to discuss them before the deadline.

Thank you very much for your consideration.

Best regards, The Authors

Comments on the paper

1 comment

Dear reviewers, with the discussion period nearing its end, could you please share your thoughts on the authors’ rebuttal and whether your concerns have been addressed? Please also update your scores if warranted. Thank you for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.