acceptodds
Under review as a conference paper at ICLR 2027

ARC-Golf Bench: Benchmarking Continual Program Optimization with Code Golf Agents

Abstract

Code benchmarks typically reward correctness, leaving agents' ability to improve already-correct programs underexplored. We introduce ARC-Golf Bench, a benchmark for continual program optimization in which coding agents solve ARC-AGI grid-transformation tasks while minimizing Python program length. ARC-Golf Bench comprises 96 tasks from the NeurIPS 2025 Google Code Golf Championship, selected through item response theory-based difficulty discrimination. We evaluate four models with two agentic harnesses, tracking the shortest verified solutions relative to human performance as output-token budgets accumulate. Analyses of solution diversity and verification behavior complement these outcome trajectories. Model rankings vary across harnesses, and early progress and final quality can favor different systems. Although agents remain behind the strongest human competitors on aggregate, they collectively improve on the best known human records on 7 tasks. We further study experience transfer through a fixed library of local simplifications, conditional rewrites, and representation strategies. On ten held-out tasks, the library reduces total program length by 20.4% for one worker. For another, combining the library with an adapted harness yields a 9.4% reduction relative to library-free parallel exploration. Both comparisons show improvements on eight of ten tasks. These findings highlight the joint role of model capabilities and harness design in effective skill transfer.

Primary area:
datasets and benchmarks
Keywords:
code generation, code golf, agent, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8ZUTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ARC-Golf Bench, a 96-task benchmark for continual program optimization. It is built from the NeurIPS 2025 Google Code Golf Championship (ARC-AGI-1 grid transformations, scored by Python program length in bytes). Tasks are selected by fitting a graded response model (GRM) to the final submissions of 48 human teams. Each task’s bytes are expressed as a ratio to the best human solution, binned into five ordinal levels, and used to estimate discrimination and difficulty.

The paper also proposes Golf-UnZip, an LLM-judge procedure that expands golfed programs one verified rewrite at a time. From it, the authors derive Trick Richness, Method Diversity (rubric-based signatures), and a verification-before-submit rate, and they report model-specific harness effects on these with bootstrap and BH-corrected tests.

The paper further studies cross-task experience transfer by distilling optimization skills from 50 source tasks. On ten held-out tasks, the library reduces total bytes by 13.2% for one worker, while combining it with a lane-conditioned harness yields a 9.4% improvement for another worker over a matched library-free parallel control.

Critical strengths
  1. A clean optimization objective and carefully engineered evaluation protocol. The byte-length objective is deterministic and avoids runtime measurement noise. The evaluation harness includes isolated, version-pinned containers, controlled information access, independent verification, saved-candidate recovery, and retained execution traces. The eight model–harness configurations are evaluated across 96 tasks with three runs per configuration, and failures are explicitly accounted for.

  2. A useful model–harness comparison with behavioral analysis. The crossed evaluation reveals substantial model-dependent harness effects, including reversals in model rankings. The trajectory and behavioral analyses offer a more informative picture of sustained optimization than terminal leaderboard scores alone.

Critical weaknesses
  1. Positioning relative to the closest prior work is missing, which weakens the novelty claims. The framing (moving beyond pass/fail toward sustained, verifier-guided optimization, anchored to human competition results, with trajectory tracking and scaffold comparisons) closely follows ALE-Bench[1], which is not cited. The overlap goes beyond motivation:
  • ALE-Bench also places agents on a human leaderboard scale, tracks score trajectories over a run, and provides a curated subset whose agreement with the full set it validates.
  • The intervention in Sec. 6.3 (parallel sessions assigned to distinct solution families with family-specific guidance, keeping the best verified candidate) resembles ALE-Agent’s diversity-oriented search with targeted improvement prompts, which ALE-Bench already identified as its main source of gains.
  1. The human-only interpretation of the competition reference is not justified, which weakens the human-relative performance claims. The reference set is not purely human. Public competition write-ups indicate that several top teams used LLMs substantially; one top-5 team’s submission was reportedly almost entirely LLM-generated, and others started from LLM-generated solutions. Yet “human teams”, “human-equivalent rank”, and “improves on the best human reference on 10 tasks” are central claims.

[1] Imajuku, Yuki, et al. "Ale-bench: A benchmark for long-horizon objective-driven algorithm engineering." Advances in Neural Information Processing Systems 2025.

Questions for authors
  1. How do you position the benchmark and the lane-conditioned harness relative to ALE-Bench’s long-horizon iterative-refinement setting and ALE-Agent’s diversity-oriented search?

  2. In line 239, you mentioned the gaps between the best and worst configurations are 4.2 on easy tasks and 8.6 on hard tasks. However, in table 1, easy: 13.0-9.1=3.9, hard: 20.6-11.3=9.3. Which one is correct?

  3. In line 240 and line 241, you mentioned "Qwen, DeepSeek, and GLM receive worse ranks as difficulty increases under both harnesses". However, in table 1, Codex·DeepSeek-V4-Flash configuration, the rank is: 15.5 (medium) -> 14.8 (hard).

  4. About the validity of the IRT subset. The tasks are selected for discriminating among human teams. Do they also discriminate among agents? Have you checked whether configuration rankings on the 96-task subset agree with rankings on a larger set of tasks?

  5. How do you position ARC-Golf Bench relative to ALE-Bench's long-horizon iterative-refinement setting, human-relative evaluation, and ALE-Agent's diversity-oriented search? What are the specific methodological contributions beyond changing the optimization objective to program length?

2 comments

Thanks for reading our paper.

Response to Q1 (and Q5, W1).

Thank you for pointing out ALE-Bench; we will cite and discuss it in the paper. Both benchmarks evaluate long-horizon iterative optimization using fine-grained, human-referenced scores. However, they differ in several aspects:

  1. Objective: ALE-Bench evaluates heuristic algorithm engineering, whereas ours optimizes verified program length, providing an exact, inexpensive, and deterministic metric for systematic comparisons.
  2. Benchmark and analysis: We use human-calibrated IRT for task selection and introduce Golf-UnZip to reconstruct compression trajectories beyond explicit CoT, enabling quantitative analysis (TR, MD, VR). We also study cross-task skill transfer.
  3. Harness: ALE-Agent employs diversity-oriented search to improve solutions within a task. Our lane-conditioned harness instead investigates cross-task skill transfer through a controlled comparison with a compute-matched, library-free baseline.

Thus, our work complements ALE-Bench by focusing on the mechanisms of iterative optimization and the transferability of optimization skills, rather than proposing another search algorithm.

Response to Q2 and Q3.

Thank you for catching these inconsistencies.

  1. The values in Table 1 are correct: the gaps should be 3.9 (easy) and 9.3 (hard). The original values (4.2 and 8.6) came from an write-up of earlier results. We inadvertently left the outdated numbers in the text and will correct them. We have also carefully rechecked the manuscript for similar inconsistencies.

  2. We agree that our statement was imprecise. We intended to describe the overall trend rather than a strictly monotonic increase. For example, Codex·DeepSeek-V4-Flash worsens from 9.4 (easy) to approximately 15 (15.5 medium, 14.8 hard), despite a slight improvement from medium to hard. We will revise the statement to clarify that these models generally perform worse on medium and hard tasks than on easy tasks, without implying monotonic degradation.

Response to Q4

(a) Agent discrimination. Yes, the selected tasks also discriminate among agents. Across the 96 tasks, the average best–worst rank gap among eight configurations is 10.83 (easy), 11.69 (medium), and 13.36 (hard) rank positions. However, human-calibrated IRT discrimination does not significantly correlate with agent discrimination (Spearman ρ = −0.13, p = 0.21). Thus, while the subset effectively distinguishes agents, we do not claim that human discrimination directly predicts agent discrimination.

DifficultyTasksMean per-task rank SD across 8 settingsMean per-task max–min rank gap
Easy163.5210.83
Medium483.6511.69
Hard324.2013.36

(b) Ranking robustness. Check the response to reviewer 2 (reviewer Hmdz) on Q2.

Response to W2

We agree that some competition teams used LLMs, so describing their submissions as purely human-generated is imprecise. Although these results likely involved human guidance and optimization, we cannot verify the degree of human contribution. We will therefore use “competition teams,” “competition-relative rank,” and “best competition reference” throughout the paper, and explicitly acknowledge potential LLM assistance.

Importantly, the validity of competition-relative comparisons does not depend on the submissions being purely human-generated, as they remain independently verified performance references.

Thank you for the clarifications. I still have two concerns:

  1. Does the 96-task subset work well for evaluating agents? The authors found no significant correlation between human IRT discrimination and agent discrimination (ρ = −0.13, p = 0.21). This makes me question whether IRT is a good way to select tasks for evaluating agents. I also checked the response to Reviewer Hmdz. The 96- vs. 400-task comparison only covers competition teams, not agents. The agent results compare different scoring methods on the same 96 tasks. So we still don't know whether agent rankings would hold on a larger set of tasks, or whether this subset is better than 96 randomly selected tasks. It would also help to compare the differences between configurations with the run-to-run variation within each configuration.

  2. I appreciate the authors' willingness to change "human" to "competition teams." The competition results are still valid reference points. However, some teams used LLMs, and their time and compute budgets were different from those of the evaluated agents. This means the results cannot be presented as a direct comparison between agents and humans.

Reviewer_HmdzOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ARC-Golf bench, which is a benchmark designed to evaluate how well AI agents can compress programs while still keep its functionality. The benchmark uses IRT methods to take 96 tasks from the NeurIPS 2025 code golf competition to form this benchmark. And evaluates agent to write functionally correct python with the fewest possible bytes. The experiments tested four models and two harnesses and analyzed how the solutions improves as the agents uses more tokens. To closely analyze the agent behavior, the paper introduce Golf-UnZip. A tool that unpacks the compressed code to identify the specific optimization tricks. The paper also explores whether providing agents with a library of skills for optimization helps them on new tasks.

Critical strengths
  • The benchmark has a clear and exact objective for the agents which is the byte length, which is good and intuitive.
  • The Golf-UnZip tool is an effective method for understanding the compression.
Critical weaknesses
  • The task selection part is overcomplicated, the paper uses a rather complex theory to subsample the 400 tasks down to 96 tasks and argues that this reduces the evaluation cost. However, these theory about how to subsample tasks is somewhat irrevant to the core contribution of this work and that I think from 96 tasks to 400 tasks is not a huge saving. why not just use the 400 tasks and report that performance, which would provide a more definitive baseline.
  • The main metric used in comparing models and harness is the human relative ranks, which is difficulty to interpret, besides, the human ranks are based on the 400 tasks which is a mismatch with this works 96 tasks. Additionally, this doesn't tell intuitively how much an agent is able to compress a program, a metric like the percentage of bytes reduces would be much clearer.
  • Success on code golf tasks relies more on discoverying syntactic quirks rather than structure software engineering skills, this great limits how well the findings in this work can reflect the actually utility of an agent in real workflows. I think in some cases, in software engineering, people would prefer long codes if they are clearer.
  • The paper claims to study how solutions improves as computation accumulated, but the paper caps the agent runs to 8 hours. It would be good to see if running the agent longer would see a plateau performance or would the agent keep pushing.
Questions for authors
  1. Could the paper report the raw percentage of bytes compressed as a metric?
  2. How does the performance on the 96 selected tasks compare to the full 400 tasks?
  3. How well would the specific python syntax tricks transfer to real software engineering workflows?
3 comments
AuthorsOpenReview ↗

Responses to Q1   Q3

Thanks for the review.

Response to Q1. No single "percentage compressed" exists: agents may start from different initial solutions, so there is no common baseline. We instead report each agent's excess bytes over the best human solution.

SettingTotal bytesExcess over human best (8,790)
Codex·Qwen10,480+19.2%
Codex·GPT10,561+20.1%
Codex·DeepSeek11,214+27.6%
Claude Code·GLM11,303+28.6%
Claude Code·DeepSeek12,123+37.9%
Claude Code·Qwen12,206+38.9%
Codex·GLM12,996+47.8%
Claude Code·GPT13,791+56.9%

Response to Q2 The 400-task set is too large to fully validate. Using available data, we show the 96-task set is a strong proxy for it, for both human teams and GRM agent ranking.

  1. Human teams: 96- vs. 400-task rankings nearly identical.

Across all public human teams, the 96- vs. 400-task total-score rank correlation is 0.9977 (Kendall τ ≈ 0.973), mean absolute rank difference 0.53. 28 teams rank identically, 42 differ by at most one, all by at most three; Top-10 sets are identical up to adjacent swaps. The 96-task ranking is thus an effective proxy.

  1. Agents: GRM and official-score rankings consistent on the 96-task set.

The original GRM estimation uses all 400 tasks; we also estimate GRM on the 96-task set. Official scores are ∑max⁡(1,2500−bytes)\sum \max(1, 2500-\text{bytes}), where bytes is the compressed size under the official scoring rule. The eight agents are ranked with the 47 human teams (55 objects total).

ConfigurationGRM θGRM rankOfficial 96-task rankΔ
Codex · Qwen3.8-Flash3.484756−1
Codex · GPT-5.6-Luna3.249798+1
Codex · DeepSeek-V4-Flash2.788011110
Claude Code · GLM-5.3-Flash2.709411110
Claude Code · DeepSeek-V4-Flash2.291314140
Claude Code · Qwen3.8-Flash2.280414140
Codex · GLM-5.3-Flash1.946516160
Claude Code · GPT-5.6-Luna1.810316160

Note: Δ = GRM rank − Official 96-task rank.

Six of eight configurations rank identically and two differ by one; mean absolute difference 0.25, maximum 1, ordering identical (Spearman = 1); task selection and GRM ranking add no substantial bias.

Response to Q3: We agree that code-golf syntax tricks have limited practical value (Section 7). Yet its benchmark value lies in rigorously evaluating agents' optimization: though recreational, code golf is a challenging optimization problem with

(1) a precise, verifiable objective free from runtime noise;

(2) an iterative, recoverable process (Golf-UnZip);

(3) high difficulty, with even strong agents trailing the best records.

Since correctness is enforced and length verified, improvements reflect shorter correct solutions, making code golf a rigorous, objective testbed for iterative reasoning and optimization.

AuthorsOpenReview ↗

Responses to W1   W4

Response to W1: We agree IRT-based selection adds complexity, but it offers a principled way to characterize difficulty and discriminative power, yielding a representative, informative subset. Evaluating 96 instead of 400 tasks cuts cost by 4.2× while preserving rankings (Q2), valuable for future work. For readability, we will keep the IRT intuition in the main text and move theory to the appendix.

Response to W2:

(a) We will clarify Eq. 1: it gives the average per-task rank over the same 96 tasks; a rank of 10.9 means about 10 of 48 human teams outperform the agent. The 400-task mismatch affects only GRM-Rank; recomputing on 96 tasks shifts rankings by at most one position, confirming consistency.

Human scopeCo·QwenCo·GPTCo·DSCC·GLMCC·DSCC·QwenCC·GPTCo·GLM
400 tasks59131315151616
96 tasks59111114141616

(b) Percentage compression has no common baseline, as agents may start from different initial solutions. We will instead report how far each agent's final length exceeds the best human solution — a more intuitive, comparable measure of the remaining optimization gap.

Response to W3. We agree code golf does not directly reflect practical software engineering, where readability and maintainability often matter more than code length. Yet ARC-AGI code golf involves optimization beyond syntactic tricks — algorithmic simplification and structural refinement — and evaluates agents' optimization via a precise, verifiable objective free from runtime noise. It is thus a challenging, objective testbed for iterative reasoning and optimization, not a measure of real-world engineering performance.

Response to W4. Figure 2 shows performance plateauing between 1M and 2M tokens. Limits rarely bind: only 4.0% of runs hit the 8-hour limit and 0.8% the 3M-token limit; the rest stop on their own. Longer runs are thus unlikely to change our findings, though some agents, e.g., Codex·GPT (27% hitting 8 hours), may benefit from more compute.

I would like to thanks the authors for their response. However, my three main concerns are still not address.

  1. Why not report some results on the 400 task sets? it is only a 4x larger evaluation and I don't think that is too large to fully validate, could you please provide results on the full tasks or give real evidences on why this is infeasible. Evaluating on the 400 tasks can save the paper reader a lot of room for understanding the real strength of the paper instead of the IRT theory.
  2. I don't think "how far agent's final length exceeds the best human solution" is a intuitive way of measuring this task, as the best human solution can also depend on a lot of factors and most importantly, if one task is just not fully explored by humans, then it becomes possible that the agent get a high score on it, also the human data comes from the kaggle challenge, can we rule out that the participant on kaggle didn't use AI at all? On my point of using the metric of "Percentage compression", I think it is more intuitive and for the purpose of this benchmark, why not only allow the agent start with one single baseline so that we get this more intuitive metric?
  3. This paper didn't show how well the python tricks could transfer to more valuable domains, the response shows that the performance plateaus between 1M and 2M tokens, I think (1) typically long horizon SWE tasks could easily reach beyond this token counts, (2) it could be that the code golf tasks are already saturated.
Reviewer_w3VxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ARC-Golf Bench, a benchmark for evaluating LLM agents' ability to iteratively optimize Python programs under functional correctness constraints and byte-length minimization objectives. It combines IRT-based task selection, large-scale model–harness comparisons, behavioral analysis through Golf-UnZip, and cross-task skill transfer experiments. The work provides a promising framework for studying program optimization beyond conventional code generation.

Critical strengths

1. A meaningful benchmark contribution. The paper establishes a well-defined evaluation framework comprising 96 ARC-Golf tasks, executable verification, byte-level optimization objectives, and human performance references. It provides a useful testbed for studying iterative program optimization.

2. Extensive empirical evaluation. The authors evaluate four LLMs across two agentic harnesses, covering 2,304 runs. The analysis reveals meaningful model–harness interactions and suggests that harness effectiveness is highly dependent on the underlying model.

3. Interesting behavioral analysis. Golf-UnZip provides a systematic approach to recovering compression mechanisms from verified programs. The proposed Trick Richness and Method Diversity metrics extend evaluation beyond final solution quality and offer a promising direction for understanding agent behavior.

Critical weaknesses

W1. Infrastructure failures compromise the reliability of the main performance comparisons.

The paper reports that 25 out of 2,304 runs failed to produce accepted solutions, of which 24 were attributed to API or harness issues, including response truncation and timeouts. However, these failures appear to be treated as unsuccessful model runs in the performance evaluation. This conflates infrastructure reliability with model capability and may bias comparisons across configurations, particularly given the uneven distribution of failures.

W2. The skill-transfer experiment raises concerns about potential test-set leakage.

Appendix G acknowledges that two process-level instructions in the lane descriptions were developed after examining trajectories from the hold-out tasks in earlier library versions. Although no task-specific solutions were directly incorporated, this introduces potential test-set information leakage and weakens the validity of the claimed cross-task generalization.

W3. The contribution of the Skill Library is not adequately isolated.

The Worker B comparison simultaneously introduces the Skill Library and lane-based parallel exploration. Consequently, the reported 9.4% improvement cannot be independently attributed to skill transfer. Furthermore, the evaluation covers only ten hold-out tasks with no repeated runs or statistical uncertainty estimates, limiting the strength of the generalization claims.

W4. The behavioral metrics lack sufficient validation of measurement reliability.

Golf-UnZip relies on a single LLM judge for program expansion and method annotation. Moreover, TR and MD are computed exclusively from completed DONE traces, excluding BLUR and ERROR traces even when they contain verified expansions. Without independent annotation validation, judge robustness analysis, and comparisons of completion rates across configurations, it remains unclear whether the observed differences reflect genuine agent behavior or judge-dependent and sample-selection effects.

Questions for authors

Q1. How robust is the IRT-based benchmark selection? Have the authors evaluated task-selection stability and difficulty categorization through bootstrap resampling of the 48 human teams or under alternative calibration thresholds?

Q2. Given the fixed token and wall-clock budgets, can the authors report actual token consumption, execution time, and API costs across configurations? Do the observed model–harness performance differences persist when optimization efficiency is compared under matched resource consumption?

2 comments
AuthorsOpenReview ↗

Responses to Q1 and Q2

Thanks for reading our paper. We answer the questions first, then the weaknesses.

Response to Q1. We retain the 59 highest-discrimination core tasks and randomly add 37 others. Across 200 bootstrap resamples of the 48 teams (GRM refit each time):

  1. Team rankings remain consistent: median Spearman 0.994 (2.5–97.5th: 0.980–0.998); median mean-rank change 1.0.
  2. The core keeps 1.53× (1.30–1.80) the mean discrimination of the other 341 tasks.
  3. Exact membership is sensitive: a re-selected top-59 shares a median of 20 (13–28) tasks with ours.
  4. Difficulty order on 96 tasks: median Spearman 0.898 (0.766–0.951); label agreement 81.3% (64.5–89.6%).

Shifting the GRM-estimation cutpoints (in A.2) from 20/40/60/80 to 15/35/55/75 or 25/45/65/85 and refitting yields team-ranking Spearman 0.992/0.989, difficulty Spearman 0.932/0.916, and label agreement 87.5%/85.4%. Fixed-benchmark measurements are therefore stable; exact core membership is not.

Response to Q2. (a) Mean ± SD per run (288 runs per setting):

ModelHarnessInput (M)Output (M)Minutes
DeepSeekClaude Code0.449 ± 1.4790.628 ± 0.48685.5 ± 61.5
DeepSeekCodex16.919 ± 13.647†1.018 ± 0.802126.2 ± 97.3
GLMClaude Code5.018 ± 2.751*0.580 ± 0.277191.4 ± 87.9
GLMCodex10.362 ± 10.3240.212 ± 0.30692.8 ± 97.7
GPTClaude Code66.645 ± 59.9910.285 ± 0.169156.8 ± 123.3
GPTCodex83.809 ± 77.8820.987 ± 0.818220.3 ± 186.6
QwenClaude Code0.913 ± 0.7410.261 ± 0.10669.3 ± 28.2
QwenCodex9.810 ± 6.3030.703 ± 0.440162.4 ± 103.1

*Reconstructed from messages (280/288 runs reported no input); †recorded for 143/288. API cost depends on cache hits and was not tracked.

(b) Δ = the Codex − Claude Code difference in mean human-relative rank (negative favors Codex); 95% intervals from 10,000 difficulty-stratified bootstraps.

ModelΔ at 1MΔ at 2M
DeepSeek−0.95 [−1.83, −0.10]−1.33 [−2.21, −0.46]
GLM+3.43 [2.40, 4.45]+3.42 [2.39, 4.45]
Qwen−4.27 [−4.98, −3.55]−4.47 [−5.22, −3.72]
GPT−4.97 [−6.06, −3.91]−5.58 [−6.78, −4.41]

The eight-setting ordering is identical at 1M and 2M (Spearman 1.000; 0.952–1.000) but differs at smaller budgets; all four harness directions hold after the reruns in W1.

AuthorsOpenReview ↗

Responses to W1   W4

Response to W1. Per Appendix D, the 24 failures came from the endpoint or harness (8 timeouts, 2 empty responses, 14 over the 128K limit). We reran all 24; conclusions unchanged:

MetricClaude Code·GLMCodex·DeepSeek
Pass rate96.2% → 100%95.5% → 100%
Mean human-relative rank14.19 → 13.2514.24 → 13.18
Total best bytes (96 tasks)11,303 → 11,26411,214 → 11,208
GRM θ / GRM-Rank0.856 / 13 (same)0.871 / 13 (same)
Record-breaking tasks1 → 13 → 3

Mean ranks improve by about 1, and these two settings swap 3rd and 4th place (previously 0.05 apart); all other positions and directions hold.

Response to W2 and W3. Thank you for raising these concerns.

Response to W2 (Potential leakage). We acknowledge this limitation, as disclosed in Appendix G. Only two process-level instructions were informed by earlier hold-out trajectories; the Skill Library, skeletons, and lane partition contain no task-specific hold-out information. To further validate generalization, we evaluated the same harness on 10 additional tasks {task007, task012, task016, task036, task044, task057, task058, task065, task080, task093}, observing a consistent 6.8% byte reduction (8 wins, 1 tie, 1 loss). These results provide additional evidence of transfer beyond the original evaluation set, although they do not fully eliminate the leakage concern.

Library-freeLibrary-enabledByte Change(%)Win / tie / loss
Worker A1023888-13.27 / 1 / 2
Worker B11201157+3.33 / 1 / 6
Worker B (adaptive harness)11171012-9.48 / 0 / 2
Worker B (adaptive harness, new tasks)13491257-6.88 / 1 / 1

Response to W3 (Isolating skill transfer). We agree that the 9.4% improvement reflects the combined benefit of the Skill Library and adaptive harness, as stated in Section 6.3. The library-only comparison yields a 13.2% reduction for Worker A but a 3.3% increase for Worker B, indicating that the effectiveness of skill transfer depends on how the library is integrated into the agent's workflow. We also acknowledge that the limited task count and single-run evaluation make these findings preliminary. We will clarify this distinction, temper our generalization claims.

Response to W4. We compared three judges with the original DeepSeek judge on 112 held-out programs (samples fixed before scoring).

JudgeTR abs. diff.TR diff ≤1MD / DeepSeek (same programs)MD label agreement
Ling-3.1-Flash0.8480.9%0.204 / 0.18796.9%
GLM-5.3-Flash1.0378.8%0.234 / 0.23095.6%
GPT-5.6-Sol0.7587.0%0.270 / 0.24194.5%

Method labels and MD are stable; TR varies by about one family per program, so we treat TR comparisons as judge-dependent. DONE rates are 94.6–98.9% for seven settings and 88.2% for Codex·GPT.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.