acceptodds
Under review as a conference paper at ICLR 2027

ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization

Abstract

Automated harness optimization can substantially improve LLM agents by iteratively updating their prompts, tool interfaces, and control logic from execution feedback. However, existing methods primarily optimize how the harness is updated while largely fixing which training scenarios generate the feedback that drives those updates. As the harness evolves, the scenarios most useful for further optimization can change, suggesting that the training curriculum itself should adapt alongside the harness. We formulate this missing dimension of harness optimization as an automated curriculum-learning problem and introduce ActiveSaddler. ActiveSaddler models the evolving curriculum as a non-stationary bandit with dynamically instantiated optimization targets. It abstracts recurring failures into reusable failure-pattern arms, estimates the potential learning progress from further targeting each pattern, and adaptively balances revisiting known weaknesses with exploring unseen scenarios for new ones. Optimization outcomes continually update both the set of discovered failure patterns and their priorities, allowing the curriculum to co-evolve with the harness. Experiments on GAIA2 and Terminal-Bench 2.0 show that ActiveSaddler consistently discovers stronger harnesses, improving test Pass@1 by 4.4 and 7.5 percentage points over the same harness optimizer using a scenario order fixed before optimization, respectively. Ablations further show that these gains depend on dynamically constructing optimization targets, estimating their evolving utility, and balancing continued optimization with new failure discovery. Together, these results establish automated curriculum learning as a new optimization dimension for harness optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Harness Optimization, Automated Curriculum Learning, Non-Stationary Bandits

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3ZSxOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ActiveSaddler, an automated curriculum-learning framework for offline LLM agent harness optimization. Existing harness optimizers mainly focus on how to update prompts, tools, and control logic, while using a training-scenario schedule fixed before optimization. ActiveSaddler instead adapts the scenario curriculum as the harness evolves. It organizes observed failures into dynamically instantiated failure-pattern arms, uses an LLM-based prioritizer to estimate the expected learning value of revisiting each arm, and employs an exploration controller to decide between repairing known weaknesses and executing unseen scenarios to discover new ones.

Critical strengths
  1. The paper identifies an underexplored limitation of existing harness optimization methods: while the harness evolves during optimization, the training scenarios used to generate feedback are typically scheduled in advance. Formulating adaptive scenario selection as a curriculum-learning problem is timely and practically relevant.

  2. ActiveSaddler combines dynamically constructed failure-pattern arms, adaptive arm prioritization, and exploration of unseen scenarios. Under the same task-agent rollout budget, it improves test Pass@1 over fixed-order AutoSaddler by 4.4 percentage points on GAIA2 and 7.5 points on Terminal-Bench 2.0.

  3. The paper includes component-wise ablations, comparisons with fixed and non-LLM curriculum strategies, an application to a second optimizer (GEPA), cost–accuracy analysis, and qualitative case studies. The detailed algorithms, prompts, data splits, and released implementation also support reproducibility.

Critical weaknesses
  1. Robustness across independent optimization runs. Most reported uncertainty comes from three test-time executions of the same optimized harness, rather than from repeated end-to-end optimization runs. GAIA2 includes only two independent optimization runs, while Terminal-Bench 2.0 and most ablations appear to rely on a single optimization trajectory. Given the stochasticity of LLM-based diagnosis, patch generation, and curriculum decisions, additional optimization seeds and statistical tests are needed to establish that the reported gains are robust.

  2. The generality and formal justification of the approach. Although ActiveSaddler is described as a non-stationary bandit method, its arm scores and exploration decisions are produced primarily through LLM judgments rather than an explicit reward estimator or a theoretically grounded bandit update. Moreover, the evaluation covers only two benchmarks and largely one model configuration. It is therefore unclear how well the method transfers across 1. different LLMs, 2. harness optimizers, and 3. task distributions.

Questions for authors

ActiveSaddler requires additional LLM calls for failure-pattern extraction, arm scoring, and exploration decisions. If all methods are given the same total cost, token budget, or number of optimizer-side LLM calls, does ActiveSaddler still outperform the strongest fixed-curriculum baseline?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3ZSx (1/3)

We sincerely appreciate your thorough review of our paper. Below are our responses to your concerns:

W1: Robustness across independent optimization runs. Most reported uncertainty comes from three test-time executions of the same optimized harness, rather than from repeated end-to-end optimization runs. GAIA2 includes only two independent optimization runs, while Terminal-Bench 2.0 and most ablations appear to rely on a single optimization trajectory. Given the stochasticity of LLM-based diagnosis, patch generation, and curriculum decisions, additional optimization seeds and statistical tests are needed to establish that the reported gains are robust.

Thank you for raising the concern regarding robustness across independent optimization runs. To address this concern, we conducted a second independent optimization run on Terminal-Bench 2.0 for ActiveSaddler and the AutoSaddler-based baselines during the rebuttal period. Each resulting harness was evaluated with three repeated test-time executions.

Harness (Type)Terminal-Bench 2.0 (40 tasks; Pass@1)
Terminus 2 (manual)64.2 ± 2.9
AutoSaddler (Run 1)72.5 ± 0.0
AutoSaddler (Run 2)70.0 ± 0.0
AutoSaddler w/ Category Acc. Order (Run 1)70.8 ± 1.4
AutoSaddler w/ Category Acc. Order (Run 2)70.8 ± 3.8
AutoSaddler w/ Scenario Acc. Order (Run 1)73.3 ± 1.4
AutoSaddler w/ Scenario Acc. Order (Run 2)70.0 ± 4.3
ActiveSaddler (Run 1)80.0 ± 2.5
ActiveSaddler (Run 2)79.2 ± 3.8

ActiveSaddler achieved 80.0% and 79.2% Pass@1 across the two independent runs, respectively. Both runs outperformed the strongest AutoSaddler-based baseline across either run (73.3%) by 6.7 and 5.9 pp, respectively. These consistent gains across independent runs on both benchmarks suggest that ActiveSaddler’s improvement is not driven by a single favorable optimization trajectory. We will include these results in the revised manuscript and extend repeated runs to the ablation studies.

AuthorsOpenReview ↗

Response to Reviewer 3ZSx (2/3)

W2: The generality and formal justification of the approach. Although ActiveSaddler is described as a non-stationary bandit method, its arm scores and exploration decisions are produced primarily through LLM judgments rather than an explicit reward estimator or a theoretically grounded bandit update. Moreover, the evaluation covers only two benchmarks and largely one model configuration. It is therefore unclear how well the method transfers across 1. different LLMs, 2. harness optimizers, and 3. task distributions.

Thank you for your thoughtful comments. To address the concern, we first present additional evaluations across different LLMs, harness optimizers, and task distributions, followed by a clarification of the motivation and empirical justification for our bandit-based formulation.

1. Generality. We evaluate ActiveSaddler across different LLMs, harness optimizers, and task distributions.

(1) Different LLMs. We replace GPT-5.5 with GPT-5.6 Sol as the underlying LLM while keeping all other settings unchanged. ActiveSaddler outperforms AutoSaddler by 3.5 pp with GPT-5.6 Sol, demonstrating that its benefits are not limited to the original LLM configuration.

Harness (Type)GAIA2 (300 tasks; Pass@1)
Default Agent w/ GPT-5.6 Sol (manual)56.8 ± 1.7
AutoSaddler w/ GPT-5.6 Sol59.6 ± 2.5
ActiveSaddler w/ GPT-5.6 Sol63.1 ± 1.4

(2) Different Harness Optimizers. We integrate ActiveSaddler with GEPA, a different offline harness optimizer, while preserving GEPA's harness update and candidate-selection procedures (Appendix E). ActiveSaddler improves GEPA by 3.0 pp, showing that its curriculum benefits extend beyond AutoSaddler.

Harness (Type)GAIA2 (300 tasks; Pass@1)
Default Agent (manual)53.6 ± 1.1
GEPA54.2 ± 2.2
GEPA w/ ActiveSaddler57.2 ± 0.4

(3) Different Task Distributions. During this rebuttal, we additionally evaluated ActiveSaddler on ClawMark [5], with OpenClaw, a real-world agent harness commonly used for personal agents, as its underlying task agent. ClawMark features multi-day workplace tasks involving interactions across multiple services, providing a distinct task distribution from GAIA2 and Terminal-Bench 2.0. ActiveSaddler outperforms AutoSaddler by 4.1 pp on ClawMark, demonstrating consistent improvements on a third task distribution.

Harness (Type)ClawMark (22 tasks; Weighted Score)
Default Agent w/ GPT-5.5 (manual)73.9 ± 1.4
AutoSaddler w/ GPT-5.576.3 ± 0.8
ActiveSaddler w/ GPT-5.580.4 ± 1.0

2. Justification of the LLM-based approach and comparison with bandit-based baselines.

Following prior bandit-based curriculum learning [1, 2, 3], ActiveSaddler models each failure pattern as an arm with a non-stationary expected reward, representing the potential improvement from further optimizing the corresponding harness weakness. As the harness evolves through successive patches, previously identified weaknesses may be resolved or reintroduced, changing the expected reward of each arm. However, unlike prior curriculum learning methods designed for neural network training, harness optimization lacks direct learning-progress signals, such as loss reduction [1] or TD error [3], for estimating arm-specific rewards. Thus, ActiveSaddler chooses to use LLM judgments grounded in the current harness and optimization history to estimate each arm's evolving improvement potential and guide exploration decisions.

To evaluate this design, Appendix D (Table 5) compares two alternatives adapted from existing bandit algorithms under the same optimization setting.

  • EMA-based Arm Prioritization, adapted from the non-stationary bandit-based curriculum of Teacher-Student Curriculum Learning [2], using exponential moving averages to track residual failure rates.
  • UCB-AIR-based Exploration Control, adapted from the infinitely many-armed bandit algorithm [4], using an arm-increasing rule to determine when to explore unseen scenarios.
Harness (Type)GAIA2 (300 tasks; Pass@1)
Default Agent (manual)53.6 ± 1.1
EMA-based Arm Prioritization56.7 ± 1.5
UCB-AIR-based Exploration Control55.6 ± 1.4
ActiveSaddler59.8 ± 1.0

EMA-based Arm Prioritization and UCB-AIR-based Exploration Control underperform ActiveSaddler (59.8%) by 3.1 and 4.2 pp, respectively. Together, these results show that, among the tested alternatives, our LLM-based modules are more effective at deciding which known failure pattern to revisit and when to explore unseen scenarios to discover new weaknesses.

Overall, these results provide empirical evidence that ActiveSaddler's benefits extend across different LLMs, harness optimizers, and task distributions. We will incorporate the new results into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 3ZSx (3/3)

[1] Graves, Alex, et al. "Automated curriculum learning for neural networks." international conference on machine learning. Pmlr, 2017.

[2] Matiisen, Tambet, et al. "Teacher–student curriculum learning." IEEE transactions on neural networks and learning systems 31.9 (2019): 3732-3740.

[3] Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. "Prioritized level replay." International Conference on Machine Learning. PMLR, 2021.

[4] Wang, Yizao, Jean-Yves Audibert, and Rémi Munos. "Algorithms for infinitely many-armed bandits." Advances in Neural Information Processing Systems 21 (2008).

[5] Meng, Fanqing, et al. "ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents." arXiv preprint arXiv:2604.23781 (2026).

Q1: ActiveSaddler requires additional LLM calls for failure-pattern extraction, arm scoring, and exploration decisions. If all methods are given the same total cost, token budget, or number of optimizer-side LLM calls, does ActiveSaddler still outperform the strongest fixed-curriculum baseline?

Thank you for this important question. ActiveSaddler achieves better end-to-end cost efficiency than the strongest fixed-curriculum baselines, even after accounting for all optimizer-side and task-agent computation. Appendix F (Figure 5) reports cumulative monetary cost, LLM calls, and token usage. The following table compares the end-to-end cost of achieving the same development accuracy.

BenchmarkTarget Dev. Accuracy (%)Strongest Fixed Curriculum End-to-End Cost ($)ActiveSaddler End-to-End Cost ($)
GAIA256.967391
GAIA258.5673298
TB268.415030
TB278.9363128

At the same development accuracy, ActiveSaddler reduces end-to-end cost by 86.5% and 55.7% on GAIA2 (673 USD to 91 USD; 673 USD to 298 USD) and 80.0% and 64.7% on TB2 (150 USD to 30 USD; 363 USD to 128 USD). Furthermore, on GAIA2, ActiveSaddler reaches 63.1% at 1,178 USD, whereas category ordering remains at 58.5% even after spending 1,479 USD. On TB2, ActiveSaddler reaches 84.2% at 181 USD, surpassing scenario ordering's peak of 78.9% at 363 USD. Similar efficiency advantages hold for cumulative LLM calls and token usage. These results demonstrate that improved rollout allocation offsets the additional curriculum overhead. We will bring these key observations and cost comparisons from Appendix F into the main text of the revised manuscript.

Reviewer_DZMHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduced an adaptive curriculum learning through a dynamic multi-arm bandit whose arms are failure patterns instantiated at the run time, with LLM-estimated scores of learning progress and an exploration controller choosing pull or draw. Improved agent performance is observed using the proposed method over public benchmarks.

Critical strengths
  • The authors proposed a curriculum learning approach for scenario selection for agent harness optimization, with selection dynamically adapting to the harness evolution.
  • The paper presents ActiveSaddler as a system to construct evolving curriculum jointly through arms of failure patterns, adaptive exploration, and adaptive prioritization.
  • The evaluation over benchmarks under the same budget shows the effectiveness of the proposed method.
Critical weaknesses
  • While the proposed curriculum learning is a principled approach, it is also worth figuring out the overhead such a learning process imposes. The authors should provide the complexity analysis and experimentally show the overhead it comes with.
  • Although it is beneficial to leverage LLMs for the estimation and learning, the capacity cost should be documented and analyzed. Especially, my hunch is that a reliable estimate may require high-quality LLMs, which may turn out to overshadow the benefits the system brings to us at an end-to-end persepctive.
  • As complex systems can suffer from chaotic effectiveness, it is not clear if the LLM-based estimate and curriculum learning would also be fluctuating significantly through various noises from execution stochasticity.
  • The authors should provide more context or rationales on the benchmark selection. The currently used benchmarks, GAIA and TB2, vary a lot in terms of the number of tasks in the test split, 300 vs 40.
  • The LLM judges in this proposed method lack validation. It is not sure how much the judgement would be consistent across LLMs, or simply how consistent the judges will be without inter-judge agreement.
  • The authors should provide rigorous analysis or discussion on the stationarity assumption for the non-stationary bandit used in the proposed solution, as it lacks systematic comparison with representative non-stationary algorithms, such as some UCB methods, making the solution more like a LLM guided curriculum selection.
  • It is not very clear how the proposed method would perform regret analysis or other related measurements to avoid the optimization process getting stuck in some local minima. The author should make some clarifications on that.
Questions for authors

n/a

10 comments
AuthorsOpenReview ↗

Response to Reviewer DZMH (1/4)

We sincerely appreciate your thorough review of our paper. Below are our responses to your concerns:

W1: While the proposed curriculum learning is a principled approach, it is also worth figuring out the overhead such a learning process imposes. The authors should provide the complexity analysis and experimentally show the overhead it comes with.

W2: Although it is beneficial to leverage LLMs for the estimation and learning, the capacity cost should be documented and analyzed. Especially, my hunch is that a reliable estimate may require high-quality LLMs, which may turn out to overshadow the benefits the system brings to us at an end-to-end persepctive.

We agree that curriculum overhead should be evaluated from an end-to-end perspective. To address these concerns, we provide both a complexity analysis of the additional curriculum operations and an empirical evaluation of their overhead.

Complexity Analysis. The additional arm-processing complexity scales linearly with the number of optimization iterations and the maximum number of discovered arms, i.e., O(T∣Amax⁡∣)O(T|A_{\max}|). As shown in Algorithm 1, ActiveSaddler leaves the underlying harness optimizer unchanged and adds at most three LLM sessions per iteration for exploration control, arm scoring, and failure-pattern extraction. At iteration tt, scoring ∣At∣|A_t| existing arms and matching ∣Ft∣|F_t| failure records against them requires O(∣At∣+∣Ft∣∣At∣)O(|A_t| + |F_t||A_t|) operations, assuming linear matching. Since each mini-batch contains at most three scenarios, ∣Ft∣≤6|F_t| \leq 6, accounting for both pre- and post-patch failures. Therefore, the total arm-processing complexity over TT iterations is O(∑t=1T∣At∣)O(\sum_{t=1}^{T}|A_t|), bounded by O(T∣Amax⁡∣)O(T|A_{\max}|). Moreover, each arm represents a distinct observed harness weakness rather than an individual failure. As optimization resolves existing weaknesses, fewer unresolved weaknesses may remain, potentially limiting the growth of the arm set in practice.

Empirical Overhead. Despite higher optimizer-side costs per generated patch, ActiveSaddler substantially reduces the end-to-end cost of achieving the same development accuracy. Appendix F (Tables 7 and 8) reports runtime, monetary cost, LLM calls, and token usage, including all curriculum operations using gpt-5.5 (xhigh). The following table summarizes the end-to-end cost comparisons from Figure 5.

BenchmarkTarget Dev. Accuracy (%)AutoSaddler End-to-End Cost ($)ActiveSaddler End-to-End Cost ($)
GAIA256.946591
GAIA258.51,360298
TB268.47330
TB278.9220128

At the same development accuracy, ActiveSaddler reduces end-to-end cost by 80.4% and 78.1% on GAIA2 (465 USD to 91 USD and 1,360 USD to 298 USD) and 58.9% and 41.8% on TB2 (73 USD to 30 USD and 220 USD to 128 USD). These savings more than offset the increase in optimizer-side cost per generated patch, from 7.87 USD to 11.44 USD on GAIA2 and from 6.77 USD to 11.37 USD on TB2. Similar efficiency gains hold for cumulative LLM calls and token usage. We will incorporate the complexity analysis and clarify these findings in the revised manuscript.

W3: As complex systems can suffer from chaotic effectiveness, it is not clear if the LLM-based estimate and curriculum learning would also be fluctuating significantly through various noises from execution stochasticity.

To address the concern, we evaluated the stability of ActiveSaddler across independent optimization runs on GAIA2 and Terminal-Bench 2.0 (TB2). As reported in Appendix C, two independent runs on GAIA2 achieved 59.8% and 61.6% Pass@1, outperforming all fixed-order baselines by 3.9 to 4.4 pp and 5.0 to 5.7 pp, respectively. During this rebuttal, we further conducted a second independent optimization run on Terminal-Bench 2.0, including the fixed-order baselines, and evaluated each resulting harness with three repeated test-time executions.

Harness (Type)TB2 (40 tasks; Pass@1)
Terminus 2 (manual)64.2 ± 2.9
AutoSaddler (Run 1)72.5 ± 0.0
AutoSaddler (Run 2)70.0 ± 0.0
AutoSaddler w/ Category Acc. Order (Run 1)70.8 ± 1.4
AutoSaddler w/ Category Acc. Order (Run 2)70.8 ± 3.8
AutoSaddler w/ Scenario Acc. Order (Run 1)73.3 ± 1.4
AutoSaddler w/ Scenario Acc. Order (Run 2)70.0 ± 4.3
ActiveSaddler (Run 1)80.0 ± 2.5
ActiveSaddler (Run 2)79.2 ± 3.8

Although stochastic execution leads each run to a different curriculum trajectory, ActiveSaddler outperforms all fixed-order baselines in both runs, by 6.7 to 9.2 pp in Run 1 and 8.4 to 9.2 pp in Run 2. These consistent gains indicate that the LLM-based estimates and the resulting curriculum are robust to execution stochasticity. We will include these results in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer DZMH (2/4)

W4: The authors should provide more context or rationales on the benchmark selection. The currently used benchmarks, GAIA and TB2, vary a lot in terms of the number of tasks in the test split, 300 vs 40.

We chose GAIA2 and Terminal-Bench 2.0 to evaluate whether ActiveSaddler generalizes across different agent interfaces. GAIA2 covers tool-based interaction in a simulated smartphone environment, and TB2 covers command-line interaction across domains such as system administration, machine learning, and cybersecurity. Both are challenging benchmarks commonly used in prior agent harness optimization work (e.g., Meta-Harness [1] and AutoSaddler [2]).

The difference in test size comes from the benchmarks themselves. TB2 contains only 89 tasks in total, so our 30/19/40 train/dev/test split is the largest test split possible while still leaving data for optimization. GAIA2 provides far more scenarios, which lets us hold out 300 test scenarios across three universes. To keep results comparable, we apply the same protocol on both benchmarks: the same rollout-budget rule, mini-batch size, and three repeated test-time executions. We also repeat optimization runs on both benchmarks (see our response to W3).

During this rebuttal, we additionally evaluated ActiveSaddler on ClawMark [3], a third benchmark with a different task distribution. ClawMark uses OpenClaw, a real-world agent harness widely used for personal-agent tasks, as its underlying task agent. We followed the same protocol with a 50/20/22 train/dev/test split.

Harness (Type)ClawMark (22 tasks; Weighted Score)
Default Agent (manual)73.9 ± 1.4
AutoSaddler76.3 ± 0.8
ActiveSaddler80.4 ± 1.0

On ClawMark, ActiveSaddler still outperforms AutoSaddler and the default agent by 4.1 pp and 6.5 pp, respectively. This shows that ActiveSaddler's gains are not tied to our choice of GAIA2 and TB2, but carry over to a third task distribution. We will add the rationale for benchmark selection and these results to the revised manuscript.

[1] Lee, Yoonho, et al. "Meta-harness: End-to-end optimization of model harnesses." arXiv preprint arXiv:2603.28052 (2026).

[2] Park, Sungho, et al. "AutoSaddler: Automatic harness optimization with durable updates from agent execution traces." arXiv preprint arXiv:2608.23041 (2026).

[3] Meng, Fanqing, et al. "ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents." arXiv preprint arXiv:2604.23781 (2026).

W5: The LLM judges in this proposed method lack validation. It is not sure how much the judgement would be consistent across LLMs, or simply how consistent the judges will be without inter-judge agreement.

Our evaluation does not rely solely on LLM judges. On Terminal-Bench 2.0 and ClawMark, task outcomes are evaluated using deterministic, task-specific verifiers without LLM judges. On GAIA2, we use the benchmark's official ARE Verifier with Llama-3.3-70B-Instruct, following the original paper [4]. The verifier checks agent actions against oracle annotations and uses an LLM only for flexible text fields. The original paper validates this exact verifier configuration on 450 human-labeled trajectories, reporting 0.98 agreement with human annotations (0.99 precision, 0.95 recall). These results support the reliability of our evaluation protocols and reduce concerns that the observed improvements are artifacts of LLM-judge inconsistency. We will clarify these protocols in the revised manuscript.

[4] Froger, Romain, et al. "Gaia2: Benchmarking LLM agents on dynamic and asynchronous environments." International Conference on Learning Representations. Vol. 2026. 2026.

AuthorsOpenReview ↗

Response to Reviewer DZMH (3/4)

W6: The authors should provide rigorous analysis or discussion on the stationarity assumption for the non-stationary bandit used in the proposed solution, as it lacks systematic comparison with representative non-stationary algorithms, such as some UCB methods, making the solution more like a LLM guided curriculum selection.

Thank you for raising this important point. We agree that the non-stationarity assumption and comparisons with existing bandit algorithms deserve clearer discussion. Following prior curriculum learning methods [5, 6], ActiveSaddler assumes non-stationary arm rewards because the expected improvement potential of each failure pattern changes as the harness evolves. Our manuscript already provides a retrospective analysis (Appendix I) and comparisons with bandit-based alternatives (Appendix D), which we summarize below.

Retrospective analysis. Our analysis (Appendix I, Figure 11) shows that pull probabilities shift across failure-pattern arms over time rather than remaining concentrated on a fixed subset. The case studies further illustrate these dynamics: P3's priority drops sharply after a successful patch resolves its weakness (Appendix I, Figure 9), whereas P8 retains high priority despite temporary successes and gains further priority when the same weakness appears in additional scenarios (Appendix I, Figures 10–11). These observations illustrate how arm priorities adapt as the harness evolves.

Comparisons with bandit-based alternatives. Appendix D (Table 5) evaluates two alternatives for non-stationary curriculum decisions on GAIA2, under the same optimization setting:

  • EMA-based Arm Prioritization, adapted from the non-stationary bandit-based curriculum of Teacher-Student Curriculum Learning [6], which tracks residual failure rates using exponential moving averages.
  • UCB-AIR-based Exploration Control, adapted from the infinitely many-armed bandit algorithm [7], which determines when to explore unseen scenarios.
Harness (Type)GAIA2 Test Pass@1 (%)
Default Agent (manual)53.6 ± 1.1
EMA-based Arm Prioritization [6]56.7 ± 1.5
UCB-AIR-based Exploration Control [7]55.6 ± 1.4
ActiveSaddler59.8 ± 1.0

Both alternatives underperform ActiveSaddler by 3.1 and 4.2 pp, respectively. Together, these results show that, among the tested alternatives, our LLM-based modules are more effective at deciding which known failure pattern to revisit and when to explore unseen scenarios to discover new weaknesses. We will clarify the non-stationarity assumption and bring the key retrospective analysis and bandit-baseline comparison results from the appendices into the main text of the revised manuscript.

[5] Graves, Alex, et al. "Automated curriculum learning for neural networks." international conference on machine learning. Pmlr, 2017.

[6] Matiisen, Tambet, et al. "Teacher–student curriculum learning." IEEE transactions on neural networks and learning systems 31.9 (2019): 3732-3740.

[7] Wang, Yizao, Jean-Yves Audibert, and Rémi Munos. "Algorithms for infinitely many-armed bandits." Advances in Neural Information Processing Systems 21 (2008).

Reviewer_31GaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ActiveSaddler, an automated curriculum learning framework for agent harness optimization. Instead of following a fixed training scenario order, the method dynamically selects optimization targets based on observed failure patterns. It introduces three components: a Failure-Pattern Extractor, an Arm Prioritizer, and an Exploration Controller. Experiments on GAIA2 and Terminal-Bench 2.0 show improvements over AutoSaddler and other harness optimization baselines.

Critical strengths
  1. The paper is well written and clearly motivated. The overall framework is easy to follow, and adapting the training curriculum during harness optimization is a reasonable direction.

  2. The empirical results are promising, with consistent improvements over fixed-order baselines on two benchmarks. The paper also includes useful ablations, cost analysis, and an additional experiment applying the curriculum to GEPA.

Critical weaknesses
  • The framework mainly combines existing ideas, including failure analysis, adaptive task selection, and exploration-exploitation strategies. Applying these mechanisms at a different stage of harness optimization does not necessarily constitute a substantial methodological contribution.

  • Most main baselines rely on fixed scenario ordering. A stronger comparison with adaptive failure-based replay, without failure-pattern clustering, would help isolate the contribution of the proposed approach.

  • The fixed exploration baseline differs from ActiveSaddler in both exploration timing and potentially exploration frequency. A comparison with fixed schedules using the same total exploration budget would better isolate the benefit of adaptive exploration.

  • Terminal-Bench 2.0 contains only 30 training and 40 test tasks, and the reported repeated evaluations do not establish robustness across independent optimization runs. Additional runs and evaluations on larger, more diverse settings would strengthen the conclusions.

Questions for authors
  1. How does ActiveSaddler compare with adaptive failure-based scenario replay under the same optimization budget?

  2. Can the authors evaluate exploration strategies with matched exploration counts?

  3. How stable are the results across independent optimization runs on Terminal-Bench 2.0?

I will be open to raise my score if the authors answers the above questions.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 31Ga (1/3)

We sincerely appreciate your thorough review of our paper. Below are our responses to your concerns:

Q1: How does ActiveSaddler compare with adaptive failure-based scenario replay under the same optimization budget?

W2: Most main baselines rely on fixed scenario ordering. A stronger comparison with adaptive failure-based replay, without failure-pattern clustering, would help isolate the contribution of the proposed approach.

Thank you for the suggestion. During this rebuttal, we conducted a direct comparison between ActiveSaddler and adaptive failure-based scenario replay in GAIA2, without failure-pattern clustering and under the same optimization budget. Specifically, each failed scenario is registered as its own arm without being clustered into failure patterns, while all other components (Arm Prioritizer, Exploration Controller, and the rollout budget) are kept identical to ActiveSaddler.

Harness (Type)GAIA2 (300 tasks; Pass@1)
Default Agent (manual)53.6±1.1
ActiveSaddler59.8±1.0
w/o Failure Pattern Clustering (adaptive failure-based scenario replay)57.7±1.7
w/o Failure Pattern Based Arm (Scenario Arm)56.2±0.2

The results in the table show that ActiveSaddler outperforms failure-based scenario replay by 2.1 pp. We attribute this gap to the fact that, without clustering, the same weakness is fragmented across individual scenario arms. Pulling one arm only addresses a particular failure case while excluding other similar failure cases, and repairs on one scenario do not update the status of other scenarios sharing the same weakness. We will add this result to the revised manuscript to more clearly isolate the contribution of failure-pattern clustering.

Q2: Can the authors evaluate exploration strategies with matched exploration counts?

W3: The fixed exploration baseline differs from ActiveSaddler in both exploration timing and potentially exploration frequency. A comparison with fixed schedules using the same total exploration budget would better isolate the benefit of adaptive exploration.

We agree that matching the exploration counts would strengthen our comparison with fixed schedules. Following your suggestion, we compared ActiveSaddler with a fixed exploration schedule that uses the same exploration count. Specifically, within the 1,400-rollout budget in GAIA2, ActiveSaddler explored unseen scenarios 17 times. We therefore built a fixed-interval baseline that explores once every 82 rollouts (1,400 / 17), so both methods explore the same number of times and differ only in timing. All other components are kept identical to ActiveSaddler.

Harness (Type)GAIA2 (300 tasks; Pass@1)
Default Agent (manual)53.6±1.1
ActiveSaddler59.8±1.0
w/o Exploration Controller (every 5 iterations)55.3±0.9
w/o Exploration Controller (matched count)56.3±0.7

The above results show that ActiveSaddler outperforms the matched-count fixed schedule by 3.5 pp. Since both methods explore equally often, this gap reflects the benefit of adaptive exploration timing. ActiveSaddler explores more often early in optimization, when few arms exist, and continues repairing known arms while unresolved weaknesses remain. We will add this result to the revised manuscript to more clearly isolate the benefit of adaptive exploration.

AuthorsOpenReview ↗

Response to Reviewer 31Ga (2/3)

Q3: How stable are the results across independent optimization runs on Terminal-Bench 2.0?

W4: Terminal-Bench 2.0 contains only 30 training and 40 test tasks, and the reported repeated evaluations do not establish robustness across independent optimization runs. Additional runs and evaluations on larger, more diverse settings would strengthen the conclusions.

Thank you for pointing this out. Terminal-Bench 2.0 contains only 89 tasks, and our 30/19/40 train/dev/test split already uses all of them. To address these concerns, during this rebuttal, we conducted two additional experiments: (i) a second independent optimization run on Terminal-Bench 2.0 for ActiveSaddler and the AutoSaddler-based baselines, evaluating each resulting harness with three repeated executions on the test split, and (ii) an evaluation on ClawMark [1], a third benchmark with a different task distribution, to assess generality in a more diverse setting.

Stability across optimization runs. The second ActiveSaddler run on Terminal-Bench 2.0 achieved 79.2% Pass@1, only 0.8 pp below Run 1 and 15.0 pp above Terminus 2, matching the strong performance observed in the initial run. It consistently outperformed the baseline settings across both runs. Specifically, ActiveSaddler outperformed AutoSaddler, category-order, and scenario-order baselines by 8.4, 8.8, and 8.0 pp on average, respectively.

Harness (Type)Terminal-Bench 2.0 (40 tasks; Pass@1)
Terminus 2 (manual)64.2±2.9
AutoSaddler (Run 1)72.5±0.0
AutoSaddler (Run 2)70.0±0.0
AutoSaddler w/ Category Acc. Order (Run 1)70.8±1.4
AutoSaddler w/ Category Acc. Order (Run 2)70.8±3.8
AutoSaddler w/ Scenario Acc. Order (Run 1)73.3±1.4
AutoSaddler w/ Scenario Acc. Order (Run 2)70.0±4.3
ActiveSaddler (Run 1)80.0±2.5
ActiveSaddler (Run 2)79.2±3.8

Together with the independent GAIA2 run in Appendix C, which covers 300 test scenarios across three held-out universes, these results suggest that ActiveSaddler's gains do not depend on a single favorable optimization trajectory.

Evaluation in a more diverse setting. ClawMark [1] uses OpenClaw, a real-world agent harness commonly used for personal-agent tasks, as its underlying task agent. We split ClawMark into 50 training, 20 development, and 22 test tasks and followed the same protocol as in our main experiments. ActiveSaddler achieved a Weighted Score of 80.4%, outperforming AutoSaddler and the default agent by 4.1 pp and 6.5 pp, respectively. These results provide additional evidence that ActiveSaddler's improvements extend beyond GAIA2 and Terminal-Bench 2.0 to a different task distribution.

Harness (Type)ClawMark (22 tasks; Weighted Score)
Default Agent (manual)73.9±1.4
AutoSaddler76.3±0.8
ActiveSaddler80.4±1.0

We will include these additional experimental results in the revised manuscript.

[1] Meng, Fanqing, et al. "ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents." arXiv preprint arXiv:2604.23781 (2026).

AuthorsOpenReview ↗

Response to Reviewer 31Ga (3/3)

W1: The framework mainly combines existing ideas, including failure analysis, adaptive task selection, and exploration-exploitation strategies. Applying these mechanisms at a different stage of harness optimization does not necessarily constitute a substantial methodological contribution.

Thank you for your valuable feedback. We agree that failure analysis, adaptive task selection, and exploration-exploitation are established ideas. However, we believe that effectively combining established ideas to address an important and challenging problem, such as harness optimization, can itself constitute a meaningful research contribution. More importantly, applying these ideas to harness optimization is not straightforward, as it introduces three distinct challenges that require nontrivial methodological designs:

  1. Failure Analysis: Conventional failure analysis often identifies what went wrong in individual executions. In harness optimization, however, failures must be organized across multiple scenarios, where one scenario can expose multiple weaknesses and the same weakness can appear across different scenarios. ActiveSaddler addresses this by dynamically constructing failure-pattern arms that capture shared weaknesses and serve as reusable optimization targets. Our ablations support the importance of this design: replacing failure-pattern arms with category-level or scenario-level arms reduces Pass@1 by 3.0/3.6 pp on GAIA2 and 10.8/6.7 pp on TB2, respectively (Table 1).

  2. Adaptive Task Selection: Conventional curriculum learning prioritizes tasks using observable learning-progress signals, such as loss reduction [2] or TD error [3]. In harness optimization, however, such signals are not directly observable for individual weaknesses, since discrete harness patches can affect multiple weaknesses simultaneously. ActiveSaddler therefore introduces adaptive arm prioritization to estimate each weakness's improvement potential from the current harness and optimization history, considering failure severity, fixability, generalization breadth, and regression risk. Our ablations support this design: removing the Arm Prioritizer reduces Pass@1 by 4.5/7.5 pp on GAIA2/TB2 (Table 1), while replacing it with TSCL-inspired EMA-based failure-persistence scoring [4] reduces Pass@1 by 3.1 pp on GAIA2 (Appendix D).

  3. Exploration-Exploitation: Conventional fixed-arm bandits balance exploration and exploitation among predefined arms. In harness optimization, however, the curriculum must decide whether to revisit known weaknesses for further repair or execute unseen scenarios to discover new optimization targets, as their relative value changes with harness updates. ActiveSaddler addresses this through adaptive exploration control. Our ablations support this design: replacing adaptive exploration with a fixed schedule reduces Pass@1 by 4.5/8.3 pp on GAIA2/TB2 (Table 1), while UCB-AIR-derived exploration underperforms ActiveSaddler by 4.2 pp on GAIA2 (Appendix D).

Together, these results demonstrate that ActiveSaddler goes beyond combining established ideas by addressing distinct methodological challenges in harness optimization, with empirical evidence supporting the effectiveness of its designs. We will clarify these contributions in the revised manuscript.

[2] Graves, Alex, et al. "Automated curriculum learning for neural networks." International Conference on Machine Learning. PMLR, 2017.

[3] Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. "Prioritized level replay." International Conference on Machine Learning. PMLR, 2021.

[4] Matiisen, Tambet, et al. "Teacher–student curriculum learning." IEEE Transactions on Neural Networks and Learning Systems 31.9 (2019): 3732-3740.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers and Area Chair

Dear Reviewers and Area Chair,

We thank the reviewers for their constructive feedback and recognition of ActiveSaddler's motivation and promising results. During the rebuttal, we added controlled comparisons, independent optimization runs, and further analyses of generality, reliability, and cost. All results and clarifications have been incorporated into the revised manuscript, with revisions highlighted in purple. Below, we summarize the main concerns and how we addressed them:

  1. Methodological Novelty (W1 of Reviewer 31Ga): We clarified three challenges that distinguish harness optimization from existing failure analysis, curriculum learning, and bandit methods. First, weaknesses can recur across scenarios, while one scenario can reveal multiple weaknesses, motivating failure-pattern arms. Second, unlike neural-network training, harness optimization lacks direct per-weakness learning-progress signals, motivating adaptive arm prioritization. Third, unlike fixed-arm bandits, harness optimization must balance revisiting known weaknesses with exploring unseen scenarios to discover new arms as the harness evolves, motivating adaptive exploration. Component ablations and comparisons with existing alternatives support all three designs (Table 1, Appendix D).

  2. Adaptive Failure-Based Replay (W2 & Q1 of Reviewer 31Ga): We added adaptive replay of failed scenarios without failure-pattern clustering, keeping the remaining components and rollout budget unchanged. On GAIA2, ActiveSaddler achieves 59.8% vs. 57.7% Pass@1 (+2.1 pp). This shows that grouping failures by shared harness weaknesses is more effective than treating each failed scenario as a separate arm.

  3. Matched Exploration Counts (W3 & Q2 of Reviewer 31Ga): We compared ActiveSaddler with a fixed schedule using the same 17 exploration decisions within 1,400 GAIA2 rollouts. ActiveSaddler achieves 59.8% vs. 56.3% Pass@1 (+3.5 pp). Matching exploration counts isolates adaptive exploration timing, rather than exploration frequency, as the source of the improvement.

  4. Non-Stationary Bandit Formalization and Empirical Justification (W6 & follow-up of Reviewer DZMH; W2 of Reviewer 3ZSx): We formally defined arm rewards as expected downstream improvements conditional on the current harness, explaining how patches change arm rewards and exploration introduces new arms. We connected this formulation to adaptive exploration (draw or pull) and stochastic arm prioritization, using LLM-based surrogate estimates grounded in the current harness and optimization history. Retrospective analyses show how arm priorities evolve as weaknesses are resolved or recur (Appendix I). Comparisons with TSCL-inspired EMA prioritization and UCB-AIR-based exploration show that both underperform ActiveSaddler by 3.1 pp and 4.2 pp, respectively (Appendix D), supporting our approach.

  5. Robustness Across Optimization Runs (W4 & Q3 of Reviewer 31Ga; W3 & follow-up of Reviewer DZMH; W1 of Reviewer 3ZSx): We added a second independent TB2 optimization run with baseline reruns. ActiveSaddler achieves 80.0% and 79.2% Pass@1 on TB2 and 59.8% and 61.6% on GAIA2, with three test-time evaluations per harness. Gains persist across both optimization runs, rather than one favorable trajectory.

  6. Generality Across Models, Optimizers, and Tasks (W4 of Reviewer 31Ga; W4 of Reviewer DZMH; W2 of Reviewer 3ZSx): We evaluated ActiveSaddler with GPT-5.6 Sol (+3.5 pp) and the cheaper GPT-5.6 Luna (+5.5 pp) over AutoSaddler on GAIA2, showing improvements across different LLM capabilities. We also integrated our curriculum with GEPA, a different harness optimizer, improving Pass@1 from 54.2% to 57.2% (+3.0 pp) on GAIA2 (Appendix E). On ClawMark, a third task distribution, ActiveSaddler outperforms AutoSaddler by 4.1 pp (80.4% vs. 76.3%). These results support generality across models, optimizers, and task distributions.

  7. Consistency of LLM-Based Arm Scoring (Reviewer DZMH's follow-up): We evaluated arm-scoring consistency across five LLMs, 10 optimization checkpoints, and three repeated queries. Pairwise arm-order agreement reached 83.2% within models and 79.0% across models, demonstrating high consistency in arm prioritization across repeated queries and different LLMs.

  8. Computational Overhead and End-to-End Cost (W1–W2 of Reviewer DZMH; Q1 of Reviewer 3ZSx; Reviewer 31Ga's follow-up): We provided a complexity analysis and detailed cost, LLM-call, and token-usage breakdowns, including all ablations. Despite additional curriculum overhead, ActiveSaddler achieves the same development accuracy at substantially lower end-to-end cost than the strongest fixed-curriculum baseline: 91 vs. 673 USD on GAIA2 and 30 vs. 150 USD on TB2. Similar savings hold with the cheaper GPT-5.6 Luna (5.23 vs. 15.05 USD against AutoSaddler), demonstrating that improved rollout allocation offsets the additional curriculum overhead.

Area_Chair_VYkZOpenReview ↗

AC Comment

The authors have provided additional experiments and detailed follow-up responses. I noticed that 31Ga and DZMH have not yet provided feedback on the latest responses. As the discussion period is approaching its end, I would appreciate it if both reviewers could review the additional results and clarifications. Please share your remaining thoughts and update your ratings if warranted before the discussion closes. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.