- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces ActiveSaddler, an automated curriculum-learning framework for offline LLM agent harness optimization. Existing harness optimizers mainly focus on how to update prompts, tools, and control logic, while using a training-scenario schedule fixed before optimization. ActiveSaddler instead adapts the scenario curriculum as the harness evolves. It organizes observed failures into dynamically instantiated failure-pattern arms, uses an LLM-based prioritizer to estimate the expected learning value of revisiting each arm, and employs an exploration controller to decide between repairing known weaknesses and executing unseen scenarios to discover new ones.
Critical strengths
-
The paper identifies an underexplored limitation of existing harness optimization methods: while the harness evolves during optimization, the training scenarios used to generate feedback are typically scheduled in advance. Formulating adaptive scenario selection as a curriculum-learning problem is timely and practically relevant.
-
ActiveSaddler combines dynamically constructed failure-pattern arms, adaptive arm prioritization, and exploration of unseen scenarios. Under the same task-agent rollout budget, it improves test Pass@1 over fixed-order AutoSaddler by 4.4 percentage points on GAIA2 and 7.5 points on Terminal-Bench 2.0.
-
The paper includes component-wise ablations, comparisons with fixed and non-LLM curriculum strategies, an application to a second optimizer (GEPA), cost–accuracy analysis, and qualitative case studies. The detailed algorithms, prompts, data splits, and released implementation also support reproducibility.
Critical weaknesses
-
Robustness across independent optimization runs. Most reported uncertainty comes from three test-time executions of the same optimized harness, rather than from repeated end-to-end optimization runs. GAIA2 includes only two independent optimization runs, while Terminal-Bench 2.0 and most ablations appear to rely on a single optimization trajectory. Given the stochasticity of LLM-based diagnosis, patch generation, and curriculum decisions, additional optimization seeds and statistical tests are needed to establish that the reported gains are robust.
-
The generality and formal justification of the approach. Although ActiveSaddler is described as a non-stationary bandit method, its arm scores and exploration decisions are produced primarily through LLM judgments rather than an explicit reward estimator or a theoretically grounded bandit update. Moreover, the evaluation covers only two benchmarks and largely one model configuration. It is therefore unclear how well the method transfers across 1. different LLMs, 2. harness optimizers, and 3. task distributions.
Questions for authors
ActiveSaddler requires additional LLM calls for failure-pattern extraction, arm scoring, and exploration decisions. If all methods are given the same total cost, token budget, or number of optimizer-side LLM calls, does ActiveSaddler still outperform the strongest fixed-curriculum baseline?
5 comments
Response to Reviewer 3ZSx (2/3)
W2: The generality and formal justification of the approach. Although ActiveSaddler is described as a non-stationary bandit method, its arm scores and exploration decisions are produced primarily through LLM judgments rather than an explicit reward estimator or a theoretically grounded bandit update. Moreover, the evaluation covers only two benchmarks and largely one model configuration. It is therefore unclear how well the method transfers across 1. different LLMs, 2. harness optimizers, and 3. task distributions.
Thank you for your thoughtful comments. To address the concern, we first present additional evaluations across different LLMs, harness optimizers, and task distributions, followed by a clarification of the motivation and empirical justification for our bandit-based formulation.
1. Generality. We evaluate ActiveSaddler across different LLMs, harness optimizers, and task distributions.
(1) Different LLMs. We replace GPT-5.5 with GPT-5.6 Sol as the underlying LLM while keeping all other settings unchanged. ActiveSaddler outperforms AutoSaddler by 3.5 pp with GPT-5.6 Sol, demonstrating that its benefits are not limited to the original LLM configuration.
| Harness (Type) | GAIA2 (300 tasks; Pass@1) |
|---|---|
| Default Agent w/ GPT-5.6 Sol (manual) | 56.8 ± 1.7 |
| AutoSaddler w/ GPT-5.6 Sol | 59.6 ± 2.5 |
| ActiveSaddler w/ GPT-5.6 Sol | 63.1 ± 1.4 |
(2) Different Harness Optimizers. We integrate ActiveSaddler with GEPA, a different offline harness optimizer, while preserving GEPA's harness update and candidate-selection procedures (Appendix E). ActiveSaddler improves GEPA by 3.0 pp, showing that its curriculum benefits extend beyond AutoSaddler.
| Harness (Type) | GAIA2 (300 tasks; Pass@1) |
|---|---|
| Default Agent (manual) | 53.6 ± 1.1 |
| GEPA | 54.2 ± 2.2 |
| GEPA w/ ActiveSaddler | 57.2 ± 0.4 |
(3) Different Task Distributions. During this rebuttal, we additionally evaluated ActiveSaddler on ClawMark [5], with OpenClaw, a real-world agent harness commonly used for personal agents, as its underlying task agent. ClawMark features multi-day workplace tasks involving interactions across multiple services, providing a distinct task distribution from GAIA2 and Terminal-Bench 2.0. ActiveSaddler outperforms AutoSaddler by 4.1 pp on ClawMark, demonstrating consistent improvements on a third task distribution.
| Harness (Type) | ClawMark (22 tasks; Weighted Score) |
|---|---|
| Default Agent w/ GPT-5.5 (manual) | 73.9 ± 1.4 |
| AutoSaddler w/ GPT-5.5 | 76.3 ± 0.8 |
| ActiveSaddler w/ GPT-5.5 | 80.4 ± 1.0 |
2. Justification of the LLM-based approach and comparison with bandit-based baselines.
Following prior bandit-based curriculum learning [1, 2, 3], ActiveSaddler models each failure pattern as an arm with a non-stationary expected reward, representing the potential improvement from further optimizing the corresponding harness weakness. As the harness evolves through successive patches, previously identified weaknesses may be resolved or reintroduced, changing the expected reward of each arm. However, unlike prior curriculum learning methods designed for neural network training, harness optimization lacks direct learning-progress signals, such as loss reduction [1] or TD error [3], for estimating arm-specific rewards. Thus, ActiveSaddler chooses to use LLM judgments grounded in the current harness and optimization history to estimate each arm's evolving improvement potential and guide exploration decisions.
To evaluate this design, Appendix D (Table 5) compares two alternatives adapted from existing bandit algorithms under the same optimization setting.
- EMA-based Arm Prioritization, adapted from the non-stationary bandit-based curriculum of Teacher-Student Curriculum Learning [2], using exponential moving averages to track residual failure rates.
- UCB-AIR-based Exploration Control, adapted from the infinitely many-armed bandit algorithm [4], using an arm-increasing rule to determine when to explore unseen scenarios.
| Harness (Type) | GAIA2 (300 tasks; Pass@1) |
|---|---|
| Default Agent (manual) | 53.6 ± 1.1 |
| EMA-based Arm Prioritization | 56.7 ± 1.5 |
| UCB-AIR-based Exploration Control | 55.6 ± 1.4 |
| ActiveSaddler | 59.8 ± 1.0 |
EMA-based Arm Prioritization and UCB-AIR-based Exploration Control underperform ActiveSaddler (59.8%) by 3.1 and 4.2 pp, respectively. Together, these results show that, among the tested alternatives, our LLM-based modules are more effective at deciding which known failure pattern to revisit and when to explore unseen scenarios to discover new weaknesses.
Overall, these results provide empirical evidence that ActiveSaddler's benefits extend across different LLMs, harness optimizers, and task distributions. We will incorporate the new results into the revised manuscript.
Response to Reviewer 3ZSx (3/3)
[1] Graves, Alex, et al. "Automated curriculum learning for neural networks." international conference on machine learning. Pmlr, 2017.
[2] Matiisen, Tambet, et al. "Teacher–student curriculum learning." IEEE transactions on neural networks and learning systems 31.9 (2019): 3732-3740.
[3] Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. "Prioritized level replay." International Conference on Machine Learning. PMLR, 2021.
[4] Wang, Yizao, Jean-Yves Audibert, and Rémi Munos. "Algorithms for infinitely many-armed bandits." Advances in Neural Information Processing Systems 21 (2008).
[5] Meng, Fanqing, et al. "ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents." arXiv preprint arXiv:2604.23781 (2026).
Q1: ActiveSaddler requires additional LLM calls for failure-pattern extraction, arm scoring, and exploration decisions. If all methods are given the same total cost, token budget, or number of optimizer-side LLM calls, does ActiveSaddler still outperform the strongest fixed-curriculum baseline?
Thank you for this important question. ActiveSaddler achieves better end-to-end cost efficiency than the strongest fixed-curriculum baselines, even after accounting for all optimizer-side and task-agent computation. Appendix F (Figure 5) reports cumulative monetary cost, LLM calls, and token usage. The following table compares the end-to-end cost of achieving the same development accuracy.
| Benchmark | Target Dev. Accuracy (%) | Strongest Fixed Curriculum End-to-End Cost ($) | ActiveSaddler End-to-End Cost ($) |
|---|---|---|---|
| GAIA2 | 56.9 | 673 | 91 |
| GAIA2 | 58.5 | 673 | 298 |
| TB2 | 68.4 | 150 | 30 |
| TB2 | 78.9 | 363 | 128 |
At the same development accuracy, ActiveSaddler reduces end-to-end cost by 86.5% and 55.7% on GAIA2 (673 USD to 91 USD; 673 USD to 298 USD) and 80.0% and 64.7% on TB2 (150 USD to 30 USD; 363 USD to 128 USD). Furthermore, on GAIA2, ActiveSaddler reaches 63.1% at 1,178 USD, whereas category ordering remains at 58.5% even after spending 1,479 USD. On TB2, ActiveSaddler reaches 84.2% at 181 USD, surpassing scenario ordering's peak of 78.9% at 363 USD. Similar efficiency advantages hold for cumulative LLM calls and token usage. These results demonstrate that improved rollout allocation offsets the additional curriculum overhead. We will bring these key observations and cost comparisons from Appendix F into the main text of the revised manuscript.
Response to Reviewer 3ZSx (1/3)
We sincerely appreciate your thorough review of our paper. Below are our responses to your concerns:
Thank you for raising the concern regarding robustness across independent optimization runs. To address this concern, we conducted a second independent optimization run on Terminal-Bench 2.0 for ActiveSaddler and the AutoSaddler-based baselines during the rebuttal period. Each resulting harness was evaluated with three repeated test-time executions.
ActiveSaddler achieved 80.0% and 79.2% Pass@1 across the two independent runs, respectively. Both runs outperformed the strongest AutoSaddler-based baseline across either run (73.3%) by 6.7 and 5.9 pp, respectively. These consistent gains across independent runs on both benchmarks suggest that ActiveSaddler’s improvement is not driven by a single favorable optimization trajectory. We will include these results in the revised manuscript and extend repeated runs to the ablation studies.