- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLM agents can translate broad capability goals into effective self-improvement without access to downstream evaluation task specifications. The proposed ASPIRE benchmark supports model-weight and agent-harness updates, with 520 hidden, expert-authored evaluation items across six goals. Experiments compare vague-goal and explicit-task settings, examine weight updates under final-only and adaptive-feedback protocols, and test harness evolution with fixed weights. Agents generally complete update workflows, but retained gains over the base model are rare, and successor harnesses remain below the reference in mean score. The main contributions are formalizing goal operationalization, introducing the benchmark, and analyzing the gap between executing updates and achieving capability gains.
Critical strengths
- Valuable problem formulation and benchmark. ASPIRE makes goal operationalization in autonomous learning measurable by connecting agent-directed data selection, updates, and self-evaluation with hidden evaluation, providing a useful setting for studying this challenge.
- Useful outcome accounting and preliminary failure evidence. The paper distinguishes local recovery, above-base improvement, and retained gains. Its trajectory analyses provide concrete, though preliminary, evidence of data-selection mismatch, narrow self-evaluation, and output-pipeline failures, helping interpret apparent progress and identify directions for further investigation.
Critical weaknesses
- Insufficient direct evaluation of goal operationalization. RQ2 mainly reports checkpoint scores and retained gains, without systematically presenting or assessing the proxy tasks, learning objectives, and agent-constructed validation criteria. The reported cases provide partial evidence, but insufficiently distinguish poor goal operationalization from downstream training failures. This limits insight into the paper’s central question of deciding what to learn and how to verify progress.
- Limited evidence of robustness. The adaptive-feedback protocol uses one run per configuration–goal cell, and RQ3 uses one harness-creation trajectory per condition. RQ2’s trainable models are also limited to two sizes from the same family. These choices leave the sensitivity of key findings to search randomness and model selection unclear.
Questions for authors
Could the authors systematically report the proxy tasks, learning objectives, and validation criteria selected or constructed in RQ2, including how often agents performed their own validation? How well do these artifacts align with the capability goals, and how do agent-validation scores relate to hidden-evaluation scores? This evidence would help distinguish goal-operationalization failures from downstream training failures.
Could the authors provide independent evolution repeats for a small set of key configurations, particularly the sole retained improvement in RQ2 and the strongest harness creator in RQ3, and report across-run variability? This would help assess whether the observed gains and gaps to the reference are stable.
Which specific completion requirement did the Qwen3.5-4B Self science run fail, and why did this require rejecting its otherwise verified, above-base checkpoint?
5 comments
Response to Reviewer qS48 (2/5)
W1 / Q1: Direct evaluation of goal operationalization; systematic reporting of proxy tasks, objectives, self-validation, and alignment
W1: Insufficient direct evaluation of goal operationalization. RQ2 mainly reports checkpoint scores and retained gains, without systematically presenting or assessing the proxy tasks, learning objectives, and agent-constructed validation criteria. …
Q1: Could the authors systematically report the proxy tasks, learning objectives, and validation criteria selected or constructed in RQ2, including how often agents performed their own validation? How well do these artifacts align with the capability goals, and how do agent-validation scores relate to hidden-evaluation scores? …
Why the hidden-set score is the primary measure. Under a vague goal, the final hidden-set score is the most direct end-to-end test of goal operationalization, i.e., of turning a broad goal into training data, learning objectives, and validation criteria. The agent cannot optimize against hidden items, so whether its chosen data and validation work is ultimately shown by whether the target capability improves (§2.2, p.3). We do not use intermediate artifacts as the primary metric, for three reasons:
- No reference plan: the same broad goal can be decomposed in many valid ways, so there is no reference learning plan to grade against.
- Incomparable validation: self-built validation varies across runs (checklists, held-out sets, scripts) and cannot be placed on a common scale.
- Subjective grading: human grading of alignment would introduce a subjective scale.
We therefore treat the hidden-set score as primary and use process statistics to diagnose failures.
Process statistics already in the paper. The appendix reports full-coverage extraction over 63,973 events, 124 training plans, and 95 evaluation attempts (Appendix C.5, p.17). Examples (Appendix C.4, p.17):
- 4B Self (30/32 imports) and 9B Self (11/15 imports) concentrate on mathematics data;
- SFT/GRPO ratios per configuration;
- search profiles of Luna, Terra, and Sol.
New results from the discussion period. We systematically audited and back-tested all 54 RQ2 runs.
| Audit dimension | Scope | Result | Notes and log limits |
|---|---|---|---|
| Rate of declared self-validation | All 54 runs (querying the hidden aggregate excluded) | Final-only: 11 / 24 (45.8%); Adaptive: 5 / 30 (16.7%) | A lower bound from controller logs: agents with external feedback build their own validation less often. A declaration does not imply successful validation. |
| Self-validation vs. hidden score | All 54 runs | No common-scale self-score series | Self-built validation metrics are heterogeneous, and the portable logs redact raw messages, so a pooled correlation is not meaningful. RQ3 cases show narrow self-evaluation diverging sharply from the hidden set (e.g., Luna scores 8/8 on its own checklist but 19.32 vs. the reference 28.64). |
| Blind labeling of proxy task vs. goal | All 54 runs, hidden scores masked | Three-way independent AI blind labeling, agreement 46 / 54 (85.2%) | Off-domain mathematics imports and the 13 single-digit-label collapses (989 single-digit answers) are the main front-end operationalization errors. |
| Public-benchmark results when the data match the goal | 42 pairs of trained model vs. matching public test set (GSM8K / MATH-500 / MMLU / SciQ) | 38 / 42 below base; mathematics goal 18 / 18 below base | Even on the mathematics goal, where the agent chose matching GSM8K/MATH data, none of the 9 models exceeds base on the public test sets. |
These numbers separate the two failure types:
- Front end: more than half of the final-only runs and more than 80% of the adaptive runs never even declare self-validation, and off-domain data choices and single-digit label mismatches are frequent.
- Back end: even with matching mathematics data, continued training quickly collapses the model without a reliable intermediate self-check.
Response to Reviewer qS48 (3/5)
W2 & Q2: Robustness; independent repeats of key configurations
W2: Limited evidence of robustness. The adaptive-feedback protocol uses one run per configuration–goal cell, and RQ3 uses one harness-creation trajectory per condition. RQ2's trainable models are also limited to two sizes from the same family. …
Q2: Could the authors provide independent evolution repeats for a small set of key configurations, particularly the sole retained improvement in RQ2 and the strongest harness creator in RQ3, and report across-run variability? …
What the paper already reports.
- The final-only protocol runs each model–goal pair twice and reports the mean, Avg@2 (§3.3 p.6; Table 4 p.17).
- The overall negative finding is consistent across both protocols, five decision models, and 54 runs (Figure 3 p.6; Figure 4 p.7).
- The paper states that the adaptive protocol uses one run per cell (§3.3 pp.6–7; Appendix C.6 p.18; §5 p.9).
- It also states that 4B vs. 9B Self changes the decision model, trainable base, and initial score together, so it is not a pure scale effect.
Our conclusions concern the Qwen3.5 family of trainable models and are not directly extrapolated to other model families.
New stability results. For the configurations the reviewer singled out, the RQ2 positive gains (especially the sole retained Terra 4B mathematics gain) and the RQ3 harnesses (especially Sol's H1, the closest to the reference), we added repeated re-evaluation, paired item-level bootstrap (10,000 resamples), and independent repeated creation of the Sol harness.
1. RQ3: standard deviation over three executions and paired item-level bootstrap for all frozen harnesses.
| Harness | Task macro (mean ± s.d.) | Example micro (mean ± s.d.) | Diff. vs. H0 [95% CI] | Conclusion |
|---|---|---|---|---|
| Qwen-Agent reference (H0) | 28.64 ± 0.69 | 27.65 ± 0.57 | — | Reference is highly stable (s.d. < 0.7) |
| Luna H1 | 19.32 ± 0.98 | 18.33 ± 1.11 | −9.32 [−16.28, −3.07] | Significantly below reference |
| Terra H1 | 20.76 ± 0.74 | 20.14 ± 0.88 | −7.88 [−15.59, −1.82] | Significantly below reference |
| Sol H1 | 27.22 ± 2.23 | 25.97 ± 2.58 | −1.42 [−5.04, +1.46] | No significant difference (Sol reverted to the H0 policy) |
Standard deviations across executions are generally small (0.69 for H0, 0.74–0.98 for Luna and Terra), enough to clearly detect the degradation of Luna and Terra. Sol does not differ significantly from H0 (its CI includes 0) because, as Appendix D notes, Sol restored the original Qwen-Agent policy after rejecting complex designs and added only a non-empty check that never fired. The two harnesses share the same runtime logic.
Response to Reviewer qS48 (1/5)
We thank the reviewer for the comments on process-level evaluation and robustness. First, the definitions of the two RQ2 protocols used below:
Qwen3.5-4B SelfandQwen3.5-9B Selfeach run twice independently on each of six goals, for 2 × 6 × 2 = 24 runs. ("Self" means the frozen initial model acts as its own decision model.)Adaptive outcomes are reported at three levels: