- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies persistent skill learning for atmospheric workflow construction. An agent converts execution feedback into reusable procedural guidance and code, and the authors compare sequential and task-parallel skill formation, training coverage, synthesis inputs, and three stages of skill refinement. Evaluation spans four atmospheric task families, with most analyses centered on regional probabilistic forecasting (RPF).
Critical strengths
The executable workflow setting and objective scientific metrics provide a concrete way to evaluate learned skills.
Critical weaknesses
- The core approach—distilling task experience into reusable skills and updating them through later interactions—closely resembles existing agent skill-learning methods. The paper does not clearly establish a new mechanism or insight beyond applying this paradigm to atmospheric workflows.
- Most detailed analyses use only four fixed RPF evaluation cases, which also informed development. The cross-family improvements are small for WPEP and PSSC. Moreover, the refinement experiment does not separate the effect of prior-skill reuse from additional exploration and accumulated history, while the trace comparison also changes the synthesis budget. These limitations make the central conclusions difficult to assess.
Questions for authors
- What specific mechanism or empirical insight distinguishes ATMOSKILLRSI from prior persistent-skill learning methods, beyond the atmospheric application?
- Can the authors provide fresh held-out evaluations and matched-budget ablations that isolate the contributions of prior-skill reuse and interaction traces?
2 comments
I appreciate the authors’ response and their careful answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.
Weakness 1 — Limited Novelty
Thank you for highlighting the need to clarify our contribution. This work is an initial empirical study of RSI in atmospheric workflows, examining how Skill performance varies across tasks, experience organizations, and training coverage. We have added the previously completed Meta Skill Learner experiments in Sections 4.3 and 5.4, with RQ4: an agent proposes learning rules U from training trajectories, and we report six evaluated candidates. This extends the investigation to learning-program modification without claiming a new general-purpose Skill consolidation algorithm.
Weakness 2 — Evaluation Cases Used During Development
Thank you for raising this concern. RPF uses 2020 cases for training and separate 2021 cases for testing; learning agents receive only training feedback. Appendices C.3 and C.7 describe training-history transfer and frozen-Skill evaluation, with test scores excluded from subsequent Skill-learning inputs. The same four held-out cases are reused across stages, so our conclusions remain limited to the cases and experimental settings studied.
Weakness 3 — Small Cross-Family Improvements
Thank you for noting the variation in improvement. Section 5.1 reports WPEP Brier-loss reductions of 1.05% and 2.06% for Sequential and TP, respectively, and a mean PSSC composite-score improvement of approximately 1.76% for both. These are modest gains and illustrate task-dependent benefits rather than consistently large improvements. The added Appendices B.4–B.5 report the corresponding methods’ observed resource usage and runtime.
Weakness 4 — Causal Attribution
Thank you for highlighting this attribution issue. We agree that the current design does not isolate the causal contribution of prior Skills. Section 5.3 reports a 3.02% decrease in mean normalized CRPS from I1 to I3 across five learning lineages. Appendix C.3 explains that successive stages receive both prior Skills and training history while collecting additional experience. The result therefore describes improvement across the complete recursive learning process.
Weakness 5 — Budget Confounding in the Trace Comparison
Thank you for identifying this budget difference. The added Appendix B.3 (Table 6) explicitly reports limits of 40 requests/900 seconds for no-trace synthesis and 48 requests/3,600 seconds for trace-enabled synthesis. Appendix C.2 (Table 10) specifies the input differences, while Appendices B.4–B.5 (Tables 7–8) report observed token usage and runtime. This comparison reflects the overall configuration difference and does not isolate the contribution of interaction traces.
Question 1 — Distinctive Mechanism or Empirical Insight
Our main empirical observations are that Skill gains depend on the task and learning conditions, broader training coverage does not guarantee improvement, and the complete recursive procedure shows aggregate improvement under the current setting. The added Sections 4.3 and 5.4 further examine how an agent modifies learning rules U using previous training trajectories. The six candidates retain a common synthesis procedure and yield mixed outcomes, providing initial evidence for extending adaptation from Skills to their learning program.
Question 2 — Held-Out Evaluation and Matched-Budget Ablations
The existing experiments include held-out cases separate from training, with frozen-Skill evaluation described in Appendix C.7. We have not added a fresh test set or strictly matched-budget ablations. The additions instead comprise method and input specifications in Appendix C.2 and budget and resource accounting in Appendices B.3–B.5. These clarify the experimental implementation and comparison scope, but do not replace causal tests of prior-Skill reuse or interaction traces.