acceptodds
Under review as a conference paper at ICLR 2027

AtmoSkillRSI: Skill Formation and Recursive Refinement in Atmospheric Workflows

Abstract

Recursive self-improvement (RSI) raises a central question for scientific agents: how can task experience be consolidated into persistent updates that participate in subsequent adaptation? In AtmoSkillRSI, we study this question in atmospheric workflow construction using Earth2Studio, where agents build executable workflows evaluated with objective scientific metrics. Agents convert task experience and evaluation feedback into persistent Skills containing reusable procedural guidance and supporting code; these Skills are retained across tasks, guide subsequent workflow exploration, and can themselves be revised using later experience. Across experiments, we hold the base coding model, execution environment, task interface, and evaluator fixed, while examining variations in experience organization, training-case coverage, synthesis evidence, and repeated Skill reuse. We compare Sequential and Task-Parallel experience organization, evaluate Skill effectiveness across atmospheric task families and learning conditions, and track repeated Skill refinement across successive stages. The available results show that both Sequential and Task-Parallel learning improve mean RPF performance over Base, that broader training coverage does not consistently yield larger gains, and that mean performance increases across successive refinement stages under a fixed learning rule. Because later stages also incorporate additional exploration and accumulated history, these stage-wise trends are descriptive rather than a causal estimate of prior-Skill feedback. Overall, AtmoSkillRSI provides a controlled execution setting for studying persistent Skill formation and recursive refinement in scientific agents.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Recursive Self-Improvement, Scientific Agents, Atmospheric Workflows, AI for Science, Agent Skills

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dmDaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies persistent skill learning for atmospheric workflow construction. An agent converts execution feedback into reusable procedural guidance and code, and the authors compare sequential and task-parallel skill formation, training coverage, synthesis inputs, and three stages of skill refinement. Evaluation spans four atmospheric task families, with most analyses centered on regional probabilistic forecasting (RPF).

Critical strengths

The executable workflow setting and objective scientific metrics provide a concrete way to evaluate learned skills.

Critical weaknesses
  1. The core approach—distilling task experience into reusable skills and updating them through later interactions—closely resembles existing agent skill-learning methods. The paper does not clearly establish a new mechanism or insight beyond applying this paradigm to atmospheric workflows.
  2. Most detailed analyses use only four fixed RPF evaluation cases, which also informed development. The cross-family improvements are small for WPEP and PSSC. Moreover, the refinement experiment does not separate the effect of prior-skill reuse from additional exploration and accumulated history, while the trace comparison also changes the synthesis budget. These limitations make the central conclusions difficult to assess.
Questions for authors
  1. What specific mechanism or empirical insight distinguishes ATMOSKILLRSI from prior persistent-skill learning methods, beyond the atmospheric application?
  2. Can the authors provide fresh held-out evaluations and matched-budget ablations that isolate the contributions of prior-skill reuse and interaction traces?
2 comments

Weakness 1 — Limited Novelty

Thank you for highlighting the need to clarify our contribution. This work is an initial empirical study of RSI in atmospheric workflows, examining how Skill performance varies across tasks, experience organizations, and training coverage. We have added the previously completed Meta Skill Learner experiments in Sections 4.3 and 5.4, with RQ4: an agent proposes learning rules UU from training trajectories, and we report six evaluated candidates. This extends the investigation to learning-program modification without claiming a new general-purpose Skill consolidation algorithm.

Weakness 2 — Evaluation Cases Used During Development

Thank you for raising this concern. RPF uses 2020 cases for training and separate 2021 cases for testing; learning agents receive only training feedback. Appendices C.3 and C.7 describe training-history transfer and frozen-Skill evaluation, with test scores excluded from subsequent Skill-learning inputs. The same four held-out cases are reused across stages, so our conclusions remain limited to the cases and experimental settings studied.

Weakness 3 — Small Cross-Family Improvements

Thank you for noting the variation in improvement. Section 5.1 reports WPEP Brier-loss reductions of 1.05% and 2.06% for Sequential and TP, respectively, and a mean PSSC composite-score improvement of approximately 1.76% for both. These are modest gains and illustrate task-dependent benefits rather than consistently large improvements. The added Appendices B.4–B.5 report the corresponding methods’ observed resource usage and runtime.

Weakness 4 — Causal Attribution

Thank you for highlighting this attribution issue. We agree that the current design does not isolate the causal contribution of prior Skills. Section 5.3 reports a 3.02% decrease in mean normalized CRPS from I1 to I3 across five learning lineages. Appendix C.3 explains that successive stages receive both prior Skills and training history while collecting additional experience. The result therefore describes improvement across the complete recursive learning process.

Weakness 5 — Budget Confounding in the Trace Comparison

Thank you for identifying this budget difference. The added Appendix B.3 (Table 6) explicitly reports limits of 40 requests/900 seconds for no-trace synthesis and 48 requests/3,600 seconds for trace-enabled synthesis. Appendix C.2 (Table 10) specifies the input differences, while Appendices B.4–B.5 (Tables 7–8) report observed token usage and runtime. This comparison reflects the overall configuration difference and does not isolate the contribution of interaction traces.

Question 1 — Distinctive Mechanism or Empirical Insight

Our main empirical observations are that Skill gains depend on the task and learning conditions, broader training coverage does not guarantee improvement, and the complete recursive procedure shows aggregate improvement under the current setting. The added Sections 4.3 and 5.4 further examine how an agent modifies learning rules UU using previous training trajectories. The six candidates retain a common synthesis procedure and yield mixed outcomes, providing initial evidence for extending adaptation from Skills to their learning program.

Question 2 — Held-Out Evaluation and Matched-Budget Ablations

The existing experiments include held-out cases separate from training, with frozen-Skill evaluation described in Appendix C.7. We have not added a fresh test set or strictly matched-budget ablations. The additions instead comprise method and input specifications in Appendix C.2 and budget and resource accounting in Appendices B.3–B.5. These clarify the experimental implementation and comparison scope, but do not replace causal tests of prior-Skill reuse or interaction traces.

I appreciate the authors’ response and their careful answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.

Reviewer_znVQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

AtmoSkillRSI studies persistent Skill formation and recursive refinement in atmospheric workflow construction using Earth2Studio, where agents build executable workflows evaluated with objective scientific metrics. Agents convert task experience and evaluation feedback into persistent Skills containing reusable procedural guidance and supporting code, these Skills are retained across tasks, guide subsequent workflow exploration, and can themselves be revised using later experience. The results show that both Sequential and Task-Parallel learning improve mean RPF performance over Base, and that mean performance increases across successive refinement stages under a fixed learning rule.

Critical strengths
  • A valuable task setting for RSI. The paper introduces a meaningful setting for RSI, i.e., atmospheric workflow optimization. I think this setting is valuable, although its scope need not be limited to workflow optimization.
Critical weaknesses
  • The experimental analysis is poorly presented. To be honest, it reads like a rebuttal. I cannot understand how existing baselines perform or how much the proposed method improves over them.
  • The writing falls below academic standards. I notice numerous unusual symbols, which makes the manuscript appear insufficiently prepared and polished.
  • The method shows no contribution. This is essentially a common approach, applying RSI to any verifiable task can yield some improvement. I cannot identify a clear technical contribution.
Questions for authors

None. I do not think a more in-depth discussion is necessary.

2 comments

Weakness 1 — Unclear Experimental Presentation

Thank you for highlighting the unclear presentation. Our Base is an agent without learned Skills. On RPF, the pooled Base reference has a mean normalized CRPS of 0.031909, compared with 0.027481 for Sequential and 0.028021 for TP, corresponding to reductions of 13.88% and 12.18%.

We have added Skill contents, synthesis inputs, and complete synthesis prompt templates in Appendix C.2, together with budget limits, observed token usage, and elapsed runtime in Appendices B.3–B.5. Sample sizes and accounting scopes accompany these statistics. Our presentation revision focuses on showing baseline definitions, absolute scores, and relative changes together, with the analysis organized by research question.

Weakness 2 — Writing Quality and Unusual Symbols

Thank you for identifying the writing and notation issues. We will standardize terminology, abbreviations, and mathematical notation, remove unnecessary special formatting, and simplify the experimental narrative and figure captions. In particular, we will clarify the distinction between scientific workflows gg, persistent Skills SS, and learning rules UU, connecting each symbol directly to its role.

Weakness 3 — Lack of Technical Contribution

Thank you for highlighting the need to clarify our contribution. ATMOSKILLRSI is an initial empirical study of RSI in atmospheric workflows, providing four executable task families with objective scientific evaluation. The experiments show that broader training coverage does not consistently improve performance and that Skill gains vary across tasks and learning conditions. The constrained DAG interface addresses the complexity, runtime, and resource demands of atmospheric experimentation.

The added Meta Skill Learner and six evaluated learning-rule candidates extend the study to modification of the learning program itself. These candidates retain a common synthesis procedure and yield mixed outcomes. We present this as an empirical extension and do not claim a new general-purpose Skill consolidation algorithm.

Thanks for the response, I will keep my score.

Reviewer_6Z8zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ATMOSKILLRSI, a controlled setting for studying persistent Skill formation and recursive refinement in atmospheric scientific agents. Workflows are represented as executable DAGs in Earth2Studio. Agents distill interaction experience and evaluator feedback into reusable Skills that persist across tasks and stages and can be revised later. The paper compares experience organization (Sequential or Task-Parallel), training-case coverage, synthesis evidence (with or without interaction traces), and recursive reuse across stages (I1-I3), showing descriptive improvements over a fixed Base on Regional Probabilistic Forecasting (RPF) and variable gains across three other task families (FIIF, WPEP, PSSC).

Critical strengths

The study operationalizes recursive self-improvement (RSI) at the Skill layer in a scientific domain: persistent procedural artifacts are learned, reused, and revised under fixed learning rules. Workflows are cast as typed DAGs with executable, objectively evaluated outputs, enabling consistent comparisons and clear separation of candidate g, persistent Skill S, and update rule U. The Task-Parallel vs Sequential experience-organization comparison and the “trace-enabled synthesis” vs “no-trace” synthesis contrast are informative design axes rarely isolated in prior AI-for-science agent work. The environment, evaluator, and base coding model are held fixed across conditions. Evaluation uses strictly proper scoring rules (CRPS, Brier) or a composite scientific metric, lending scientific objectivity. Repeated runs (N = 3–5) and explicit pooled Base references are used. Recursive refinement is tracked across three stages with frozen Skill assessments at each stage. Studying persistent skill formation and RSI for scientific workflows is timely and important for AI4Science, where reproducible, auditable, and improvable agents are increasingly needed. Providing a shared, executable setting for accumulating and evaluating agent skills in atmospheric workflows could benefit the community and stimulate more rigorous RSI research.

Critical weaknesses

The paper fixes the learning rule U and studies only how experience is organized. It does not propose or analyze alternative consolidation algorithms beyond synthesis inputs, limiting technical depth on “how to learn Skills.” The recursive gains are explicitly described as descriptive. The setup cannot isolate the causal effect of prior Skill feedback versus added exploration and accumulated history. The DAG style has been widely used in related Agents works. Evaluation cases were used during development. There is no fresh held-out set, and no significance testing or uncertainty quantification is provided for the reported improvements. Baselines are limited: there is no comparison to simple non-persistent alternatives (e.g., best-of-prior-DAG retrieval, DSPy/TextGrad-style pipeline tuning) or to principled online aggregators for probabilistic forecasting (e.g., AdaWeather/VT‑MOS) that could match or exceed agent-learned mixtures under CRPS. The number of evaluation cases per run is small (four RPF cases), and task-family results are reported with minimal replication (e.g., two or three successful runs in some cells), limiting generality. While thorough overall, some critical design choices are under-specified: the exact Skill contents (size, code vs rules, selection criteria), the synthesizer prompts, and how the “trace-enabled” signals are operationalized for synthesis. Resource usage, exploration budgets, and cost fairness across conditions are not fully quantified, making efficiency and compute-normalized comparisons hard to interpret. Connections to recent graph-structured agent orchestration and compilation frameworks (e.g., GRASP skill graphs, Agint DAG compilers, LangGraph workflow orchestration) are not discussed in depth; these provide alternative formalisms for reusable, verifiable procedures and might serve as baselines or integration points.

Questions for authors

See weaknesses.

2 comments

Weakness 1 — Limited Novelty and Technical Depth

Thank you for highlighting the need to clarify our contribution and technical scope. We agree that I1–I3 fix UU and do not introduce a new Skill consolidation algorithm. ATMOSKILLRSI is an initial empirical study of RSI in atmospheric workflows. Its contribution is an executable setting spanning four task families and initial findings on how Skill performance varies with experience organization, training coverage, and synthesis inputs.

Atmospheric workflows pose substantial challenges in complexity, runtime, and resource demand: routine modeling and forecasting tasks can involve data volumes on the order of 10 GB and complete workflows lasting several hours. Unrestricted exploration can further expand into model/data acquisition, environment setup, and debugging, making repeated experiments difficult to scale. We therefore use established DAG representations with registered scientific operators, preconfigured resources, and explicit input/output and resource limits. This constrains exploration to scientific decisions—including model combinations, ensemble construction, initial-field repair, and forecast correction—and supports repeated execution and consistent evaluation. Maintaining tractable resource use in more open-ended exploration remains an important research direction.

The added Meta Skill Learner further explores modification of the learning program itself. A meta agent proposes executable learning rules UU from previous training trajectories, and we report six evaluated candidates. These retain a common synthesis procedure and yield mixed outcomes, extending the investigation from Skill refinement under a fixed rule to exploratory learning-rule search.

Weakness 2 — Causal Attribution

Thank you for highlighting this point. We agree that the current design does not isolate the causal contribution of prior Skills. Our empirical finding is that Skills guide subsequent exploration and are updated using the resulting experience; across five learning lineages, mean normalized CRPS decreases by 3.02% from I1 to I3. This demonstrates a descriptive improvement across stages in the complete recursive learning process, which combines Skill reuse, accumulated history, and additional exploration under the current evaluation setting.

Weakness 3 — Evaluation Validity

Thank you for raising this concern. Our experiments include held-out cases separate from training: for example, RPF uses 2020 cases for training and 2021 cases for testing. Each Skill is evaluated after learning completes and it is frozen; learning agents receive feedback only on training cases. In recursive refinement, I2 inherits I1’s Skill, training experience, and training synthesis report, but not its test scores. The same held-out cases are used across stages to compare performance. We have not reported evaluation on an additional fresh test set, formal significance tests, or uncertainty estimates. We therefore present the results as initial empirical findings under this fixed held-out evaluation setting.

Weakness 4 — Insufficient Baselines

Thank you for highlighting the limited baseline coverage. Our current Base is an agent without learned Skills, allowing us to compare workflow performance with and without Skills. We agree that comparisons with workflow retrieval, DSPy/TextGrad-style tuning, and probabilistic forecasting aggregators are missing. The current results provide initial evidence on Skill formation, reuse, and performance across learning conditions; they do not establish superiority over these alternatives.

Weakness 6 — Insufficient Method Specification

Thank you for highlighting these missing details. We have expanded Section 4.1.2 and Appendix C.2 with the Skill structure, an example containing five files totaling 23,022 bytes, complete synthesis prompt templates, and a comparison of TP and TP_NI inputs. We clarify how the synthesizer uses training feedback to retain, combine, revise, or discard content and records its rationale separately. Trace inputs comprise recorded prompts, conversations, tool events, diagnostics, and failures. A concrete example shows how a missing-parameter failure informs procedural guidance and supporting validation code.

Weakness 7 — Resource Accounting and Budget Fairness

Thank you for highlighting the incomplete resource accounting. We have added synthesis-budget comparisons, observed LLM token usage, and end-to-end elapsed time to Appendices B.3–B.5, with sample sizes and accounting scopes specified in the table notes. The budget comparison explicitly reports 40 requests/900 seconds for no-trace synthesis and 48 requests/3,600 seconds for trace-enabled synthesis. Accordingly, the results describe performance under each configuration’s resource settings and do not establish matched-budget or compute-normalized efficiency advantages.

Weakness 8 — Connections to Graph-Based Agent Frameworks

Thank you for highlighting these connections. Our primary purpose in using DAGs is to address the complexity, runtime, and resource demands of atmospheric workflows. Each DAG represents an agent-generated scientific computation, with registered operators connected by data dependencies. Together with preconfigured models, data, and resource limits, this turns open-ended development into a bounded search over workflows that can be repeatedly executed and consistently evaluated. Here, the graph defines and constrains the scientific task space, whereas the cited frameworks emphasize skill composition, program compilation, or agent orchestration.

These works also offer useful directions for future extensions. GraSP’s dependency modeling and execution verification could inform more complex Skill composition; Agint’s graph compilation could inform automatic scientific program construction; and LangGraph’s state and execution management could support longer, multistage agent workflows.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.