acceptodds
Under review as a conference paper at ICLR 2027

ROSE: Rollout-Free Skill Evolution via Static Task–Skill Analysis

Abstract

Skill evolution offers a practical way to improve agent performance without updating model weights. Existing methods often rely on repeated task rollouts and evaluator feedback to guide skill revision, making skill construction costly and time-consuming. Yet procedural deficiencies, such as formatting errors and omitted steps, can often be identified from task requirements before execution. Motivated by this observation, we introduce ROSE, a rollout-free framework for efficient agent skill evolution inspired by static program analysis. ROSE organizes an initial skill, training-task specifications, and visible artifact metadata into a shared code-style intermediate representation. It then audits this representation for unmet requirements, contract mismatches, and predictable execution risks, and uses the findings to guide localized skill revisions while preserving useful procedures. Construction requires only three LLM calls, without task rollouts or evaluator feedback. The resulting skill is frozen and reused across held-out tasks without test-time adaptation. Across SpreadsheetBench, LiveMathematicianBench, DocVQA, and OfficeQA with four agent models, ROSE achieves competitive performance, obtaining a macro-average score of 58.5%, compared with 56.8% for SkillOpt. On matched SpreadsheetBench construction instances, ROSE uses 9,183 tokens and completes construction in 32.7 seconds, compared with 3.86 million tokens and approximately 100 minutes for SkillOpt. These results demonstrate that static task evidence can support competitive held-out performance with substantially reduced skill-construction cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, skill evolution, program analysis, rollout-free optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nSXTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method for agentic skill evolution through program synthesis. Exisiting methods mostly need to gather feedback to to optimize the skills. The proposed method can observe the evidence and directly optimize the skills without costly feedback gathering. The method organizes initial skills and task evidence into a program-like representation that make the task requirement and skill procedure directly comparable, and audits for missing requirements. Extensive experimental results show that the proposed method can optimize the skills without rollout-based optimization.

Critical strengths
  1. The presentation of this paper is clear.
  2. The idea of optimizing the skills without rollout is interesting and might be useful for resource constrained settings.
  3. The experimental results show the effectiveness of the proposed method.
Critical weaknesses
  1. While the idea of optimizing the skills without actual rollouts seems interesting, more types of mistakes require actually running the skills to discover. Simply auditing the static skills may be sufficient to discover some easy cases, but may not be sufficient for more complex cases.
  2. The types of information to be auditted seem fixed and hand crafted. It is unknown if this information is sufficient to detect the errors in the skills, or if the framework can scale to more complex settings, where more aspects of information needs to be considered in order to audit the skills.
  3. Simply auditing the skills and improve it, without actual feedback and stronger optimization algorithms (e.g., feedback with reinforcement learning) may limits how far the skill can improve. In some cases, its improvement upper bound may be shallow compared to more principled skill optimization approaches. Also if the audit is not accurate, the errors may accumulate over optimization iterations.
Questions for authors

Please refer to the weakness sections and try to answer the questions.

Reviewer_joiKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ROSE aims to improve a reusable agent skill without running each task over and over. It takes an initial skill, the training task specifications, and visible file metadata, arranges them into a shared code-style intermediate representation, and audits that representation for unmet requirements, interface mismatches, and predictable failure points. The findings drive a single localized edit to the skill, which is then frozen and reused on held-out tasks. The premise is that many procedural flaws can be read off the task before execution, so the costly rollout loop can be skipped. Experiments report accuracy close to a rollout-based baseline at far lower construction cost.

Critical strengths
  1. ROSE makes a genuinely attractive cost–performance argument. It reaches comparable accuracy while spending orders of magnitude fewer tokens and far less wall-clock time on construction, and the framing itself, read the task rather than roll it out, is clean and reusable.
  2. The ablation backs up the design. Cutting the pipeline down to three ordinary revision prompts drops performance from 60.36% to 35.00%, so the intermediate representation and the audit appear to be doing real work rather than just three cheap calls.
Critical weaknesses
  1. The reported accuracy cannot establish even the parity the paper's thesis needs. ROSE trails SkillOpt on three of four benchmarks, yet Table 1 gives a single number per cell with no uncertainty for either method, and for the only model with stability runs its values match favorable individual runs rather than the three-run means, so the reader cannot tell genuine parity from a small consistent deficit.
  2. Fair-comparison and reproduction details are also underspecified. I would want the paper to say where the initial skill comes from, whether every method gets the same data access and tuning budget.
Questions for authors
  1. Please state how the Table 1 numbers were aggregated and report repeats with ±SD for both ROSE and SkillOpt. For GPT-5.4-Nano, the only model with stability runs, the Table 1 values (60.4, 29.0, 77.8, 50.0) coincide with single runs from Table 5: Run 1 on SpreadsheetBench, LiveMath, and OfficeQA, and the best Run 3 on DocVQA. They do not match the three-run means (60.12, 26.61, 74.33, 48.45), and on LiveMath the mean falls below SkillOpt's 27.2. Were the Table 1 cells single runs, and how were they selected?
  2. What is the provenance of the initial skill S0, and are initialization, data access, evaluator, and tuning budget aligned across methods?
  3. Since the 50/24/172 split sums to 246 items — OfficeQA Full rather than the cited 133-question Pro set — and OfficeQA has no official split, could you explain how this partition was constructed, relabel the evaluation as OfficeQA Full, and confirm that all baselines share the same 172-item test pool?
  4. Reference details: (a)he LiveMathematicianBench paper's actual title is "LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches," but the bibliography says "Livemathematicianbench: A live benchmark for mathematician-level reasoning with proof sketches"; (b) the WebArena entry lists "volume 2024, pp. 15585–15606," but ICLR proceedings have no page number; (c) the TextGrad title contains garbled quotation marks. Please correct these in the bibliography.
Reviewer_mWDfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes ROSE, a rollout-free framework for agent skill evolution. Inspired by static program analysis, ROSE first converts the initial skill, task specifications, and artifact metadata into a structured intermediate representation, then audits potential procedural deficiencies and synthesizes an updated skill. The entire process requires only three LLM calls without task rollouts or evaluator feedback. Experiments across four benchmarks demonstrate competitive performance compared with rollout-based skill evolution methods, while substantially reducing construction costs.

Critical strengths

The paper is clearly written and well-organized, making the proposed framework easy to understand and follow. The experiments cover multiple benchmarks and agent models, with ablation studies and efficiency comparisons providing empirical support for the approach. The substantial reduction in skill construction costs is also a practical advantage.

Critical weaknesses
  1. Insufficient research contribution and weak motivation

The research contribution of this paper is highly limited, and I find the motivation fundamentally unconvincing. The authors motivate their approach by arguing that agent failures often arise from basic formatting errors, omitted steps, or violations of output requirements. However, these are largely elementary instruction-following and prompt-engineering issues rather than substantive challenges in agent skill acquisition or evolution. Addressing such deficiencies by explicitly incorporating task requirements into a skill document does not, by itself, constitute a meaningful research contribution.

More importantly, the paper fails to justify why its proposed three-stage framework is necessary in the first place. Since ROSE already has access to detailed task specifications, artifact metadata, and execution constraints, a much simpler solution would be to directly instruct the agent to inspect the relevant task files and follow their requirements during execution. Alternatively, the same static information could be directly provided to an LLM to revise the initial skill in a single step.

Overall, the proposed method appears to be an unnecessarily elaborate procedure for incorporating readily available task requirements into agent instructions. The paper neither establishes a compelling need for this additional complexity nor demonstrates that ROSE enables meaningful skill improvements beyond basic instruction refinement. Consequently, I find both the novelty and the significance of the proposed contribution seriously questionable.

  1. Questionable motivation for completely eliminating rollouts

Another major concern is that the paper does not adequately justify why skill evolution should be entirely rollout-free. While reducing the computational cost of skill evolution is a reasonable objective, completely eliminating task execution appears to be an unnecessarily restrictive design choice rather than a well-motivated research goal.

More fundamentally, many deficiencies in agent skills cannot be identified through static analysis alone. In realistic agentic tasks, failures often arise from unexpected tool behavior, incorrect strategy selection, dynamic environment states. Such problems may only become apparent when the agent actually attempts to execute the task. Without observing any execution trajectories or feedback, it is unclear how ROSE could identify or address these deficiencies.

Questions for authors
  1. Could the authors justify the necessity and significance of the proposed framework over simply instructing agents to follow available task specifications and constraints?

  2. Could the authors explain why completely eliminating rollouts is necessary or preferable to using limited execution feedback for more effective skill evolution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.