acceptodds
Withdrawn from ICLR 2027

SkillSpec: Consensus‑Gated Agent Skill Evolution via Representation Specialization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Natural-language skills are textual procedural memories through which large language model (LLM) agents retain reusable task knowledge without updating model weights. Existing methods typically treat skills as either static artifacts or monolithic documents optimized using aggregate validation scores as feedback. However, representing a skill as a monolithic document restricts optimization to its textual content, without explicitly modeling the structure through which procedural knowledge is retrieved and executed. We identify a key distinction between learning what knowledge to retain and determining how to organize it: textual updates should first be validated through execution evidence, after which the retained knowledge should be structured according to its procedural dependencies and retrieval requirements. To this end, we introduce SkillSpec, a two-phase framework comprising consensus-gated evolution and representation specialization. In the consensus-gated phase, complementary editing intents generate complete candidate skills, which are accepted as updates only when paired evaluations reach consensus: an overall improvement over the current skill with no performance degradation in any individual evaluation run. In the specialization phase, signals of process and redundancy sensitivity derived from the optimization trajectory guide the selection of a flat, graph, or hybrid representation while preserving the validated backbone as an immutable lineage source. Across six benchmarks and several frontier language models, SkillSpec improves average success rate over prior state-of-the-art skill optimization methods by 4.8%. These results demonstrate that reliable skill evolution and representation specialization address complementary objectives: deciding what knowledge to retain and how to structure it for inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agents, Agent Skill Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fjY6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper propose a two stage skill selection process which use gate to control what skill should accept and what should be remove out. Two stages are specifically 1. generate the cadidate skills and select skills 2. update the skill representation by different kind of structures. Paper varified on 6 different benchmarks and 3 different base models.

Critical strengths

This paper propose a two stage skill selection process which use gate to control what skill should accept and what should be remove out. Two stages are specifically 1. generate the cadidate skills and select skills 2. update the skill representation by different kind of structures. Paper varified on 6 different benchmarks and 3 different base models.

Critical weaknesses

Paper organization needs to be improved; repeated context makes the paper's structure not very tight strucrure and loses concentration. The efficiency of the method should be carefully evaluated. The method in stage one is not very novel; it only calculates the successful rate gain and uses a threshold to cut off skills below the threshold. The literature review is not comprehensive; the gate-based selection is not a new idea; the method should carefully position its contribution.

Questions for authors

1.How do sensitive scores work? In the paper, where is the definition of the sensitive function f? Not only f, many notations lack clear definitions, and it is unclear how they work. For example, "p", I see it as a very important score, but how do you get it? How do you know if the definition or the collection process is reasonable? Authors should explain p r, c, and carefully about what are they and how can it works.

  1. Following up on question 1, the definition of the function f is not clear; the following equation 6 looks confusing and not reasonable.

  2. The experiments part uses only GPT. It might be good to show more base model results because it will give a good evaluation of the method's performance stability and will provide a cost and efficiency evaluation of different methods.

Reviewer_KYPKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SkillSpec, a two-phase framework for optimizing large language model (LLM) agent skills without updating model weights. Existing methods optimize skills as static monolithic text using aggregate scores, which often masks performance regressions across task instances and ignores procedural dependencies. SkillSpec decouples content learning from structural organization: Consensus-Gated Evolution (Phase I) and Representation Specialization (Phase II). Across six benchmarks, SkillSpec achieves significant performance improvements over SOTA methods like SkillOpt.

Critical strengths
  1. Rigorous, Regression-Aware Content Validation (Consensus Gate) Instead of relying on unstable aggregate scores that can hide instance-level regressions, Phase I evaluates candidates via paired instance comparisons and requires non-negative aggregate paired gain across multiple seeded executions. This ensures that adopted skill edits reflect robust, persistent improvements rather than random evaluation variance.

  2. Principled Decoupling of Skill Content and Structural Representation The framework explicitly separates what knowledge to retain from how to structure it for execution. By leveraging telemetry signals (Process Sensitivity & Redundancy Sensitivity) to dynamically route skills into Flat, Graph, or Hybrid representations, SkillSpec avoids the pitfalls of fixed monolithic prompts and optimizes procedural retrieval for diverse task demands.

Critical weaknesses
  1. High Computational Overhead and Inefficiency in Validation. Phase I requires running every candidate skill across the entire validation dataset for screening, followed by multiple additional validation runs across all confirmation finalists (KK extra runs per candidate). Evaluating whole-agent execution trajectories over multi-step benchmarks repeatedly creates significant computational and inference cost bottlenecks. For real-world applications or longer-horizon benchmarks, this exhaustive multi-pass validation gate makes the evolution framework largely impractical compared to lighter-weight optimizers or direct offline editing.

  2. Heuristic Thresholding and Arbitrary Structural Routing Policy. Phase II relies heavily on hand-crafted heuristic cutoffs to decide how to structure the skill. The router uses arbitrary fixed bootstrap threshold parameters (ζψ=0.50\zeta_{\psi} = 0.50, ζρ=0.50\zeta_{\rho} = 0.50) to assign skills into Flat, Graph, or Hybrid representations based on Process Sensitivity (PSS) and Redundancy Sensitivity (RSS). The paper provides little theoretical or empirical justification for why a simple majority threshold (0.500.50) is optimal across distinct domain types, making the structural selection mechanism feel ad-hoc rather than fundamentally grounded.

  3. Limited Technical Novelty and Overlap with Concurrent Work. The core concept of refining LLM agent skills through trajectory feedback and candidate generation is heavily incremental. Specifically, the framework shares substantial conceptual and procedural similarities with recent skill optimization paradigms (such as Skill-RL presented at ICLR 2026 workshops), which also leverage execution feedback and reinforcement/trajectory-conditioned signals to iteratively edit and update agent skills.

Questions for authors

see Weaknesses

Reviewer_KmoWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to improve reusable textual skills for a frozen agent. SkillSpec first proposes edits under four intents and accepts candidates using repeated paired validation. It then estimates process and interference sensitivity from the edit history, chooses a flat, graph, or hybrid representation, and further optimizes that artifact. Experiments cover six benchmarks and three target models. The full method improves average success over the reported skill-optimization baselines; the representation study compares adaptive routing with three fixed choices.

Critical strengths
  1. The evaluation covers quite different tasks, from question answering to spreadsheet editing and ALFWorld. Table 2 reports a higher macro-average success rate than SkillOpt-Lite for all three models, with a mean gain of 4.80 percentage points. This makes the overall approach worth investigating.

  2. Table 3 compares representation choices starting from a shared Phase I skill for each benchmark. Adaptive selection reaches 50.27%, versus 49.10% for fixed Hybrid, and the table includes a continued Phase I control labelled equal compute. These are useful controls for studying the second stage.

Critical weaknesses

I could not determine whether the main comparisons use comparable optimization resources and starting skills. Phase I screens several complete candidates and repeats validation for finalists, followed by four Phase II epochs. Table 4 lists SkillSpec's settings, but not rollout or token budgets for each external baseline. It also says the initial skills are method- and cell-specific and reported in Table 2, yet Table 2 does not identify them. The equal-compute control in Table 3 does not settle this question for the external baselines. Without this information, it is hard to separate the method's contribution from differences in resources or initialization.

The contribution of consensus-gated evolution is not isolated by the ablations. Under the stated binary-accuracy metric, the paired net gain in Eq. (1), n+ - n-, is exactly the difference in total successes. The acceptance rule therefore does not distinguish updates with the same aggregate improvement but different numbers of regressions; its additional restriction comes from repeated confirmation. Table 3 studies representation choices, but does not compare consensus gating with an ordinary score-based gate using the same candidate pool and evaluation budget, or isolate the four-intent search. These controls are needed to establish what drives the Phase I gain.

Questions for authors

Could you provide a Phase I ablation that compares consensus gating with an ordinary aggregate-score gate using the same candidate pool, initialization and total evaluation budget, and separately tests the four-intent search at a matched candidate and rollout budget? Please also explain how the checkpoints, seeds and budgets in Tables 2 and 3 correspond, since the GPT-5.4 Nano results differ. These comparisons would materially affect my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.