- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CtrlSkill improves external agent skills without updating model weights. It organizes execution traces into structured evidence, proposes scoped skill edits, and admits candidates only after paired comparisons with the active policy on subsequent matching tasks. Experiments report improvements on SpreadsheetBench and WikiTableQuestions with two skill-author models, alongside component ablations, transfer to Kimi K3, and a DocVQA evaluation.
Critical strengths
-
A coherent approach to controlling persistent updates. Separating candidate generation from admission addresses a practical failure mode: plausible lessons can harm later execution. Grounding edits in tool behavior and saved artifacts makes the proposed feedback more actionable and auditable.
-
Substantial reported improvements. On held-out SpreadsheetBench, CtrlSkill improves over Trace2Skill by 26.5 and 12.5 percentage points in GPT-5.5 and DeepSeek Deepening, respectively. Removing individual components reduces GPT-5.5 Deepening accuracy by 17–19.5 points. These results provide useful initial evidence for the design.
Critical weaknesses
- The incremental contribution over closely related methods is not established experimentally. SkillCAT [1] already filters candidate patches through replay, while SkillOpt [2] accepts bounded edits only after held-out validation improves. Direct comparisons with these methods are needed to establish CtrlSkill's incremental benefits. Subsequent-task admission and structured evidence may offer additional value, but comparisons only against Trace2Skill do not establish this advantage. Missing compute accounting also prevents assessing the benefit of additional validation relative to its cost.
- The core admission mechanism is insufficiently specified and validated. Equation (9) defines clustering similarity, but the signature extraction procedure and task–candidate matcher m(z,s) are not operationally specified. These determine which examples support admission. The detailed analysis covers only six candidates and 18 matched-task validations; ablations cover one configuration, with no reported variability across runs or task orders. The reliability of admission from such limited evidence remains uncertain.
References:
[1] SkillCAT: Contrastive, Assessment-Augmented and Topology-AwareSkill Self-Evolution for LLM Agents
[2] SkillOpt: Executive Strategy for Self-Evolving Agent Skills
Questions for authors
-
Comparison and cost — targeted validation. Can you compare against SkillCAT and a held-out-validation method such as SkillOpt under matched models, splits, and computation budgets? Please report total evolution tokens/calls or time, including evidence construction, editing, and paired validation.
-
Matching — clarification. How are structural signatures and m(z,s) computed in practice? Please provide the extraction/routing algorithm or prompts and clarify which inputs are available before versus after baseline execution.
-
Metric consistency — clarification. Table 1 reports Soft = 23.8% and Hard = 28.6% for DeepSeek Human-Written. The official SpreadsheetBench definitions imply Hard ≤ Soft on the same evaluated tasks. Is this a reporting error or a difference in evaluation protocol? Please reconcile the scores and any affected averages.
-
DocVQA — clarification. Section 4.1 specifies 50 DocVQA examples for evolution, whereas Section 5 describes transfer beyond spreadsheet tasks used during evolution. Does Table 4 evaluate frozen spreadsheet skills or skills adapted on DocVQA? The distinction changes the generalization claim.
-
Admission robustness — targeted validation. Do the promotion decisions and ablation gains persist across task orders or repeated evolution runs? C1 and C2 are admitted after repairing only one matching failure each, so variability could materially affect the reliability conclusion.
Summary of Responses
Summary of Responses
Dear Chairs and Reviewers,
We thank you and the three reviewers for the careful evaluation of our work and the valuable suggestions. These comments have helped us clarify the method, the experimental evidence, and the scope of our conclusions. Since several concerns overlap across the reviews, we summarize our main responses and the corresponding manuscript locations below.
◆ Whether CtrlSkill’s contribution comes only from cached-value or formula repairs
We explain how CtrlSkill updates skills through structured execution evidence, scoped candidate edits, and delayed admission. The failures it addresses include not only missing cached values but also non-empty yet semantically incorrect outputs, such as errors in source-data and target-range mapping, auxiliary-list usage, and sorting rules.
Corresponding locations: Sections 3.2–3.4, Section 5, and Appendices A.2 and A.5.
◆ Whether a candidate is accepted after a single success, and how task matching works
We clarify that a single success does not directly trigger admission. Candidates must accumulate paired-validation evidence on subsequent matched tasks and jointly satisfy the requirements for sample size, baseline-failure count, repair rate, and mean gain. The matching set is determined by baseline routing before candidate execution, and candidate results do not modify this set. We also provide pseudocode covering structural-signature extraction, candidate editing, and admission.
Corresponding locations: Sections 3.2–3.4, Equations (8)–(26), Section 4.2, and Appendices A.1 and A.6.
◆ Admission-threshold sensitivity and whether thresholds were retuned for DocVQA
We analyze the admission decision boundaries using existing paired-validation records and explicitly distinguish candidate decisions under fixed evidence from guarantees about final accuracy after rerunning the entire evolution process. For DocVQA, the minimum validation sample size remains 3 and the mean ANLS gain threshold remains 0.02, while the additional binary repair-rate conditions used for SpreadsheetBench are not imposed.
Corresponding locations: Section 4.2 and Appendix A.7.
◆ Stability under different training-task orders
We conducted supplementary experiments with three random seeds under the DeepSeek-V4-Flash Deepening setting. CtrlSkill achieves 93.7% ± 0.29% accuracy, compared with 82.7% ± 1.00% for Trace2Skill + Comb. The responses also describe the common operational themes of the experiences retained across runs, while distinguishing this observation from a guarantee that the skill texts are identical.
Corresponding locations: Table 5 reports accuracy across repeated runs. Table 2 and Appendices A.1 and A.4 provide ablation results and process analyses for admission control and evidence accumulation. Observations about experience-content consistency are provided in the reviewer responses.
◆ Runtime, token costs, and comparisons with related methods
We report accuracy, evolution and testing token costs, and runtime for Trace2Skill and CtrlSkill, with paired-validation costs listed separately. We additionally compare with SkillOpt to address the request for a method using held-out validation. For SkillCAT, the responses explain why a direct reproduction was not conducted.
Corresponding locations: Section 5 and Table 5 provide the overall comparison. Appendix A.8 and Table 9 provide stage-level costs and the SkillOpt configuration. SkillOpt’s accuracy and runtime are reported in the reviewer responses. Appendix A.3 compares the mechanisms of related methods.
◆ Evaluation metrics, data splits, and the interpretation of the DocVQA experiment
We corrected the swapped values in Table 1 to Soft = 28.6% and Hard = 23.8%. The responses clarify that the Vrf evolution and test sets are disjoint, while the full Soft/Hard evaluation scope includes some evolution tasks.
For DocVQA, we clarify that 50 samples are used for task-specific Creation and 100 independent samples are used for evaluation. This experiment demonstrates applying the framework to a new task, rather than directly transferring frozen SpreadsheetBench skills across domains.
Corresponding locations: Table 1, Section 4.1, Section 5, and Table 4. The overlap between the Soft/Hard evaluation scope and evolution tasks is explained in the reviewer responses.
◆ Positioning of the control-theoretic framework
We clarify that control theory describes the functional relationships among observation, candidate editing, feedback, and delayed updates. The paper does not claim formal stability or convergence guarantees.
Corresponding location: Section 3.1.