- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SkillAdam optimizes natural-language skills for frozen LLM agents. It maintains an Evolving Issue Tracker (EIT) to record problems and previous repair attempts, and adjusts the edit budget using the variance of case-level score improvements. Candidate skills are evaluated before being accepted. The paper reports results on seven benchmarks, with larger gains on long-horizon planning, lower optimization costs, and improved cross-model transfer compared with SkillOpt.
Critical strengths
S1. The reduction in optimization cost is substantial. On DeepPlanning, token usage drops from 226.6M to 74.0M while DP-Avg improves from 21.7% to 28.3%.
S2. The evaluation covers seven benchmarks and includes cross-model transfer. On GPT-5.4-mini, the transferred skills outperform SkillOpt on five of six benchmarks, with the average score increasing from 63.1% to 67.8%.
Critical weaknesses
W1. I find EIT an incremental change to the memory mechanisms already used by SkillOpt. Tracking issues and repair histories is a reasonable design, but I do not see clear evidence that this organization preserves useful corrections better than existing optimizer memory. The ablation removes EIT and conversation history together, so it does not establish EIT's distinct contribution. I also find the Adam framing overstated: the connection is mainly conceptual and makes the method appear more novel than the demonstrated advance warrants.
W2. The variance-based edit rule is not sufficiently justified. The comparison with a fixed budget shows that the proposed scheme can help, but does not explain why variance is the right signal. No comparison with other improvement signals is reported. It is also unclear whether the number of requested diff hunks reflects the scale of changes in the skill's strategy or behavior. Without establishing this connection, I am not convinced that the budget provides the claimed control over update magnitude, particularly when several parts of a skill need to change together.
Questions for authors
Q1. What evidence separates EIT's contribution from that of conversation history and shows that issue-level tracking helps preserve successful repairs across revisions? Does it improve subsequent editing decisions compared with SkillOpt's optimizer memory?
Q2. Using improvement variance to control the edit budget seems heuristic. Why was this signal chosen, and how does it compare with other improvement signals?
Q3. Does changing the hunk budget reliably change the extent of revisions to the skill's strategy or behavior? How does the method handle improvements that require coordinated changes across the skill, including substantial restructuring?
6 comments
Response to Reviewer 1BFV (2/2)
Q3: Whether the hunk budget controls the extent of strategy and behavior changes, and how coordinated revisions are handled.
A3: Thank you for this insightful question. Each hunk is intended to be one logical change, so the hunk count reflects how many parts of the strategy a revision modifies, and the extent of revisions follows the budget:
| Requested | #Patches | Avg. realized hunks | Max realized hunks | Within budget |
|---|---|---|---|---|
| 1 | 33 | 1.33 | 3 | 72.7% |
| 2 | 22 | 1.64 | 3 | 95.5% |
| 3 | 21 | 1.67 | 3 | 100% |
| 4 | 12 | 2.08 | 3 | 100% |
Coordinated changes are not blocked. Related edits within a section fit into one hunk, the budget returns to once improvements become consistent, and since the budget is soft, a necessary change is never truncated. A large restructuring may take several iterations under inconsistent evidence, which we now note as a limitation in Section 6 of the revised paper (marked in blue).
Q4: Clarifying the connection between SkillAdam and Adam.
A4: Thank you for this comment. We agree that the connection to Adam is a conceptual analogy, and Adam serves as the inspiration for our design.
Skill self-evolution faces two problems similar to those in stochastic optimization. Successive revisions often overwrite earlier corrections, causing a seesaw effect, and revisions without guidance on their scope keep adding rules that overfit a few cases. Following Adam's principles of aggregating history and scaling the step by signal variability, we designed the EIT and the volatility-driven edit budget to address these two problems.
Our experiments confirm that the design is effective: SkillAdam achieves higher accuracy at lower cost (Tables 2 and 5), keeps its accepted skills stable (Figure 4), relies on the adaptive budget (Table 3), and produces higher-quality skills that transfer better (Figure 3 and Table 4).
Thanks again for your careful and valuable comments, hope our response addresses your concerns. If our responses answer your questions well and reassure your concerns, we would appreciate if you could reassess our work and increase the rating.
Follow-up on our response
Dear Reviewer 1BFV,
Thank you again for your detailed and constructive review. As the discussion period is coming to an end, we would like to kindly check whether our response has addressed your concerns. In particular, we added an ablation that removes only the EIT while keeping the conversation history, which shows that the history cannot substitute for the EIT, explained the rationale for the variance-based budget signal, showed that the extent of revisions follows the hunk budget, and clarified that the connection to Adam is a design inspiration rather than a mathematical correspondence. All revisions are marked in blue in the updated paper.
If any question remains, we would be glad to clarify it before the discussion closes. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your rating.
Best regards, The Authors
Response to Reviewer 1BFV (1/2)
Thanks for your careful and constructive comments, and for recognizing the substantial cost reduction and the cross-model transfer of SkillAdam. We have revised the paper following your suggestions, with all changes marked in blue.
Q1: Distinct contribution of the EIT beyond the conversation history, and whether issue-level tracking preserves repairs and improves editing decisions compared with SkillOpt's optimizer memory.
A1: Thank you for this critical question. We agree that the original ablation removes the EIT only together with the fixed budget. The conversation history is a global component kept in all variants, so we ran an ablation on DeepPlanning that removes only the EIT while keeping the conversation history, the adaptive budget, and all other settings unchanged:
Removing only the EIT leads to the largest drop among all variants, even though the conversation history is still available, so the history cannot substitute for the EIT. This variant even falls below the one that removes both the EIT and the adaptive budget. Without the EIT, the patch generator still sees earlier attempts in the history but has no record of which issues are already resolved, so the adaptive budget narrows each revision without knowing which corrections must be preserved. Issue-level tracking is therefore what allows successive revisions to build on earlier repairs.
The EIT also differs from SkillOpt's optimizer memory, which records the outcomes of rejected edits. The EIT is organized by issue, tracking each problem's status and all attempts, so the optimizer knows which problems are solved and avoids overwriting them. The comparison with SkillOpt is consistent with this difference: SkillAdam reaches higher accuracy at a lower optimization cost (Tables 2 and 5), its accepted skills stay above the initial accuracy while SkillOpt's fluctuate (Figure 4), and its skills are rated higher in the blind skill-quality evaluation (Figure 3). We have added this ablation to Table 3 and Section 4.4 of the revised paper (marked in blue).
Q2: Rationale for using improvement variance as the budget signal, and comparison with other signals.
A2: Thank you for this question. The choice of variance is inspired by Adam, which scales its step size by the variability of recent gradients. In skill revision, we measure this variability by the variance of per-case improvements. A high variance means a revision helps some cases but hurts others, so the next edit should be narrower; a low variance supports a broader edit. This complements the acceptance gate, which uses the mean improvement.
Table 3 shows that this signal is effective: replacing it with a fixed budget clearly lowers accuracy, most of all on DeepPlanning Travel. We have not compared other signals, since our goal is a principled and effective design rather than a search over signals.