- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DeltaSkill , a framework for adaptive skill use in long-horizon terminal agents. The authors observe that poorly adapted skill use can prolong task execution and waste tokens. DeltaSkill consists of two primary components: capability-aware residual skill compilation and evidence-guided execution control. It first constructs a capability profile for the agent, then compiles task related skill sets by retaining, merging, reinforcing, or removing existing skill instructions. During execution, the evidence-guided execution control checks the agent's results, saves the best validated output, and stops execution when further attempts fail to make progress, reducing unnecessary model calls. The authors evaluate DeltaSkill on three long-running tasks: Generals, Spot, and DuckDB. Experimental results demonstrate that DeltaSkill improves rewards on certain tasks while reducing execution time, the number of model requests, and observed token consumption.
Critical strengths
- The research problem is practical and important.
- Experiments show substantial efficiency improvements.
Critical weaknesses
-
Insufficient comparison with related work. For example, SkVM has already proposed skill compilation based on model-harness capability profiling, while SkillAligner has explored task conditioned skill adaptation. Not sure if the Residual Skill Compilation proposed by this paper is better than these baselines.
-
Limited evaluation. The main experiments only run with GPT-5.4-mini+Terminus2, while the profile matching experiments only run with GPT-5.5+Terminus2/OpenCode. Not sure if the improvement could be generalized to other models/harness.
-
The paper claims that DeltaSkill significantly enhances efficiency. However, it primarily reports token consumption during the agent execution phase, which is after skill compilation is complete, not covering the cost of capability profiling and skill compilation. Also, it's not clear whether the profiling results and compiled skills could be reused for multiple tasks.
Questions for authors
-
Do Capability Profiling and Skill Compilation need to be re-executed for every individual task, or can they be reused across multiple tasks? could you do some experiment results to show whether the profiling results and compiled skills could be reused for multiple tasks.
-
could the results be expanded to a broader range of Model+Harness combinations? why do you only choose GPT-5.4-mini+Terminus-2 for main experiments?
-
What are the fundamental differences between DeltaSkill’s Residual Skill Compilation and existing works like SkVM or SkillAligner? Can you do some ablation study experiments to show the specific contribution of DeltaSkill’s Residual Skill Compilation?