- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces SkillCred, an online framework for assessing and managing agent skills as they evolve. The framework updates Bayesian value estimates using local progress signals collected during task execution, selects among skill versions through Thompson sampling, and governs version retention with a predictive criterion. These components support skill management without requiring separate evaluation rollouts.
Critical strengths
-
Reliable skill assessment is crucial for continual skill evolution, as newly generated versions can introduce errors and degrade agent performance.
-
Learning from normal execution trajectories avoids dedicated validation rollouts and reduces the overhead of evaluating competing skill versions.
-
Experiments include diverse models on different benchmarks exhibit consistent improvements, supported by component ablations.
Critical weaknesses
-
Local progress labels are crucial to value estimation, but their accuracy and attribution reliability are not directly validated. The LLM-only and synthetic-noise experiments provide useful evidence, yet fixed-checkpoint replays do not capture downstream effects, and symmetric label corruption does not address systematic errors such as misattributing upstream failures to downstream skills.
-
The paper characterizes skill versions as arms and local progress to rewards, but does not clearly clarify what is the difference between the bandit formulation and simple version-selection strategies. Comparisons with greedy selection or uniform evaluation would help demonstrate the value of explicitly balancing exploration and exploitation.
-
In this work, skill-version routing is formulated as a multi-armed bandit problem, but the choice of Thompson sampling is not supported by comparisons against alternatives such as UCB. The experiments compare retention criteria rather than routing algorithms.
Questions for authors
Why do the experiments use Qwen3.5-9B and Qwen3.6-35B-A3B rather than models from the same version? Could differences in model generation and architecture confound the interpretation of results across model scales?
2 comments
-
The clarification does not resolve my concern, as none of the four listed components appears novel. Local progress assessment offers limited performance improvement over directly using the DeepSeek Harness. Bayesian value estimation and posterior-sampling routing are simply components of a standard multi-armed bandit algorithm, and comparisons with alternatives such as UCB are still missing. Predictive lifecycle control likewise appears to apply existing bandit evaluation and elimination ideas. Combining these components into a framework does not sufficiently establish methodological novelty.
-
I know your comparison is fair when using the same model, but my question is: “Could differences in model generation and architecture confound the interpretation of results across model scales?” It is unusual to use two models with different versions for experimental evaluation. So my main concern is that the proposed method is not applicable to other model sizes in Qwen 3.5 or 3.6, which raises severe concerns on the generalization of this method, and adding the experimental results of other sizes will address this.
Thanks for the rebuttal. However, the following questions remain unsolved.
The methodological novelty remains unclear. CRV appears to estimate retention value by sampling possible futures and averaging their returns, a standard Monte Carlo approach. What is new beyond applying this established multi-arm bandit technique to skill lifecycle management?
The model selection rationale remains incomplete. Qwen3.5 already includes a 35B-A3B MoE model, so covering both a 9B dense configuration and a 35B-A3B MoE configuration does not sufficiently explain the cross-generation pairing of Qwen3.5-9B and Qwen3.6-35B-A3B. This raise serious concerns on the soundness of the proposed method.