- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SkillHellix proposes iterative optimization of skills for black-box LLMs using an open-weight model as a proxy evaluator of the combinations of proposed edits. The decision model computes the teacher-forced likelihood on datasets and select the edit subset that most increases likelihood on failed questions, without lowering on solved questions. The same subset scores are also converted into Shapley values, which the optimizer model uses to optimize its own skill. Experiments cover six benchmarks and five reasoning backbones.
Critical strengths
-
Evaluating every combination of edits with the black-box model would be expensive, so leveraging token probabilities from a open-weight model is a simple and practical way to get feedback at the level of individual edits.
-
The method is evaluated on six benchmarks, with five different reasoning backbones. It outperforms strong baselines such as GEPA and SkillOpt, so the gains do not seem limited to a particular task or model.
Critical weaknesses
-
(Prensentation) The title is not intuitive, since it does not say what the method actually does. In Figure 1 (right), it is unclear what each axis exactly measures. In Figure 2, appears in the caption but not in the figure, and the text uses “reasoning model” and “decision model” while the figure uses “Reasoner” and “Decider.”
-
The paper mostly reports final scores and does not explain much about why the method works. Without this kind of analysis, it is hard to understand where the gains come from. For example, The method relies on the assumption that raising the decision model’s answer likelihood on failed questions also raises the reasoning model’s accuracy. The paper states this assumption but does not show it experimentally.
-
With , there are only 16 subsets, so selecting among them directly by the reasoning model’s accuracy seems feasible. This baseline is not included, so it is not clear whether the decision model actually helps compared with direct black-box selection.
-
The paper does not clearly explain how spans are defined in span evolution. There is also no analysis of which spans were selected or how they were changed.
-
Shapley values are used to credit each edit, but the paper does not compare them with other ways of assigning credit. So it is hard to tell whether Shapley values are actually important here.
-
The decision model computes the likelihood by teacher-forcing the answer after the question, without any reasoning. The reasoning models (ex. GPT 5.5) answer after long reasoning, so it is not clear that the proxy score would change in the same direction.
Questions for authors
Refer to the weaknesses.
Clarifying the SkillOpt Baselines: Repeated Runs and Independent Evidence
We sincerely thank the reviewer for the careful and detailed comparison with the original SkillOpt results. Your scrutiny highlights an important issue in baseline reporting, and we appreciate the opportunity to clarify the evidence and improve the transparency of our comparisons. Independent studies have also reported reproduction difficulties or obtained SkillOpt baseline scores different from those originally published [1–3]. These discrepancies are therefore not unique to our experiments.
We conducted three full GPT-5.5 SkillOpt runs using the released implementation and four-epoch configurations. The mean and sample standard deviation are:
The repeated results remain close to the lower scores reported in our table. In particular, SearchQA, SpreadsheetBench, and LiveMath remain below the published scores of 87.3, 80.7, and 66.9, respectively. The gaps persist across repeated runs and substantially exceed the observed run-to-run standard deviations, supporting a persistent reproduction discrepancy within our setting.
The external evidence is consistent with this observation. COBRA-Skills and SoftSkill document reproduction difficulties or evaluation discrepancies [1,2]. SkillOpt-Lite also reports different SkillOpt baseline scores, including 78.8 on GPT-5.5 SearchQA and 76.2 on SpreadsheetBench [3]. These studies use their respective evaluation settings and provide context for the discrepancies.
We agree with the reviewer that readers should be able to distinguish published results from our reproduced measurements. We will make the provenance of the baseline entries explicit and present published and reproduced results separately. As your review notes, SkillHelix still leads in 25 of 30 settings when the original SkillOpt scores are substituted, although the GPT-5.5 improvement margins become smaller.
We appreciate the time and care invested in this assessment. We respectfully ask that the repeated-run evidence, independent reports, and clarified evaluation settings be considered when reassessing the baseline concern and the associated ethics flag. Your feedback will help us make the experimental reporting clearer and more useful to readers.
[1] COBRA-Skills: Contextual Bandit-Guided Evolution for Agent Skill Optimization. arXiv:2609.11682, 2026.
[2] SoftSkill: Behavioral Compression for Contextual Adaptation. arXiv:2606.20333, 2026.
[3] SkillOpt-Lite: Better and Faster Agent Self-evolution via One Line of Vibe. arXiv:2607.03451, 2026.