W1. The paper never measures what it is an alternative to. The motivation is explicitly "many deployed models cannot be retrained," and Table 1 positions SAT against distillation. Yet there is no distillation baseline (SFT, or OPD per Agarwal et al. 2024, which the paper cites) and — more damaging — no teacher-direct baseline. Across 26 pages, the teacher's own task accuracy is never reported. The entire practical case therefore rests on a number that is absent. Appendix K even defines the amortization threshold in terms of C_strong, then leaves it unfilled. Three further points make this a structural, not incidental, gap:
OPD requires teacher logits, which a closed teacher (Claude-Sonnet-5) cannot provide. The flagship configuration is thus incapable by construction of supporting the comparison.
The authors already ran an open-weight pair (Qwen3-14B → Qwen3-8B, their largest gain at +15.0) with matched splits. LoRA SFT or OPD there is a few GPU-days, not infeasible.
The evidence base contradicts the premise. Largest gains (+15.0, +11.7) occur on open-weight students that users can fine-tune; on the two genuinely closed students (GPT-5.4-Mini, +1.7 ties agnostic; Gemini-3.8-Flash, +4.2 with agnostic ahead at 92.5 vs 91.7) the benefit vanishes. The headline student, DeepSeek-V4-Flash, is itself open-weight. Headroom and weight-openness are confounded here, and only the missing distillation arm can disentangle them.
W2. The headline number is not statistically supported, and the paper's own control says so. Per-run aware−agnostic is +10.00, +2.50, +0.83, +8.33: sign test p = 0.0625, paired t = 2.44 (df=3, p ≈ 0.09). Mean +5.42 points is ≈3.25 of 60 tasks. Meanwhile Appendix H.2 reports a fixed six-skill library varying 71.7–82.5% (10.8-point spread) across four seeds and calls it "the scale of evaluation noise"; Table 23 shows the identical agnostic procedure reading 85.0 / 73.3 / 80.8 across three jobs. The seed noise exceeds the effect. Since the same frozen 60-task split is shared across conditions, a task-level paired bootstrap — already used in Appendix G — would be far more powerful and is simply missing from Table 2.
W3. The aware/agnostic contrast confounds modeling with more information and compute. Aware receives σ, probe outcomes (i.e. additional student rollouts), and opens with 2.2× the teacher input tokens (119.4k vs 54.7k). No equal-budget control exists: an agnostic teacher given the same number of extra rollouts but no learner model. The 2×2 is also incomplete (no-probes/no-model cell). The main comparison is additionally against an author-designed ablation, not against published skill-optimization methods (SkillOpt, EvoSkill, Trace2Skill) that the paper itself cites — and SkillOpt's reported transfer-across-model-scales result directly tensions with the student-specificity claim here.
W4. The gain is one small task family. Decomposing Table 30 (S2–S4, 180 records, weighted): +2.22 of the +3.89 advantage (57%) comes from extension rename, a 6-record category, while aware is worse on the largest category, dependent workflow (94.44 vs 98.61, 36 records), and on mixture arithmetic (82.14 vs 92.86). Remove extension rename and the advantage is +1.66. Table 13 shows 6 of 12 aware skills target renaming. The extended run shows the same pattern: aware wins only on terminal (73.3 vs 50.0), and ties or loses elsewhere (inventory 55.0 vs 56.7). A single rename skill, not a general modeling capability, is carrying the result.
W5. The admission gate is too weak to bear the claims placed on it. Each task runs once with the skill and once empty; 8 of 12 aware skills were admitted on a single development task, and each terminal category has exactly one dev task reused across candidates. At p≈0.6, a useless skill is scored +1.0 with probability ≈0.24. The flagship rename result (0→100) therefore rests on one binary comparison. The gate also measures standalone utility against an empty library, never marginal utility in the deployed 2-skill configuration — likely why L2 student-selection is −0.8 and Qwen2.5-7B flat injection is −14.2. Generated/admitted/rejected counts and utility distributions are not reported.
W6. The library-swap control cannot isolate student-specificity. Both libraries were selected by a gate evaluated on the owning student's dev tasks. "Own library wins" is therefore equally explained by selection overfitting; a same-student, different-seed swap baseline is needed to set that floor. Two of six intervals have a lower bound at exactly 0.0, with no multiplicity correction.