- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper:
- Introduces the Writer–Executor Gap, showing that the utility of a reusable skill depends on the model executing
- Proposes SKILLDIAG to analyze potential sources of transfer failure, and SKILLTUTOR to adapt skill presentation using executor feedback.
Experiments across multiple benchmarks, model scales, and model families demonstrate executor-dependent skill utility and show improvements in action prediction and, in some settings, end-to-end task success. However, minor clarifications and corrections would improve the understandability and readability of some specific parts of the paper.
Critical strengths
The major strength is the breadth of the empirical evaluation, which covers several benchmarks, writer–executor matrices, model families, and both local and end-to-end metrics. The study also employs shared-source comparisons, controlled interventions, held-out selection and evaluation sets, repeated executions, matched baselines, and source-faithfulness audits. These choices make the central evidence for executor-relative skill utility convincing and demonstrate substantial experimental rigor and transparency.
Critical weaknesses
Overclaiming
Some conclusions appear stronger than the evidence supports. The diagnostic measures do not directly estimate the theoretical failure components, diagnosis-based treatment selection is not consistently validated, and repair gains vary substantially across task families and benchmarks. The authors should therefore distinguish aggregate diagnostic insights from reliable instance-level diagnosis and avoid implying that the repair method is uniformly effective or broadly generalizable.
Questions for authors
Clarification requests
- Sec. 2: The cited works appear to provide related foundations in informativeness and decoding with side information, but do they directly support the claimed distinction between information retention and decoder mismatch? Especially the two oldest, Blackwell (1953) and Wyner–Ziv (1976). I am not convinced that they should be cited in this context.
- Evaluation: How should the action-NLL improvements be interpreted in relation to end-to-end task success? The former appear more consistent, while the latter are heterogeneous across benchmarks and task families.
- Appendix: Is SKILLDIAG intended to support instance-level intervention selection or only aggregate analysis? Most instances receive no dominant diagnosis, and the preliminary matched-treatment experiment does not validate diagnosis-based treatment selection.
Suggestions
- Sec 3.1: The paper should define the first p explicitly, especially because pϕ is separately introduced.
- Sec 3.1: Please define H, I, the expectation distributions, and the interpretation of p(⋅∣S,Ye) more explicitly. It would also help to distinguish standard information-theoretic quantities from the paper’s proposed labels for them. It would improve readability.
- The aforementioned claims should be narrowed
3 comments
Q3: Is SkillDiag used for single-instance diagnosis or for group analysis?
Most instances have no dominant diagnosis, and the early matched-intervention trial did not validate treatment selection. At what level is SkillDiag used?
It is used mainly for group analysis. In Appendix E, 85.5% of state–cell instances have no dominant component. The single-instance matching trial and its limitations will be reported together. The new calibrated selection chooses an intervention per task family × writer × executor and does not depend on the answer at a single instance.
| Setting | Description |
|---|---|
| Discovery and selection | 60 discovery tasks supply the three diagnostic scores; 30 separate selection tasks supply the realized intervention benefits |
| Selection rule | Fit a fixed linear predictor to the three scores and choose the positive-benefit intervention with the highest predicted benefit; otherwise fall back to Base |
| Candidate interventions | Add independent training evidence / restore content from training experience / reorder the original skill |
| Evaluation | Exclude the 60 tasks in the original routing Final set. Evaluate 214 held-out tasks and 497 critical states; nine writer–executor pairs; 54 group decisions |
| Comparator | Action-NLL advantage of calibrated selection, nats [95% CI] |
|---|---|
| vs Base | +0.1049 [+0.0683, +0.1432] |
| vs uniform random repair | +0.0488 [+0.0328, +0.0660] |
| vs the original maximum-score rule | +0.0688 [+0.0359, +0.1022] |
| vs the content repair fixed on the selection set | +0.021 [+0.013, +0.029] |
S1: Define the joint distribution and the writer distribution
Define at its first use in §3.1, and distinguish it from .
Accepted. Before the Strict Skill Protocol, we add the following definitions. denotes the fixed source distribution, so that it is not confused with the writer distribution.
| Symbol | Definition |
|---|---|
| Fixed joint distribution of source, target action, and side information | |
| Conditional distribution with which the writer generates a skill from the source | |
| Full joint distribution induced by a fixed writer | |
| Bayesian posterior of the target action given the skill and side information | |
| Action distribution of the actual executor |
S2: Define , , the distribution inside the expectation, and the theoretical labels
Define entropy, mutual information, and the distribution used in expectations, and distinguish standard mathematical quantities from the interpretive labels in this paper.
Accepted. In §3.1, is conditional entropy and is conditional mutual information. Both are computed from the joint distribution and its marginals, with the natural logarithm and units of nats.
| Standard quantity | Interpretive label | Meaning |
|---|---|---|
| Source insufficiency | Uncertainty about the target action that remains given the full source and side information | |
| Writer compression loss | Action-relevant information in the source that the skill has not retained | |
| Executor decoding mismatch | Extra log loss of the executor relative to the Bayesian posterior |
The expectation will be written uniformly as
§4.1 will state the conditions under which the empirical diagnostics correspond to the theoretical terms: canonical action loss, a Bayes-optimal reference, and other ideal conditions. Identifying absolute decoding mismatch from EDM further requires .
S3: Align the abstract and conclusion with the evidence
Narrow the claims above that are stronger than the evidence.
Accepted. The abstract and conclusion will use one statement: skill utility depends on the writer–executor pair; SkillDiag supports group-level diagnostic analysis; the local-decision and task-success gains of SkillTutor are reported separately by dataset, task family, and method version. The locations of these edits are given in W1.
Concerns addressed
Thanks for the additional experiments. I also notice that the claims have been sufficiently narrowed. My concerns are therefore addressed. Soundness and overall scores have been raised accordingly.
Best regards
We thank Reviewer w8bk for the careful reading and constructive suggestions.
W1: Scope of the claims about diagnosis, intervention selection, and repair gains
We appreciate the distinction among these three levels. We will revise the abstract, §§3–4, §6.4, and the conclusion as follows.
New end-to-end evaluation after expansion. The evaluation sets are disjoint from the development, confirmation, and repair-construction sets. Intervals use 20,000 paired task-bootstrap samples under greedy decoding.
Units are percentage points. For all three datasets, the P-Base, P-Generic, and P-Shuffled intervals lie entirely above zero.
Q1: Do Blackwell and Wyner–Ziv directly support the decomposition?
The reviewer's judgment is correct. The two papers are conceptual motivation. The mathematical basis of our decomposition is the standard conditional cross-entropy identity, and it does not depend on the specific results of those papers.
Q2: How do action-NLL gains relate to end-to-end success?
Using records from the original method, we compute paired associations on the three benchmarks. The standardized pooled correlation between the NLL decrease and the success increase is +0.1200 [+0.0640, +0.1724], entirely above zero. The pooled correlation between low NLL and high success is also above zero.
The records use the original skill, capsule, selection, and fallback policy. ALFWorld and WebShop use action NLL at fixed oracle-prefix states. Math adds 2,640 deduplicated answer NLLs, averaged over the correct tokens. We remove configuration means, standardize within each dataset, and average the three datasets with equal weight. Intervals use 20,000 paired task-bootstrap samples.
NLL measures the probability assigned to the correct action. Task success measures the completed execution. Raising the probability of the same correct action from 0.6 to 0.8 lowers NLL without changing the greedy choice, and a full trajectory also depends on later states and decisions. NLL is evidence of a local change in decision probability. Task success is the direct measure of its task-level return. The two are not interchangeable, and their directions agree. We will add this analysis and explanation in §6.4.