- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies why skills do not port cleanly across LLM deployments even when their “meaning” is unchanged, and proposes that models have stable, deployment-specific preferences over semantically equivalent skill representations. The authors introduce SkillProbeBank, a dual-track set of matched, controlled probes isolating 14 representation factors (instruction and executable), and SkillProbe, a budgeted, hierarchical Bayesian diagnosis that adaptively selects probe blocks to estimate a model’s preference profile and uses decision-aware regret to choose what to KEEP/TRANSFORM/DEFER in downstream skill adaptation. Empirically, sparse diagnosis recovers nearly 84% of full-bank representation decisions and, when plugged into strong skill learners, yields consistent gains across ALFWorld, WebShop, SkillsBench, SWE-Skills-Bench, and additional cross-family transfers, indicating broad practical value.
Critical strengths
-
The paper addresses an important problem. Skills that work well for one model may transfer poorly to another, even when their underlying procedures remain unchanged. Diagnosing representation compatibility under a limited budget and reusing that diagnosis across skills is a practically valuable direction.
-
The reported improvements are encouraging. In particular, the gains on SkillsBench and SWE-Skills-Bench suggest that representation diagnosis can provide benefits beyond those offered by existing skill learning and optimization methods.
-
The evaluation covers a broad range of settings. The experiments examine model-specific representation preferences, budgeted diagnosis, cross-model migration, cold start, skill generation, and adaptation to version drift. This breadth helps demonstrate the potential usefulness of the proposed profiles.
Critical weaknesses
My main concern is the clarity and organization of the presentation. The paper introduces many concepts and methodological components, but their motivations and connections are not always sufficiently explained. This makes it difficult to assess how the framework operates as a whole and which components are essential to its contribution.
-
Some transitions between definitions and examples are abrupt. For example, at line 215, the paper defines a probe and introduces as the set of selectable probes. The next sentence begins “Only organization changes” and discusses the example in Table 1, without a clear transition establishing which comparison is being described. This requires readers to reconstruct the connection between the general definition and the specific example. Introducing the example explicitly before interpreting its effect would make the paragraph easier to follow.
-
The density of auxiliary terminology creates a substantial reading burden. Readers must distinguish among semantic cores, variants, probe blocks, fixtures, sentinels, anchors, and coverage gates, among other concepts. The glossary in the appendix is helpful, but the main text would benefit from a running example that establishes their relationships and distinguishes essential concepts from implementation details.
-
The connections between methodological stages would benefit from more intuitive explanation. The paper presents equations for effect extraction, Bayesian profiling, representation decisions, and adaptive probe selection. However, it does not consistently explain how the output of one stage becomes the input to the next. A worked example tracing probe outcomes through posterior inference to a representation recommendation would substantially improve accessibility.
-
The rationale for selecting the four confirmatory interactions is unclear. The paper specifies T1×T5, T2×T3, T4×T7, and E1×E2 as preregistered interactions, but does not clearly explain why these particular pairs were selected over other combinations or why the confirmatory evaluation is limited to four pairs. The ablation removing interaction probes supports their collective usefulness, but does not establish the adequacy of this particular selection. Please clarify the selection criteria and supporting evidence, and discuss how restricting the analysis to these four confirmatory interactions may affect the resulting profiles. A sensitivity analysis using alternative or additional interaction pairs would help assess robustness to this design choice.
Questions for authors
-
Beyond preregistration, what theoretical considerations, prior evidence, or pilot experiments supported choosing T1×T5, T2×T3, T4×T7, and E1×E2 over other combinations? What determined the decision to include four pairs?
-
Starting from a semantic core and its variants, please illustrate how probe outcomes produce effect estimates, update the Bayesian profile, guide subsequent probe selection, and ultimately yield a representation recommendation or DEFER. Such an example would help clarify the roles of the notation and auxiliary terminology, as well as how uncertainty and task-family heterogeneity affect the final decision.
2 comments
Official Comment by Reviewer mwbZ
Thank you for your additional efforts and experiments to clarify the ambiguous aspects of the paper and investigate different interactions. While I acknowledge the importance of the problem addressed and the promising improvements demonstrated throughout the experiments, I still believe that the paper requires substantial improvements in terms of clarity and soundness. Therefore, I will maintain my original rating.
Author Response to Reviewer mwbZ
Thank you for your thoughtful comments. We have carefully revised the manuscript to address your concerns and suggestions. All references below are to the revised paper.
[w1] “Some transitions between definitions and examples are abrupt.”
Section 4 now introduces the Table 1 example before interpreting its effect. The core extracts three fields, variant A uses prose, and variant B uses numbered steps. Both use the same four inputs and exact-JSON verifier. Their paired score difference estimates the organization effect T1. This connects the general probe definition to the concrete comparison.
[w2, w3, q2] “a running example that establishes their relationships” / “how probe outcomes produce effect estimates, update the Bayesian profile, guide subsequent probe selection, and ultimately yield a representation recommendation or DEFER”
Sections 3–5 and App. E.5 now follow one extraction example through the method. Its semantic core fixes the required fields and behavior. Variants express that core differently, fixtures supply shared inputs, and a matched block contains the paired runs. Anchors repeat blocks to measure variability. Sentinels audit independent cores after diagnosis, and coverage gates check completion before early stopping.
In the illustrative calculation, prose scores 0.50 and numbered steps score 0.75, giving a task effect of 0.25. Together with observations from other fixtures, repeats, and cores, this effect updates the hierarchical posterior in Eq. (5). The posterior separates global effects, family deviations, and observation noise. It then evaluates representation-decision regret in Eq. (6) and predicts how a new or repeated block would reduce it in Eq. (7). Predicted benefit, family uncertainty, coverage, reliability, and cost determine the next test.
A numbered-step preference requires posterior probability at least 0.95 beyond the task ROPE of 0.03, evidence from at least three cores and two families, and the coverage and heterogeneity gates (App. C.2). The illustrated block contributes to that accumulated evidence. A confident preference leads to TRANSFORM for a prose source and KEEP for a numbered-step source. Neutral evidence yields KEEP. Insufficient evidence, missing coverage, ambiguous source fingerprints, or confident opposing family effects yield DEFER. This makes uncertainty and task-family heterogeneity explicit in the final action.
[w4, q1] “why these particular pairs were selected over other combinations or why the confirmatory evaluation is limited to four pairs”
App. E.3 now explains the structural roles. T1×T5 couples organization with contract encoding, T2×T3 couples hierarchy with step granularity, T4×T7 couples control flow with validation placement, and E1×E2 couples code location with decomposition. The four-pair allocation gives dedicated four-cell measurements to three instruction couplings and one executable coupling, defining which interactions directly inform the profile.
We also test an alternative allocation on Qwen3.5-9B. It retains T1×T5 and T4×T7 and replaces the other pairs with T1×T2 and E2×E4. Both allocations share the absolute diagnostic token cap, main-effect coverage, source skills, rewriter, final tests, and downstream budget (Table 22).
The replacement changes are small and their paired intervals span zero, quantifying sensitivity to this alternative. The no-interaction ablation loses 6.4 recovery points (Table 18), supporting the contribution of interaction information.