acceptodds
Under review as a conference paper at ICLR 2027

One Skill, Many Models: Learning Representation Compatibility from Sparse Behavioral Tests

Abstract

Agent skills can transfer poorly across large language models (LLMs) even when their procedures are unchanged: models may respond differently to semantically equivalent choices in instruction structure and executable organization. We introduce SkillProbe, a budgeted framework that diagnoses these model-specific representation preferences before skill adaptation.SkillProbeBank provides 156 matched probes and 360 semantically equivalent variants spanning 14 instruction and executable factors. SkillProbe uses a hierarchical Bayesian profile to track preference effects and uncertainty, then selects probes according to their expected improvement to representation decisions per unit cost. The resulting profile can condition existing methods for skill migration, generation, cold start, and version-drift repair. At 20% of the full-bank budget, SkillProbe achieves 84.3% agreement with full-bank representation decisions using 198K tokens, outperforming adaptive low-budget baselines at lower cost. Integrated with strong skill learners, it achieves the best average rank across ALFWorld, WebShop, SkillsBench, and SWE-Skills-Bench. Across seven additional transfer settings involving Gemma, Llama, Qwen, and a GPT-4o target, SkillGraph+SkillProbe outperforms SkillGraph in 27 of 28 paired benchmark comparisons. These results suggest that sparse behavioral diagnosis provides a reusable signal for adapting skills across model deployments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Agent Skills, Skill Portability, Adaptive Probing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mwbZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies why skills do not port cleanly across LLM deployments even when their “meaning” is unchanged, and proposes that models have stable, deployment-specific preferences over semantically equivalent skill representations. The authors introduce SkillProbeBank, a dual-track set of matched, controlled probes isolating 14 representation factors (instruction and executable), and SkillProbe, a budgeted, hierarchical Bayesian diagnosis that adaptively selects probe blocks to estimate a model’s preference profile and uses decision-aware regret to choose what to KEEP/TRANSFORM/DEFER in downstream skill adaptation. Empirically, sparse diagnosis recovers nearly 84% of full-bank representation decisions and, when plugged into strong skill learners, yields consistent gains across ALFWorld, WebShop, SkillsBench, SWE-Skills-Bench, and additional cross-family transfers, indicating broad practical value.

Critical strengths
  1. The paper addresses an important problem. Skills that work well for one model may transfer poorly to another, even when their underlying procedures remain unchanged. Diagnosing representation compatibility under a limited budget and reusing that diagnosis across skills is a practically valuable direction.

  2. The reported improvements are encouraging. In particular, the gains on SkillsBench and SWE-Skills-Bench suggest that representation diagnosis can provide benefits beyond those offered by existing skill learning and optimization methods.

  3. The evaluation covers a broad range of settings. The experiments examine model-specific representation preferences, budgeted diagnosis, cross-model migration, cold start, skill generation, and adaptation to version drift. This breadth helps demonstrate the potential usefulness of the proposed profiles.

Critical weaknesses

My main concern is the clarity and organization of the presentation. The paper introduces many concepts and methodological components, but their motivations and connections are not always sufficiently explained. This makes it difficult to assess how the framework operates as a whole and which components are essential to its contribution.

  1. Some transitions between definitions and examples are abrupt. For example, at line 215, the paper defines a probe and introduces P\mathcal{P} as the set of selectable probes. The next sentence begins “Only organization changes” and discusses the example in Table 1, without a clear transition establishing which comparison is being described. This requires readers to reconstruct the connection between the general definition and the specific example. Introducing the example explicitly before interpreting its effect would make the paragraph easier to follow.

  2. The density of auxiliary terminology creates a substantial reading burden. Readers must distinguish among semantic cores, variants, probe blocks, fixtures, sentinels, anchors, and coverage gates, among other concepts. The glossary in the appendix is helpful, but the main text would benefit from a running example that establishes their relationships and distinguishes essential concepts from implementation details.

  3. The connections between methodological stages would benefit from more intuitive explanation. The paper presents equations for effect extraction, Bayesian profiling, representation decisions, and adaptive probe selection. However, it does not consistently explain how the output of one stage becomes the input to the next. A worked example tracing probe outcomes through posterior inference to a representation recommendation would substantially improve accessibility.

  4. The rationale for selecting the four confirmatory interactions is unclear. The paper specifies T1×T5, T2×T3, T4×T7, and E1×E2 as preregistered interactions, but does not clearly explain why these particular pairs were selected over other combinations or why the confirmatory evaluation is limited to four pairs. The ablation removing interaction probes supports their collective usefulness, but does not establish the adequacy of this particular selection. Please clarify the selection criteria and supporting evidence, and discuss how restricting the analysis to these four confirmatory interactions may affect the resulting profiles. A sensitivity analysis using alternative or additional interaction pairs would help assess robustness to this design choice.

Questions for authors
  1. Beyond preregistration, what theoretical considerations, prior evidence, or pilot experiments supported choosing T1×T5, T2×T3, T4×T7, and E1×E2 over other combinations? What determined the decision to include four pairs?

  2. Starting from a semantic core and its variants, please illustrate how probe outcomes produce effect estimates, update the Bayesian profile, guide subsequent probe selection, and ultimately yield a representation recommendation or DEFER. Such an example would help clarify the roles of the notation and auxiliary terminology, as well as how uncertainty and task-family heterogeneity affect the final decision.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer mwbZ

Thank you for your thoughtful comments. We have carefully revised the manuscript to address your concerns and suggestions. All references below are to the revised paper.

[w1] “Some transitions between definitions and examples are abrupt.”

Section 4 now introduces the Table 1 example before interpreting its effect. The core extracts three fields, variant A uses prose, and variant B uses numbered steps. Both use the same four inputs and exact-JSON verifier. Their paired score difference estimates the organization effect T1. This connects the general probe definition to the concrete comparison.

[w2, w3, q2] “a running example that establishes their relationships” / “how probe outcomes produce effect estimates, update the Bayesian profile, guide subsequent probe selection, and ultimately yield a representation recommendation or DEFER”

Sections 3–5 and App. E.5 now follow one extraction example through the method. Its semantic core fixes the required fields and behavior. Variants express that core differently, fixtures supply shared inputs, and a matched block contains the paired runs. Anchors repeat blocks to measure variability. Sentinels audit independent cores after diagnosis, and coverage gates check completion before early stopping.

In the illustrative calculation, prose scores 0.50 and numbered steps score 0.75, giving a task effect of 0.25. Together with observations from other fixtures, repeats, and cores, this effect updates the hierarchical posterior in Eq. (5). The posterior separates global effects, family deviations, and observation noise. It then evaluates representation-decision regret in Eq. (6) and predicts how a new or repeated block would reduce it in Eq. (7). Predicted benefit, family uncertainty, coverage, reliability, and cost determine the next test.

A numbered-step preference requires posterior probability at least 0.95 beyond the task ROPE of 0.03, evidence from at least three cores and two families, and the coverage and heterogeneity gates (App. C.2). The illustrated block contributes to that accumulated evidence. A confident preference leads to TRANSFORM for a prose source and KEEP for a numbered-step source. Neutral evidence yields KEEP. Insufficient evidence, missing coverage, ambiguous source fingerprints, or confident opposing family effects yield DEFER. This makes uncertainty and task-family heterogeneity explicit in the final action.

[w4, q1] “why these particular pairs were selected over other combinations or why the confirmatory evaluation is limited to four pairs”

App. E.3 now explains the structural roles. T1×T5 couples organization with contract encoding, T2×T3 couples hierarchy with step granularity, T4×T7 couples control flow with validation placement, and E1×E2 couples code location with decomposition. The four-pair allocation gives dedicated four-cell measurements to three instruction couplings and one executable coupling, defining which interactions directly inform the profile.

We also test an alternative allocation on Qwen3.5-9B. It retains T1×T5 and T4×T7 and replaces the other pairs with T1×T2 and E2×E4. Both allocations share the absolute diagnostic token cap, main-effect coverage, source skills, rewriter, final tests, and downstream budget (Table 22).

BenchmarkOriginal fourAlternative fourDifference [95% CI]
SkillsBench62.461.8-0.6 [-1.32, 0.11]
SWE-Skills47.547.0-0.5 [-1.23, 0.23]

The replacement changes are small and their paired intervals span zero, quantifying sensitivity to this alternative. The no-interaction ablation loses 6.4 recovery points (Table 18), supporting the contribution of interaction information.

Reviewer_mwbZOpenReview ↗

Official Comment by Reviewer mwbZ

Thank you for your additional efforts and experiments to clarify the ambiguous aspects of the paper and investigate different interactions. While I acknowledge the importance of the problem addressed and the promising improvements demonstrated throughout the experiments, I still believe that the paper requires substantial improvements in terms of clarity and soundness. Therefore, I will maintain my original rating.

Reviewer_mBxmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(−1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to adapt agent skills across language models with different preferences for instruction structure and executable organization. The authors introduce SkillProbeBank, a collection of controlled representation variants, and SkillProbe, a framework that uses Bayesian profiling and adaptive testing to identify model-specific representation preferences under a limited budget. The resulting profiles guide skill migration, generation, and adaptation. Experiments across multiple models and benchmarks show improvements in diagnosis efficiency and downstream skill performance.

Critical strengths
  1. Interesting and practical research problem. The paper addresses the important challenge of transferring agent skills across different language models. Studying how models respond to different representations of the same skill is an interesting direction, and identifying these preferences before skill adaptation has clear practical value.

  2. Well-designed framework and broad evaluation. SkillProbeBank uses controlled representation variants and held-out tests to study model preferences. The framework also considers uncertainty and evaluation costs. Experiments across multiple model families, benchmarks, and skill adaptation methods provide encouraging evidence for its practical usefulness.

Critical weaknesses
  1. Limited evidence for the value of model-specific profiles:
    SkillProbe improves downstream performance, but it is unclear whether these gains come from learning each model's preferences or simply from better skill rewriting. Most representation factors show similar preferences across models. The main migration experiments do not compare SkillProbe with using the same representation rules for all models or using a profile learned from another model. These comparisons would help show whether model-specific profiles are truly necessary.

  2. Unclear ablation results and limited statistical reporting:
    Removing the decision regret component barely changes the main diagnosis metrics, but the reported migration score drops substantially. However, the paper does not clearly explain how this migration score is calculated. The main results also lack confidence intervals, and the reported average ranks appear to include reference models that the authors state should be excluded. These issues make it difficult to interpret the ablation results and assess the reliability of the reported comparisons.

Questions for authors
  1. How does SkillProbe compare with using the same representation rules for all models, or using a preference profile learned from another model, while keeping the skill rewriting process unchanged?

  2. What exactly does the Migr. metric in Table 4 and Table 17 measure? Why does removing decision regret have little effect on diagnosis agreement but cause a large drop in migration performance?

  3. Could the authors provide uncertainty estimates for the main results and clarify the average rank calculation in Tables 2 and 12? The reported ranks appear to include reference models that Appendix D.1 states should be excluded.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer mBxm

Thank you for your thoughtful comments. We have carefully revised the manuscript to address your concerns and suggestions. All references below are to the revised paper.

[w1, q1] “whether these gains come from learning each model’s preferences or simply from better skill rewriting”

We now hold source skills, rewriter, validation, and downstream budget fixed and vary only the guidance. Shared rules are generated randomly and manually refined. Donor profiles come from Gemma 2 27B for Qwen3.5-9B and Qwen3.5-27B for Gemma 2 9B (Table 5, App. E.1).

TargetBenchmarkShared rulesDonor profileTarget profile
Qwen3.5-9BSkillsBench58.656.762.4
SWE-Skills44.042.647.5
Gemma 2 9BSkillsBench48.947.252.7
SWE-Skills37.836.242.1

Target profiles improve scores by 3.5–4.3 points over shared rules and 4.9–5.9 over donor profiles. All eight paired 95% gain intervals lie above zero (Table 19). The comparison supports the incremental value of target-matched guidance under the same rewriting process. Similar preferences on many factors can coexist with consequential deployment-specific differences.

[w2, q2] “What exactly does the Migr. metric in Table 4 and Table 17 measure?” / “Why does removing decision regret have little effect on diagnosis agreement but cause a large drop in migration performance?”

Migr. is SkillsBench migration gain recovery for Qwen3.5-27B→9B using SkillGraph. It is 100(Pc−P0)/(PF−P0)100(P_c-P_0)/(P_F-P_0), where PcP_c, P0P_0, and PFP_F are pass rates with the tested configuration, no profile, and a Full-Bank profile. Tasks are averaged equally within each seed, then across seeds 11, 23, and 47. The definition appears in the Table 4 caption and App. E.4, Eq. (8). The full ablation table is now Table 18.

Removing regret changes agreement from 84.3% to 84.1% and recovery from 92.4% to 83.2%, a 9.2-point recovery decrease [7.61, 10.81]. Agreement counts matching decisions equally. Regret weights each choice by the effect beyond the ROPE, with loss ∣eq∣|e_q| for neutrality and 2∣eq∣2|e_q| for the opposite direction. Similar agreement can therefore accompany different effect-weighted losses. Recovery also normalizes the downstream difference by the Full-Bank gain, so 9.2 recovery points correspond to 0.092(PF−P0)0.092(P_F-P_0) in pass rate. Section 6.5 and App. E.4 now explain both the weighting and scale.

[w2, q3] “provide uncertainty estimates for the main results and clarify the average rank calculation”

Table 23 now reports paired differences and 95% intervals. Migration compares SkillGraph+SkillProbe with SkillGraph, and diagnosis compares SkillProbe with Continuous CAT.

ComparisonMetricDifferencePaired 95% CI
MigrationALF All0.4[-0.14, 0.95]
Web success-0.2[-1.21, 0.84]
SkillsBench7.5[5.49, 9.59]
SWE-Skills6.4[4.29, 8.47]
DiagnosisError ↓-18.6[-22.58, -14.54]
Agreement17.0[12.57, 21.17]
Sentinel accuracy7.3[5.19, 9.57]

Pairing retains semantic cores for diagnosis and skills for migration. For recovery, matched skill clusters are resampled jointly and the ratio is recomputed in each resample (App. D.2). The SkillsBench and SWE-Skills gain intervals lie above zero, while those for ALFWorld and WebShop span zero.

The original ranks incorrectly included the three family-reference rows. We recomputed all 17 comparable methods. Each benchmark rank is one plus the number of strictly better methods, so ties share the minimum rank. Average rank is the mean across ALFWorld aggregate success, WebShop success, SkillsBench, and SWE-Skills. SkillGraph+SkillProbe changes from 1.50 to 1.25 and SkillRL+SkillProbe from 2.75 to 2.25. Tables 2 and 13 and their discussion now use the corrected values and candidate set (App. D.1–D.2).

Thank you for the clarifications. However, the evidence for model-specific profiles remains incomplete. The donor comparisons confound model family and size, and the competitiveness of the shared-rule baseline remains unclear. I therefore maintain my original rating.

Reviewer_xq6bOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper recognizes that even though skills files are readable across language models, perturbations that preserve the semantic core of the skill still result in different resulting behaviors of the same model. This indicates that text skills written / optimized for 1 model might not be optimally transferrable as is. The paper first contributes SKILLPROBEBANK for investigating the preference profile of any particular language model and the SKILLPROBE for ascertaining this profile and leveraging it to adapt skills to a target LLM.

Critical strengths
  1. The bayesian model allows for capturing the run-to-run model randomness (even at fixed configuration) thus removing that as a confounder
  2. The adaptive evaluation means the procedure cost can be controlled when used on a new family / set of models
  3. The problem explored is interesting -- though I am unsure if it will remain an issue in newer generations of the model
Critical weaknesses
  1. The models surveyed are older model generations (gemini 2.5, gpt 4o, claude 3.7) -- and whilst I understand resource and access may have been an issue, it is very possible that the newer generation of models do not suffer that much from sensitivity to changes to skills that still preserve the original semantics.
  2. The writing quality / prose of the sections after the introduction make the paper quite hard to parse. For example, the problem formulation as expressed by equations (1), (2) and can be better conveyed in simpler, natural language. Terms like Diagnostic channel, task family, ROPE and evidence D are not defined before use, making the problem setting section very hard to parse. The sentences tend to be short statements about configurations -- making it quite a hard read
Questions for authors
  1. Why the particular choice of models to explore. Is there any evidence that the performances of newer models are significantly impacted by these biases.
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer xq6b

Thank you for your thoughtful comments. We have carefully revised the manuscript to address your concerns and suggestions. All references below are to the revised paper.

[w1, q1] “The models surveyed are older model generations” / “Why the particular choice of models to explore.”

The initial evaluation covers four Qwen scales and three closed-model families to compare preferences within and across families. We have added gpt-6.1-sol and claude-sonnet-5.5, using the original evaluation setting. Profile conditioning improves SkillGraph on both procedural benchmarks (App. E.2, Table 20).

ModelBenchmarkSkillGraph+SkillProbeGain [95% CI]
gpt-6.1-solSkillsBench64.869.34.5 [2.26, 6.64]
SWE-Skills52.157.45.3 [3.10, 7.50]
claude-sonnet-5.5SkillsBench68.271.83.6 [1.70, 5.50]
SWE-Skills55.659.74.1 [1.96, 6.26]

Gains are percentage points with paired 95% confidence intervals. Direct matched probes also show sensitivity to equivalent representations (Table 21). External versus inline code improves scores by 7.3 points [1.44, 13.02] on gpt-6.1-sol and 5.2 [0.35, 10.31] on claude-sonnet-5.5. Numbered steps versus prose gives +4.8 [0.09, 9.05] and +2.8 [-0.30, 6.20], respectively. The latter interval spans zero. These results establish representation effects and adaptation benefits in the tested newer deployments, with factor-specific variation.

[w2] “Terms like Diagnostic channel, task family, ROPE and evidence D are not defined before use.”

We rewrote Section 3 to explain the problem before introducing its notation. DD contains paired outcomes from shared test inputs, a diagnostic channel measures an outcome such as task success, and a task family groups related procedures. ROPE is expanded at first use as the region of practical equivalence, defining a negligible effect. Equation (1) estimates representation effects and uncertainty. Equation (2) selects tests that reduce expected representation-decision loss under a budget.

Sections 4–5 now connect the stages through one extraction example. Prose and numbered-step variants share the same inputs and JSON verifier. Their paired effects update the posterior, which guides the next test and the final representation recommendation. Comparing that recommendation with the source skill yields KEEP, TRANSFORM, or DEFER. App. E.5 follows the complete process and explains anchors, sentinels, and coverage gates in context.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response for all reviewers

We thank all reviewers for your comments. We have added experiments and clarified the method and result reporting. References below are to the revised paper, where changes are marked in blue.

With source skills, rewriter, validation, and downstream budget fixed, target profiles improve SkillsBench and SWE-Skills scores by 3.5–4.3 points over shared rules and 4.9–5.9 over donor profiles across Qwen3.5-9B and Gemma 2 9B. All eight paired 95% gain intervals lie above zero (Table 5, App. E.1). On gpt-6.1-sol and claude-sonnet-5.5, profile conditioning improves SkillGraph by 3.6–5.3 points, with positive paired intervals in all four comparisons. Matched probes also show representation sensitivity on the newer deployments (App. E.2, Tables 20–21).

We define Migr. as the fraction of Full-Bank downstream gain recovered, 100(Pc−P0)/(PF−P0)100(P_c-P_0)/(P_F-P_0). Removing regret reduces recovery by 9.2 points [7.61, 10.81], while agreement changes by 0.2 points. Section 6.5 and App. E.4 explain the different scales and effect weighting. Table 23 reports the main paired intervals. SkillGraph+SkillProbe gains 7.5 points on SkillsBench [5.49, 9.59] and 6.4 on SWE-Skills [4.29, 8.47], while its ALFWorld and WebShop intervals span zero. We corrected ranks to exclude the three reference rows, giving average ranks of 1.25 for SkillGraph+SkillProbe and 2.25 for SkillRL+SkillProbe (Tables 2 and 13, App. D.1–D.2).

Sections 3–5 now explain the method through one extraction example, from paired outcomes to posterior inference, test selection, and KEEP/TRANSFORM/DEFER (App. E.5). App. E.3 explains the four interaction pairs and tests a two-pair replacement under the same budget and main-effect coverage. The replacement changes SkillsBench by -0.6 points [-1.32, 0.11] and SWE-Skills by -0.5 [-1.23, 0.23] (Table 22).

AuthorsOpenReview ↗

Revised PDF with changes in blue

We have carefully revised the manuscript in response to your comments. Added and modified passages are marked in blue. All references in our responses refer to this revised version.

Changes to the paper.

  • Sections 3–5 clarify the terminology and connect probe outcomes, posterior inference, adaptive selection, and representation decisions through an extraction example.
  • Section 6 adds the profile controls in Table 5 and clarifies the Migr. metric. Tables 2 and 13 report corrected average ranks.
  • Appendix E.1–E.3 reports the profile comparisons, newer-model experiments, and interaction-allocation sensitivity analysis.
  • Appendix E.4 defines migration gain recovery and reports paired confidence intervals for the principal comparisons.
  • Appendix E.5 provides the running example, including uncertainty, task-family heterogeneity, and KEEP/TRANSFORM/DEFER decisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.