acceptodds
Under review as a conference paper at ICLR 2027

Beyond Similarity: A PID-Inspired Diagnosis of Skill Composition

Abstract

Modern large language models increasingly rely on retrieving and injecting reusable skills from external libraries. The prevailing paradigm often assumes a positive correlation between skill quantity and performance gain, directly concatenating the top-ranked similar skills into the prompt context. However, our study reveals significant limitations in this approach: aggregating highly related skills frequently leads to severe diminishing returns and information redundancy, or even induces negative interference that degrades model performance. The primary contribution of this paper is the introduction of a novel analytical framework based on Partial Information Decomposition (PID) to systematically investigate the discrepancy between “retrieval similarity” and “actual compositional utility.” By decomposing the model's log-likelihood into shared evidence, skill-specific evidence, and non-additive interactions, this framework enables precise quantitative evaluation of the complementary or interfering effects among multiple skills. Our extensive evaluations across seven math, question-answering, and code generation benchmarks reveal that top-ranked skill combinations typically suffer from information redundancy, yielding a marginal gain that is frequently negative relative to the best single skill. Furthermore, as an extension of this analytical framework, we conduct a preliminary exploration of using PID as a guiding strategy for skill selection; empirical results indicate that filtering skills via PID can mitigate inter-skill interference to some extent, offering a promising pathway for optimizing multi-skill composition in the future.

Primary area:
interpretability and explainable AI
Keywords:
partial information decomposition, skill composition, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BPXSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether jointly injecting related retrieved skills into an LLM provides benefits beyond the better standalone skill. The authors propose a framework based on PID to decompose changes in reference-answer log-likelihood into shared, unique, and interaction contributions, and measure marginal value as the joint gain minus the better single-skill gain. Experiments with Qwen3-8B across multiple benchmarks show that most retrieved skill pairs fail to outperform the better single skill, while constructed complementary hints can yield positive gains. Results also show improvements from using PID interaction terms to guide skill merging.

Critical strengths
  • The paper addresses a meaningful question: a relevant retrieved skill may provide little additional value given the existing context. Comparing no-skill, single-skill, and joint-skill conditions offers an interesting perspective on the conditional benefit of adding a second skill.
  • The empirical analysis is broad. It spans mathematics, question answering, code generation, and interactive tasks, and provides useful checks on the robustness and interpretation of the findings.
Critical weaknesses
  • The signed decomposition may not distinguish shared from unique information. Two skills may each increase the reference-answer log-likelihood by 1 while conveying either the same knowledge or entirely different useful clues. In both cases, the definition gives R=1R=1 and Ua=Ub=0U_a=U_b=0, which indicates no unique evidence. This raises concerns about the validity of interpreting these terms as shared and unique contributions.
  • The behavioral marginal Δa∣bbeh=pass(ab)−max⁡pass(a),pass(b)\Delta^{\mathrm{beh}}_{a\mid b}=\text{pass}(ab)-\max\\{\text{pass}(a),\text{pass}(b)\\} is susceptible to bias from taking the maximum of two estimated pass rates: As an example, if all three conditions have the same true pass rate and the estimates are unbiased, then E[Δa∣bbeh]=−12E[∣pass(a)−pass(b)∣]\mathbb{E} [\Delta^{\mathrm{beh}}_{a\mid b}]=-\frac{1}{2}\mathbb{E}\left[\left|\text{pass}(a)-\text{pass}(b)\right|\right]. Sampling variation between the two single-skill estimates can therefore make this expectation negative even when composition causes no actual harm. The reported negative marginals may partly reflect this estimation bias, which weaken the evidence for harmful skill interactions.
Questions for authors
  • Could you address the concerns raised in the Weaknesses section?
Reviewer_WEh2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether adding a second retrieved skill to a prompt helps beyond the better single skill. It measures the log-likelihood of the reference answer under no-skill, single-skill and two-skill prompts with a frozen Qwen3-8B, decomposes the joint gain into redundancy / unique / interaction terms (a signed PID), and reports that on seven benchmarks the second skill usually adds negative value, with free-generation and AlfWorld results pointing the same way.

Critical strengths

The question is practically relevant: most agent systems just concatenate the top-k retrieved skills, and nobody checks whether the second one helps.

Many controls (order swap, length-matched placebos, constructed hints, several redundancy definitions) and an interactive AlfWorld check, reported honestly.

Critical weaknesses

My main concern is that the headline result does not tell a practitioner anything actionable. The paper compares the two-skill prompt against the better single skill chosen per task after the fact. If the joint prompt simply behaves like the average of the two singles, this difference is negative by construction, so a negative median on all seven benchmarks cannot distinguish "the second skill interferes" from "the second skill dilutes".

The comparison an agent system actually faces is top-1 vs. top-1 + top-2, and the paper's own numbers for that (the "original" column in the placebo table) are positive on all three datasets. For the free-generation and AlfWorld results the "better single" is picked from 3 runs or 10 rollouts, which adds a selection bias of roughly the same size as the reported effects.

The measurement itself is also questionable: shuffled or irrelevant text raises the reference-answer likelihood more than the real skill does, so I am not sure the redundancy / unique / interaction terms are measuring skill information at all. The PID framing adds little in the end, since the quantity the paper acts on does not depend on the decomposition.

Questions for authors

Could you report the marginal of adding the second skill on top of the first-ranked skill (and relative to the average of the two singles) on all seven datasets, instead of relative to the per-task best single? Does the sign still hold?

Given that content-free text scores higher than a real skill under your evidence measure, why should the decomposition terms be read as skill redundancy or complementarity?

Reviewer_M3g3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors explore whether injecting multiple relevant skills into a prompt actually improves an LLM’s performance. They run a systematic study of whether pairs of skills are complementary, comparing the model with no skill, with each skill on its own, and with both together across seven math, multiple-choice and coding benchmarks using Qwen3-8B. They find that in most cases the skills aren’t complementary, and a second retrieved skill usually fails to beat the better of the two skills used alone, and sometimes noticeably lowers performance. Similarity between skills does little to predict whether they’ll work well together, while hand-crafted hints covering different parts of a problem do combine productively. The authors conclude that retrieval is useful for finding candidate skills, but whether to add another should depend on what it contributes beyond what’s already in the prompt.

Critical strengths
  • They have a nice experimental design that allows them to isolate the value of adding a second skill. Each pair is run with no skill, with each skill alone, and with both together, which makes it possible to ask whether the second skill adds anything beyond the first. This is a sharper question than prior work, which mostly varies how many skills are loaded, and the result holds up across seven benchmarks and in interactive settings like AlfWorld.
  • They run some good sanity checks on their own measurements, like swapping the order of the skills and replacing the second skill with irrelevant or scrambled text of the same length. To their credit, they report results that don’t fit their story, such as filler text helping more than the real second skill on the coding tasks. They don’t fully follow through on what this means, but it gives readers a fair picture of the method’s limits.
Critical weaknesses
  • The main comparison is against whichever single skill turned out better, which can only be known after the fact. That seems more like an oracle comparison. The fairer comparison, against the top-ranked skill, only shows up in side experiments, and there we see mixed results since the second skill helps a little on the evidence measure but hurts slightly in the GSM8K and τ²-Bench depth sweeps.
  • The decomposition doesn’t clearly tell us how the skills interact. The main result is just the pair’s score minus the better single score, which you can get without any of the PID machinery. The “positive synergy” on the coding tasks may not mean the skills are working together. On those benchmarks most pairs seem to be ones where both skills hurt, and in that case the formula reports positive synergy whenever the pair lands somewhere between the two, even though it still does worse than the better skill alone. The placebo controls raise further doubts, since on GSM8K and HumanEval swapping the second skill for irrelevant or scrambled text raises the score more on average than the real skill does. The authors acknowledge in passing that the interaction term picks up prompt-context effects, but the main text still treats it as evidence of how the skills work together, and the evidence that undercuts this is mostly left to the appendix.
Questions for authors
  • How does the pair compare with just the top-ranked skill across all seven benchmarks? Right now that comparison only appears for three datasets and in the GSM8K and τ²-Bench depth sweeps.
  • For the coding tasks, how many of the pairs with positive synergy are ones where both skills individually lower the score? If most are, the positive synergy may just mean the pair did better than the worse skill rather than the skills working together?
  • Since irrelevant or scrambled text raises the score more than the real second skill on GSM8K and HumanEval, how do you separate the effect of a skill’s content from the effect of simply adding text to the prompt? Could the decomposition be measured relative to a placebo baseline?
  • How does this work relate to SkillsBench, which already found that loading four or more skills does worse than two or three? What does the pairwise analysis show beyond that result?

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the 18 November Revised PDF

This note updates our 11 November summary. The revised paper separates three questions about adding top-2 after top-1: Does the pair improve the final answer? How does it change the reference-answer score? What information about the correct answer appears in the two single-skill predictions?

Comparing prompts

Top-1 is chosen before the model is evaluated, so pair versus top-1 is now our primary behavioral comparison. Choosing the better of the two single skills after seeing their results provides a retrospective oracle reference. The revised paper labels it accordingly.

Interpreting the measurements

The signed terms describe how prompt scores overlap and interact. They do not identify information shared by the skill texts. Williams–Beer PID instead measures answer information in the two single-skill predictions. The revised title, abstract, and Figure 1 caption make this distinction explicit.

Table 2 (p. 5) adds two-slot controls with equal complete-prompt token lengths across seven benchmarks. On MMLU, the score interaction is +1.67 nats [1.28, 2.09] against shuffled words, but +0.15 nats [−0.16, 0.48] against irrelevant skills. Appendix G.1 gives the four scores behind each comparison. The positive shuffled-control result therefore depends on which text serves as the control.

Explaining the changed PID numbers

Our earlier PID correction shuffled individual skill-pair records. Several records come from the same question and share its correct answer, so Table 3 (p. 7) now shuffles labels by question while keeping those records together. On the full MMLU set, the corrected redundancy, information unique to the second prediction, and synergy estimates change from 0.168/0.049/0.054 to 0.149/0.041/0.027 bits. On the 67 questions answered incorrectly without a skill, corrected redundancy is 0.002 bits (p=0.364).

AuthorsOpenReview ↗

further response to reviewer bpxs

Thank you for revisiting the paper. Your example highlights a distinction that we now make explicit. Our Williams–Beer PID analysis measures information about the gold answer in the two single-skill predictions. On MMLU, the corrected estimates show 0.168 bits of redundancy, 0.049 bits unique to the second prediction, and 0.054 bits of synergy. Controls with known information structure recover these patterns.

The signed equations address a related question: how does adding the second skill change the joint prompt’s score for the reference answer? The conditional increment is D₁₂-D₁=U₂+S, independent of the overlap convention. We retain the equations for this purpose and describe their terms as score overlap, residuals, and interaction. Equal score gains are not interpreted as evidence that two skill texts contain the same clue.

We have also made the fixed-rank comparison the primary behavioral result. Retrieval determines top-1 before evaluation; adding top-2 is then tested on the same tasks. In the new 379-task MMLU audit, the pair scores 3.17 percentage points above top-1 and 3.43 points above a shuffled second slot (95% CI [0.79, 6.33] for the latter). The observed-best comparison remains a supplementary oracle reference. We would appreciate your reconsideration in light of these revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.