- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether jointly injecting related retrieved skills into an LLM provides benefits beyond the better standalone skill. The authors propose a framework based on PID to decompose changes in reference-answer log-likelihood into shared, unique, and interaction contributions, and measure marginal value as the joint gain minus the better single-skill gain. Experiments with Qwen3-8B across multiple benchmarks show that most retrieved skill pairs fail to outperform the better single skill, while constructed complementary hints can yield positive gains. Results also show improvements from using PID interaction terms to guide skill merging.
Critical strengths
- The paper addresses a meaningful question: a relevant retrieved skill may provide little additional value given the existing context. Comparing no-skill, single-skill, and joint-skill conditions offers an interesting perspective on the conditional benefit of adding a second skill.
- The empirical analysis is broad. It spans mathematics, question answering, code generation, and interactive tasks, and provides useful checks on the robustness and interpretation of the findings.
Critical weaknesses
- The signed decomposition may not distinguish shared from unique information. Two skills may each increase the reference-answer log-likelihood by 1 while conveying either the same knowledge or entirely different useful clues. In both cases, the definition gives and , which indicates no unique evidence. This raises concerns about the validity of interpreting these terms as shared and unique contributions.
- The behavioral marginal is susceptible to bias from taking the maximum of two estimated pass rates: As an example, if all three conditions have the same true pass rate and the estimates are unbiased, then . Sampling variation between the two single-skill estimates can therefore make this expectation negative even when composition causes no actual harm. The reported negative marginals may partly reflect this estimation bias, which weaken the evidence for harmful skill interactions.
Questions for authors
- Could you address the concerns raised in the Weaknesses section?
Summary of the 18 November Revised PDF
This note updates our 11 November summary. The revised paper separates three questions about adding top-2 after top-1: Does the pair improve the final answer? How does it change the reference-answer score? What information about the correct answer appears in the two single-skill predictions?
Comparing prompts
Top-1 is chosen before the model is evaluated, so pair versus top-1 is now our primary behavioral comparison. Choosing the better of the two single skills after seeing their results provides a retrospective oracle reference. The revised paper labels it accordingly.
Interpreting the measurements
The signed terms describe how prompt scores overlap and interact. They do not identify information shared by the skill texts. Williams–Beer PID instead measures answer information in the two single-skill predictions. The revised title, abstract, and Figure 1 caption make this distinction explicit.
Table 2 (p. 5) adds two-slot controls with equal complete-prompt token lengths across seven benchmarks. On MMLU, the score interaction is +1.67 nats [1.28, 2.09] against shuffled words, but +0.15 nats [−0.16, 0.48] against irrelevant skills. Appendix G.1 gives the four scores behind each comparison. The positive shuffled-control result therefore depends on which text serves as the control.
Explaining the changed PID numbers
Our earlier PID correction shuffled individual skill-pair records. Several records come from the same question and share its correct answer, so Table 3 (p. 7) now shuffles labels by question while keeping those records together. On the full MMLU set, the corrected redundancy, information unique to the second prediction, and synergy estimates change from 0.168/0.049/0.054 to 0.149/0.041/0.027 bits. On the 67 questions answered incorrectly without a skill, corrected redundancy is 0.002 bits (p=0.364).