acceptodds
Under review as a conference paper at ICLR 2027

What to Distill and What to Retrieve: Knowledge-Split Distillation for LLM–SLM Collaboration

Abstract

Large–small language model (LLM–SLM) collaboration enables SLMs to leverage the stronger knowledge and reasoning capabilities of LLMs while reducing dependence on repeated LLM assistance. Across different collaboration paradigms, LLM-provided information is ultimately either internalized into SLM parameters or retained externally for on-demand access. Yet existing approaches largely treat this placement as a consequence of the collaboration paradigm, leaving a fundamental question under-explored: what should the SLM learn, and what should it retrieve? We find that LLM-provided information can be broadly divided into domain-specific knowledge and generalizable reasoning capabilities, which favor different placement strategies. Domain-specific knowledge primarily serves particular domains or tasks. Parameterizing such knowledge in a capacity-limited SLM may introduce knowledge interference without necessarily improving its generalizable capabilities, making external storage more suitable. In contrast, generalizable reasoning capabilities, such as the ability to select, organize, and apply knowledge, can transfer across problems and tasks and are therefore better internalized into SLM parameters than repeatedly retrieved during inference. However,such differentiated placement is challenging because domain-specific knowledge and generalizable reasoning are often tightly coupled within the same reasoning trajectory, and their roles depend on the dependencies among reasoning steps. The key challenge is therefore to disentangle them based on these dependencies while preserving a complete reasoning path. We propose KSplit, a type-aware knowledge-placement framework for LLM–SLM collaboration. KSplit represents LLM reasoning as a typed dependency graph connecting knowledge units and reasoning operations through their functional dependencies. Dependency-Aware Knowledge Partitioning (DAKP) exploits these dependencies to disentangle domain-specific knowledge from generalizable reasoning while preserving complete reasoning paths. Functional Knowledge Placement (FKP) then internalizes generalizable reasoning through knowledge-free reasoning supervision, while compiling domain-specific knowledge into an external retrievable repository. Experiments across multiple benchmarks show that KSplit consistently improves LLM–SLM knowledge transfer, with controlled studies further validating its type-aware partitioning and placement strategy.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Large-Small Language Model Collaboration, Knowledge Distillation, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VEKPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KSplit, a type-aware knowledge replacement framework for LLM-SLM collaboration. To be specific, KSplit represents LLM reasoning as a typed dependency graph connecting knowledge units and reasoning through their functional dependencies. Dependency aware knowledge portioning is introduced to preserve complete reasoning paths during separation and function knowledge placement is utilized to internalize reusable reasoning while retaining domain knowledge externally. Extensive experiments show the effectiveness of the proposed method.

Critical strengths
  1. This paper identifies a key knowledge placement principle for LLM and SLM collaboration, where domain specific knowledge generalizable reasoning capabilities exhibit different transfer characteristics. This observation is interesting.
  2. LLM-SLM collaboration is a well-motivated and practical research direction.
  3. Extensive experiments demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. It would be beneficial to add a further comprehensive experiment to analyze the necessity of the placement strategy. As shown in Table 12, only the OpenbookQA dataset shows positive statistical support, and the remaining datasets do not show positive support..
  2. For the CAKR variant, the performance gains are modest. It would be beneficial to clarify the reason behind introducing CAKR module.
  3. Parameterizing R involves the raw question and answer, so it introduces domain-specific information to some content. It would be better to clarify this.
  4. It would be beneficial to polish Figures 1 and 3 to improve readability.
Questions for authors

Please refer to the weaknesses section. Main concerns involve the contribution and necessity of each component.

2 comments

We thank the reviewer for the positive assessment of our research question, the practical value of LLM–SLM collaboration, and our experimental results. We address your concerns about placement, CAKR, R supervision, and figure readability below.

1. Placement strategy

We added RR and KK controls. The first letter denotes internalized content, the second external content; ₀ denotes no external content. RK−KR compares the proposed and reversed placements. RK−RR holds parameterized R fixed to compare external K with R; RK−KK holds external K fixed to compare internalized R with K. Accuracy is in %, differences in percentage points (pp).

DatasetNR₀RRRKK₀KRKKRK−KRDIDRK−RRRK−KK
OB40085.6784.9286.8385.9284.6786.002.172.421.920.83
ARC117284.1684.2284.4483.9683.7383.900.710.510.230.54
Med90058.1957.6759.5258.0757.8958.701.631.521.850.81
Strategy22971.1870.3171.9171.6271.3270.890.581.021.601.02
QuaRel55293.0692.4592.8791.9192.2192.390.66−0.480.420.48

Differences use unrounded results. DID=(RK−R₀)−(KR−K₀) compares incremental gains relative to each branch's baseline. We compute paired differences within each task and average them equally across five tasks:

ContrastMean [unadjusted 95% CI], pp
RK−KR1.15 [0.22,2.08]
DID1.00 [0.07,1.97]
RK−RR1.20 [0.44,2.00]
RK−KK0.74 [−0.16,1.67]

RK has higher mean accuracy than KR, RR, and KK on all five tasks. The average RK−KR, DID, and RK−RR intervals exceed zero. These unadjusted intervals provide supplementary evidence for an average benefit. The task-level results show variation in that benefit. On QuaRel, RK−KR≈0.66 pp but R₀−K₀≈1.14 pp, giving DID≈−0.48 pp: RK has higher final accuracy, while KR gains more from its corresponding external content. These controls support internalizing R and keeping K external, with task-dependent gains.

2. Motivation for CAKR

R and RK can complement each other because questions differ in their knowledge requirements and retrieval quality varies. First, questions relying mainly on general knowledge may need little additional domain-specific K. Second, retrieval may return related content that fails to support the answer and disrupts an otherwise correct prediction. Trained on held-out or out-of-fold outcomes, CAKR uses question, retrieval, and candidate-prediction signals to select between the executed paths.

On OpenBookQA, Fixed RK achieves 91.40%, while the candidate oracle reaches 93.00%, leaving 1.60 pp of selection headroom. Across 1,500 decisions from three seeds, five corrections and one introduced error yield four additional correct answers: a gain of approximately 0.267 pp, capturing 16.7% of the 1.60 pp oracle headroom. The oracle uses gold labels to measure the upper bound; actual gains reflect both corrections and introduced errors.

For OpenBookQA question 1530 (seed 20260801), “Which of these travels through the air?”, the options are planets, thoughts, automobile, and music. R correctly selects music; RK selects automobile after retrieving content about airplanes and air surrounding planets. CAKR retains R's correct answer. This illustrates how selection helps when general knowledge suffices and retrieval provides insufficient support.

In the main table, CAKR improves mean accuracy by 0.06–1.17 pp across five tasks, including 0.59 pp on ARC-Challenge and 1.17 pp on StrategyQA. It is an optional enhancement. Fixed RK provides a complete separation-and-placement configuration.

3. Domain information in question–answer supervision

Original questions and answers carry task-relevant knowledge and provide shared supervision during R adaptation. KSplit studies how to allocate the additional facts, rules, and reasoning operations supplied by the teacher alongside this supervision.

DAKP separates this content by function. FKP compiles explicit K-node text into the external repository and R operations into interface targets—the allocation described as “K-text-free R supervision.” Students may retain knowledge acquired during pretraining and learn task-relevant information through shared question–answer supervision.

The separation controls hold question–answer supervision fixed and assess factual probes and task accuracy. Across all five tasks, 95% CIs for factual-probe differences lie below zero, while task-accuracy lower bounds exceed the −1 pp non-inferiority margin. The probes measure correct-answer log-odds: these results support reduced confidence in the probed facts while preserving task utility. We clarified the training components and their destinations in §3.3; results appear in §4.4, with settings in Appendix D.

4. Figure readability

Thank you for the suggestion. We increased the text size in Figures 1 and 3 and refined their annotations and layout to improve readability in the revised manuscript.

Thank you for your response. Your responses have resolved part of my concerns, but I still have concerns regarding the necessity of placement strategy and CAKR module. Therefore, I tend to maintain my current rating.

Reviewer_CFpnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors address an under-explored question: where LLM-provided information should reside in LLM–SLM collaboration. To this end, the authors propose KSplit, which distinguishes domain-specific knowledge (K) from generalizable reasoning operations (R) via a typed dependency graph (DAKP), then places R parametrically and K externally (FKP), with an optional instance-wise router (CAKR). Experimental results verified the effectiveness of the proposed method.

Critical strengths

This paper identifies a interesting but under-explored question: the information placement decision of domain-specific knowledge and reasoning operations. And using a typed dependency graph to disentangle K from R is reasonable, and the experimental results are competitive and convincing.

Critical weaknesses

Although this paper provides methods to distinguish between K and R, sometimes it is hard and vague between K and R. Furthermore, some reasoning abilities are high-dependent to knowledge within a LLM, e.g., the decomposition of a big problem will depend on the problem-specific knowledge.

For experimental results, all experiments use Qwen2.5-3B-Instruct as the student, the generalization ability should be further verified and discussed.

Questions for authors

See weaknesses in above.

1 comment

We thank the reviewer for recognizing the research question, the dependency-graph design, and the experimental results. We address your concerns about K/R separation and student-model generalization below.

1. The K/R boundary and the dependence of reasoning on knowledge

The knowledge dependencies you highlight are precisely the relationships KSplit aims to preserve. We distinguish K from R according to their functions in solving a problem: facts, definitions, and rules belong to K, while operations that organize, apply, and check this content belong to R. Facts or rules that are reusable across questions may still be classified as K based on their function.

For problem decomposition, domain facts or rules that determine the basis for decomposition are represented as K, while operations that organize and solve subproblems using these premises are represented as R. The graph explicitly records the dependencies between them. Thus, the reusability of R refers to applying operations to different evidence, drawing on the knowledge required by those operations.

For example, consider the question, “A hawk is chasing a mouse. The mouse suddenly darts away. How should the hawk respond?” Identifying the pursuer, the target, and the change in the target’s state belongs to R. The rule that “a pursuer needs to adjust its speed or direction when the target changes position” is the K needed to make the judgment. Applying this rule to the current situation and selecting an answer belongs to R. DAKP represents the rules and operations separately and records their dependencies.

FKP then uses R for parametric learning and retains K externally for use during reasoning. This preserves the support that knowledge provides to reasoning while specifying where each type of content should reside.

To assess the reliability of the resulting separation, we added a human audit of complete K/R graphs from OpenBookQA and MedQA, examining their type assignments and dependencies according to the functional criteria above.

DatasetGraphs auditedAcceptableUnacceptableUncertainAcceptance rate
OpenBookQA40355087.50%
MedQA40361390.00%
Total80716388.75%

We manually audited 80 complete K/R graphs from OpenBookQA and MedQA, of which 71 were judged acceptable, yielding a graph-level acceptance rate of 88.75%. This provides direct human assessment of the teacher-assisted construction. We view this separation as a functional representation for organizing supervision and storage, allowing reasonable ambiguity in how some natural-language steps are segmented.

2. Generalization across student model families and sizes

We expanded the student-model evaluation on OpenBookQA Test500 to cover three families—Qwen, Llama, and Ministral—and three Qwen2.5 sizes: 1.5B, 3B, and 7B.

The tables report accuracy (%).

Student modelBaseRKRKRouter
Qwen2.5-3B79.287.47±0.4685.93±1.2987.47±1.1589.13±0.81
Llama-3.2-3B62.282.60±1.0474.87±0.1282.67±1.5182.67±1.51
Ministral-3-3B69.685.80±1.3185.53±0.9086.67±1.1086.87±1.27
Qwen2.5 sizeBaseRKRKRouter
1.5B72.681.80±1.0682.93±0.4283.00±1.2283.47±0.12
3B79.287.47±0.4685.93±1.2987.47±1.1589.13±0.81
7B86.892.73±0.4291.93±0.5092.80±0.5394.80±0.60

Among the evaluated pipelines, the complete pipeline achieves the highest mean accuracy in every group, including ties; with the 7B student, it reaches 94.80%. Results with error terms report the mean and sample standard deviation across three seeds. These results extend the evaluation beyond a single Qwen2.5-3B student to multiple model families and capacities, supporting the framework’s applicability across students.

Reviewer_DEYAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KSplit, a framework for allocating teacher-derived information between small language model (SLM) parameters and external knowledge storage. It introduces Dependency-Aware Knowledge Partitioning (DAKP) to separate domain-specific knowledge from reusable reasoning operations using typed dependency graphs, and Functional Knowledge Placement (FKP) to externalize knowledge while internalizing reasoning capabilities. An optional Capability-Aware Knowledge Routing (CAKR) module selects between reasoning-only and retrieval-augmented predictions. Experiments on five benchmarks demonstrate competitive performance and investigate the effectiveness of the proposed knowledge placement strategy.

Critical strengths
  1. The paper addresses an important question in collaboration between large and small language models: what information should be learned and what should remain externally accessible. KSplit provides a structured approach to separating factual knowledge from reasoning operations through typed dependency graphs, with distinct mechanisms for parameter adaptation and external knowledge storage.
  2. KSplit achieves competitive results across five benchmarks. Fixed RK outperforms the strongest evaluated baseline outside KSplit on three datasets and remains close on the other two. The authors also conduct controlled experiments on knowledge placement and separation, providing useful evidence beyond overall system accuracy.
Critical weaknesses
  1. In Section 4.3, RK outperforms the reversed KR configuration in point estimates across all five tasks, but none of the reported 95% confidence intervals for these direct comparisons lies strictly above zero. The adjusted DID provides positive statistical support only on OpenBookQA and is negative, though uncertain, on QuaRel. These results do not yet establish a consistent advantage for the proposed allocation across tasks.
  2. The Same-K evaluation in Section 4.4 is limited to OpenBookQA. Although Graph-R reduces K acquisition while maintaining noninferior accuracy, Bag-R targets are 57.8% longer, and differences in organization and expression remain uncontrolled. Thus, the results support the overall supervision design rather than the specific contribution of graph dependencies. Moreover, comparisons with relevant methods such as KARD [1] are missing, limiting evidence of the proposed approach's advantages over existing methods.

[1] Kang, Minki, et al. "Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks." NeurPIS 36 (2023): 48573-48602.

Questions for authors
  1. Since K/R typing and validation rely on teacher assistance, could the authors provide human annotation results or analyze ambiguous cases to assess the reliability of the separation?
  2. The Same-K experiment changes target organization, length, and expression simultaneously. Could the authors clarify which of these factors is most likely to explain the observed reduction in K acquisition, or provide a more controlled comparison?
  3. Since the experiments use Qwen2.5-3B-Instruct as the student model, is there evidence that the proposed placement principle generalizes to other student model sizes or architectures?
  4. Given that CAKR selects answers after retrieval and candidate execution, could the authors quantify its additional inference cost relative to Fixed RK and discuss whether the observed accuracy gains justify this overhead?
3 comments

We thank the reviewer for the thoughtful feedback and positive assessment of our research question, structured separation method, and experimental results. We address your concerns below.

1. Placement and statistical evidence

We added RR and KK controls. The first letter denotes internalized content, the second external content; ₀ denotes no external content. RK−KR tests reversed placement; RK−RR fixes parameterized R to compare external K with R; RK−KK fixes external K to compare internalized R with K. DID=(RK−R₀)−(KR−K₀) compares gains relative to each branch's baseline. Accuracy is in %, differences in percentage points (pp).

DatasetNR₀RRRKK₀KRKKRK−KRDIDRK−RRRK−KK
OB40085.6784.9286.8385.9284.6786.002.172.421.920.83
ARC117284.1684.2284.4483.9683.7383.900.710.510.230.54
Med90058.1957.6759.5258.0757.8958.701.631.521.850.81
Strategy22971.1870.3171.9171.6271.3270.890.581.021.601.02
QuaRel55293.0692.4592.8791.9192.2192.390.66−0.480.420.48

Differences use unrounded results. Equal-weight averages of within-task paired differences:

ContrastMean [unadjusted 95% CI], pp
RK−KR1.15 [0.22,2.08]
DID1.00 [0.07,1.97]
RK−RR1.20 [0.44,2.00]
RK−KK0.74 [−0.16,1.67]

RK has higher mean accuracy than KR, RR, and KK on all five tasks. The average RK−KR, DID, and RK−RR intervals exceed zero. These support an average effect, with per-task uncertainty and multiple comparisons still relevant. On QuaRel, RK−KR≈0.66 pp but R₀−K₀≈1.14 pp, giving DID≈−0.48 pp: RK has higher final accuracy, while KR gains more from external content.

2. Same-K controls

Original Bag-R targets are 57.8% longer than Graph-R, so this evaluates the overall supervision design. Our content-matched structural controls reuse identical knowledge content and R-node text, changing only explicit dependencies or bindings to control node-text content, wording, and length.

Cross-task settings include an OB content-scope control, ARC full-test follow-up on frozen models, and MedQA evaluation after a shared task update. Across all five tasks, 95% CIs for factual-probe differences lie below zero and task-accuracy lower bounds exceed the −1 pp non-inferiority margin. Differences are DAKP-derived condition minus control. Probes measure correct-answer log-odds. Full results appear in §4.4, with settings in Appendix D.

3. Human validation

We audited 80 complete K/R graphs, checking functional typing and dependencies: K supplies facts, definitions, and rules; R organizes, applies, and checks them. Counts for acceptable, unacceptable, and uncertain graphs were 35/5/0 on OpenBookQA and 36/1/3 on MedQA. Overall, 71/80 graphs were acceptable (88.75%), complementing the model-assisted diagnostic.

4. Student generalization

On OpenBookQA Test500, complete-pipeline means are 89.13%, 82.67%, and 86.87% for Qwen, Llama, and Ministral, and 83.47%, 89.13%, and 94.80% for Qwen2.5 at 1.5B, 3B, and 7B. Each is the highest mean in its group, including ties. RK matches or exceeds both R and K in each family. Appendix E gives means and sample SDs across three seeds.

5. KARD

Our original evaluation covered distillation, retrieval, and collaboration baselines, with KARD discussed in related work. Following your suggestion, we added a direct comparison with KARD in §4.2. KARD combines external knowledge with reasoning distillation; KSplit further separates teacher content, preserves dependencies, and evaluates its placement. Fixed RK achieves higher mean accuracy on four tasks and is 0.11 pp lower on MedQA. CAKR matches KARD’s reported mean on MedQA and exceeds it on the other four tasks.

6. CAKR: benefits and cost

CAKR addresses two sources of complementarity between R and RK. First, questions relying mainly on general knowledge may need little additional domain-specific K. Second, retrieval may return related content that fails to support the answer and disrupts a correct prediction. CAKR selects between executed paths using question and candidate signals.

The OB candidate oracle reaches 93.00% versus Fixed RK's 91.40%, showing headroom for selection. The oracle uses gold labels. On OB question 1530 (seed 20260801), “Which of these travels through the air?”, R correctly selects music; RK selects automobile after retrieving content about airplanes and planets; CAKR retains R. This illustrates limited need for domain-specific K and retrieval interference.

In the main table, CAKR improves mean accuracy by 0.06–1.17 pp, including 0.59 pp on ARC-Challenge and 1.17 pp on StrategyQA. In a separate runtime measurement on 500 questions at batch size 8, Fixed RK and CAKR take 86.25 and 105.83 seconds, respectively, an increase of 22.70%. Fixed RK provides a complete separation-and-placement configuration. CAKR can be enabled as an optional enhancement according to accuracy and latency requirements.

Thank you for the detailed responses and substantial revisions. I appreciate the additional experiments, which have addressed several of my original concerns. However, two concerns regarding the core technical contributions remain:

  1. Knowledge placement: After multiplicity correction, RK−RR remains statistically supported, whereas RK−KK, RK−KR, and DID do not. Could the authors clarify how the current evidence supports both aspects of the proposed asymmetric placement principle?

  2. DAKP contribution: Given the different experimental settings in Table 3, could the authors specify which comparisons isolate the contribution of graph dependencies while controlling knowledge content, target wording, and length?

Thank you for your positive assessment of the additional experiments and revisions. We address your two follow-up questions below.

1. Evidence for both sides of the asymmetric placement principle

We measure the average paired gains of different placement choices across five tasks. We compute paired differences within each task, then average them with equal weight across tasks. We report mean effects and their uncertainty using 95% confidence intervals.

Equal-weight average across five tasksMean difference [95% CI] (pp)
RK−KR+1.15 [0.22, 2.08]
DID+1.00 [0.07, 1.97]
RK−RR+1.20 [0.44, 2.00]
RK−KK+0.74 [−0.16, 1.67]

These comparisons assess the full configurations, gains over the respective parameter-branch baselines, and the two placement decisions. The mean effects for RK−KR and DID are positive, with 95% intervals entirely above zero. They support the average gain of the proposed configuration and its incremental gain relative to the branch baselines.

RK−RR holds internalized R fixed. Providing K externally in place of R improves accuracy by 1.20 pp on average. Both the ordinary and Bonferroni-adjusted intervals lie above zero, directly supporting the choice of external K. RK−KK holds external K fixed. Internalizing R in place of K yields a mean gain of 0.74 pp, with positive point estimates on all five tasks. This consistent direction provides empirical support for internalizing R, and the interval in the table quantifies uncertainty in the mean estimate.

The Bonferroni analysis uses a stricter criterion for simultaneous inference across six comparisons. Our primary aim is to report the magnitude and uncertainty of each mean gain. The revised manuscript consistently reports mean effects and 95% intervals and explains how they are calculated. Our assessment of the full placement design draws on these estimates, the consistent configuration results across tasks, and observations about the storage mechanisms. The comparison for external K has clear statistical support. The positive mean estimate and task-level results support internalizing R. The main text now explains how each comparison informs its placement decision.

2. Controls for isolating the contribution of graph dependencies

We added the comparison settings to the appendix. The paired structural controls retain the corresponding content and wording while varying the explicit dependency representation. The table lists the matched content, lengths, and supervision positions, along with the structural intervention in each comparison.

ComparisonContent and wording controlsLength and position controlsStructural intervention
OpenBookQASame questions, answers, and K inventory. Same R-node text, order, and format.Target token counts and token multisets match per example.Vary explicit R dependencies.
ARC-ChallengeSame source examples and R-node text.R text length matches within each pair.Vary explicit R–R dependency supervision.
StrategyQASame training examples and QA supervision. Same R-node text and order.Sequences have equal length. Supervised tokens, labels, masks, and absolute positions match.Vary explicit R dependency information through source-to-port bindings.
MedQASame R-node text. Both models receive the same task adaptation update.Supervised R tokens, masks, and absolute positions match in the original structural comparison.Vary explicit R dependency information through R–R binding context.
QuaRelSame relation and property values and R text. Matched subsequent task adaptation.R text length matches within each pair.Retain or flatten the explicit organization of relations and properties.

These comparisons share the same separation objective and encode dependencies in each task’s native representation. Matching the R text controls the wording and length of the operations. Matching the full sequence length and supervision positions further controls for positional shifts introduced by the structural context. The table specifies the level of matching for each comparison.

In all five paired evaluations in Table 3, the factual-probe difference intervals lie below zero, and the lower bounds of the task-accuracy difference intervals exceed the −1 pp non-inferiority margin. This supports separation supervision as a way to reduce the parametric expression of the probed facts while preserving task utility. The appendix provides further details on the experimental setup for each structural comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.