- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes KSplit, a type-aware knowledge replacement framework for LLM-SLM collaboration. To be specific, KSplit represents LLM reasoning as a typed dependency graph connecting knowledge units and reasoning through their functional dependencies. Dependency aware knowledge portioning is introduced to preserve complete reasoning paths during separation and function knowledge placement is utilized to internalize reusable reasoning while retaining domain knowledge externally. Extensive experiments show the effectiveness of the proposed method.
Critical strengths
- This paper identifies a key knowledge placement principle for LLM and SLM collaboration, where domain specific knowledge generalizable reasoning capabilities exhibit different transfer characteristics. This observation is interesting.
- LLM-SLM collaboration is a well-motivated and practical research direction.
- Extensive experiments demonstrate the effectiveness of the proposed method.
Critical weaknesses
- It would be beneficial to add a further comprehensive experiment to analyze the necessity of the placement strategy. As shown in Table 12, only the OpenbookQA dataset shows positive statistical support, and the remaining datasets do not show positive support..
- For the CAKR variant, the performance gains are modest. It would be beneficial to clarify the reason behind introducing CAKR module.
- Parameterizing R involves the raw question and answer, so it introduces domain-specific information to some content. It would be better to clarify this.
- It would be beneficial to polish Figures 1 and 3 to improve readability.
Questions for authors
Please refer to the weaknesses section. Main concerns involve the contribution and necessity of each component.
2 comments
Thank you for your response. Your responses have resolved part of my concerns, but I still have concerns regarding the necessity of placement strategy and CAKR module. Therefore, I tend to maintain my current rating.
We thank the reviewer for the positive assessment of our research question, the practical value of LLM–SLM collaboration, and our experimental results. We address your concerns about placement, CAKR, R supervision, and figure readability below.
1. Placement strategy
We added RR and KK controls. The first letter denotes internalized content, the second external content; ₀ denotes no external content. RK−KR compares the proposed and reversed placements. RK−RR holds parameterized R fixed to compare external K with R; RK−KK holds external K fixed to compare internalized R with K. Accuracy is in %, differences in percentage points (pp).
Differences use unrounded results. DID=(RK−R₀)−(KR−K₀) compares incremental gains relative to each branch's baseline. We compute paired differences within each task and average them equally across five tasks:
RK has higher mean accuracy than KR, RR, and KK on all five tasks. The average RK−KR, DID, and RK−RR intervals exceed zero. These unadjusted intervals provide supplementary evidence for an average benefit. The task-level results show variation in that benefit. On QuaRel, RK−KR≈0.66 pp but R₀−K₀≈1.14 pp, giving DID≈−0.48 pp: RK has higher final accuracy, while KR gains more from its corresponding external content. These controls support internalizing R and keeping K external, with task-dependent gains.
2. Motivation for CAKR
R and RK can complement each other because questions differ in their knowledge requirements and retrieval quality varies. First, questions relying mainly on general knowledge may need little additional domain-specific K. Second, retrieval may return related content that fails to support the answer and disrupts an otherwise correct prediction. Trained on held-out or out-of-fold outcomes, CAKR uses question, retrieval, and candidate-prediction signals to select between the executed paths.
On OpenBookQA, Fixed RK achieves 91.40%, while the candidate oracle reaches 93.00%, leaving 1.60 pp of selection headroom. Across 1,500 decisions from three seeds, five corrections and one introduced error yield four additional correct answers: a gain of approximately 0.267 pp, capturing 16.7% of the 1.60 pp oracle headroom. The oracle uses gold labels to measure the upper bound; actual gains reflect both corrections and introduced errors.
For OpenBookQA question 1530 (seed 20260801), “Which of these travels through the air?”, the options are planets, thoughts, automobile, and music. R correctly selects music; RK selects automobile after retrieving content about airplanes and air surrounding planets. CAKR retains R's correct answer. This illustrates how selection helps when general knowledge suffices and retrieval provides insufficient support.
In the main table, CAKR improves mean accuracy by 0.06–1.17 pp across five tasks, including 0.59 pp on ARC-Challenge and 1.17 pp on StrategyQA. It is an optional enhancement. Fixed RK provides a complete separation-and-placement configuration.
3. Domain information in question–answer supervision
Original questions and answers carry task-relevant knowledge and provide shared supervision during R adaptation. KSplit studies how to allocate the additional facts, rules, and reasoning operations supplied by the teacher alongside this supervision.
DAKP separates this content by function. FKP compiles explicit K-node text into the external repository and R operations into interface targets—the allocation described as “K-text-free R supervision.” Students may retain knowledge acquired during pretraining and learn task-relevant information through shared question–answer supervision.
The separation controls hold question–answer supervision fixed and assess factual probes and task accuracy. Across all five tasks, 95% CIs for factual-probe differences lie below zero, while task-accuracy lower bounds exceed the −1 pp non-inferiority margin. The probes measure correct-answer log-odds: these results support reduced confidence in the probed facts while preserving task utility. We clarified the training components and their destinations in §3.3; results appear in §4.4, with settings in Appendix D.
4. Figure readability
Thank you for the suggestion. We increased the text size in Figures 1 and 3 and refined their annotations and layout to improve readability in the revised manuscript.