- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Intervention-to-Skill (I2S), a framework that evolves small-large language model collaboration in interactive tasks. Rather than merely routing requests dynamically, I2S lets the small language model (SLM) attempt tasks first; when confidence is low, a large language model (LLM) intervenes. Crucially, I2S extracts the successful LLM interventions and grounds them with environment feedback to formulate reusable, structured procedural "skills" (JSON cards). These skills are then added to a shared external state, enabling the SLM to handle similar difficulties independently in future tasks and cumulatively reducing reliance on the expensive LLM over a deployment stream.
Critical strengths
- Novel Shift from Transient Routing to Cumulative Adaptation: While most existing literature focuses on stateless query-level or step-level routing, this work pioneers the concept of converting transient LLM interventions into persistent, cross-task skills. This is a highly practical and insightful approach for continuous agent deployment.
- Rigorous Anti-Hallucination Skill Induction: The framework does not rely on free-form text summaries for skills. Instead, it utilizes strict JSON schema templating () coupled with a physical Evidence-grounded admission mechanism (
CheckEvidence). This effectively prevents the LLM from hallucinating generic, ungrounded task recipes.
Critical weaknesses
- Presentation Ambiguity Regarding Confidence Scores: In Figure 2, the pipeline illustrates a continuous confidence score (e.g., ), yet the methodology implies the LLM outputs discrete labels (Prompt D.2 outputs
CONFIDENTorUNCERTAIN). While the mechanism (applying softmax over target token logits) is explained deep in Appendix A.1, its omission from the main text (Section 4.2) causes significant confusion regarding how the continuous threshold () is actually computed during inference. - Cost Inversion in Complex Environments: In Table 1 and Figure C.1, the complete-stream weighted token usage () for I2S with DeepSeek-V4-Flash on ScienceWorld (k) is astonishingly higher than simply using the Large-Only model (k). The "propose-first" design incurs a massive cost penalty when the SLM is highly incompetent for a complex task, as the system must pay for the SLM's generation, the confidence scoring, and the subsequent LLM intervention at nearly every step. The paper lacks a critical analysis of this cost inversion phenomenon.
- Unexplored Lower Bound of SLM Capabilities: The framework relies on the SLM (Qwen3-8B) to accurately parse long contexts containing complex JSON skill cards (
skill_refs), interpretanalysisrationales, and adhere to strict action grammars. An 8B model is already highly capable. It remains entirely unclear whether this I2S pipeline would completely collapse if paired with a genuinely "small" model (e.g., 1.5B or 3B parameters) due to catastrophic forgetting or instruction-following failures.
Questions for authors
- Clarification on Confidence Calculation in Main Text: Could you briefly add a sentence in Section 4.2 explaining that the continuous confidence score is derived from the softmax of the log-probabilities of the
CONFIDENTandUNCERTAINtokens? This would immediately resolve the apparent contradiction between Figure 2 and Prompt D.2. - Analysis of Cost Inversion in ScienceWorld: For the DeepSeek-V4-Flash + ScienceWorld setup, I2S consumes more weighted tokens (k) than the Large-Only baseline (k). Does this suggest that when the capability gap between the SLM and the task complexity is too large, the "SLM proposes first + confidence scoring" mechanism becomes an active liability? Would a "fail-fast" or dynamic thresholding mechanism be necessary to prevent this triple-billing of tokens in highly complex environments?
- Capability Requirements for the Small Model: The framework expects the SLM to parse and utilize injected JSON procedural skills effectively. Qwen3-8B is relatively powerful. Have you experimented with truly small models (e.g., 1B-3B parameters like Qwen2.5-1.5B or LLaMA-3.2-3B)? Is there a fundamental capability threshold below which the SLM fails to utilize the induced skills, thereby causing the I2S loop to break down? Adding a smaller model ablation would significantly strengthen the paper's claims about "Small-Large" collaboration.