acceptodds
Withdrawn from ICLR 2027

Diagnose, Allocate, and Bind: Conditional Knowledge Injection into Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Although language models can assimilate target knowledge through continued pre-training, conditional knowledge injection remains fundamentally limited by uniform token exposure and condition-independent recall shortcuts, where models often recall family-consistent values without binding them to their governing conditions. Under a fixed training-token budget, this exposes two coupled challenges: identifying facts requiring additional exposure and ensuring recalled values depend strictly on their queried conditions. This motivates a natural question: Can fact-level internal states be diagnosed to dynamically allocate exposure while explicitly binding values to their applicability conditions? To this end, we propose DAB (Diagnose, Allocate, and Bind), a framework linking fact-level memory diagnosis to adaptive token allocation and condition-dependent supervision. DAB exactly decomposes the gold-value negative log-likelihood into family-mass and within-family concentration terms, isolating under-supported facts from those confused with sibling values via behavior-calibrated routing. After a shared uniform stage, DAB reallocates a bounded token budget toward facts with diagnosed learning needs while strictly preserving minimum exposure for every fact. Furthermore, a late-stage Condition Necessity objective enforces a reference-relative margin between full-condition and condition-masked inputs, explicitly penalizing condition-independent shortcuts. Experiments on domain-specific technical standards (GB-Standard) and SQuAD-derived closed-book QA across Qwen3.5-4B-Base and Llama-3.1-8B-Instruct demonstrate the generality of DAB. On GB, DAB improves closed-book question-level accuracy by up to 8.23 percentage points over the same-renderer uniform baseline. On SQuAD, DAB reaches 64.06% with Qwen and 69.64% with Llama, outperforming the strongest baseline, Knowledge-Instruct (KI), by 6.22 and 19.50 percentage points, respectively. Factorial ablations confirm that diagnosis-guided allocation and conditional supervision provide complementary gains, framing conditional knowledge injection as the joint problem of allocating exposure and learning condition–value bindings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Injection, Continual Pre-training, Adaptive Data Allocation, Synthetic Data Augmentation, Condition-Value Binding, Factual Recall

Related papers

Discussion (0)

Sign in to comment.