acceptodds
Under review as a conference paper at ICLR 2027

The Reach of Interventions: What Can Be Written into a (Semi-)Frozen Language Model?

10 views

Abstract

Writing new knowledge into a partially frozen language model means changing an answer's reachability through one of three routes: carrying and placing content into weights, carrying and placing content through activations, or reweighting or tilting the base model's existing sampling support. Fine tuning, RL, steering, retrieval, and editing span these routes, and the route determines what each can reach. We develop a predictive theory of what each intervention can write. Using sampling support, verifiability, and a prior-support decomposition, the theory separates items into display, verifiable closure, and underdetermined layers, and derives an inheritance law for reachable answers. It predicts learning limits and per item learnability before training. Across synthetic and public benchmarks, the theory explains channel specific reach and failure modes, matches observed ceilings, and turns post training from trial and error into measurement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge injection, activation steering, reinforcement learning with verifiable rewards, knowledge editing, long-term memory, frozen language models, limits of post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MHpgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops a predictive framework for what each class of post training intervention (steering, retrieval and context injection, SFT, DPO, reward only RL) can write into a partially frozen language model. Each knowledge item is placed into one of three measurable layers via two scalars: the probability that temperature one sampling hits the gold answer, and the hit rate once a premise containing the answer is supplied. The three layers correspond respectively to items the base model knows but does not reliably produce, items it can verify but not sample, and items that lie outside the prior. The authors state an inheritance law: answers reachable after any intervention lie in the union of the budget widened base support and the content the protocol explicitly carries. A theorem proves, for on policy reward only RL under an item symmetric policy class, that items with zero sampling support receive zero gradient along the sample path, that the expected gradient scales with support, and that the asymptotic test accuracy ceiling is an inner product of training and test answer marginals that can be computed before training with no fitted parameters. Four channel propositions extend the picture to steering, content injection and gradient writing, each supported by empirical constants on a controlled synthetic bed. Validation combines the synthetic bed on Qwen3-8B and 14B, cross family replication on four additional models, and a new benchmark assembled from three public sources. A no context identity audit is introduced to distinguish genuine writes from distillation leakage, and a decision table operationalises the framework.

Critical strengths

1.A training free and parameter free upper bound on reward only RL. The paper derives the asymptotic test accuracy ceiling of on policy reward only RL as an inner product of two frequency tables known before training. The predicted number is matched empirically within about one percentage point and reproduced by rejection sampling fine tuning, which supports the authors' claim that the ceiling reflects the learning signal rather than any particular optimiser. In the current discussion of the capability boundary of RLVR, this is a rare pre training computable, falsifiable prediction.

2.A useful public benchmark. The paper assembles ReachBench from three existing public sources and, more importantly, labels each item by measurement (sampling support and verifiability under a fixed sampling budget) rather than by its source dataset. The resulting layer assignments are shown to replicate across five model families, which makes the benchmark immediately usable both for diagnosing where a given base model stands and for prospectively evaluating future post training methods against a common coordinate system. The release of per item records alongside the benchmark adds practical value.

Critical weaknesses

1.The paper is dense and heavy on terminology, which makes it difficult to read. Beyond readability, once the formalism is set aside, several non RL claims closely restate established findings from the steering, knowledge editing and retrieval literatures. The genuinely novel component appears to be the RL ceiling together with the measurement protocol; the authors should state more explicitly which claims are new and which are repositionings of existing results. On a presentational note, none of the figures in the submission are vector graphics, and the resulting raster rendering makes several of the quantitative curves difficult to read at normal zoom.

2.The unification is of notation rather than of proof strength. The theorem is rigorous only for the reward only RL case. The remaining three channel propositions are supported by empirical constants on a single base model under a short LoRA training schedule, without closed form reach bounds comparable to the RL ceiling. The framework is therefore predictive in a strong sense for one of its four quarters and observational for the other three, in tension with the paper's framing as a unified predictive theory.

Questions for authors

1.RLVR is now primarily applied to reasoning models, whose generations are long token sequences rather than short factual spans. How is the proposed training accuracy upper bound to be computed in that regime?

2.Is the proposed framework restricted to knowledge question answering with direct factual answers, or is it intended to apply more broadly, for instance to multi step reasoning, program synthesis, or agentic tasks?

1 comment

We thank the reviewer for the careful reading and for the two questions. We have uploaded a revised version of the paper, and every change is marked in blue.

The paper starts from a definition. The knowledge it measures is knowledge that is written into the model and can be read back, either by sampling an answer or by accepting one that a premise supplies. Knowledge that a model derives by reasoning is outside this object. We exclude it for two reasons: a derived answer can be reached through a chain that does not establish what the model holds, and it becomes impossible to attribute what pretraining and post-training contributed, because the same stored fact can support different derivations in different models. Fixing the object is what makes the comparison possible at all. With the object fixed, the four channels are measured on one coordinate, and the proof differs by what the object admits. For reward-only training the object reduces to the answer marginal, which is known before training, so the ceiling is a closed-form number. For steering and content the effect depends on the model's representation and on how the content is supplied, so we measure the constants and label the statements descriptive. The unification is of the measured object rather than of notation, and the closed-form result is the case where the object is simple enough to admit one.

The revision's experiments test this object rather than the notation, and they test it in the way the reviewer identified as the contribution. The prediction is computed before training from the base model alone: for items a base does not answer, the acceptance measured before training ranks the items a later update moves, so the coordinate is prospective in the same sense as the reward-only ceiling. The six pairs place a released model and its update on one coordinate without modification, and they cover four families: Qwen3-8B-Base and its instruction-tuned model, Qwen3.5-9B-Base and its instruction-tuned model, OLMo-2-7B and OLMo-2-7B-Instruct, Llama-3.1-8B and Llama-3.1-8B-Instruct, GLM-5.2 and GLM-5.3, with a size-matched Qwen3-4B and Qwen3.5-4B control. Two of the pairs are full-parameter updates, the GLM pair is a same-base update with no new pretraining data, and the Qwen3.5 pair is a newer generation of the same family. GLM-5.3 is a near-frontier model, measured through the vendor API because the weights are unavailable, which is the case the black-box probes are designed for; a probe that reads internal states could not measure it. The revision measures nine models on the same 99 items, with 64 samples per item for support and 8 for acceptance, and re-measures 430 items at a larger budget for the reliability check. The validity control establishes what the label measures, since a wrong premise is accepted almost as readily as a true one and a matched two-alternative test is answered at chance. The null test places the one run above the bound within chance, so the bound is measured against a null. The benchmark ships the per-item labels, which is what lets another group place a new model on the same coordinate. The revision also reduces the terminology, states in Section 4 which findings are prior work, and replaces the main figures with vector versions.

The two questions follow from the same definition. A reasoning mode is not enabled by design: it would turn the object from a retrievable fact into a derivable conclusion, and the layer would record derivability rather than retrievability. For a model with long generations, the answer class is the final answer, so the formula applies, but the cost of estimating support rises with the length of the generation. On scope, the framework assumes an item with a checkable answer and a premise that can supply it; multi-step reasoning and agentic tasks introduce intermediate steps that are not answer classes and can reach a right answer through a wrong chain, so we state the scope as factual question answering with checkable answers.

Reviewer_JiEzOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an integrated analytical framework for selecting an intervention method for each knowledge item. It categorizes knowledge items into display, verifiable closure, and underdetermined layers, and argues that each method’s reach is restricted by the base model’s probability of generating the correct answer and the information provided by the intervention. These claims are evaluated using ReachBench.

Critical strengths
  • This paper compares different intervention methods from a common perspective. By distinguishing between changing the model’s preferences among existing outputs and providing new information, the authors clearly explain the differences between steering, RLVR, retrieval, and parameter updates.

  • The experimental setting is well controlled: the authors investigate model outputs with the memory cleared and try to distinguish the effects of prior knowledge, information leakage, and context injection.

  • The controlled experiments are extended to ReachBench, which is built from EntityQuestions, PopQA, and FreshQA. This suggests a practical direction: measuring per-item learnability before training.

Critical weaknesses
  • The limitations observed in the experiments are overgeneralized to post-training methods as a whole. The authors state that the GRPO experiments use short training schedules with LoRA fine-tuning. It is difficult to extend observations from a specific setting, such as performance saturation and performance drops after removing hints, to general limitations.

  • There is insufficient evidence to claim that pre-diagnosis leads to better method selection. The effectiveness of diagnosis-based method selection is mainly evaluated in the closure setting. Therefore, comparisons with a retrieval-only baseline or a fixed intervention method are needed, taking computational costs, including the cost of diagnosis, into account.

  • The reliability of item categorization is not sufficiently validated. Diagnosis uses 64 generations per item by default, so an item whose correct answer is rarely generated could be treated as unsupported. Although the authors acknowledge this measurement limitation, additional experiments with more diagnosis trials are needed.

Questions for authors
  • Can an item’s category change during training? For example, could a closure or underdetermined item change categories as the model trains on other items? Could the authors measure the stability of the initial categorization at several training checkpoints?
3 comments

We thank the reviewer for the careful reading and for the three points. We have uploaded a revised version of the paper, and every change is marked in blue.

The reviewer notes that the limits are overgeneralized from short LoRA GRPO schedules. We agree, and the revision narrows the claim. The controlled runs are one setting, and the paper now says so: the channel constants come from a single base and do not transfer without remeasurement, while the layer assignment is what replicates. To test whether the pattern survives outside it, we compare six pairs of a base model and its post-trained version on the same questions, from four families: Qwen3-8B and Qwen3.5-9B with their instruction-tuned models, OLMo-2-7B and its Instruct version, Llama-3.1-8B and its Instruct version, and GLM-5.2 with GLM-5.3, plus a size-matched Qwen3-4B and Qwen3.5-4B control. Two are full-parameter updates, and the GLM pair is a same-base update that the vendor states uses no new pretraining data. The movement goes in the same direction in all of them, and its size grows with the number of items the base cannot answer, so the claim is about the tested intervention set rather than post-training in general.

The reviewer also notes that the diagnosis is evaluated mainly in the closure setting, and that a retrieval-only or fixed-intervention baseline is needed with the cost of the diagnosis included. We added the cost-matched comparison. The diagnosis costs 72 generations per item, while supplying the answer costs one and reaches the same accuracy on the closure testbed, and the diagnosis selects that rule. We therefore do not claim that the diagnosis beats a fixed intervention there. What it provides is the layer assignment that tells a practitioner which of several expensive routes can work, and the comparison shows where that is worth the cost and where it is not.

On reliability, the reviewer notes that 64 generations may classify a rare item as unsupported. We report the bound that the observation supports: an item with no hit in k samples has support at most about 3/k, which is 4.7% at the standard budget. We then re-measured 430 items at 1024 samples. The labels agree on 84% of them, and every disagreement moves toward display, so a misclassified item is one that is in fact reachable. The paper reports the layer together with its budget.

The reviewer asks whether an item's category can change during training, and whether the initial categorization is stable at later checkpoints. We measured it. A three-epoch training run was interrupted after each epoch, and the 99 items were re-measured at each checkpoint. Consecutive checkpoints agree on 85 to 87 of the 99 items, and the changes go in both directions, which is what the sampling resolution at the budget predicts; the display share does not drift systematically. Categories do change across a complete post-training update, and the pair analysis measures that: in the pairs with a raw base, tens of items move, always toward reachability, and the items that move are those the base could already accept. Training on other items left most items in their layer, with the changes going in both directions.

Thank you for the detailed response and the additional experiments. I appreciate the efforts to address my concerns. However, my concern about the practical value of diagnosis remains.

The closure-testbed comparison shows that the always-premise baseline achieves the same accuracy without the diagnosis overhead. I appreciate the clarification that diagnosis is not claimed to outperform this baseline in that setting. However, it remains unclear when diagnosis-based method selection provides a practical benefit over a fixed intervention after accounting for diagnosis costs.

For this reason, I will maintain my original score.

We thank the reviewer for the precision of this question, and we agree with it. We also apologize that our earlier response left the cost scale implicit, which made the comparison read as a loss for the diagnosis.

We do not claim a benefit for inference-time routing, because supplying the answer costs one generation and reaches the same accuracy on the closure testbed, so a fixed supply policy cannot be beaten there; our own comparison shows this, and the revision says so.

The benefit we claim is at a different cost scale: a post-training run costs hours of compute and a curated dataset, while the probe costs tens of generations per item, and the question it answers is whether that run will move the items it is aimed at. On our 400-item bed, items at zero sampled support produced zero training signal, and reward-only training recovered only closure items, so a run aimed at underdetermined failures yields nothing and the data plan has to carry the content instead. The probe identifies this before the budget is committed.

Formally, the gate pays if d < F times C_post, where d is the diagnosis cost per item, F is the fraction of a planned run's items that the gate would exclude and that would otherwise yield no gain, and C_post is the all-in per-item cost of the run. We report d and the zero-signal evidence, and we do not claim that the inequality holds on any particular mix; it is a condition to audit, and an equal-budget comparison of gated and ungated runs would settle it. The revised text states this condition.

We are grateful for the reviewer's insistence on cost accounting, which is the right question and which changed how we state the claim.

Reviewer_czbLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a framework for characterizing what different interventions can make a partially frozen language model answer. It distinguishes display, verifiable-closure, and underdetermined items using sampling support, premise-conditioned success, and prior-preference measurements. The proposed inheritance law connects intervention reach to existing model support and explicitly supplied content. Synthetic experiments and ReachBench evaluate these predictions across steering, content injection, fine-tuning, reinforcement learning, and retrieval.

Critical strengths
  1. Useful unifying perspective. The framework connects intervention methods that are often studied separately and distinguishes eliciting existing knowledge from supplying new content. This provides an interesting perspective on choosing an intervention based on the model’s initial knowledge state.

  2. Thoughtful diagnostic methodology. The no-context audit, prior-support decomposition, and targeted controls help distinguish genuine content transfer from leakage or generic preference changes. These tools have practical value beyond the particular theoretical claims.

  3. Substantial controlled evidence. The evaluation combines synthetic testbeds with natural-data experiments, multiple model families, and repeated training runs. Controls such as wrong-answer steering and premise-token ablation provide informative evidence about the tested mechanisms.

Critical weaknesses
  1. The empirical results do not establish the full generality of the proposed limits. The complete channel-by-layer comparison primarily uses synthetic tasks on Qwen3-8B, while other models and natural-data experiments validate selected relationships. The acknowledged use of LoRA and short GRPO schedules further limits extrapolation. In particular, a gap left unclosed by the tested interventions does not establish that closing it necessarily requires changing pretraining or permanently supplying content. Broader validation or more carefully scoped conclusions would strengthen the paper.

  2. Some theoretical connections require additional justification. Theorem 1(iii) establishes marginal matching under likelihood optimization, but this does not itself establish the reported marginal-overlap ceiling for reward-only GRPO: maximizing expected binary reward need not reproduce the training-answer distribution. Similarly, Appendix C.5’s probability-proportional logit-drift argument assumes independent logit updates, whereas parameter updates couple logits through shared gradients. The additional assumptions needed for these conclusions should be stated, and empirical regularities should be clearly distinguished from proved limits.

  3. Finite-budget support estimates complicate practical predictions. With the usual 64-sample budget, zero observed hits do not establish zero underlying support. The authors acknowledge this distinction, and Appendix B even reports improvement on an observed-zero-support PopQA subset. Larger-budget sensitivity checks and uncertainty-aware layer assignments would help establish how reliably the framework can predict intervention success before training.

Questions for authors

See weakness

1 comment

We thank the reviewer for the careful reading and for the three points. We have uploaded a revised version of the paper, and every change is marked in blue.

The reviewer notes that the full channel-by-layer comparison uses one model on synthetic tasks, so the limits may be a property of that setting, and that our own runs use LoRA and short GRPO schedules. The revision adds four experiments. The pair comparison tests whether the layer structure survives outside that setting: we compare six pairs of a base model and its post-trained version, all released by other teams, from Qwen3-8B and Qwen3.5-9B to a size-matched Qwen3-4B and Qwen3.5-4B control, OLMo-2-7B, Llama-3.1-8B, and GLM-5.2 with GLM-5.3. The pairs differ in team, size, and recipe, so a pattern that holds across them is not tied to one model or one method; the movement goes in the same direction in every pair, and its size grows with the number of items the base cannot answer. The prediction test asks whether the measurement is useful before training: for items a base cannot answer, the acceptance measured before training tells which items a later update moves, and the prediction uses the base model alone. The validity control asks what the premise probe measures, since a prediction is only as good as its input: a wrong answer in the premise is accepted almost as readily as the true one, and a matched two-alternative test is answered at chance, so the closure category records acceptance. The null test asks whether the one run above our bound is real: nine seeds against a random-route null place it within chance, so the ceiling is reported as holding within noise.

The reviewer also notes that Theorem 1(iii) is a likelihood result and cannot establish the overlap ceiling for GRPO, and that Appendix C.5 assumes independent logits. The first point is correct, and it matters because the paper used the overlap as the reward-only ceiling. We now scope part (iii) to updates that maximize expected log-likelihood, and the text states that the overlap reported for GRPO is an empirical reference. What the theory does establish for reward-only training is parts (i) and (ii): zero-support indistinguishability holds for any deterministic update, and the rate law is proportional to the hit probability. The measured saturation is consistent with those two parts, which is the sense in which the paper now claims a reward-only limit. On the second point, the first-order step in C.5 moves each non-target logit by a probability-proportional term and so treats the logits as independent; shared parameters couple them, so we state the assumption and use the drift only for the direction and ordering of the effect.

The reviewer further notes that a zero at 64 samples does not establish zero support, so the layer labels may be unstable. We address this in two steps. First, we report the bound that the observation supports: an item with no hit in k samples has true support at most about 3/k at 95%, which is 4.7% at the standard budget. Second, we re-measured 430 items at 1024 samples to see how much the label changes. The labels agree on 84% of them, and every disagreement moves toward display, so the descriptive layer does move with the budget and the paper reports it with its budget. What does not move is the quantity the prediction uses: the rank correlation between support and training signal changes by less than 0.01 across the three budgets. The practical claim therefore survives the objection, and the descriptive claim is stated more carefully.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.