- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies whether small language models of 7–8B scale can follow a conditional rule, answer stable facts from memory, answer volatile facts from retrieved context, and defer when a fact is volatile and no context is given. It defines “volatile” as a fact whose value changed between two EvoWiki snapshots. The authors show that prompting and five fine-tuning objectives fail to install this rule. Through probing, patching and steering, they demonstrate that volatility is linearly decodable but unused by the deferral decision. Based on this observation, the paper proposes BRIDGe, an inference-time hook that steers along the context-presence direction only when probes indicate the fact is volatile and no context is present. On three backbones, BRIDGe improves deferral discrimination over the best fine-tuned model, at a modest cost in stable retention and with contextual obedience nearly unchanged.
Critical strengths
- The paper addresses practically relevant problem with a careful evaluation protocol. Knowing when parametric memory may be stale, as distinct from simply following context or abstaining, matters for deployed RAG system.
- The intervention pipeline that separates volatility and presence is simple and interpretable.
- The ablation of random and shuffled-label directions do not raise DD supports the design within paper's own setting.
- The released code aids reproducibility.
Critical weaknesses
- The central mechanistic claims, and the framing of volatility as “the model’s own estimate,” are not sufficiently supported. Volatility is defined by whether a value changed between two arbitrary snapshots. It is unclear how a model could know which facts will change beyond correlates such as relation type or entity category. If so, the volatility probe may detect question type rather than an internal belief about staleness. The retail experiment on Section 6 states that near-ceiling deferral there comes from “a small, learnable tag set,” which reinforces this concern.
- The strength of the diagnosis is not clear enough. The causal conclusion rests largely on activation patching, however but the patching uses only 7 valid volatile/stable pairs on Llama, and the steering comparison of \hat_v versus \hat_c is run only on Llama. Also in Mistral, the ablation has no significant effect on either split.
- Conditional activation steering is well studied, and closely related methods are cited, e.g. CAST (Lee et al., 2025), SHIFT (Li et al., 2026) , but the paper didn't compare theirs with those existing method. The baselines are mostly focused on authors’ own fine-tuning variants, a presence-only ablation of BRIDGe. Without such comparisons, it is hard to judge whether the design choices are well-founded.
- The method contains hyperparameters of which selection criteria is unclear. Section 4.2 reports “the setting with the best trade-off,” but it is unclear whether \tau_v, \tau_c, \alpha and the layer were chosen on a split separate from the reported evaluation set. Also, Table 3, only the Llama-3.1-8B uses a manually set. Clarification is needed on which configuration produced each number.
- Main results (Tables 2–3) come from a single run without confidence intervals.
- In absolute terms, VRR under BRIDGe remains at 44-60%, and SDR rises roughly 3–5x (e.g., 6.1 -> 20.8 on Llama). The model still answers many volatile questions from stale memory and now declines more stable ones.
- BRIDGe requires gold volatility labels to fit its probe. In practical setting, if such labels exist at deployment time, a classifier on the question text or a lookup may be a simpler alternative, and the paper should compare against it. Also, in the retail scenario presented as “the common production scenario,” CA-AVMD alone reaches 99.5% VRR and 94.3% CO, and BRIDGe is not applied. This weakens the case that the method is needed where the paper says it matters most.
[minor]
- Many abbreviations are introduced quickly or before their definitions. Table 1 appears before the metrics are defined in Section 2.1. As all column names and row names are abbreviation, an illustrative figure would help greatly. Potentially, one concrete EvoWiki example per cell of the rule, showing the expected output and which metric scores it.
- These are used in Equation 1 with only a brief introductory definition. More concrete explanation on the snapshots should be stated in the main text.
- Specific versions are omitted in Table 2 (“Llama,” “Mistral,” “Qwen”) and named inconsistently elsewhere (“Mistral3-7B” vs. “Mistral-7B-v0.3”). Please state exact versions consistently, and indicate base or instruct.
- Redundancy. “A deferral denoted by def(\hat{y}) = 1 if \hat{y}is an output that declines to give a value” is followed by identical definition, “def(\hat{y}) = 1 if \hat{y} is a deferral and 0 otherwise.”
Questions for authors
- How can a model know which facts are volatile, given that volatility is defined by changes between two snapshots? Does the volatility probe generalize to held-out relation types, or mainly detect relation or question category? Please report probe accuracy and BRIDGe results by relation type and on the semi-volatile held-out split.
- On what data were \tau_v, \tau_c, \alpha and the hook layer selected, and is it disjoint from the evaluation set behind Tables 2–3?
- How does BRIDGe compare with an adapted conditional steering baseline such as CAST?
- Which exact checkpoints are used (base or instruct)? Can confidence intervals or multi-seed results be provided for all the main tables, including Qwen?
- From my understanding, the rule rewards following context (CO) even when it contradicts memory, while the introduction motivates the work by noting that retrieval can be missing or incomplete. How does BRIDGe behave with irrelevant, incomplete or incorrect context?
- Does the representation–policy gap persist at larger scales? If volatility is linearly decodable, larger models might use it without intervention. If the paper’s scope is restricted to SLMs, this could be stated more explicitly in the title or abstract
5 comments
Thank you for the detailed response and the additional experiments. In particular, I appreciate the cross-validated threshold selection and the paired bootstrap CIs (Table C), which show that the DD gain over CA-AVMD remains statistically significant on all three backbones. However, several concerns remain, and some of the new results make them stronger.
- The held-out relation-type experiment (Table A) addresses part of my question, but the setup is unclear. The no-intervention row (18.8 DD, 53.5 SR) is identical to the full-evaluation numbers in Table 3. This suggests it may not have been re-evaluated on the held-out subset, so it is hard to interpret the comparison with the BRIDGe rows.
- The response acknowledges that the probe picks up relation-type and entity cues. This affects the framing, not only the wording. The paper’s novelty is gating “on what the model knows, not only on what it sees,” a rule “neither a prompt nor a fine-tuning objective can express.” If the gate largely reads question properties, it is closer to an input-property gate such as CAST. Table B supports this reading. A TF-IDF router on the question text reaches cross-validated DD of 64.2/62.9/63.0 vs. 24.7/43.4/26.0 for BRIDGe at comparable SR, and swapping in a TF-IDF gate improves BRIDGe on every backbone. I acknowledge that on unseen relation types (Table A) the probe gate degrades less than TF-IDF (22.2 vs. 13.9 DD after re-tuning). Still, this advantage is modest, and hard to interpret as the “probe gate” variant is not explained relative to the Table 3 configuration, which underperforms no intervention (15.6 vs. 18.8 DD, 39.8 vs. 53.5 SR). The requested per-relation probe accuracy and semi-volatile split (Table 6) results are also missing. in addition, none of the new tables report CO, BRIDGe’s main stated advantage. This matters especially for the router, which replaces outputs with a deferral string.
- Recasting the contribution as a “lightweight, interpretable intervention demonstrating coupling” is narrower than the manuscript’s claims. Also, \tau_v and \tau_c were selected on the evaluation questions and revised during rebuttal, so Table 3 should report the cross-validated numbers, which are lower on all backbones. With some several changes, given it is allowed to upload revised pdf during rebuttal, I would like to see the revised manuscript reflect this. Also, Qwen’s \alpha was also set after \alpha=20 was observed to be inert on the evaluation set, this choice was therefore informed by evaluation results as well.
- The response states that the paper makes no claim of superiority over CAST or SHIFT. However, the paper proposes a new conditional steering method, and to evaluate it as such, readers need to know what it adds over existing methods in the same family. The presence-only gate is an internal ablation of BRIDGe, so it cannot fill this role. This comparison has become more important after the response. Since the volatility gate is acknowledged to rely mainly on relation and entity cues in the input, CAST, which conditions steering on input properties, is the most direct point of comparison. Without it, it is unclear whether the dual-gate design contributes beyond an established approach.
- On the other points; Qwen still has no multi-seed results, and the Table 2 baselines are still single runs. The concern about absolute performance (VRR 44–60%, SDR up 3–5x) is acknowledged but not mitigated. Table C confirms that the SR loss is significant on all backbones. The presentation issues were not addressed, including abbreviations, a concrete example per cell of the rule, an explanation of the snapshots in the main text, and consistent model names.
CAST Comparison
Thank you for the follow-up.
We applied CAST to our CA-AVMD model. It needs two conditions: volatile and no context. We fitted both condition vectors with CAST’s own search over layer and threshold (best F1 on held-out pairs), and used CAST’s refusal vector (our deferral string vs. the model’s own answer) as the steering direction. CAST was fitted on the same questions as BRIDGe and tuned with the same selection rule.
The no-context condition is perfect, but the volatile one fires on 79–85% of stable questions, against 16–19% for BRIDGe. So CAST either has no effect or costs stable answers, and its DD stays below BRIDGe’s (Table D). This holds against BRIDGe’s cross-validated numbers too (24.7 / 43.4 / 26.0, Table B). Pairing CAST’s gate with BRIDGe’s steering doesn’t help either: the model then defers on 70% of stable questions. So the gap comes from the gate, not from how CAST steers.
We agree that BRIDGe belongs to the same family as CAST, and we will frame it that way rather than as “what the model knows”. What it adds is a volatility gate selective enough to be useful; on these models, CAST’s condition is not.
Table D. CAST vs. BRIDGe on CA-AVMD. SR = stable retention, DD = deferral discrimination, CO = context obedience. "Fires on stable" = how often steering triggers on stable questions.
| Model | Method | SR | DD | CO | Fires on stable |
|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | No intervention | 53.3 | 18.5 | 62.7 | – |
| BRIDGe | 50.0 | 27.0 | 62.7 | 16.5 | |
| CAST | 52.6 | 18.7 | 62.7 | 85.1 | |
| CAST gate + BRIDGe steering | 11.2 | 22.4 | 62.7 | 85.1 | |
| Mistral-7B-Instruct-v0.3 | No intervention | 55.6 | 28.6 | 62.0 | – |
| BRIDGe | 50.7 | 45.0 | 62.0 | 18.8 | |
| CAST | 46.7 | 28.8 | 62.0 | 83.8 | |
| CAST gate + BRIDGe steering | 6.5 | 23.2 | 62.0 | 83.8 | |
| Qwen2.5-7B-Instruct | No intervention | 43.9 | 12.2 | 54.9 | – |
| BRIDGe | 40.5 | 27.4 | 54.0 | 17.5 | |
| CAST | 43.7 | 12.5 | 54.9 | 79.4 | |
| CAST gate + BRIDGe steering | 4.1 | 18.6 | 54.7 | 79.4 |
CAST strength chosen with BRIDGe’s selection rule; no strength meets it, so the best DD is shown. All rows come from one re-run with the released code; they differ from Table 3 by up to 1.7 DD (BRIDGe) and 0.4 DD (no intervention).
Thank you for the detailed and useful review. We first address the main concern around volatility and then clarify the other ones, and report the results the reviewer has asked for.
Volatility is a concept (facts that go stale) and whether the fact has changed between 2 timestamps is simply a way to label the data, not a definition of this concept. And as the reviewer intuited, the model is picking up on cues of the relationship type and entities. However the model would still fail to defer. And if labels are available, to distinguish volatile facts, then simpler classifiers (like tf-idf) can be good alternatives - see table B. If these cues are not seen in training, then both gates lose most of their accuracy, but the TF-IDF router ends up worse than no intervention, while BRIDGe with a logistic-probe gate fires rarely and still slightly improves (we ran this and results are in Table C). The retail case contain these cues as well, but the cues are not the bottleneck, but prior knowledge. We do not position BRIDGe as the diagnosis or the only router, but as a lightweight, interpretable intervention demonstrating how volatility-associated and context-presence features can be coupled. Our wording ("the model's own estimate") is confusing, and we will clarify it.
Another important aspect raised in the reviews is the causal claim: our diagnosis is based on steering while patching is used mostly for locating the signal. Presence is the strongest signal on Llama and Qwen (Mistral is comparable as well. However we agree that volatility survives ablation so we cannot measure deferral without it - we’ll add this clarification.
Hyperparameters alpha and layer are fixed while tau_c and tau_v were selected on the evaluation questions, which were held from fine-tuning not evaluation (we will correct it in A4). Qwen uses a larger alpha to match Llama activation magnitude. However, re-picking the parameters with 5-fold cross-validation keeps the gain significant on all three backbones (Table C). Regarding the baselines, both SHIFT and CAST are relevant related approaches. In the paper we did try conditioning on a single gate (see Tables 3 and 13) which motivates our choice of combining both context presence and volatility signals together. However, we did not directly benchmark against CAST or SHIFT and therefore make no claim of superiority over them.
Nonetheless, as the reviewer pointed out, there are limitations and tradeoffs. Even though BRIDGe significantly improves deferral discrimination over the fine-tuned baselines, we did not achieve 100% coverage. This remains a trade-off, and other mechanisms such as RL or stronger base representations can be further studied. We will state the 7–8B scope in the title and abstract. Further experiments could explore a relevance-aware gate: what happens when the context is also irrelevant (not only missing) as the gate right now doesn’t make that distinction.
We hope this shows that the paper’s claim (a behavioural gap that fine-tuning does not close, and a conditional intervention that narrows it) holds with the new data, and we would be grateful if the reviewer would reconsider the soundness scores.
Table A. Llama, on relation types not seen in training (higher is better).
Table B. DD (SR), cross-validated; higher is better. The CA-AVMD row is the reference without intervention.
Table C. BRIDGe vs CA-AVMD, paired bootstrap 95% CIs. ΔDD is a gain in deferral discrimination; a negative ΔSR is a loss in stable retention.