- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies explicit forgetting in reasoning-distilled language models, focusing on whether a forgetting instruction remains effective after intervening conversation turns. The authors introduce ICF-DSM, evaluate four base/distilled model pairs, and propose Forgotten-State Contrastive Routing (FSCR). FSCR generates an answer from each model and routes to the base-model answer when three conditions on lexical leakage scores are met.
The paper reports positive held-out gains across all four pairs, including experiments using dialogue-derived forgotten states. The problem is relevant, and the results suggest that complementary model outputs can be used to improve forgetting compliance. My main reservations concern the interpretation of the delayed-forgetting results and whether the proposed gate adds value over a simpler, calibrated selector.
Critical strengths
1、The consistent held-out gains suggest that the base and distilled models make complementary errors that can be usefully exploited. Table 2(c) reports improvements of 1.00–5.76 percentage points across four model pairs, with dialogue-clustered confidence intervals excluding zero. Using the same candidate answers across routing rules also makes the comparison cleaner. Together with the positive dialogue-derived-state results, this supports FSCR as a practical, training-free intervention that does not rely solely on manually annotated forgetting targets.
- The distinction between immediate and delayed forgetting is worth studying. ICF-DSM tests whether a forgetting instruction remains effective after intervening dialogue, while also evaluating retained-fact use and adversarial reconstruction. This is more informative than checking only whether the model suppresses a fact immediately after the request. The matched-query control and blind human audit are useful additions, even though their scope limits the conclusions that can be drawn from them.
Critical weaknesses
-
I am not convinced that FSCR’s three-condition gate offers a better trade-off than a simpler, properly tuned selector. At the reported settings in Table 4, margin-only achieves larger net gains in every model pair, while FSCR routes fewer answers and introduces fewer regressions. Appendix D.4 makes the trade-off concrete: the extra checks avoid 8 regressions but also give up 88 fixes. This may be reasonable when regressions are particularly costly, but a higher margin-only threshold might achieve a similar trade-off. The current results do not establish whether FSCR’s additional conditions help beyond implementing a more conservative operating point. This matters for judging the method’s contribution.
-
The evidence for a delay-related control deficit is narrower than the original comparison suggests. The immediate and delayed probes differ in both query wording and intervening dialogue. For Qwen-7B, base-model SMS rises from 5.43% to 39.64%, while distilled-model SMS falls from 34.00% to 22.43% (Table A4). The relative contrast therefore cannot be attributed solely to the distilled model losing the instruction over time. The matched-query control is a useful step, but it is limited to Qwen-7B; the zero-versus-two-exchange interaction is estimated at zero with a wide confidence interval (Table A6). This does not rule out a delay effect, but leaves open how much of the original result reflects intervening turns rather than differences between the probes. The other model pairs lack the same controlled comparison.
Questions for authors
- Could you tune the margin-only threshold on the development set to match FSCR’s coverage, then compare both methods on the held-out set? This would help determine whether the extra conditions improve the trade-off when both methods operate at a similar coverage level.
- How do you interpret the original two-exchange contrast in light of the zero estimated interaction in Table A6? Do matched-query results show a similar pattern for any of the other model pairs?
3 comments
Response to Reviewer YStk (2/2)
W2 & Q2. Matched-query controls and the two-exchange contrast
To separate query wording from intervening dialogue, we expanded the matched-query analyses in Appendices B.3 and B.4. The zero interaction in Table A6 is a point estimate for one contrast in 100 dialogues; the expanded controls examine the trajectory across multiple delays while holding the final question fixed.
In the original matched-query control, Qwen-7B's distilled model scored 36%, 51%, 34% and 23% SMS after 0, 1, 2 and 4 intervening exchanges, respectively; its base model scored 11%, 8%, 9% and 7%. With the wording fixed, the base model no longer shows the rise from 5.43% to 39.64% seen in Table A4. From 0 to 2 exchanges, both models lose exactly 2%, yielding the zero interaction. This two-point contrast does not use the distilled model's rise to 51% at one exchange or its decline to 23% at four exchanges. Its 95% CI of [−13, +13]% also accommodates effects of a few percentage points. Using all four delays, the distilled model's SMS slope was 3.86% per exchange more negative than the base model's (p = 0.0094).
We then extended the control to 300 source dialogues with 0, 2, 4, 8 or 16 intervening exchanges, again keeping the final question fixed within each dialogue. Qwen-7B's distilled model lost 1.37% more SMS per exchange than its base model (95% CI [1.00, 1.73]). Under this fitted linear trend, a two-exchange contrast would be about 2.7%, within the original interval; this is an implication of the fitted slope rather than a separately measured two-exchange effect.
The expanded matched-query controls also identify faster declines in other model pairs. Across all four pairs, four of six prespecified slope comparisons survived Holm correction (adjusted p = 0.0003 each): Qwen-7B and Llama-8B with short exchanges, and Qwen-14B and Llama-8B with long exchanges. All six point estimates indicate a faster decline for the distilled model. The Qwen-1.5B estimate (−0.07% per exchange) and Qwen-14B estimate with short exchanges (−0.37%) were not significant.
We have added the interpretation of the two-exchange contrast and the expanded matched-query results in Appendices B.3 and B.4 of the revised manuscript. To make comparisons across model pairs easier, we revised Figure 2 to show the distilled–base gap on immediate and delayed probes (a) and the size and direction of each behavioral score difference (b).
Dear Reviewer,
We appreciate the time and effort you have devoted to reviewing our work. We would like to briefly follow up on our responses to your comments and ask whether you have any further feedback.
We hope our rebuttal has helped clarify the points you raised. If any aspects remain unclear or require additional discussion, we would welcome the opportunity to elaborate.
Thank you again for your thoughtful feedback and your time.
Best regards, The Authors
Response to Reviewer YStk (1/2)
Thank you for your careful review and constructive suggestions. We have added routing comparisons calibrated on development data and extended the matched-query controls to address your questions.
W1 & Q1. Comparisons with calibrated margin-only routing
Following your suggestion, we calibrated margin-only routing to match FSCR's coverage on the development set and compared the frozen rules on held-out data. We also evaluated the more conservative operating point you proposed by matching FSCR's development regression count. Both thresholds were selected on ICF-DSM development dialogues and frozen before test evaluation: t = 0.32 for development coverage and t = 0.447 for the development regression count.
The table reports FSCR minus margin-only in net gain (pp), pooled over the four DeepSeek pairs, with dialogue-clustered 95% CIs and shared three-vote labels. The comparison with coverage matched on development data shows a small observed difference on ICF-DSM and a positive difference on MQuAKE. With the threshold matched to the development regression count, FSCR achieves larger net gains on both test sets.
On ICF-DSM, FSCR routed 11.5% of answers and margin-only at t = 0.32 routed 11.3%. FSCR made 217 fixes and 17 regressions, compared with 226 fixes and 15 regressions for margin-only. At t = 0.447, margin-only made 166 fixes and 4 regressions: relative to FSCR, this more conservative rule gave up 51 fixes to avoid 13 regressions. The +0.73% difference therefore describes the held-out performance of rules matched on the development regression count, with their realized test regression counts reported explicitly.
On MQuAKE, the same frozen FSCR rule made 72 fixes and 1 regression with 144 switches. Margin-only at the threshold matched to development coverage switched 336 answers, producing 66 fixes and 4 regressions. We also swept every shared margin threshold on the test set as a descriptive check of the attainable trade-off. At the same observed regression cap of one, FSCR fixed 72 errors, whereas the best margin-only threshold fixed 47. Reaching 72 fixes with margin-only required at least six regressions. This comparison directly tests whether increasing a single margin threshold can reproduce FSCR's operating point on these transfer samples.
The small observed difference on ICF-DSM at matched coverage can be traced to the individual routing decisions. When the distilled answer leaks heavily and the base answer does not, their leakage gap is also large, so the two rules mostly select the same answers. Of FSCR's 600 switches, 501 are shared with margin-only and contribute the same 202 fixes and 12 regressions to both rules. FSCR's 99 additional switches have gaps between 0.15 and 0.32 and yield 15 fixes and 5 regressions. The 87 switches made only by margin-only have gaps of at least 0.32 but fail one absolute check; they yield 24 fixes and 3 regressions. These differing decisions account for the 11 net corrections out of 5,208 evaluations (0.21%), whose interval includes zero.
Together, these comparisons characterize both the shared decisions and the contribution of the absolute checks. The higher threshold matched to development regressions gives 0.73% fewer net corrections on ICF-DSM and 0.65% fewer on MQuAKE, while the MQuAKE threshold sweep shows that the additional conditions select an operating point that a single margin threshold does not reproduce under the same observed regression cap.
We have added these comparisons calibrated on development data, routing-decision analyses, and transfer results in Section 5.5, Table 4(c), and Appendices D.4 and F.3 of the revised manuscript.