- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether preservation differences between slow- and fast-learning-rate factual rewriting programs persist after matching the mean optimization-prompt NLL of their terminal states. The two suffix programs start from the same saved model, optimizer, and sampling state. The authors compare the checkpoints at which each program first crosses an editing-loss threshold with fractional states selected within the final update. In a controlled Transformer, large slow–fast preservation differences remain after matching. On Qwen3-8B-Base and CounterFact, coarse-step differences largely disappear, while fine-step experiments retain much smaller differences. The held-out analysis further shows that lower old-fact cost for the slow suffix can coexist with higher test NLL and equal test accuracy.
Critical strengths
-
The paper identifies an important evaluation issue in factual rewriting: a shared first-crossing threshold can select states with substantially different achieved editing losses. Comparing first-crossing states with directly evaluated mean-loss-matched states provides a useful diagnostic for separating endpoint overshoot from preservation differences.
-
The experiments are detailed and extensive with respect to this evaluation question. The authors examine controlled and natural-fact settings, coarse- and fine-step programs, matching failures, per-edit differences, and held-out evaluation; in particular, the attenuation of the coarse-step Qwen differences and the held-out test-NLL trade-off make the conclusions more transparent and informative.
Critical weaknesses
-
The controlled study defines factual load using 200 or 3,000 base-training associations, but does not report full-set pre-edit accuracy or NLL, including performance on the 20 edited subjects and the additional 2,800 high-load associations. These additional facts are also excluded from the main preservation pools. Thus, the analysis does not verify whether nominal factual load reflects acquired knowledge or whether the reported preservation results characterize forgetting across the high-load setting.
-
The matched-state experiments focus on mean optimization NLL near a single target of 0.01, without establishing whether the conclusions hold at other editing-strength levels. Moreover, matching the mean does not ensure equivalent per-edit or held-out effectiveness, as the authors also acknowledge. only narrows the tolerance around the same target and therefore does not address either limitation.
Questions for authors
-
Can the authors report pre-edit accuracy and NLL over all 200 or 3,000 base associations, separately for the shared facts, the 20 edited subjects, and the additional 2,800 high-load associations? How many of these associations are reliably mastered before editing?
-
What motivates the target NLL of 0.01 and the matching bands centered at 0.00995? Do the slow–fast preservation contrasts persist when terminal states are matched at other editing-strength levels?
-
How do the conclusions change when effectiveness is compared at the per-edit level or on held-out expressions, rather than only through batch-mean optimization NLL?
-
Do the main attenuation results persist for another model, a larger edit batch, or a less restricted parameter-update setting?