- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed PLACEBU to solve the utility preservation in a remain data free setting in LLM unlearning. This is a polished an empirically strong paper with an intuitive matched-control idea. However, the central theoretical assumptions are not validated, the implementation is only indirectly connected to the theory, and the "retain-free" claim is weakened by retain-side hyperarameter selection.
Critical strengths
-
The paper introduces an interesting request-specific matched-control construction.
-
Its empirical analysis is further strengthened by random-seed experiments and adversarial evaluations, and the overall presentation is clear and well organized.
Critical weaknesses
-
The central claim that the forget set reveals what should remain unchanged is not well substantiated. PLACEBU defines the target contrast as a mixture of forget-specific and preservation-relevant effects and assumes that a synthetically constructed non-target relation recovers the latter. However, these components are never independently observed, and the paper neither measures the proposed control mismatch nor demonstrates that semantically different target and placebo relations induce comparable nuisance variation.
-
The theoretical results are largely assumption-driven. Proposition 3 follows directly after assuming . More importantly, Lemma 1 assumes that the assistant residuals approximate the abstract target and placebo contrasts, which is precisely the nontrivial connection that needs justification. The current theory does not establish that the training objectives recover these contrasts.
-
The end-to-end retain-free status of PLACEBU is unclear. Appendix C states that “retain-side scores may be used” to select the inference coefficients and mask parameters, while Appendix E.6 does not specify whether the reported configurations were selected using such scores. If retain-side validation performance informed this selection, the complete pipeline is not strictly retain-free, even though no retain examples are used to optimize the assistants.
-
On MUSE, PLACEBU produces substantially lower memorization than the retrained reference while also losing utility. Likewise, TOFU’s aggregate metric can reward stronger-than-retrain suppression. Distance to the retrained model should be reported across all settings, rather than treating maximal forget scores as uniformly desirable.
Questions for authors
- Most headline improvements are small, while uncertainty across baseline runs is not reported. PLACEBU also requires the deployed model plus multiple assistant forward passes at every decoding step.
2 comments
Response to Weakness 4 & Question 1
W4. Stronger-than-retrain suppression
We agree that stronger suppression than Retrain should not automatically be interpreted as better unlearning. Our claim is that PLACEBU provides a controllable forgetting–utility trade-off without retain data, rather than that maximal forgetting is always desirable.
For example, on 8B Forget10, Retrain obtains MEM./UTIL. , while the default PLACEBU setting obtains , corresponding to absolute differences of . However, the existing coefficient sweep shows that PLACEBU can move toward the Retrain operating point: with and , it obtains MEM./UTIL. , reducing the corresponding differences to .
In the updated manuscript, we have added the distance to Retrain alongside the direction-specific metrics for all settings and revise the text to avoid treating maximal suppression as uniformly preferable. The stronger suppression observed on MUSE, particularly for Books KnowMem, are described as a calibration limitation rather than an unconditional improvement in our revised paper.
Q1. Magnitude, uncertainty, and inference cost
We respectfully clarify that the improvements are not uniformly small. On 8B TOFU, PLACEBU achieves the highest AGG. at all three forget ratios, exceeding the strongest baseline by , , and on Forget1/5/10. The Forget10 margin is modest, but the advantage is consistent across model scales and forget ratios. The gains are larger in other settings: on Qwen Forget5, PLACEBU improves AGG. from to ; on MUSE-Books, it reduces KnowMem from to relative to the utility-matched SimNPO baseline while maintaining comparable utility ( vs. ).
PLACEBU is stable across three independent assistant-training seeds. On 8B Forget10, the standard deviations of AGG., MEM., and UTIL. are , , and , respectively. Since some baselines are available only through reported results or single checkpoints, we cannot reliably estimate their training-seed variance without fully reproducing them. We therefore avoid interpreting small margins, such as the Forget10 gain, as statistically significant and distinguish them from the larger improvements observed on MUSE.
PLACEBU follows the auxiliary-model decoding paradigm established by ULD, which also constructs an unlearned distribution through logit differences between the deployed model and a lightweight assistant. ULD explicitly acknowledges the resulting inference latency and notes that it can be mitigated by executing the deployed model and assistant in parallel.
PLACEBU extends this paradigm with separate deletion and compensation residuals. Each decoding step requires one deployed-model pass and four lightweight residual passes. However, each PLACEBU assistant contains only four Transformer layers, compared with the eight-layer assistant used by ULD. The four residual passes are independent, support KV caching, and can be batched or executed concurrently with the deployed model. Thus, four residual passes do not imply a four-fold increase in wall-clock latency, although PLACEBU does incur additional total computation and memory.
The additional computation implements the matched-control compensation that improves the forgetting–utility balance: on 8B Forget10, PLACEBU obtains AGG. , compared with for ULD, without using original retain examples to train the assistants. We have revised the paper accordingly to report end-to-end decoding latency, throughput, and peak memory under both sequential and parallel execution, making this quality–efficiency trade-off explicit.
Response to Weaknesses 1 - 3
W1. validation of the matched-control assumption
We agree that the latent nuisance component and the exact control mismatch ϵctrl,i cannot be observed directly. We therefore test their observable implication: whether placebo compensation reduces deletion-induced changes on matched non-target information.
To separate compensation from overall forgetting strength, we compare the closest available MEM. operating points while fixing β−=2.1. Deletion-only uses β+=0, whereas the diagnostic compensated setting uses β+=0.25, which gives the closest MEM. among the tested compensation strengths. This setting is used only for this analysis.
With only a 0.011 MEM. difference, compensation reduces non-target KL by 21.2% and residual RMS by 12.6%. KL decreases on all 800 controls, and RMS decreases on 799/800 controls.
The existing control ablation further shows that the matched structure matters:
Together, these results show that matched placebo compensation cancels a substantial portion of deletion-induced non-target change, beyond what is obtained from random or semantic-nearest controls.
W2. Connection between the theory and assistant training
Logit-difference correction is an established unlearning mechanism: ULD derives an unlearned distribution by subtracting an assistant model’s logits, while Offset Unlearning learns an unlearning offset by contrasting the logits of smaller models. PLACEBU builds on this established mechanism; its new contribution is to obtain two separately supervised residuals corresponding to the target and matched-placebo interventions.
For each branch, the trained assistant and its frozen reference share the same initialization. The deletion branch is trained on C11 and constrained on C01, while the compensation branch is trained on C10 and constrained on C00. Therefore, the formula below rσ=ℓaσ−ℓa0σ isolates the training-induced logit change associated with that intervention pair. PLACEBU subtracts the target-derived residual and restores the matched-placebo residual.
We agree that Lemma 1 is a conditional error decomposition rather than a guarantee of exact recovery: ei− and ei+ explicitly represent the remaining approximation errors. We have clarified this point in the updated manuscript.
Empirically, the learned compensation residual reduces non-target KL by 21.2% and residual RMS by 12.6% at a closely matched forgetting level, supporting the intended correspondence between the supervised branches and their operational roles.
W3. End-to-end retain-free status
Thank you for noting that “retain-side scores may be used” in Appendix C makes the protocol ambiguous. It refers to an optional operating-point procedure, not the protocol used for the reported PLACEBU results.
PLACEBU does not require a retain set. Retain examples are used only after configuration selection to report UTIL. They are not used for quartet construction, assistant training, checkpoint selection, or inference-parameter selection.
Calibration of β− and β+ is also retain-free. We first select the smallest β− satisfying the prescribed forget-side MEM. requirement. We then initialize β+ by matching the deletion and compensation residual scales:
β+init=β−RMS(r+)RMS(r−).
This normalization accounts for differences in the magnitudes of r− and r+. Around this initialization, we select β+ and the support-mask parameters by minimizing residual changes on the constructed C10/C00 controls while maintaining the MEM. requirement. This uses only C11 forget queries and forget-derived C01, C10, and C00 controls. The resulting parameters are fixed before retain-side evaluation and shared across benchmark requests.
Table 22 shows an operating-point trade-off, not dependence on retain data. Increasing β− strengthens forgetting, while increasing β+ restores non-target behavior but may moderately weaken forgetting. All settings use the same frozen assistants, with no retain examples used in training or calibration.
We have removed the ambiguous sentence from Appendix C, added the calibration procedure, and distinguished parameter selection from post-hoc utility evaluation. Under the reported protocol, PLACEBU remains end-to-end retain-free.