- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper describes how a language model responds to a fact using six probes. It then matches the facts to be unlearned onto never-seen facts through conditional optimal transport. Training and model selection need no access to the retrained model. The method improves some per-fact likelihood metrics. However, sampling leakage becomes worse, and the independent benefit of joint matching and its stability across models remain insufficiently supported.
Critical strengths
- The multi-probe view has value. It shows that a single answer probability cannot fully measure unlearning.
- The internal ablations are fairly thorough, and the paper reports metric conflicts and failed results honestly.
- Training and model selection do not depend on the retrained model, and the procedure is clear.
Critical weaknesses
- The metric gains do not turn into better behavioral unlearning. JOINT reaches a 73% sampling leakage rate. That is 40.6 percentage points above the retrained model, and it also fails the Forget Quality test.
- The core mechanism lacks direct support. Against conditional per-coordinate matching and against pairing shuffled within condition, JOINT shows no significant advantage on either. The necessity of joint coupling is therefore hard to confirm.
- The reference target carries an unresolved bias. Never-seen facts and the retrained model sit at different likelihood levels. A more accurate match to that reference can therefore retain more of the information marked for deletion.
- The cross-model advantage is not stable. A simple scalar objective does better on Qwen, and full joint matching has not been verified on larger models.
- The comparison with the closest recent method is not direct enough. REGUN ports only the reference target. That cannot replace a full comparison against the original ReGUn method.
Questions for authors
- At the same leakage rate, does JOINT still reach a lower per-fact error?
- Which result directly supports joint coupling over conditional information alone?
- Why does the simple scalar objective do better on Qwen?
- Can the likelihood bias of the reference target be corrected without the retrained model?
- For the roughly 25% of forget facts that never enter training, is leakage more severe?
- How does a change in the stopping threshold affect leakage and retention?
- Can the comparison against the original ReGUn use the same budget and the same stopping rule?
- When a real deletion request has no never-seen facts from the same distribution, how does the method build its reference set?
4 comments
Response to Reviewer Jnqn (2/2): questions
This comment answers your eight questions, each with data from the three further seeds. Locations refer to the revised PDF; every arm is in Table 16 (p. 25).
Q1. At the same leakage rate, does JOINT still reach a lower per-fact error? We scored every epoch of JOINT and of the published objectives (Appendix E.7 (p. 26), "Equal leakage"). Against SimNPO no comparison exists: along its epochs JOINT never comes down to SimNPO's leakage. Against UNDIAL, where the leakage ranges overlap, JOINT is closer per fact by 0.20 nats.
Q2. Which result directly supports joint coupling over conditional information alone? No result does on per-fact error, and the paper now says so. The within-strata shuffle is not separated from JOINT, REGRESS (no transport) is not separated from JOINT, and REGRESS-ANCHORED matches JOINT-ANCHORED. What the evidence supports is the conditioning, which now includes the fact's own base profile (Section 5.6 (p. 9)).
Q3. Why does the simple scalar objective do better on Qwen? We have one measured difference, not a full explanation. On Qwen2.5-1.5B JOINT brings the forget facts up to the reference level only at epoch 5 or 6 of its six epochs, against epoch 2 at 1B, so its budget ends as the level is reached and little is left for matching individual facts (Section 5.6 (p. 9)). On the two further seeds the scalar's lead is also not reproduced on both seeds (W4). We ran the arms with twice the epochs as well; the ordering stayed inconsistent across the two seeds, so we report no claim from it.
Q4. Can the likelihood bias of the reference target be corrected without the retrained model? Largely, yes, by anchoring each target on the fact's own profile under the model before fine-tuning (Section 3.5 (p. 6)). Before training, the DIRECT target error falls from 0.26 to 0.06 nats, and the per-fact target error from 0.613 to 0.520 (six seeds). After training, the per-fact distance to the retrain falls from 0.799 to 0.675 nats, and to 0.570 with per-token targets, which is significant after Holm correction.
Q5. For the roughly 25% of forget facts that never enter training, is leakage more severe? Yes. On the 101 forget facts that lack a core probe and enter no arm, JOINT's excess leak is +0.49 against +0.35 on the others, higher on each of six seeds (Appendix E.7 (p. 26), "Facts without a profile").
Q6. How does the stopping threshold affect leakage and retention? It does not, within a wide range. For every threshold from 0.02 nats to no limit, the rule selects the same epoch in each JOINT run, so leakage and retention do not move (Appendix E.7 (p. 26), "Stopping threshold").
Q7. Can the comparison against the original ReGUn use the same budget and stopping rule? Yes; see W5 in (1/2). The row "ReGUn's objective" of Table 16 (p. 25) uses our budget and stopping rule.
Q8. When a deletion request has no never-seen facts from the same distribution, how does the method build its reference? We tested a generated reference. Another model (Qwen2.5-7B-Instruct) invents 36 fictitious people who are not authors and writes question-answer pairs about them; after the probe checks, 123 facts build targets and 55 feed the stopping rule. This reference costs JOINT 0.266 nats, but JOINT-ANCHORED only 0.044, because anchoring asks the reference only for the shift that fine-tuning induces, not for the level (Appendix E.7 (p. 25), "Generated reference").
Presentation (rated 2). We agree. A readability pass rewrote sixty-two passages of the main text (counted by a script, in blue): Section 1 (p. 1) states the paper in one sentence, one example fact runs through Sections 1, 3.1 and 5.1, captions open with their takeaway, and Algorithm 1 (p. 5) has inputs, an output and a comment on every step. One term that your review uses changed: "coupling" now always means the transport plan, and what you call joint coupling is "sharing one coupling across coordinates" (Section 5.4 (p. 8), Section 6 (p. 9)). An example from Section 5.3 (p. 8). Before: "COORDINATE's monotone maps keep the forget set's pairing between phrasings in its targets". After: "COORDINATE's increasing maps keep the forget set's correlation in its targets".
In summary. We accept the review's central criticisms, and the paper is now explicit about each. In response, the revision adds a retrain-free estimate of the level that lowers the per-fact distance significantly, a measured cause of the behavioural gap, a direct comparison with ReGUn's objective, and an answer with data to each of your eight questions. We hope that these additions address a good part of your concerns about soundness, and we would be grateful if you would take them into account in your final assessment. Thank you again; the review changed the paper substantially.
Dear Authors,
Thank you for the detailed rebuttal and the additional experiments, including three new seeds with corresponding retrains. I read the full rebuttal and revised PDF. The new results answer most of my questions and leave the main concerns below unresolved.
The experiments do not establish a benefit from joint transport. The title, abstract, and Section 3 present entropic OT as the main contribution. The controls give similar results.
- REGRESS scores 0.831 nats versus 0.799 for JOINT (p = 0.72, Table 16).
- REGRESS-ANCHORED scores 0.687 versus 0.675 for JOINT-ANCHORED (Table 16).
- The within-strata shuffle and COORDINATE-COND show no significant difference from JOINT (p = 0.75 and 0.54, Table 6).
COORDINATE is the only control that differs significantly from JOINT, and it removes both coupling and conditioning. As the authors acknowledge in Q2, the experiments leave the contribution of transport unresolved. DIRECT scores for memorised facts are near the floor (mean 0.07 nats, Table 7), which could explain much of the drop in its correlation with QPARA from 0.91 to 0.15. SCALAR recovers most of the correlation with one supervised coordinate (0.740 versus 0.092 for the untouched model and 0.873 for the retrain, Table 5). Proposition 1(b) applies only to the targets (Section 5.3).
Therefore, I will keep my rating.
Response to Reviewer Jnqn (follow-up)
Thank you for reading the rebuttal and the revised PDF in full, and for this precise follow-up. We agree with your reading of every number you cite; they are the paper's own controls. We differ only on what the paper rests on, so we state that as exactly as we can.
1. Transport is not shown to be necessary, and the paper does not rest on it. We agree that no control separates JOINT from REGRESS, from the within-strata shuffle or from COORDINATE-COND. The revised paper says so in Section 5.4 (p. 8), in Section 5.6 (p. 9), where the gain is "not from the transport", and at the head of the Limitations (Section 6 (p. 9)). We also accept that Section 3 gives the coupling most of its space.
What the method proposes is the target. Each forgotten fact gets a target for its whole profile, conditioned on what is known about the fact apart from its memorisation and estimated from never-seen facts; a two-sided loss approaches that target, and training stops by a rule that needs no retrain. Transport is one way to compute the target and ridge regression is another. That the two are not separated suggests that the target, not the solver, carries the result. Two results about the target are significant after Holm correction:
- Conditioned whole-profile targets against unconditional per-probe targets: 0.785 against 0.906 nats on five seeds, closer on each (p = 0.01, Table 2 (p. 7)).
- Targets conditioned further, on each token's likelihood under the base model: 0.570 against 0.799 nats on three further seeds, closer on each (p < 0.01, Table 16 (p. 25)), at the cost of truncated samples.
The loss must also be two-sided: a variant that only raises NLL ends at 2.749 nats (two seeds, Table 4 (p. 18)).
2. The floor. We agree that this is the likely mechanism, and the paper claims no other: memorisation drives DIRECT to the floor (mean 0.07 nats, Table 7 (p. 20)), and its correlation with QPARA falls from 0.91 to 0.15. The consequence is what the method acts on. At the floor, a fact's own value no longer says where that fact would sit had it not been trained on, so that place has to come from information taken before memorisation. The difficulty covariate is exactly that, DIRECT under the base model, and it is the one penalty whose removal hurts (0.866 against 0.797 nats at seeds 3 and 4, Table 4 (p. 18)). Anchoring on the fact's base profile (Section 3.5 (p. 6)) takes the same step further. In our reading, your observation is therefore the reason why conditioning matters.
3. SCALAR and Proposition 1(b). We agree on both. Lifting DIRECT off the floor restores much of the correlation by itself, and Section 5.3 (p. 8) states that the proposition constrains targets, not trained models. This is why the paper's criterion is the per-fact distance, with correlation as the diagnostic that motivates it: SCALAR regains a correlation of 0.740 and still ends 1.031 nats from the retrain per fact, against 0.811 and 0.785 for JOINT (five-seed means, Table 5 (p. 19); this pair was not tested).
We understand your decision and do not ask you to credit the transport. We hope that the narrower claim, a retrain-free per-fact target whose gain comes from what it conditions on, can be weighed on its own evidence in the final discussion. Thank you again; your review made the paper more accurate.
Response to Reviewer Jnqn (1/2): weaknesses
Thank you for the detailed and direct review. It is the review that changed the paper most: your points on the reference bias (W3, Q4) and on ReGUn (W5, Q7) led to new experiments on three further seeds with their own retrains (in blue: Section 3.5 (p. 6), Section 5.6 (p. 9), Appendix E.7 (p. 24) with Table 16 (p. 25)). We agree with most of your points and say so plainly below. This comment answers the weaknesses; (2/2) answers the questions.
W1. The metric gains do not turn into better behavioural unlearning; JOINT leaks most and fails Forget Quality. We agree with the observation: on the reported score JOINT leaks most among the arms of Table 2 (p. 7), and it fails Forget Quality. The five-seed results are unchanged. Three things are new.
No arm of ours passes Forget Quality; Section 5.6 (p. 9) and the Limitations (Section 6 (p. 9)) say so.
W2. The core mechanism lacks direct support. We agree, and the new arms support your reading rather than ours. A regression on relation, length and difficulty with no transport (REGRESS) is not separated from JOINT (0.831 against 0.799 nats, closer on 1 of 3 seeds), and with anchoring the regression matches the coupled arm (0.687 against 0.675). We changed the paper accordingly: Section 5.6 (p. 9) states that the gain comes from what the targets condition on, not from the transport, the Limitations open with this point, and the abstract's conclusion now says "indicate" where it said "establish".
W3. The reference target carries an unresolved bias. We agree that it was unresolved; the submitted abstract called it "the missing piece". This was the most productive point of the review for us, because Section 3.5 (p. 6) now estimates the level without a retrain. Part of the bias is composition: under the base model the forget facts are 0.32 nats harder than the reference facts. JOINT-ANCHORED therefore starts each target from the fact's own profile under the base model and takes from never-seen facts only the shift that fine-tuning induces (Eq. (3) (p. 6)). Before training, these targets sit 0.06 nats from the retrain on DIRECT against 0.26 for JOINT's. After training, JOINT-ANCHORED is closer to the retrain than JOINT on 3 of 3 seeds (0.675 against 0.799 nats, not significant at three seeds). The remaining DIRECT bias of every arm is in the "bias" column of Table 16 (p. 25).
W4. The advantage is not stable across models; larger models are unverified. We agree, and the new runs do not remove this limit; we prefer to state it plainly. On two further Qwen2.5-1.5B seeds, SCALAR, JOINT, JOINT-ANCHORED and TOKEN-ANCHORED reach 1.117, 1.127, 1.141 and 1.039 nats, with no arm ahead on both seeds, so the gain from anchoring at 1B does not transfer within the same six epochs (Section 5.6 (p. 9)). Models above 1.5B remain untested (Section 6 (p. 9)).
W5. REGUN ports only the reference target, not the original ReGUn objective. We agree, and we ran it. ReGUn is defined for classifiers, so we transcribed its loss to next-token prediction: every answer position of a forget question is distilled onto the frozen model's next-token distribution, averaged over held-out answers of the same relation class, with our budget and stopping rule (Appendix E.7 (p. 25)). It reaches 3.678 nats per fact against JOINT's 0.799, because it overshoots the retrain's level. We note that this is our transcription, not the authors' code.
Deleted text. The abstract's and the conclusion's "a retrain-free estimate of that level is the missing piece" is replaced by the anchoring result.