acceptodds
Under review as a conference paper at ICLR 2027

RAMP: Residual-Guided Route Weighting for Vision-Language Unlearning

Abstract

Suppressing a target answer does not necessarily prevent a vision-language model from revealing the same fact through alternative wording or visual evidence. We introduce RAMP (Residual-Adaptive Multimodal Penalties), a two-stage unlearning method that uses residual answer scores to determine both when and where to apply further forgetting updates. Stage-I suppresses target answer spans. Stage-II scores verified alternative answers, activates representation penalties when those scores remain high, and allocates their strength across text, image, and fusion representations using gradient sensitivity. Alias suppression and preservation losses jointly constrain the updates. On CLEAR, Stage-II reduces held-out paraphrase recovery from 0.181 to 0.072 and indirect recovery from 0.174 to 0.089. On UMU-Bench, RAMP achieves lower remaining target performance and higher utility than the reported unlearning baselines at all three forget rates. Fixed-budget routing controls and baselines with shared supervision distinguish the contribution of adaptive weighting from that of additional data. Unseen attacks, output audits, and public-data substitution further characterize the method's effectiveness and practical requirements. Together, these results support using residual recovery to guide the allocation of unlearning updates.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Vision-Language Models, Multimodal Learning, Knowledge Unlearning, Semantic Erasure, Large Multimodal Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RF7ROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses residual factual recovery in vision-language unlearning: suppressing a target answer’s original wording may leave the same fact accessible through paraphrases, indirect questions, or visual evidence. It proposes RAMP, a two-stage method. Stage I suppresses verified target answer spans. Stage II uses teacher-forced scores of verified alternative answers to activate additional representation penalties and allocates their weights across text, image, and fusion intervention sites using gradient sensitivity. Alias suppression and several preservation objectives remain active alongside these penalties. Experiments on LLaVA-1.5-7B cover CLEAR and UMU-Bench. On CLEAR, Stage II reduces held-out paraphrase recovery from 0.181 to 0.072 and indirect recovery from 0.174 to 0.089. Equal, shuffled, and single-route controls assess adaptive weighting, while shared-supervision baselines examine the contribution of additional data. The paper also evaluates unseen attacks, output quality, public utility-data substitution, and computational cost.

Critical strengths
  1. The problem formulation is supported by quantitative evidence, and the evaluation goes beyond direct-answer suppression. The observed gap between direct recovery and paraphrased or indirect recovery provides a concrete motivation for residual-guided updates. Held-out questions and images, multiple attack families, and output audits test whether improvements extend beyond the training aliases and whether reduced disclosure merely reflects refusal or degenerate generation. This is consequential because direct-answer accuracy alone can substantially overstate successful unlearning. The paper also appropriately distinguishes its empirical suppression results from complete information erasure or equivalence to retraining.
  2. The controlled experiments provide meaningful support for target-dependent route weighting. Adaptive, equal, shuffled, and single-route variants share initialization, supervision, other losses, and Stage-II update budgets. In particular, shuffling preserves the distribution of weight vectors while breaking their association with targets, making it a useful control for the proposed adaptive mechanism. Shared-supervision baselines separately show that span and alias construction help existing methods but do not fully reproduce RAMP’s results. Together with the forgetting–preservation measurements, these experiments make the methodological contribution more convincing than a comparison against unaugmented baselines alone.
Critical weaknesses
  1. The rationale and robustness of the residual-guided routing rule are not fully established. The gap between direct and semantic recovery convincingly motivates further unlearning, but does not by itself justify allocating representation penalties according to cross-route gradient magnitudes. Text, image, and fusion sensitivities are computed over different tensors using different operations, without dimension normalization, so their relative values may reflect representation scales or token counts as well as target-relevant sensitivity. Equal and shuffled controls demonstrate the usefulness of target-dependent weighting in the evaluated setting, but comparisons with normalized sensitivities or validation-tuned constant weights are needed to substantiate the particular allocation rule. Evaluation on only LLaVA-1.5-7B further limits confidence in the robustness of this architecture-dependent design.

  2. The evaluation leaves target-level protection under repeated extraction attempts unclear. The paper provides substantial evidence of reduced average recovery through held-out questions, reserved images, and unseen attacks. However, its metrics average query outcomes within identities rather than measuring whether an identity can be recovered by at least one attempt. Consequently, low average recovery may coexist with a substantial proportion of compromised targets. The attack suite covers 20 identities with three prompts per family, leaving protection under larger query budgets insufficiently characterized. Although the paper appropriately avoids claiming complete erasure, identity-level compromise rates are important for assessing the practical significance of its unlearning gains.

Questions for authors

How well does the teacher-forced residual score predict actual output-based recovery, particularly for examples below the activation threshold? Do dimension-normalized sensitivities or validation-tuned constant route weights preserve RAMP’s advantage? How sensitive are the adaptive weights to question length, image-token count, and the selected inspection layers?

3 comments
AuthorsOpenReview ↗

Response to Reviewer RF7R (1/2): why allocating by residual sensitivity helps

Thank you for your positive assessment and for highlighting the shuffled control and the multi-condition evaluation. We agree with your key distinction: showing that recovery must be reduced under several access conditions does not by itself justify raw cross-route gradient magnitudes as the allocation signal. Below we explain the rationale, what the existing controls establish, and where the limits are.

1. Rationale. The residual score is a soft maximum of the teacher-forced log-likelihoods of the verified alternative answers (aliases), i.e., how easily the model can still state the fact after Stage I has suppressed the direct answer. Its gradients with respect to the question embeddings, the projected image tokens, and answer-to-image attention (attention × gradient) measure how strongly this remaining likelihood depends, locally, on each part of the input. Each sensitivity weights a penalty that moves the corresponding features away from those of the original model until a similarity margin is reached: image-free question states (text), vision-encoder and projected image features (image), and answer states and attention patterns while producing the alias (fusion). The weights are detached and smoothed per example, so the optimizer changes the model under a fixed allocation rather than learning the allocation (App. C.1, C.3).

2. The penalties are distinct levers, and adaptive weights use them better. All rows share the Stage-I checkpoint, data, other losses, and 1,200 Stage-II updates (three seeds; Tables 15, 16):

Route weightsQuestion only ↓Image + neutral prompt ↓Question + image ↓Worst condition ↓
Text only0.0350.1450.1830.183
Image only0.0730.0560.1580.158
Fusion only0.0690.1230.0610.123
Equal0.0500.0890.1120.112
Shuffled0.0630.1110.1390.139
Adaptive (RAMP)0.0410.0680.0810.081

Each single-route penalty is best in its own condition and weak elsewhere, so which lever a target needs matters. Adaptive weights beat equal weights in all three conditions at once, rather than trading one condition against another, and have the lowest worst-condition recovery (0.081 vs. 0.112). Relative to training without the penalties (route-loss weight λ6 = 0; question+image recovery 0.151, Table 26), equal weights remove 0.039 of question+image recovery at a cost of 0.019 in native utility (fixed examples from the original instruction-tuning distribution), whereas adaptive weights remove 0.070 at a cost of 0.006.

3. Target signal or scale artifact? (a) A fixed scale difference between routes, whether from feature dimensions, the number of layers, or the operation type, affects every target alike, so it is also present in the shuffled control, which reuses the same weight vectors for other identities. Shuffled weights are worse than equal weights on every metric (paraphrase recovery 0.113 vs. 0.094, against 0.072 for adaptive; UMU-10% UC, the remaining target performance, 0.5918 vs. 0.5712, against 0.5474). The benefit therefore comes from matching weights to their own target, not from the distribution of weights. (b) Image-token count cannot drive the image sensitivity here: LLaVA-1.5 always supplies 576 projected tokens per image. (c) Following the sensitivities more closely helps up to a point. Equal weights are the τa → ∞ limit, and question+image recovery is 0.112 there, 0.105 at τa = 4, 0.081 at the default τa = 2, and 0.090 at τa = 1 (τa = 1 and 4 use one seed; Table 26).

4. Limits. Question and alias lengths vary and enter the text and fusion sums. We have not measured how the weights correlate with them, nor tested dimension-normalized scores, validation-tuned constant weights, or other inspection layers (fixed at decoder layers {8, 16} and {20, 24, 28, 31}). Among constant allocations, we tested the simplex center (equal) and its three corners (single routes); adaptive weights are better than all four on held-out paraphrase and indirect recovery and on worst-condition recovery. We therefore present the rule as an empirically supported, locally measured heuristic for LLaVA-1.5-7B, not as a localization of stored facts, and will state these limits in the paper.

AuthorsOpenReview ↗

Response to Reviewer RF7R (2/2): the residual score, repeated attempts, and outputs

1. Does the training score predict output leakage, especially below the threshold? We have not measured a per-example calibration curve or a below-threshold false-negative rate, and we will state this. The design limits the consequence of a miss: (i) log(0.02) is a cut on the length-normalized score, not a 2% leakage rate; (ii) below it, aliases are still suppressed at one-fifth strength and all preservation losses stay on; (iii) the score is recomputed at every later visit, so an example whose aliases regain likelihood returns to the active branch; and (iv) checkpoints are selected on recovery rates for separate paraphrased and indirect validation questions, so leakage that the score misses is penalized at selection.

What the paper shows is that updates guided by the score reduce generated-answer recovery on prompts the score never sees. The post-training attack suite uses 20 identities, three prompts per family, and reserved images, and none of its prompts was used for training or checkpoint selection (Table 19). "No route penalty" is Stage II with alias suppression but without the representation penalties:

Attack familyStage I ↓No route penalty ↓RAMP ↓
Long-tail paraphrase0.2600.1370.091
Visual clues0.2680.1600.098
Two-hop relations0.2860.1830.111
Candidate disambiguation0.2240.1310.084
Two-turn dialogue0.3100.1990.124

The last two columns start from the same Stage-I checkpoint and suppress the same aliases; the score-weighted penalties lower recovery in every family. This supports the update that the score guides, not the calibration of the score itself.

2. Average recovery vs. identity-level compromise. You are right. Our metric averages prompts within each identity and then across identities, so it cannot determine the share of identities recovered by at least one of several attempts; a low mean is compatible with many compromised identities. Only the two-turn family applies an OR, and only within one dialogue. We will (i) describe the results as reduced average recovery under the specified tests, not as protection against repeated attempts, and (ii) report identity-level any-hit rates and query-budget curves computed from the stored attack generations.

3. Lower recovery is not refusal or fluent error. On CLEAR forget queries (Table 20), from Stage I to RAMP, refusals fall from 7.1% to 4.1%, semantic-only disclosure from 15.6% to 2.9% (CLEAR's Gold model, never trained on these identities: 3.0%), and wrong non-target facts from 13.6% to 8.6% (Gold: 8.3%). Accuracy on harmless questions about the same images rises from 88.6% to 90.3% (Gold: 89.6%).

Ethics flag. All forgotten identities are fictitious benchmark characters (CLEAR, UMU-Bench), and the attacks are used only to measure residual disclosure. We are happy to clarify further.

Thank you again for these precise questions. We hope the rationale for following the residual, the evidence that its benefit is target-specific and efficient, and the explicit metric boundaries strengthen your confidence in the contribution.

AuthorsOpenReview ↗

Follow-up: happy to clarify any remaining questions

Dear Reviewer RF7R,

Thank you again for your time and thoughtful feedback. With only a few hours remaining in the discussion period, we would greatly appreciate your thoughts on our responses and additional evidence. We are eager to address any remaining questions and discuss any points that are still unclear, and will respond promptly.

Best regards,

The Authors of Submission #36225

Reviewer_QMsrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RAMP is a two-stage full-parameter unlearning method for VLMs (LLaVA-1.5-7B). Stage 0 uses gpt-4o-2024-08-06 to extract indispensable answer spans K(x)K(x) and verified alias answers A(x)A(x) for each forget example and to build retain and benign-neighbour sets; the default utility set is selected from the original instruction-tuning mixture by overlap filtering, embedding-based coverage ranking and quotas (Appendix A.2, lines 604-644), with a filtered public-data alternative. Stage I applies a length-normalised unlikelihood loss to the spans (Eqs. 1-3). Stage II computes a residual score ssem(x)s_{\mathrm{sem}}(x), a soft-max over alias scores (Eq. 4), gates representation penalties on ssem>log⁡(0.02)s_{\mathrm{sem}} > \log(0.02), and weights three cosine-margin penalties on text, image and fusion representations (RTR_T, RIR_I, RFR_F) by a softmax over sensitivities: ℓ1\ell_1 gradient norms of ssems_{\mathrm{sem}} with respect to question embeddings (gTg_T) and projected image tokens (gIg_I), and for fusion (gFg_F) the ℓ1\ell_1 norm of attention weights multiplied entrywise by their gradients (Eq. 5, Appendix C.1 lines 972-980; Eqs. 6-9); alias suppression and preservation losses run throughout (Eq. 10).

Validation on CLEAR 10% and UMU-Bench 5/10/15%: RAMP reaches Forget 0.08 / Retain 0.52 / Real 0.50 against published baselines (Table 1); three-seed reruns give Forget 0.081 [0.045, 0.124] vs SCRUB 0.359 (Holm p=0.0006p = 0.0006) and not significantly different from NPO (p=0.444p = 0.444) (lines 284-293). Stage II lowers held-out paraphrase recovery from 0.181 to 0.072 and indirect from 0.174 to 0.089 (Fig. 2a). Fixed-budget routing controls give paraphrase recovery 0.072 adaptive vs 0.094 equal vs 0.113 shuffled (lines 345-346); shared-supervision baselines reach 0.126 (NPO+P+T) without retuning (Table 4). Unseen-attack families, an output-category audit (κ\kappa = 0.884), public-utility-data substitution and cost (22.6 h on 8xA100) are reported. The evaluation is unusually thorough and honest; the contribution is weakened because the method's distinctive component, adaptive routing, shows a measured 0.022 difference within the fixed RAMP recipe with no interval, the fraction of the total gain attributable to it is undetermined, the shared-supervision baselines are untuned, and all results are on a single model.

Critical strengths
  • Soundness: The evaluation is far more careful than is typical for unlearning papers. Three-seed reruns with identity-block bootstrap intervals and Holm-corrected permutation tests (lines 270-273, 284-293, Table 9); a common training-filter control that re-applies RAMP's exclusions to NPO and SCRUB (lines 404-411: 0.074 +- 0.041 vs 0.206 +- 0.075 vs 0.341 +- 0.087); five families of post-training attacks on reserved images and never-trained prompts (Table 5); and a six-category output audit showing refusals fall from 7.1% to 4.1% while semantic-only disclosure falls from 15.6% to 2.9% (lines 424-428). The authors also state plainly what their tests do not show (lines 293-294, 403, 445-447, 482-485). This substantially raises my trust in the reported trade-off.
  • Significance: The semantic-residual framing is the right diagnostic for VLM unlearning: Stage I alone leaves paraphrase and indirect recovery at 0.181 and 0.174 while direct recovery is 0.084 (Fig. 2a), and single-route penalties each leave a different input condition open (Fig. 2b, text-only 0.035 without image vs 0.183 with the full input). The cross-condition table is a clean demonstration that one access test is not enough.
Critical weaknesses
  • Significance: The paper's distinctive mechanism is only partly isolated, and what is isolated is small and un-intervalled. Three separate comparisons bear on it and should not be combined. (a) The controlled routing comparison, with the Stage-I checkpoint, data, losses and 1,200 updates fixed (lines 342-349, Appendix E.5), gives paraphrase recovery 0.072 adaptive vs 0.094 equal vs 0.113 shuffled, a 0.022 adaptive-vs-equal effect; the routing controls use three seeds (line 271, Appendix E.5 lines 1462-1465) and the printed values are point estimates over them (Fig. 2(b) caption), with no interval or paired test for the difference. (b) The shared-supervision comparison (Table 4) shows that giving NPO the Stage-0 spans and aliases lowers its paraphrase recovery from 0.232 to 0.126 and that the remaining gap to RAMP's 0.072 is against baselines that "are not separately retuned" (line 374); this measures the recipe plus data, not routing. (c) The gate h(x)=1[ssem(x)>δ]h(x)=\mathbb{1}[s_{\mathrm{sem}}(x)>\delta], the other half of the residual-guided mechanism (activation vs allocation, lines 173-176), is never tested on its own: every routing variant keeps the same gate (Appendix E.5, lines 1462-1470) and the no-route control removes the penalty rather than comparing gated with always-on penalties (Table 26, lines 1766-1777). Contribution 2 ("isolate target-dependent weighting") is supported for allocation only, by point estimates.
  • Soundness: Generality and evaluation provenance. Every number is on one model, LLaVA-1.5-7B, with Stage 0 bound to one closed snapshot (gpt-4o-2024-08-06, line 147) and a cost of 22.6 h on eight A100-80GB per CLEAR run (line 462). The held-out paraphrase and indirect questions are "independently constructed rewordings" (Appendix E.1, lines 1403-1405) with no statement of who or what constructed them; Appendix A.1 constructs alternative answers (lines 582-589) while Appendix E.4 evaluates reworded and indirect questions on inputs reserved before Stage 0 (lines 1401-1405), so shared generator provenance does not by itself establish overlap; the undisclosed generation procedure is a documentation gap. The concrete unresolved overlap is between checkpoint selection and evaluation: Appendix D.3 selects checkpoints on held-out paraphrase and indirect success (lines 1215-1244) and the paper does not state that those prompts are disjoint from the reported evaluation prompts (question 3 below).
Questions for authors

Questions that could change my assessment

  1. Please report the routing comparison (adaptive / equal / shuffled / single-route) with three seeds and identity-block bootstrap intervals and a paired test of adaptive vs equal, and add a matched control in which the route-penalty gate is forced on (h(x)=1h(x)=1 in the λ6h(x)∑rwr(x)Rr(x)\lambda_6 h(x)\sum_r w_r(x)R_r(x) term of Eq. 9) while the original h(x)h(x) is kept in γ(x)=0.2+0.8h(x)\gamma(x)=0.2+0.8h(x) for alias suppression (so only one loss changes), compared at matched utility, so that activation is tested separately from allocation. Score-change criterion: if adaptive beats equal weights with a Holm-corrected p<0.05p < 0.05 and the gate improves on always-on penalties at equal utility, Soundness rises to 3 and I would raise the Rating to 3; if either interval covers zero, the validated mechanism narrows accordingly and the Rating stands.
  2. How were the held-out paraphrase and indirect evaluation questions constructed (human annotators, a different LLM family, or the same gpt-4o prompt as the training aliases A(x)A(x))? Score-change criterion: human-written or different-family rewordings leave the paraphrase result standing; same-prompt-family rewordings would require a re-evaluation on CLEAR's native paraphrase probes before the 0.072 figure can carry the claim, and would lower Soundness.
  3. Appendix D.3 selects checkpoints on held-out paraphrase and indirect-description success (lines 1215-1244) and Appendix E.4 reports Paraphrase and Indirect on the reserved 25% (lines 1401-1405), without stating that the two prompt sets are disjoint. Are the checkpoint-selection prompts and the reported evaluation prompts disjoint? Score-change criterion: if they are disjoint, no change; if they overlap, the headline 0.072/0.089 are selected on evaluation data, Soundness falls to 1 and the Rating to 1 until the scores are re-reported on an untouched test set.
  4. Can the shared-supervision baselines in Table 4 be given a learning-rate and loss-weight sweep of the same size as RAMP's, and can one additional VLM (e.g. Qwen2-VL-7B or LLaVA-1.6) be run on CLEAR 10% with the full RAMP recipe? Score-change criterion: if a tuned NPO+P+T comes within 0.02 of RAMP, the central weakness is confirmed (the gain is the data) and the Rating stays at 2; if the gap persists under matched tuning and a second VLM reproduces the Stage I to Stage II drop, Rating rises to 3.

Clarifications and minor points (do not affect my assessment)

  1. The sensitivities gTg_T, gIg_I (gradient ℓ1\ell_1 norms) and gFg_F (attention-times-gradient ℓ1\ell_1 norm, Eq. 5) are over tensors of different sizes, so their raw magnitudes are not commensurable; the default τa=2.0\tau_a = 2.0 (Appendix D.1, line 1182) and the one-seed τa=1\tau_a = 1 and 44 rows (Table 26, lines 1771-1772) should be promoted to a seeded sensitivity result in the main text.
  2. Table 2's claim of "higher utility ... at all three forget rates" should be qualified by Table 11 (generation utility 0.1052 vs NPO 0.1406 at 10%).
  3. RWKU (Jin et al. 2024) and MUSE (Shi et al. 2024) established paraphrase and indirect probes for LLM unlearning; position the semantic-residual evaluation relative to them.
  4. RMU (Li et al. 2024) is the closest single-modality representation-level method; it appears only as a published row in Table 1 and would be a natural recipient of the shared supervision in Table 4.
  5. Release the Stage-0 outputs (spans K(x)K(x), aliases A(x)A(x), benign neighbours) as data so the recipe is reproducible without GPT-4o access.
  6. Future work: a cost-reduced Stage II (gate only, no attention extraction) is suggested by the 41.2% activation rate (line 465).
4 comments
AuthorsOpenReview ↗

Response to Reviewer QMsr (1/3): Q3 and Q2, evaluation protocol and question provenance

Thank you for an exceptionally careful review and for stating explicit criteria. We also appreciate your view that the evaluation is unusually thorough and honest, and we keep that standard here: we answer each question directly, using only experiments in the submission, and say plainly which criteria remain unmet. In brief:

  • Q3: The checkpoint-selection prompts and the reported test prompts are disjoint (no change under your criterion).
  • Q2: The test questions come from the same GPT-4o snapshot as Stage 0, but from a separate question-rewording step on reserved inputs, not from the alias prompt. Benchmark-native scores give a generator-independent check.
  • Q1: Allocation is isolated under matched training and supplies 41–44% of the representation-penalty effect, but without a routing-specific interval. The gate is not isolated, and we narrow Contribution 2 accordingly (Part 2).
  • Q4: The tuned baselines and the second VLM were not run; Part 3 states what the data-matched controls do establish.

Q3: Checkpoint selection vs. reported evaluation. The D.3 validation reformulations used for checkpoint selection and early stopping and the E.4 prompts behind the reported Paraphrase/Indirect scores are separate, non-overlapping question sets. No E.4 prompt was used for selection or stopping, so 0.072/0.089 were not selected on test prompts. Both sets probe the same forgotten identities, as unlearning requires. In addition, E.4 reserves 25% of each identity's QA pairs and images before Stage 0, outside training, alias construction, and neighbor construction. In D.3, "held-out" meant held out from the alias bank; we will rename it and state the separation explicitly.

Question setUsed for selection/stoppingUsed for reported results
D.3 validation reformulationsYesNo
E.4 paraphrase/indirect test promptsNoYes (0.072 / 0.089)
E.7 post-training attacksNoYes (separate table)

Q2: Provenance. Both the Stage-0 aliases and the E.4 questions were produced with gpt-4o-2024-08-06. None are human-written or come from another model family, and we will document the E.4 prompt. They are not the same prompt family, however: A(x) contains alternative answers for training examples, whereas E.4 contains reworded questions about reserved QA pairs and images that never entered Stage 0, and no E.4 question is in A(x). Three existing results do not depend on our generator:

  1. Benchmark-native probes. CLEAR's official Forget and Forget Quality (FQ) come from the released evaluator on the benchmark's own questions and answer candidates. Among other components, a truth ratio compares a paraphrased correct answer with five perturbed ones, and FQ compares the distribution of this ratio with that of the Gold model (trained without the forget set). RAMP obtains Forget 0.081 and FQ 0.889 [0.856, 0.920], above NPO's 0.761 (Holm p = 0.0021; not significant against SCRUB's 0.851, p = 0.296). These scores are likelihood-based, so they complement rather than replace generated-answer tests. On UMU-Bench's own classification, cloze, and generation probes, RAMP's UC (remaining target performance) is below every published baseline at all three forget rates, and adaptive rather than equal weights still lower UC (0.5712 → 0.5474) and narrow all three modality gaps (Table 12).
  2. Mechanism comparisons. Every variant that isolates our mechanism (equal, shuffled, and single-route weights; λ6 = 0; NPO and SCRUB given the same spans, aliases, and preservation data) was trained on the same aliases. Any advantage from a shared generator is available to all of them, so it cannot by itself explain their differences.
  3. Structurally different attacks. The E.7 two-hop, candidate-disambiguation, and two-turn prompts are not rewordings, and there RAMP is below the no-route model in all five families (e.g., two-hop 0.183 → 0.111).

We will therefore present 0.072 as a held-out, same-generator estimate, not as evidence of cross-generator robustness.

AuthorsOpenReview ↗

Response to Reviewer QMsr (2/3): Q1, how much allocation contributes, and the gate

The paper isolates allocation within matched training. All rows below are Stage-II runs from the same selected Stage-I checkpoint, with identical Stage-0 data, other losses, optimizer, 1,200 updates, three seeds, and evaluation; only the representation-penalty term differs (Tables 15, 16, 26). The λ6 = 0 row comes from the sensitivity sweep but follows this exact protocol, so this is not a combination of the separate contrasts you cautioned against. Native utility is accuracy on fixed examples from the original instruction-tuning distribution.

Route termQuestion only ↓Image + neutral prompt ↓Question + image ↓Native utility ↑Refusal on harmless questions ↓
None (λ6 = 0)0.0620.1190.1510.5073.6%
Equal weights0.0500.0890.1120.4885.8%
Shuffled weights0.0630.1110.1390.4777.5%
Adaptive (RAMP)0.0410.0680.0810.5014.1%

1. Share of the effect. The penalties lower recovery by 0.021, 0.051, and 0.070 in the three access conditions. Replacing equal by adaptive weights supplies 0.009, 0.021, and 0.031 of this, i.e., 41–44% in every condition. For the representation-penalty component, this quantifies the fraction attributable to allocation. On held-out paraphrase recovery, the adaptive–equal difference you note (0.094 → 0.072) is a 23% relative reduction, accompanied by +0.013 native utility (Table 15).

2. Both sides of the trade-off. Relative to no penalty, equal weights cost 0.019 native utility and add 2.2 points of harmless-question refusal while removing 0.039 of question+image recovery. Adaptive weights cost 0.006 and add 0.5 points while removing 0.070. The same penalties become more effective and less damaging when allocated per target.

3. Target dependence. Shuffled weights reuse the same set of weight vectors but assign them to other identities. They are worse than equal weights on every metric, and on question-only recovery they are no better than no penalty. Across all 18 recovery, preservation, and modality-gap metrics in Tables 12, 15, and 16, the ordering adaptive > equal > shuffled holds without exception, including in the separately trained UMU-Bench 10% comparison (UC 0.5474 / 0.5712 / 0.5918; classification, cloze, and generation gaps narrow by 0.015 / 0.014 / 0.016 relative to equal weights).

What remains unestablished. These are three-seed point estimates. We report no paired interval or Holm-corrected adaptive–equal test, so the submission does not meet your significance criterion. The 18 metrics share checkpoints and are not independent replications; only the CLEAR and UMU-Bench comparisons are trained separately. The gate is not isolated either: every variant uses it, and λ6 = 0 removes the penalty rather than forcing it on. We will therefore (i) revise Contribution 2 to "isolate target-dependent allocation", (ii) describe activation as a design choice with measured compute savings (route computation on 41.2% of visits) rather than as a validated performance component, and (iii) list the missing routing-specific test and gate control as limitations.

AuthorsOpenReview ↗

Response to Reviewer QMsr (3/3): Q4 and the minor points

Q4: Tuned baselines and a second VLM. We could not run the matched learning-rate and loss-weight sweep or a second backbone within the discussion period, so your Q4 criterion is not met; we will list both as limitations. Two existing data-matched results bear on the hypothesis that "the gain is the data." Below, P is the same benign-question and utility supervision, and T is the same span and alias suppression; all rows use the same Stage-0 data.

Method (CLEAR)Question+image recovery ↓Native utility ↑
NPO + P + T (one stage, not retuned)0.2260.478
SCRUB + P + T (one stage, not retuned)0.3090.493
RAMP without route penalties (λ6 = 0)0.1510.507
RAMP0.0810.501

First, within RAMP, with data, checkpoint, and budget fixed, the route penalties alone cut question+image recovery from 0.151 to 0.081 (Table 26). This part of the gain cannot come from the data, which both rows share. Second, the remaining gaps to NPO + P + T are 0.054 in paraphrase recovery (0.126 vs. 0.072), 0.145 in question+image recovery, and 0.084 in UMU-10% UC (0.6315 vs. 0.5474). Tuning may narrow these gaps, and we do not claim that it could not.

Minor points.

  • Commensurability of sensitivities. Agreed: the raw ℓ1 sums are taken over tensors of different sizes and are not normalized. A fixed route bias is common to all targets and therefore also present in the shuffled control, which is worse than equal weights. This rules out the marginal weight distribution as the explanation, but not every nuisance factor, and normalized scores remain untested. Equal weights are the τa → ∞ limit of the softmax; question+image recovery is 0.112 there, 0.105 at τa = 4, 0.081 at τa = 2, and 0.090 at τa = 1. The τa = 1 and 4 rows are one-seed and will be labelled descriptive.
  • UMU utility. We will qualify the claim as aggregate UT: at 10%, multimodal generation utility is 0.1052 for RAMP vs. 0.1406 for NPO, although classification is 0.5269 vs. 0.4357 (Table 11).
  • RWKU and MUSE. We will cite them as establishing paraphrase and indirect probes for LLM unlearning. Our evaluation applies this principle to VLMs, and our methodological claim is narrower: using residual alias scores to activate and allocate multimodal representation penalties.
  • RMU. We will discuss it as the closest representation-level predecessor. It perturbs activations at a fixed layer in the same way for every target, whereas RAMP gates and allocates its penalties per target across three sites. Its CLEAR row (Forget 0.00 with Retain 0.00) is a published value, not a shared-supervision run.
  • Release. We will release the spans, aliases, benign neighbors, D.3/E.4 question lists, prompts, and split records, so the recipe is reproducible without GPT-4o access; distillation still requires the original VLM.
  • Cheaper Stage II. Agreed. With route computation on 41.2% of visits and attention extraction at 7.9% of Stage-II time, a gate-only variant is a promising but untested direction.

In sum: Q3 is resolved; Q2 is answered transparently, with generator-independent benchmark evidence; allocation's share of the penalty effect is quantified from matched runs, though without an interval; and the route penalties remove recovery that the shared data alone do not. We hope this sharper account of what is and is not isolated supports reconsidering your Soundness and overall rating.

AuthorsOpenReview ↗

Follow-up: happy to clarify any remaining questions

Dear Reviewer QMsr,

Thank you again for your time and thoughtful feedback. With only a few hours remaining in the discussion period, we would greatly appreciate your thoughts on our responses and additional evidence. We are eager to address any remaining questions and discuss any points that are still unclear, and will respond promptly.

Best regards,

The Authors of Submission #36225

Reviewer_bQPEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes RAMP, a two-stage vision-language unlearning method that addresses residual factual recovery after direct-answer suppression. Stage I lowers the likelihood of verified target answer spans, while Stage II scores alternative expressions of the same facts to determine whether further representation penalties are needed. Gradient sensitivities adaptively allocate these penalties across text, image, and fusion representations, alongside answer-level suppression and preservation objectives. Experiments on CLEAR and UMU-Bench show a favorable forgetting–utility trade-off, with reduced recovery under held-out paraphrases, indirect questions, and unseen attacks. Controlled comparisons further support the contribution of adaptive weighting beyond additional supervision, although the evaluation is limited to one model backbone and the method incurs substantial training overhead.

Critical strengths
  1. The paper highlights semantic residuals, where targeted facts remain recoverable through alternative expressions or input modalities despite direct-answer suppression. This is an important issue that has received limited attention in prior unlearning work.

  2. Fixed-budget routing comparisons and shared-supervision baselines help isolate the benefits of adaptive weighting, while held-out recovery tests, unseen attacks, and output audits provide evidence beyond direct-answer suppression.

Critical weaknesses
  1. RAMP requires an external model to extract and verify target spans, generate and verify aliases, and construct and validate benign same-image questions. It additionally relies on utility data and a frozen reference model. This elaborate pipeline increases implementation costs and raises questions about scalability and sensitivity to annotation errors.

  2. The method combines multiple loss coefficients, gating thresholds, temperatures, representation margins, and smoothing settings. Although local sensitivity analyses are provided, the large hyperparameter space makes implementation and tuning challenging. It remains unclear whether the gains transfer to new settings without extensive retuning.

  3. The training residual score measures alternative-answer likelihoods under the original query rather than recovery across differently worded queries. Although held-out paraphrases and indirect questions are evaluated, the paper does not clearly establish how reliably this score predicts such leakage. Full response examples would also help distinguish successful suppression from incorrect but fluent answers, beyond aggregate output audits.

  4. Experiments are restricted to LLaVA-1.5-7B and do not include the commonly used MLLMU-Bench. Evaluations across additional model architectures and benchmarks would strengthen the evidence for general applicability.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bQPE (1/3): construction, preservation data, and cost

Thank you for your careful review, and for recognizing that semantic residuals are an important problem with limited prior attention, and that our controls, held-out tests, attacks, and output audits go beyond direct-answer suppression.

For reference, RAMP has three steps. Stage 0 uses an external model to extract the answer spans to forget, generate verified alternative answers ("aliases"), and build harmless questions about the same images. Stage I suppresses the spans. Stage II suppresses the aliases and adds penalties on text, image, and fusion representations that are switched on and weighted by the remaining alias scores (the "residual"). Retain, harmless-question, and utility losses act in both stages. To make the contribution easier to see, the paper's three central findings are:

  1. Direct suppression shifts disclosure into other wordings. On CLEAR forget queries, exact-string disclosure falls from 33.2% (original model) to 8.0% after Stage I, but disclosure of the fact in other words more than doubles, from 6.9% to 15.6% (Table 20).
  2. Residual-guided updates remove most of this. Held-out paraphrase recovery falls from 0.181 after Stage I to 0.072 after Stage II (Fig. 2a).
  3. Where the updates go matters. With data and training budget fixed, allocating the same penalties by residual sensitivity rather than equally nearly doubles the reduction they achieve in question+image recovery relative to training without them (0.070 vs. 0.039), at about one-third of the utility cost (0.006 vs. 0.019) (Tables 15, 26).

Your concerns are about practicality (W1, W2), the link between the training score and actual leakage (W3), and scope (W4). Part 1 answers W1, Part 2 answers W2 and W4, and Part 3 answers W3, all with experiments already in the paper.

(a) Does careful construction matter, and how sensitive is RAMP to annotation quality? Table 21 replaces the whole construction recipe on CLEAR. Native utility is accuracy on fixed examples from the original instruction-tuning distribution.

Forget targetsExact recovery ↓Native utility ↑Refusal on harmless questions ↓
Whole answers (no spans)0.0740.45212.6%
Spans + naive paraphrase bank0.0560.4845.7%
Spans + verified aliases (RAMP)0.0510.5014.1%

A cruder alias bank changes recovery only slightly (0.051 → 0.056); the verified bank mainly improves selectivity. Suppressing whole answers is what damages harmless behavior. These are complete-recipe comparisons, not a controlled noise study.

(b) Do construction failures inflate the results? No failure is removed from evaluation. In the held-out run, 6.2% of examples use fallback aliases and 3.8% receive Stage I only, so the headline numbers already include imperfect annotations. A separate full-pool run (779 rows) excludes the 11 rows without a usable span from forget training but evaluates all 779. When NPO and SCRUB receive the same exclusions (Table 23; three seeds, 95% half-widths), paraphrase recovery is 0.206 ± 0.075 (NPO), 0.341 ± 0.087 (SCRUB), and 0.074 ± 0.041 (RAMP).

(c) Are an extra model and the original instruction data required? The frozen reference is the original VLM itself; its predictions and features can be cached (App. B.3, C.5), and no additional model is trained. Replacing only the utility data with 50k filtered public examples keeps most of the preservation benefit (Tables 6, 25):

Utility training dataNative utility ↑Paraphrase recovery ↓UMU-10% UC ↓ / UT ↑
Original mixture0.5010.0720.547 / 0.481
Filtered public data0.4930.0800.555 / 0.475
None0.4210.0810.539 / 0.430

(d) Measured cost (8×A100-80GB). Stage 0 takes 1.85 h for CLEAR's 584 examples, and its per-example cost is nearly constant across all four settings (2.7–3.2 h per 1,000 examples; Table 28). Stage II adds 58–65% to Stage-I training time (CLEAR: 13.8 h + 8.8 h). Attention extraction is only 7.9% of Stage II because the gate activates on 41% of visits, and peak memory rises from 62.7 to 67.4 GB. The cost is real but bounded and measured; we will summarize it in the main text.

AuthorsOpenReview ↗

Response to Reviewer bQPE (2/3): necessity of components, hyperparameters, and scope

Is each part of the pipeline necessary? Each component has a controlled removal in the paper (CLEAR; the first two rows are complete-recipe comparisons):

ComponentWithout it → with it (RAMP)Source
Span-level targets (vs. whole answers)native utility 0.452 → 0.501; harmless-question refusal 12.6% → 4.1%Table 21
Two stages (vs. one merged stage)exact recovery 0.093 → 0.051; utility 0.463 → 0.501Table 14b
Representation penalties (vs. none, λ6 = 0)question+image recovery 0.151 → 0.081Table 26
Adaptive route weights (vs. equal)paraphrase recovery 0.094 → 0.072; utility 0.488 → 0.501Table 15
Utility supervision (vs. none)native utility 0.421 → 0.501Table 25

The harmless same-image questions are not ablated inside RAMP alone, but adding them, together with utility distillation, to NPO raises its accuracy on these questions from 81.6% to 87.8% and nearly halves its refusals (13.6% → 7.2%; Table 17).

W2: Do the gains need retuning? All loss weights, thresholds, margins, temperatures, and learning-rate multipliers were chosen once on a pilot development split and kept fixed for CLEAR and all three UMU-Bench forget rates; only the step budgets differ by setting (App. D.1–D.2). With this single set, RAMP has lower UC (remaining target performance, ↓) and higher aggregate UT (utility, ↑) than every published baseline at all three forget rates (UC 0.609/0.547/0.506 vs. best baseline 0.701/0.622/0.582; Table 2).

The gains are also not knife-edge. The sensitivity sweep changes one parameter group at a time from the same Stage-I checkpoint and training budget (Table 26):

Setting changed (CLEAR)SeedsQuestion+image recovery ↓Native utility ↑
Default30.0810.501
Route-loss weight λ6 = 0.4 / 1.230.101 / 0.0740.504 / 0.494
All preservation weights ×0.5 / ×210.069 / 0.1040.477 / 0.512
Routing temperature τa = 1 / 410.090 / 0.1050.496 / 0.491
Reference: NPO given all of RAMP's extra supervision (Table 18)0.2260.478

Every setting keeps recovery below half that of the strongest augmented baseline. On UMU-Bench 10% (Table 27, one seed each), all six variations, including removing the route loss, still have lower UC and higher UT than every published baseline (UC 0.528–0.602 vs. ≥ 0.622; UT 0.459–0.492 vs. ≤ 0.449). The knobs also act predictably: the route-loss weight trades recovery for utility, preservation strength moves the trade-off the other way, and both sharper and softer routing temperatures are slightly worse than the default. In practice, adapting RAMP means adjusting these two interpretable scales rather than searching the full space; we do not claim robustness to every parameter.

W4: One backbone; MLLMU-Bench. We agree that this is the main limitation and will say so in the abstract and conclusion. What is tested is one model (LLaVA-1.5-7B) across four settings (CLEAR 10%; UMU-Bench 5/10/15%) and three answer formats (classification, cloze, generation), all with one coefficient set. UMU-Bench shares MLLMU-Bench's profile scale (500 fictitious and 153 real individuals), task formats (classification, cloze, and generation under text-only and image+text inputs), and forget rates, and it additionally scores cross-modal consistency, which is the property RAMP targets. We nevertheless make no claim about MLLMU-Bench or other architectures.

AuthorsOpenReview ↗

Response to Reviewer bQPE (3/3): the training score versus generated outputs

W3: How does the training score relate to actual leakage? We agree that the two must be kept apart. The residual score is a training signal: a soft maximum of the teacher-forced log-likelihoods of the verified aliases under the original question and image. It decides only which examples receive representation penalties and how those penalties are weighted. It is never used to report results or to select checkpoints. Checkpoints are selected on recovery rates for separate paraphrased and indirect validation questions (App. D.3), and every reported recovery score judges generated answers to prompts that the score never sees. The threshold log(0.02) is a cut on the score scale, not a 2% leakage guarantee; below it, aliases are still suppressed at one-fifth strength, and the score is recomputed at every later visit. We do not report a per-example calibration curve between the score and leakage, and we will state this as a limitation.

What the paper does establish is that updates guided by the score reduce leakage in generated answers to reserved questions and images:

Generated-answer recovery ↓After Stage IRAMP (after Stage II)
Direct questions (exact string)0.0840.051
Paraphrased questions0.1810.072 (−60%)
Indirect questions0.1740.089 (−49%)

Against NPO, paraphrase and indirect recovery are also significantly lower (0.072 vs. 0.196 and 0.089 vs. 0.183; Holm-adjusted p = 0.0056 and 0.0104; Table 10). On the separate post-training attacks, removing only the representation penalties (aliases still suppressed) raises recovery in all five attack families, e.g., two-hop relations 0.111 → 0.183 and two-turn dialogue 0.124 → 0.199 (Table 19).

Real suppression or fluent errors? Every benchmark forget-query output is assigned to one of six mutually exclusive categories (Table 20; a human audit of 200 outputs gives Cohen's κ = 0.884). CLEAR's Gold model, trained without the forget set, shows what a model that never learned these identities produces:

Share of forget-query outputsGoldStage INPORAMP
Wrong non-target fact8.3%13.6%19.4%8.6%
Semantic-only disclosure3.0%15.6%7.6%2.9%
Refusal2.0%7.1%15.1%4.1%
Degenerate text0.3%0.5%1.2%0.3%

If RAMP replaced disclosures with fluent mistakes, its wrong-fact rate would rise. Instead, it falls to Gold's level, as do semantic-only disclosure and degenerate text, while refusals drop below Stage I's. On harmless questions about the same images, accuracy rises from 88.6% after Stage I to 90.3% (Gold: 89.6%). We will add representative outputs from each category, taken from the human-audited set, to the appendix.

Presentation. We are sorry that the paper was hard to follow. In the revision, we will add a notation table, move the component and robustness results above into the main text, and fix two issues that may have caused confusion: the appendix tables label RAMP by an internal name ("TRACE"), and one table reference in App. E.1 is broken.

We hope these answers show that each of your concerns (cost, tuning, the training score, and scope) is quantified with controlled evidence in the paper, and that the contribution summarized in Part 1 is clearer. We would be grateful if you would reconsider your assessment in light of them.

AuthorsOpenReview ↗

Follow-up: happy to clarify any remaining questions

Dear Reviewer bQPE,

Thank you again for your time and thoughtful feedback. With only a few hours remaining in the discussion period, we would greatly appreciate your thoughts on our responses and additional evidence. We are eager to address any remaining questions and discuss any points that are still unclear, and will respond promptly.

Best regards,

The Authors of Submission #36225

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: three clarifications and planned revisions

We thank all reviewers for careful and constructive reviews. We are encouraged that they find semantic residuals an important, under-studied problem (bQPE, QMsr, RF7R), consider the evaluation unusually thorough and honest (QMsr), and see the fixed-budget routing and shared-supervision controls as meaningful support for target-dependent weighting (bQPE, RF7R). Every number below comes from experiments already in the submission, and we state where the evidence stops.

1. Final test prompts were not used to select checkpoints. The validation questions used for checkpoint selection and early stopping (App. D.3) are a separate question set from the prompts behind the reported Paraphrase/Indirect scores (App. E.4) and from the post-training attacks (App. E.7). We will state this explicitly.

2. What the routing controls isolate. Every row starts from the same selected Stage-I checkpoint and uses identical data, other losses, 1,200 Stage-II updates, three seeds, and evaluation (Tables 15, 16, 26). Native utility is accuracy on fixed examples from the original instruction-tuning distribution.

Representation-penalty term in Stage IIQuestion+image recovery ↓Native utility ↑Refusal on harmless same-image questions ↓
None (λ6 = 0; alias suppression kept)0.1510.5073.6%
Equal route weights0.1120.4885.8%
Adaptive route weights (RAMP)0.0810.5014.1%

The penalties remove recovery that alias suppression leaves behind. Adaptive allocation supplies 41–44% of that reduction in each of the three access conditions (question only, image with a neutral prompt, question+image), at about one-third of the utility cost of equal weights. With shuffled weights (the same weight vectors assigned to other identities) as a third control, the ordering adaptive > equal > shuffled holds on all 18 recovery, preservation, and modality-gap metrics in Tables 12, 15, and 16. These are point estimates: we claim neither routing-specific significance nor an isolated outcome effect of the activation gate.

3. Robustness without retuning. One coefficient set is used for CLEAR and all three UMU-Bench forget rates. Across every tested change of route-loss weight, preservation strength, and routing temperature, CLEAR question+image recovery stays within 0.069–0.105, less than half that of NPO trained with all of RAMP's extra supervision (0.226). On UMU-Bench 10% (one seed per variation), every variation still has lower UC and higher UT than every published baseline.

Planned revisions. State the checkpoint/test separation and the provenance of the evaluation questions (D.3/E.4); narrow Contribution 2 to target-dependent allocation; qualify UMU utility as aggregate UT; add limitations on the single backbone, identity-level (any-hit) compromise, and calibration of the training score; add representative human-audited outputs; release the Stage-0 artifacts; and fix presentation issues (appendix tables label RAMP as "TRACE", a broken table reference in App. E.1, and the illustrative aliases in Fig. 1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.