- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RevokeBench, a benchmark for testing whether an LLM can selectively revoke information in its context. It studies three operations: ERASE, REPLACE, and SUPERSEDE. The paper compares two settings. In the instruction setting, the old information remains in the context and the model receives a revocation instruction. In the canonical setting, the old information is directly edited or removed from the context.
The main point is that correct answers to direct questions are not enough to show successful revocation. The paper separately evaluates current-state accuracy, retention of other valid information, and later emission of the invalid value. The experiments include 960 episodes, multiple model checkpoints, human evaluation, attention interventions, an external dataset study, and suffix recomputation.
Critical strengths
-
The paper studies an important and well-motivated problem. The separation between current-state accuracy, retained utility, and forbidden-value emission is simple but useful. These three aspects can behave very differently, so evaluating only direct accuracy is not enough.
-
The evaluation is careful and transparent. The paper reports that the main confirmatory tests do not pass Holm correction. It also checks whether each attack can produce the value when the value is still valid. The authors report negative and mixed results instead of only showing results that support one conclusion. This makes the paper more convincing to me.
Critical weaknesses
-
FVER does not separate using a value from mentioning a value. For example, if a model says that an old value was revoked and should not be used, this still counts as emission. Also, the old string is present in the instruction condition but removed in the canonical condition. Therefore, the FVER difference should be interpreted as literal re-emission risk, not necessarily as the model believing or using the old state.
-
Some average results hide important differences. ERASE, REPLACE, and SUPERSEDE can have different or opposite effects. The Choice attack almost never produces the value even in the positive control, but it is still included in the macro FVER. Also, the external MemOps experiment is small and shows a different direction for retention. These issues limit how broadly the main findings can be stated.
Questions for authors
-
Can the authors separate FVER cases into two types: using or asserting the invalid value, and only mentioning it to reject or discuss it? Do the main results remain after removing mention-only cases?
-
Can the authors report macro FVER without the Choice attack? The elicitation test suggests that this attack has very little measurement power.
-
Can the authors report the main results separately for ERASE, REPLACE, and SUPERSEDE, with uncertainty estimates in the main paper?
-
The global-history-drop setting has relatively high non-direct accuracy but almost zero retained utility. Does this mean that some non-direct questions can be answered mainly from the answer format or the model's prior knowledge, without using the supplied state?
3 comments
Q4 - P2 has a constant gold label, and removing it reverses the pooled current-state comparison. The original global-history-drop control retains active replacements for REPLACE and SUPERSEDE. A separate true-empty-context check keeps the query and answer instructions but removes all memory records. It obtains 55.56% non-direct accuracy: 100% on P2, whose gold label is always INVALID, and 33.33% each on P3 and P4. Part of the pooled score is therefore attainable without supplied state. This baseline does not identify which shortcut, if any, produced the paired instruction-canonical differences.
Under the original cap-as-failure convention, removing P2 changes the fixed ten-checkpoint comparison as follows:
| Split | Original P2-P4 difference | P3-P4 difference | P3-P4 95% CI |
|---|---|---|---|
| IID | -3.51 pp | +5.46 pp | [3.42, 7.61] |
| OOD | -3.28 pp | +5.82 pp | [3.64, 8.09] |
This reversal materially changes the interpretation of the pooled endpoint. The manuscript now identifies P2 as one-sided invalid-claim rejection, distinguishes global drop from true-empty context, and reports the P3-P4 sensitivity in Section 6.1. The four original Holm-corrected primary non-direct tests remain non-rejections; the additional diagnostics are reported separately.
P5 retention and attack emission are independently scored, and their estimates remain numerically unchanged after P2 exclusion. The existing ten-checkpoint P5 differences are -2.31 pp IID and -12.92 pp OOD (Table 11); blinded human evaluation also supports OOD retention loss in the two anchor models (Table 24). In the post-review P2-exclusion sensitivity analysis on the fixed synthetic panel, instruction has higher aggregate P3-P4 accuracy under the cap-as-failure convention. The separately scored results show lower same-entity retention and higher literal emission. These findings show opposing aggregate differences across the obligations.
A separate post-review balanced VALID/INVALID diagnostic reduces the constant-INVALID baseline to 50%. Under its recorded-score convention, IID instruction-minus-canonical differences are +8.85 pp for Qwen3-8B and +16.56 pp for Qwen2.5-14B; both OOD confidence intervals include zero. All 69 capped outputs occur under Qwen2.5 canonical. These results use a different scoring convention from the cap-as-failure P2-removal analysis and are presented as a supplementary diagnostic.
Thank you and follow-up
Thank you for your careful review and support for the paper. Your questions led to clearer reporting of operator-specific results, the analysis excluding the Choice attack, the P2 sensitivity analysis. We have also made explicit that FVER measures literal emission and that the semantic-use distinction remains unresolved. We hope this addresses your concerns about the interpretation of our findings. Please let us know if anything further needs clarification.
Thank you for your careful review and your questions about emission semantics, attack validity, operator differences, and P2. We address each concern below.
W1/Q1 - FVER measures literal emission; the requested semantic-use decomposition remains unresolved. FVER records complete forbidden-answer atoms, including quotations and rejections. A validated mention-only versus asserted-use partition is not available, so the requested semantic decomposition remains unresolved.
The additional response-form analysis distinguishes exact forbidden-atom answers from other emitting responses. On the fixed ten-checkpoint panel, discarding every non-exact instruction emission while retaining every canonical emission leaves instruction-minus-canonical differences of +11.52 pp IID and +20.44 pp OOD, using 480 shared episodes per split. Thus non-exact instruction responses cannot fully account for the observed literal-emission gap.
The source record remains in instruction contexts and is removed by canonical editing. These results measure literal output exposure under those paired representations. Exact forbidden-atom output does not independently establish semantic use of invalid state.
W2/Q2 - The emission contrast persists when Choice is excluded. Averaging Recovery, Premise, and Derived gives instruction-minus-canonical macro FVER differences of +21.96 pp IID (95% episode-bootstrap CI [20.51, 23.42]) and +33.44 pp OOD ([31.88, 35.01]) across the fixed ten-checkpoint panel. Bootstrap resampling preserves the pairing of the 480 shared episodes per split across checkpoints.
The positive-control results identify weak Choice elicitation, so low Choice emission cannot establish resistance. The positive three-attack contrasts show that including Choice does not create the reported emission gap. We retain the original four-attack endpoint and report this exclusion as a sensitivity analysis.
W2/Q3 - Operator-specific point estimates have different signs. ERASE favors instruction in all four anchor-model/split combinations; REPLACE and SUPERSEDE favor canonical editing in their point estimates, with several intervals including zero. The descriptive paired differences in original P2-P4 accuracy are below. Each cell contains 160 episodes; differences are instruction minus canonical in percentage points, with 95% episode-bootstrap intervals.
The main-text operator results in Section 6.2 now include these contrasts and their uncertainty. P5 retention is lower under instruction in all twelve operator/model/split point estimates, although not every descriptive interval excludes zero. The MemOps bridge marks a further boundary: the retained-utility point estimate is +3.33 pp on its 30/120 executable cases. The synthetic operator results and external bridge therefore support population-specific comparisons.