acceptodds
Under review as a conference paper at ICLR 2027

RevokeBench: Testing Selective In-Context Revocation

Abstract

A language model can comply with an in-context request to revoke a fact on a direct query while losing valid neighboring information or later emitting the invalid value. We introduce RevokeBench, a benchmark of ERASE, REPLACE, and SUPERSEDE state transitions with paired instruction and physically edited contexts. It measures current-state accuracy, retention of valid facts, and literal forbidden-value emission as separate obligations. On 960 test episodes, none of four pre-specified aggregate non-direct contrasts in the two-checkpoint confirmatory study survives Holm correction. A separately specified ten-checkpoint panel nevertheless reveals an obligation-specific pattern: instruction-mediated revocation has 2.31 and 12.92 percentage points lower same-entity retention on IID and held-out grammar-and-domain OOD splits, respectively, and 16.47 and 25.12 points higher macro forbidden-value emission than canonical editing. Human judgments corroborate OOD retention loss. An external memory-operation bridge, though coverage-limited, further shows that current-state accuracy and retention need not move together. No-revocation controls establish broad elicitation for recovery and premise attacks, but the indirect-choice attack is largely ineffective. Qwen3-8B attention interventions and suffix recomputation provide scoped evidence about prefill sensitivity and edit-aware execution. Thus direct compliance alone does not establish selective in-context revocation.

Primary area:
datasets and benchmarks
Keywords:
in-context revocation, large language models, selective forgetting, benchmark evaluation, retained utility, forbidden-value emission

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_c6JhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RevokeBench, a benchmark for testing whether an LLM can selectively revoke information in its context. It studies three operations: ERASE, REPLACE, and SUPERSEDE. The paper compares two settings. In the instruction setting, the old information remains in the context and the model receives a revocation instruction. In the canonical setting, the old information is directly edited or removed from the context.

The main point is that correct answers to direct questions are not enough to show successful revocation. The paper separately evaluates current-state accuracy, retention of other valid information, and later emission of the invalid value. The experiments include 960 episodes, multiple model checkpoints, human evaluation, attention interventions, an external dataset study, and suffix recomputation.

Critical strengths
  1. The paper studies an important and well-motivated problem. The separation between current-state accuracy, retained utility, and forbidden-value emission is simple but useful. These three aspects can behave very differently, so evaluating only direct accuracy is not enough.

  2. The evaluation is careful and transparent. The paper reports that the main confirmatory tests do not pass Holm correction. It also checks whether each attack can produce the value when the value is still valid. The authors report negative and mixed results instead of only showing results that support one conclusion. This makes the paper more convincing to me.

Critical weaknesses
  1. FVER does not separate using a value from mentioning a value. For example, if a model says that an old value was revoked and should not be used, this still counts as emission. Also, the old string is present in the instruction condition but removed in the canonical condition. Therefore, the FVER difference should be interpreted as literal re-emission risk, not necessarily as the model believing or using the old state.

  2. Some average results hide important differences. ERASE, REPLACE, and SUPERSEDE can have different or opposite effects. The Choice attack almost never produces the value even in the positive control, but it is still included in the macro FVER. Also, the external MemOps experiment is small and shows a different direction for retention. These issues limit how broadly the main findings can be stated.

Questions for authors
  1. Can the authors separate FVER cases into two types: using or asserting the invalid value, and only mentioning it to reject or discuss it? Do the main results remain after removing mention-only cases?

  2. Can the authors report macro FVER without the Choice attack? The elicitation test suggests that this attack has very little measurement power.

  3. Can the authors report the main results separately for ERASE, REPLACE, and SUPERSEDE, with uncertainty estimates in the main paper?

  4. The global-history-drop setting has relatively high non-direct accuracy but almost zero retained utility. Does this mean that some non-direct questions can be answered mainly from the answer format or the model's prior knowledge, without using the supplied state?

3 comments

Thank you for your careful review and your questions about emission semantics, attack validity, operator differences, and P2. We address each concern below.

W1/Q1 - FVER measures literal emission; the requested semantic-use decomposition remains unresolved. FVER records complete forbidden-answer atoms, including quotations and rejections. A validated mention-only versus asserted-use partition is not available, so the requested semantic decomposition remains unresolved.

The additional response-form analysis distinguishes exact forbidden-atom answers from other emitting responses. On the fixed ten-checkpoint panel, discarding every non-exact instruction emission while retaining every canonical emission leaves instruction-minus-canonical differences of +11.52 pp IID and +20.44 pp OOD, using 480 shared episodes per split. Thus non-exact instruction responses cannot fully account for the observed literal-emission gap.

The source record remains in instruction contexts and is removed by canonical editing. These results measure literal output exposure under those paired representations. Exact forbidden-atom output does not independently establish semantic use of invalid state.

W2/Q2 - The emission contrast persists when Choice is excluded. Averaging Recovery, Premise, and Derived gives instruction-minus-canonical macro FVER differences of +21.96 pp IID (95% episode-bootstrap CI [20.51, 23.42]) and +33.44 pp OOD ([31.88, 35.01]) across the fixed ten-checkpoint panel. Bootstrap resampling preserves the pairing of the 480 shared episodes per split across checkpoints.

The positive-control results identify weak Choice elicitation, so low Choice emission cannot establish resistance. The positive three-attack contrasts show that including Choice does not create the reported emission gap. We retain the original four-attack endpoint and report this exclusion as a sensitivity analysis.

W2/Q3 - Operator-specific point estimates have different signs. ERASE favors instruction in all four anchor-model/split combinations; REPLACE and SUPERSEDE favor canonical editing in their point estimates, with several intervals including zero. The descriptive paired differences in original P2-P4 accuracy are below. Each cell contains 160 episodes; differences are instruction minus canonical in percentage points, with 95% episode-bootstrap intervals.

Split / checkpointERASEREPLACESUPERSEDE
IID / Qwen3-8B+25.63 [20.83, 30.21]-8.13 [-11.25, -5.00]-16.88 [-21.67, -12.29]
IID / Qwen2.5-14B+15.42 [10.63, 20.21]-0.42 [-3.33, 2.71]-7.92 [-12.50, -3.54]
OOD / Qwen3-8B+30.21 [25.21, 35.21]-4.17 [-7.71, -0.83]-15.21 [-19.58, -10.83]
OOD / Qwen2.5-14B+7.50 [1.04, 13.96]-2.29 [-6.25, 1.46]-0.42 [-4.58, 3.75]

The main-text operator results in Section 6.2 now include these contrasts and their uncertainty. P5 retention is lower under instruction in all twelve operator/model/split point estimates, although not every descriptive interval excludes zero. The MemOps bridge marks a further boundary: the retained-utility point estimate is +3.33 pp on its 30/120 executable cases. The synthetic operator results and external bridge therefore support population-specific comparisons.

Q4 - P2 has a constant gold label, and removing it reverses the pooled current-state comparison. The original global-history-drop control retains active replacements for REPLACE and SUPERSEDE. A separate true-empty-context check keeps the query and answer instructions but removes all memory records. It obtains 55.56% non-direct accuracy: 100% on P2, whose gold label is always INVALID, and 33.33% each on P3 and P4. Part of the pooled score is therefore attainable without supplied state. This baseline does not identify which shortcut, if any, produced the paired instruction-canonical differences.

Under the original cap-as-failure convention, removing P2 changes the fixed ten-checkpoint comparison as follows:

SplitOriginal P2-P4 differenceP3-P4 differenceP3-P4 95% CI
IID-3.51 pp+5.46 pp[3.42, 7.61]
OOD-3.28 pp+5.82 pp[3.64, 8.09]

This reversal materially changes the interpretation of the pooled endpoint. The manuscript now identifies P2 as one-sided invalid-claim rejection, distinguishes global drop from true-empty context, and reports the P3-P4 sensitivity in Section 6.1. The four original Holm-corrected primary non-direct tests remain non-rejections; the additional diagnostics are reported separately.

P5 retention and attack emission are independently scored, and their estimates remain numerically unchanged after P2 exclusion. The existing ten-checkpoint P5 differences are -2.31 pp IID and -12.92 pp OOD (Table 11); blinded human evaluation also supports OOD retention loss in the two anchor models (Table 24). In the post-review P2-exclusion sensitivity analysis on the fixed synthetic panel, instruction has higher aggregate P3-P4 accuracy under the cap-as-failure convention. The separately scored results show lower same-entity retention and higher literal emission. These findings show opposing aggregate differences across the obligations.

A separate post-review balanced VALID/INVALID diagnostic reduces the constant-INVALID baseline to 50%. Under its recorded-score convention, IID instruction-minus-canonical differences are +8.85 pp for Qwen3-8B and +16.56 pp for Qwen2.5-14B; both OOD confidence intervals include zero. All 69 capped outputs occur under Qwen2.5 canonical. These results use a different scoring convention from the cap-as-failure P2-removal analysis and are presented as a supplementary diagnostic.

AuthorsOpenReview ↗

Thank you and follow-up

Thank you for your careful review and support for the paper. Your questions led to clearer reporting of operator-specific results, the analysis excluding the Choice attack, the P2 sensitivity analysis. We have also made explicit that FVER measures literal emission and that the semantic-use distinction remains unresolved. We hope this addresses your concerns about the interpretation of our findings. Please let us know if anything further needs clarification.

Reviewer_GpCeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

RevokeBench evaluates whether a language model can selectively honor a validity-changing event over information in its prompt. It defines ERASE, REPLACE, and SUPERSEDE operations and contains 960 test episodes, split equally between IID and held-out grammar-and-domain OOD settings. Authors leverage RevokeBench to empirically investigate whether language models can selectively honor validity-changing events.

Critical strengths

S1: Novel and practically grounded research topic.

The paper addresses a creative and timely problem—selective in-context revocation—that closely mirrors real-world user interaction scenarios, where users frequently correct, update, or retract previously provided information during multi-turn conversations. This practical relevance distinguishes the work from more abstract knowledge-editing benchmarks and strengthens its potential impact.

S2: Comprehensive experimental design.

The evaluation is thorough, spanning multiple model checkpoints, IID and OOD generalization settings, human validation, attention interventions, and an external memory-operation bridge. The breadth of the experimental protocol provides a well-rounded assessment of the proposed benchmark.

Critical weaknesses

W1: Writing needs substantial improvement.

While the technical contribution is sound, the manuscript is under-polished in ways that materially raise the reader's burden and reduce the verifiability of the reported results. Many sentences lack clear logical connections to their surrounding context. A thorough pass for coherence and paragraph-level structure is strongly recommended. The experimental sections present numerical results without first stating each metric's formal definition, value range, the interpretation of an increase versus a decrease, and why the metric serves as an appropriate proxy for the claimed capability. Readers unfamiliar with the specific statistical conventions adopted by the authors are left to guess their precise meaning.

W2: Dataset construction is described too briefly.

The primary contribution of this work is the construction of a new benchmark; however, Section 4 does not provide sufficient detail for the reader to clearly understand how the dataset was built or what the resulting data looks like in practice. Key aspects, such as the generation pipeline, concrete example instances, are either missing or only mentioned in passing. This makes it difficult to assess the dataset's reliability and to reproduce or extend the benchmark.

Questions for authors

In fact, several existing benchmarks already incorporate knowledge updates and abstention over interaction histories, such as [1] and [2]. What specifically distinguishes RevokeBench from these prior efforts? The authors should more clearly articulate the unique positioning and incremental contribution of their benchmark relative to these works, rather than treating them only as peripheral related work.

[1] Do LLMs Forget What They Should? Evaluating In-Context Forgetting in Large Language Models.

[2] LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

2 comments

Thank you for your careful review and your suggestions on metric definitions, benchmark construction, and the relation to prior work. We address each concern below.

W1 - Each metric corresponds to an explicit behavioral obligation. Every scored response has a specified answer contract and a score in [0, 1]. The probe roles and score directions are:

Probe or metricEvaluated behaviorBetter
P0-P1Direct and paraphrased current-target correctnessHigher
P2Rejection of a stated invalid claim; gold is always INVALIDHigher
P3-P4Policy choice and derived-answer correctnessHigher
P5Retention of a valid same-entity factHigher
P6-P7Retention of a graph-neighbor fact and an unrelated factHigher
FVER under A0-A3Emission of a registered forbidden-answer atomLower

The original non-direct endpoint averages P2-P4; aggregate retained utility averages P5-P7. P2 is a one-sided rejection probe, not balanced validity discrimination. Forbidden-answer contracts include historical values and incorrect policy or derived labels. FVER counts literal output, including quotation or rejection, and does not measure semantic belief.

None of the four original primary non-direct contrasts survives Holm correction. The retention and emission analyses address separate obligations and testing families. RevokeBench's central result is that these outcomes require separate evaluation because their representation comparisons can differ. Section 4 now gives the probe definitions and metric directions before the numerical results; Section 5 distinguishes the testing families.

W2 - The construction pipeline specifies the paired contexts, answer keys, and validation checks before model evaluation. State records, relations, and an ERASE, REPLACE, or SUPERSEDE operator determine the post-event state. The pipeline then generates the instruction and edited contexts, answer contracts, probes, and deterministic validation checks.

Seeded selection, independent of model outputs, draws training/development/IID/OOD splits of 720/240/480/480 episodes from 5,760 indexed candidates: two profiles × three operators × sixteen structural cells × three content families × twenty slots. Each test split contains 160 episodes per operator. The sixteen structural cells cross surface style, target position, semantic similarity, and reasoning depth.

The construction verifier reproduced the candidate index, all four frozen splits, and the audit sample byte-for-byte against stored hashes. This verifies deterministic reproduction of the frozen construction outputs; it is not an end-to-end rerun of GPU predictions. In the human construction audit, two annotators assessed four validity criteria for each of 240 episodes, yielding 960/960 affirmative paired criterion decisions.

A concrete ERASE episode records a registry's signal color as coral, its valid meeting day as Monday, and its audit code as V95129. Instruction retains the source record and explicitly revokes the color; canonical editing removes only the revoked color record. Both conditions require UNKNOWN for the current color and derived route, while Monday and V95129 remain answerable.

Section 4.2 now presents this pipeline and worked example together. We have also corrected the probe and control descriptions in Section 4 and Appendix A: P2 tests invalid-claim rejection, P3 tests policy choice, and global history drop retains active replacements for REPLACE and SUPERSEDE.

Q1 - RevokeBench contributes a paired evaluation of explicit post-event state contracts. ICF-Bench studies selective forgetting and retention in NoForget/Forget settings; LongMemEval evaluates knowledge updates and abstention over interaction histories. RevokeBench connects three elements within each episode:

  1. An explicit ERASE, REPLACE, or SUPERSEDE transition with specified post-event answers.
  2. Instruction and physically edited contexts representing the same intended current state.
  3. Separate current-state, retained-neighbor, and attack-specific forbidden-answer probes.

This design supports controlled comparisons of how a validity-changing event's representation affects each obligation. The contribution is the evaluation unit and the comparisons it enables. The experiments establish results on RevokeBench's populations; they do not establish head-to-head performance superiority over the prior benchmarks.

AuthorsOpenReview ↗

Thank you and follow-up

Thank you for your constructive review and support for the paper. Your suggestions helped us improve the metric definitions, explain the construction pipeline with a worked example, clarify RevokeBench’s contribution relative to prior benchmarks. We hope these revisions address your concerns about clarity and verifiability. We would appreciate your confirmation or any remaining points you would like us to address.

Reviewer_2gWfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the problem of in‑context revocation for large language models: giving instructions within‑context to require the model to invalidate a piece of factual information. In practice, although models can comply with revocation directives for direct queries, they frequently suffer from two types of failures: collateral forgetting, where valid information about the same entity is unintentionally damaged, and the re‑emission of invalid revoked values in subsequent reasoning or adversarial probing. The authors construct the RevokeBench evaluation benchmark, designing three types of state transitions: ERASE, REPLACE, and SUPERSEDE. It employs paired contexts of instruction‑mediated revocation (keeping original records plus revocation instructions) and canonical editing (physically removing invalid content as a counterfactual reference), and defines three independent evaluation metrics: current‑state accuracy, retained utility for valid facts, and forbidden‑value emission rate (FVER).

Critical strengths

Past benchmarks for in‑context forgetting often only judge correctness on direct‑query responses. This paper points out that correct answers to direct queries do not equate to successful selective revocation. It decomposes the task into three non‑substitutable obligations: correct current‑state output, preservation of valid information, and suppression of re‑emergence of old invalid values, correcting the pitfalls of single‑metric evaluation.

It conducts a broad multi‑model evaluation over ten checkpoints covering dense, Mixture‑of‑Experts (MoE), and multimodal architectures, rather than restricting experiments to a single model family. This enables observations of both the consistency of the phenomenon across models and model‑specific individual discrepancies.

Critical weaknesses

Canonical editing serves as a counter‑factual reference, yet it is implicitly treated as an “ideal gold standard” in parts of the experiments. The paper also presents counterexamples where instruction‑mediated revocation yields better performance in certain ERASE scenarios. Nevertheless, it lacks systematic analysis of the circumstances under which physical editing itself can introduce new errors, and the discussion regarding the limitations of canonical editing remains insufficient.

Out of the original 120 dialogue samples, only 30 usable samples remain after manual content filtering. The operator distribution is imbalanced; hence this subset can only serve as secondary supporting evidence and cannot establish that the observed phenomena hold generally for real‑world dialogue scenarios.

P5 retained utility, FVER (forbidden‑value emission rate), and non‑direct accuracy are reported separately. These three metrics often exhibit trade‑offs against one another. The paper does not propose a composite metric for comparing different models or prompting strategies, which creates obstacles for future researchers conducting comparative studies.

Questions for authors

Mechanistic experiments reveal that masking invalid tokens during the prefill phase improves performance, whereas masking after decoding has nearly no effect. Does this imply that most contamination stemming from revoked facts occurs during the prefill context‑encoding stage? Can this explanation be generalized to other models?

The marker‑free editing removes the edit header while preserving the state produced by canonical editing. How large is the performance gap between marker‑free editing and canonical editing? What fraction of the behavioral difference can be attributed to the prompt header marker?

2 comments

Thank you for your careful review and your questions about the canonical reference, joint evaluation, and intervention controls. We address each concern below.

W1 - Canonical editing represents the post-event state; model performance depends on its representation. On the original P2-P4 endpoint, instruction outperforms canonical editing for ERASE in all four anchor-model/split combinations, while REPLACE and SUPERSEDE favor canonical editing in their point estimates. For Qwen3-8B on IID ERASE, the 480 matched probe pairs from 160 episodes contain 224 instruction-only successes and 101 canonical-only successes. In an audited example, the required derived answer is UNKNOWN: instruction returns UNKNOWN, whereas canonical editing returns route-V.

ERASE exposes the relevant distinction: an explicit declaration that a field is unavailable and the omission of its record encode the same stipulated state, yet models can interpret them differently. Canonical editing therefore supplies a state reference whose behavioral consequences must be measured. The operator contrasts characterize this representational boundary; they do not identify a unique internal cause.

W2 - The external bridge shows a different trade-off among the three obligations. On the executable MemOps subset, instruction-minus-canonical differences are -8.33 pp for current-state accuracy, +3.33 pp for retained utility, and +6.00 pp for registered forbidden-answer emission. The retention point estimate has the opposite sign to the synthetic aggregate and reinforces the need to report the outcomes separately.

The evaluated subset contains 30 of 120 screened candidates: 9 Forget and 21 Update cases. The content gate assessed construction validity; model performance was not an exclusion criterion. As specified in Appendix F.1, the source is model-generated task-oriented dialogue. These results characterize obligation separation on the executable bridge population; the 25% coverage and operator imbalance preclude representative real-dialogue conclusions.

W3 - Joint success provides an all-obligations comparison for the fixed probe bundle. An episode passes when every P0-P7 answer is correct and none of the four attack responses emits its registered forbidden answer. This criterion combines separately generated responses; it does not measure a single dialogue rollout. The additional ten-checkpoint analysis gives:

SplitInstructionCanonicalDifference (pp)95% CI
IID26.75%40.94%-14.19[-16.63, -11.69]
OOD9.85%30.42%-20.56[-22.75, -18.35]

Excluding the one-sided P2 probe leaves differences of -13.08 pp IID and -20.44 pp OOD. These rates use recorded scorer decisions; a separate cap-as-failure check preserves the direction. Confidence intervals resample the 480 shared episodes per split jointly across all ten checkpoints. Joint success summarizes satisfaction of this probe bundle, while the component metrics identify its trade-offs.

Q1 - Full-prefill masking has a larger effect than first-answer-inclusive masking on Qwen3-8B. Relative to no-hook instruction, first-answer-inclusive masking changes non-direct accuracy by +1.94/+1.74 pp on IID/OOD; full-prefill masking changes it by +12.50/+7.99 pp. This comparison supports sensitivity to prompt computation beyond first-token access. It does not localize the dominant source of contamination.

Additional fixed-budget checkpoint checks show heterogeneous effects and no consistent P3-P4 benefit after removing P2. Generation truncation, particularly for the tested Llama repack, limits these checks. The supported finding is computational sensitivity in the original Qwen3-8B setting; cross-family improvement remains unestablished.

Q2 - Header-removal contrasts are endpoint-dependent. With edited records held fixed, marker-free-minus-canonical non-direct differences are -1.18/-1.39 pp for Qwen3/Qwen2.5 on IID and -0.14/-0.49 pp on OOD. Retained-utility differences range from +0.14 to -3.06 pp across the four cells. These contrasts estimate sensitivity to header removal while holding the edited records fixed. They do not identify a causal percentage of the full instruction-canonical gap.

AuthorsOpenReview ↗

Thank you and follow-up

Thank you for your thoughtful review and support for the paper. Your feedback helped us clarify the role of canonical editing, provide a joint-success analysis, and define the scope of the intervention findings. We hope our response has addressed your concerns. As the discussion draws to a close, please let us know if any points still need clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.