acceptodds
Under review as a conference paper at ICLR 2027

"Nothing to See Here”: Unintended Disclosure through Revision Traces of LLM Deliverables

Abstract

Large language model (LLM) assistants increasingly help users draft content for third-party recipients. During private drafting, the user or the model may introduce an item and later remove or replace it. The model may remove the item from the intended content but reveal it again when stating the edit. We call such statements revision traces. For example, after a user removes the password before sharing a configuration file, the model may delete it but leave a comment saying, "Removed the password 'No****4!' as requested." A third-party recipient who sees only the delivered file can therefore recover the withdrawn password from the comment. In an in-the-wild analysis of three public conversation corpora, we identify 26,753 revision requests, of which 2,363 (8.8%) leave revision traces. We study them in greater depth under controlled conditions by introducing , a benchmark of 100 tasks across five scenarios with a conversation track and an agent track. We measure trace occurrence, withdrawn-item recovery, trace position, and required-content retention. Across six models, about half of the deliverables in both tracks state the edit after a revocation, and a reader that sees only the deliverable can recover the withdrawn item from about 13% of them. Telling the model that its entire reply will be forwarded to the recipient still leaves revision traces in 36.4% of the deliverables. We compare prompt defenses and a delivery boundary, and propose an output-side filter that sharply reduces recovery with little loss of required content. We believe our work can benefit efforts to understand and mitigate unintended disclosure in LLM interactions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Revision Traces, Unintended Disclosure, Privacy Evaluation, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_raDrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies revision traces: statements in an LLM’s output that an item was removed, excluded, or replaced during private drafting, which can reveal the withdrawn item to a third party who receives the output. The motivating example is a model that removes a password from a configuration file but leaves a comment naming it. The authors analyze three public conversation corpora (WildChat, ShareChat, DevGPT) and report that 8.8% of 26,753 revision requests leave a trace. They then introduce RevLeakBench, 100 tasks across five scenarios adapted from existing benchmarks, with a conversation track and an agent track, and three revision situations (revocation, replacement, model-added).

Critical strengths

S1. The phenomenon is real and easy to grasp. The password example in Figure 1 is compelling.

Critical weaknesses

W1. Are revision traces actually harmful? The paper treats revision traces as dangerous in themselves, but most of them appear to be innocuous. Of the three disclosure levels introduced in section 5.2, only identifying traces are a real disclosure; descriptive and process-only traces reveal little or nothing. The motivating password example is not representative of the paper's own evidence. The in-the-wild public-repository example (Section 3, l. 176-181) is a model stating "I have removed the onclick attributes from your buttons", which reveals nothing secret. Most examples in Appendix A are similar. In the benchmark, security credentials are only 6 of the 200 withdrawn items, and the reader recovers none of them (Table 20). The paper should show how often a trace discloses something that actually matters, rather than how often a trace exists.

W2. The deliverable should be the body, not the full reply. This is my main methodological concern. Figure 4 and Section 4.3 make clear that the authors treat the entire model reply (preface, body, afterword) as the deliverable, and the headline numbers are computed on it. The justification (L182-184) is that users may forward the reply as-is. If a user copies the model's conversational remarks into a message to a third party, the error lies with the user, not with the model. The relevant measure is what ends up in the content meant for the recipient, and in that case the rate of leakage drops from 56.9% to 4.6%. All the main claims in the paper should be about the Body.

W3. Results by disclosure level are mostly missing from the main text. Given W1, the identifying level is the one that matters, yet the main text reports it in a single sentence (L322). The breakdown by level is only in Appendix J (Tables 12 to 14) and Table 18. The central result of the paper should arguably be the rate of identifying traces inside the body, per model.

Minor

m1. The detection pipeline is not shown where it is used. Section 3 relies on a "keyword rule" (L151) to select revision requests, which I assume is a regex, but it is never specified, not even in the appendix. The rule-based scanner (L288-290) is also described only in words (Appendix E). If these are regular expressions, they should be reported at least in the appendix. Furthermore, Section 3 applies "the same detectors that Section 5.2 defines" (L155-156), so the reader meets the in-the-wild results without knowing how traces or categories are defined. The revision categories (revocation, replacement, other) are similarly used in Section 3 before being properly introduced. The definitions should be moved before their first use.

m2. The validity of RevLeakBench depends on how realistic its tasks are, yet the main text never shows a complete task. Figure 4 shows a partial reply only. At least one full example (material, conversation, withdrawn item, deliverable) should appear in the main text.

m3. Six models from three providers is a small panel. Table 2 does not state which models it reports; I infer from n=240 that it pools all six. It is also unclear why a full identifier is given only for GPT-5.6.

m4. Presentation: (m4a) The figures are too small to be read, in particular Figures 3 and 5. (m4b) The introduction is long and repetitive: the research questions are stated, then restated with results, and the contributions largely repeat them. (m4c) The tweet anecdote (L52-56) adds nothing and is not appropriate for a research paper.

Questions for authors

All of the above, and also:

Q1. What is the rate of identifying traces inside the body, per model and per information type?

Q2. Of the 2,363 in-the-wild traces, how many disclose sensitive information (credentials, personal information) rather than a description of the edit?

6 comments
AuthorsOpenReview ↗

Response to Reviewer raDr (1/3)

Thank you for the care in reviewing our work, in particular the line-level pointers and the time spent reading the appendix tables, and for recognizing that the phenomenon is real and easy to grasp and that the password example in Figure 1 is compelling. We note that your main concerns relate to whether revision traces disclose information that matters, whether the body or the full reply should count as the deliverable, and the results by disclosure level. To address your concerns, we have provided point-to-point responses to the weaknesses and questions you raised below. Besides, we have carefully revised our manuscript and highlighted the changes in blue. We have also polished the main text for readability.

W3&Q1: Results by Disclosure Level in the Main Text

Thank you for your suggestion. We have moved the per-model results into the main text as Table 3 in Section 5.2, which replaces the per-model heatmap and reports full-reply traces, body traces confirmed by two annotators, identifying traces, and recovery for each model and track. Every model leaves a full-reply trace in at least 29.5% of its revocation deliverables, and recovery reaches 24.5% for DeepSeek-V4-Flash in the conversation track and 29.5% for MiniMax-M3.1-Flash in the agent track.

Regarding Q1, the scanner places 43 of the 424 identifying traces of the eight models inside the body. In the conversation track they occur in 2 of 1,598 deliverables, while in the agent track, where the model writes the document itself, they occur in 2.6% of deliverables and in 9.0% for MiniMax-M3.1-Flash. By information type, personal information has the highest rate inside the body, at 2.1% over the two tracks. These rates are lower bounds, because the identifying level of the scanner has a recall of 0.45 against the human labels. The other identifying traces sit in the sentences around the body, which still reach the recipient when the reply is forwarded (see our response to W2). We have added these rates by model and by information type to Appendix J of the revision.

m1: Detection Rules and Order of Definitions

Thank you for your careful review. The keyword rule includes a regular expression that matches removal and replacement verbs, negated requests such as "don't mention", and separable phrases such as "leave ... out", which Appendix A now gives. The rule-based scanner is also a set of regular expressions, and Appendix E now lists its rule families with their core patterns and an example for each, with the complete expressions in the supplementary material. We have also moved the definitions of revision traces, the three disclosure levels, revocation and replacement, and the two checks to the beginning of Section 3, so that the in-the-wild analysis uses them after they are defined.

AuthorsOpenReview ↗

Response to Reviewer raDr (2/3)

W2: Body or Full Reply as the Deliverable

Thank you for your insightful comment. We agree that body traces should be reported on their own. Table 3 in Section 5.2 now reports body traces confirmed by two annotators for every model and track. When only the body is delivered, as in our body-only boundary, recovery falls from 13.7% to 0.6% in the conversation track and from 13.4% to 2.9% in the agent track. We keep the full reply as the deliverable of our main results for three reasons.

First, prior privacy benchmarks evaluate the entire response that the model writes for a recipient. CIMemories [1] checks the whole message, and ConfAIde [2] tells the model that its meeting summary will be automatically sent to all attendees.

Second, we respectfully disagree that the error lies only with the user, because revision traces remain when the user tells the model what the recipient will receive. On four models, under a forwarding notice such as "I'll forward your entire reply as-is", 36.4% of deliverables still contain a trace and the reader recovers the withdrawn item from 5.6% of all deliverables. Under a system message that asks for the deliverable only, with no note, preface, or explanation, 27.5% still contain a trace and recovery is 4.7%. In this setting, the trace is the model's own addition to what the recipient receives.

Third, models also write revision traces into the delivered content itself. In at least 15 of the 1,186 in-the-wild revocations, the delivered code or document names the withdrawn item, and Ruff [3] and a Claude Code issue [4] ask coding agents to keep such history out of what they deliver. Similar to Non-Adversarial Reproduction [5], which measures text that benign users may inadvertently redistribute, we therefore report the full reply as what a recipient sees when the reply is forwarded without editing, together with the body.

In the revision, the Conditions paragraph of Section 4 states that the full reply is the deliverable as in CIMemories [1], Section 5.4 reports recovery under both instructions, and Section 6 states that our main rates apply to replies forwarded without editing.

m2: Complete Task in the Main Text

Thank you for your suggestion. Figure 4 now shows a complete task: the material excerpt, all five conversation turns, the withdrawn and replacement items, the deliverable with its preface, body, and afterword marked and the trace highlighted, and the reader's answers with and without the deliverable, with the elided parts marked in grey. Appendix I keeps a second example.

m3: Model Panel and Identifiers

Thank you for your careful reading. Table 2 reports all six models pooled, and its caption now says so. We gave the full identifier only for GPT-5.6 because its API identifier, gpt-5.6-sol, differs from the model name. Appendix E now lists every model with its identifier, access route, temperature, reasoning effort, and generation dates. We also add MiniMax-M3.1-Flash from MiniMax and Qwen3.8-Flash-Next from Alibaba, which extends the panel from six models from three providers to eight models from five providers. We evaluate the two models on the revocation conditions of all 100 tasks and add them to Table 3.

AuthorsOpenReview ↗

Response to Reviewer raDr (3/3)

W1&Q2: Harm of Revision Traces and Sensitive Information in the Wild

Thank you for your insightful comment. You are correct that our traces rarely disclose credentials: models restate a credential in only 1.4% of the credential deliverables, and the reader recovers none of them. You are also correct that the onclick example in Section 3 reveals nothing secret, and it serves only to show that a trace can reach a public repository. However, many withdrawn items are ones that prior benchmarks already treat as sensitive, such as the must-not-share items of AgentCIBench [6] and the policy-violating propositions of RedacBench [7]. For credentials, personal information, and organizational information, the reader recovers the withdrawn item from 15.8% of deliverables on average and from 30.5% for DeepSeek-V4-Flash in the conversation track. For example, GLM-5.2 tells the recipient that it will "leave the Maya IVF retrieval out", which reveals a family member's medical appointment (Appendix I). The trace sentence carries this disclosure: identifying traces account for 74.3% of all recoveries and descriptive traces for a further 11.2%, and deleting only the trace sentence reduces recovery from 303 of 1,239 deliverables to 24. The revision now reports disclosure by information type in Section 5.2.

Regarding Q2, as suggested, we perform a human annotation study on the 2,363 in-the-wild traces. Two authors independently label a stratified random sample of 331 traces (95% confidence, ±5% margin) and agree at κ = 0.88 on whether a sentence is a trace and at κ = 0.85 on information type. With sampling weights, both confirm 79.6% of the sentences as revision traces, and 48.3% name the edited item, rising to 65.0% among revocations. Of the same sentences, 1.6% (95% CI [0.4%, 3.2%]) disclose credentials or personal information, and the single credential trace names a removed token field without its value. This rate is a lower bound, because the public corpora contain only conversations that users chose to share, and WildChat [8] and ShareChat [9] were de-identified with Microsoft Presidio before release, so a personal value that a trace restated appears only as a placeholder. This limitation of public data is why RevLeakBench annotates each withdrawn item with its information type. We have added this study to Appendix A and the confirmation rate to Section 3.

m4: Presentation

Thank you for your suggestions. (a) We moved the screening funnel from Figure 3 to Appendix A, which enlarges the remaining three panels, and replaced the per-model heatmap in Figure 5 with the per-model table, which enlarges the remaining two panels. We also redrew Figure 2 so that it gives an overview of the whole study. (b) We removed the introduction paragraph that restated the results and now state each result once in the contributions, organized by research question. (c) We removed the tweet anecdote.

We thank you again, and we hope you will reconsider your original score in light of these clarifications. We would be glad to answer any further questions.

References

[1] Niloofar Mireshghallah, Neal Mangaokar, Narine Kokhlikyan, Arman Zharmagambetov, Manzil Zaheer, Saeed Mahloujifar, and Kamalika Chaudhuri. CIMemories: A Compositional Benchmark for Contextual Integrity in LLMs. In International Conference on Learning Representations (ICLR), 2026.

[2] Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory. In International Conference on Learning Representations (ICLR), 2024.

[3] Astral. Ruff: AGENTS.md. https://github.com/astral-sh/ruff/blob/main/AGENTS.md, 2026.

[4] [FEATURE] Keep development history out of code comments/docstrings by default (put it in git, not the file). https://github.com/anthropics/claude-code/issues/85130, 2026.

[5] Michael Aerni, Javier Rando, Edoardo Debenedetti, Nicholas Carlini, Daphne Ippolito, and Florian Tramèr. Measuring Non-Adversarial Reproduction of Training Data in Large Language Models. In International Conference on Learning Representations (ICLR), 2025.

[6] Anmol Goel and Iryna Gurevych. Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity? CoRR abs/2606.23189, 2026.

[7] Hyunjun Jeon, Kyuyoung Kim, and Jinwoo Shin. RedacBench: Can AI Erase Your Secrets? In International Conference on Learning Representations (ICLR), 2026.

[8] Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. WildChat: 1M ChatGPT Interaction Logs in the Wild. In International Conference on Learning Representations (ICLR), 2024.

[9] Yueru Yan, Tuc Nguyen, Bo Su, Melissa Lieffers, and Thai Le. ShareChat: A Dataset of Chatbot Conversations in the Wild. CoRR abs/2512.17843, 2025.

Reviewer_eYVHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a practical problem in which an AI removes information from a draft as the user requested, then reveals it again while explaining what it removed. The authors call this revealing behaviour a revision trace, it is a statement explaining that something was removed, excluded, or replaced.

Critical strengths
  1. The failure is clearly defined and practically relevant. The example is easy to understand, and the problem can arise during ordinary work.

  2. Their results give strong evidence that the trace sentence often carries the information enabling recovery. They also compare withdrawal with instructions that exclude an item from the beginning. Additional controls account for conversation length and when the exclusion arrives. These help show that the result is not explained simply by having a longer conversation or mentioning an exclusion late. This reveals a promising future direction.

Critical weaknesses
  1. Some conversations are constructed rather than naturally generated. This provides experimental control, but may not reproduce the full drafting behaviour of deployed assistants. This might also relate to the results on WildChat, ShareChat, and DevGPT where only 8% contain revision traces, much lower than RevLeakBench.

  2. Key information is missing for terms. For example, little description of the methodology for the "Output-side filter", but it takes one third of the results for defense. "The output-side filter removes every scanner-detected trace by construction". That's all I can find, but what is "scanner-detected"? It only shows once in the whole paper. Is it a manually designed list or sth?

Questions for authors

The filter uses benchmark-provided aliases and required-content annotations when deciding what to remove and whether to deliver the result. How would these constraints be obtained automatically in deployment? For example, the system may need to resolve “that appointment,” recognise a paraphrase of the withdrawn detail, and preserve an availability statement in the same sentence. Could you evaluate the full pipeline using automatically extracted constraints, and report how extraction errors affect disclosure, content retention, and delivery failures?

3 comments
AuthorsOpenReview ↗

Response to Reviewer eYVH (1/2)

Thank you for reviewing our work and recognizing that the failure is clearly defined and practically relevant, that the trace sentence often carries the information enabling recovery, and that our controls rule out a longer conversation or a late exclusion as explanations. We note that your main concerns relate to the constructed conversations and the gap to the in-the-wild rate, and to the description and deployment of the output-side filter. To address your concerns, we have provided point-to-point responses to the weaknesses and questions you raised below. Besides, we have carefully revised our manuscript and highlighted the changes in blue. We have also polished the main text for readability.

W1: Constructed Conversations and the Lower In-the-Wild Rate

Thank you for your insightful comment. First, the 8% and the benchmark rate use different denominators. The 8.8% covers all 26,753 revision requests, most of which reformat, expand, or correct text. Restricted to revocations, the setting of our benchmark, 36.8% of the revocations in WildChat, 32.0% in ShareChat, and 56% in DevGPT leave a trace, compared with 56.9% and 50.3% in our two tracks. On a stratified sample of 331 in-the-wild traces, two annotators confirm 85.9% of the revocation traces, close to the precision of 0.89 that the same check reaches on our benchmark, so the two rates are comparable.

Second, Non-Adversarial Reproduction [1] checks its controlled results against real WildChat conversations and notes that these come from older models. We therefore replay 291 of the 1,186 revocation events with current models, keeping each conversation verbatim up to the withdrawal. Our six models leave a trace in 40.4% of these replies, from 22.8% for GPT-5.6 to 54.5% for DeepSeek-V4-Flash, close to the 46.9% of the original replies. Following FormatSpread [2], we also replace our fixed withdrawal with the three most common named withdrawals in the corpora, "Remove X.", "Delete X.", and "Can you remove X?", and no paired change in recovery is significant. Revision traces are therefore not unique to older assistants or to our constructed conversations, and our benchmark adds what such data cannot provide, namely annotated withdrawn items for measuring recovery, matched controls, and defenses.

We have added the replay to Section 3 and a new Appendix P, and the withdrawal wordings to Section 6 and Appendix M of the revision.

W2: Missing Description of the Output-Side Filter

Thank you for your careful review. You are correct that the original submission did not describe the scanner adequately, and yes, the scanner is a manually designed set of regular-expression rules. It marks a sentence as a revision trace once the sentence contains both a removal action, such as "omitted" or "leave ... out", and an edit attribution that ties the action to the current deliverable, such as a first-person subject ("I'll leave that finding out.") or a reference to the user's request ("as you asked"). Offers without a completed edit, such as "Let me know if you'd like any of those removed", do not count, and the rules contain no task names or withdrawn items. The output-side filter deletes every sentence the scanner marks, keeps the rest of the deliverable byte for byte, and withholds the deliverable when the deletion would remove a required item that appears verbatim or would leave nothing. It makes no model call and reduces recovery from 13.7% to 1.3% in the conversation track and from 13.4% to 3.1% in the agent track.

We have rewritten the Output-Side Filter paragraph in Section 5.4 with this description, and added the scanner rules with an example for each rule family to Appendix E of the revision.

AuthorsOpenReview ↗

Response to Reviewer eYVH (2/2)

Q: Automatically Extracted Constraints in Deployment

Thank you for this constructive question. We evaluate the filter with automatically extracted constraints. We adapt the data minimizer of AirGapAgent [3] into an LLM extractor that reads only the conversation history and decides, for every item the user asked to include, remove, keep, or replace, whether it may be revealed. Items marked No become the aliases of the withdrawn item, and items marked Yes become the required content. The extractor identifies the withdrawn item in 399 of 400 conversations, the filtered output is byte for byte identical to the annotated version for 2,389 of the 2,397 deliverables, and recovery, utility, and delivery each stay within 0.4 percentage points of the annotated version.

Regarding your three cases, the anaphoric reference "that one" is resolved in every withdrawal that uses it. Paraphrases barely matter, since removing all aliases changes the filtered output for only one deliverable. The availability case ("don't say what that one is, just keep the time") is the single conversation that the extractor misreads, and the filtered output of its six deliverables is still identical to the annotated version.

We have added this result to the Output-Side Filter paragraph in Section 5.4, removed the corresponding limitation from Section 6, and added the extractor, its prompt, and the full results to a new Appendix O of the revision.

We thank you again, and we hope you will reconsider your original score in light of these clarifications. We would be glad to answer any further questions.

References

[1] Michael Aerni, Javier Rando, Edoardo Debenedetti, Nicholas Carlini, Daphne Ippolito, and Florian Tramèr. Measuring Non-Adversarial Reproduction of Training Data in Large Language Models. In International Conference on Learning Representations (ICLR), 2025.

[2] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying about Prompt Formatting. In International Conference on Learning Representations (ICLR), 2024.

[3] Eugene Bagdasarian, Ren Yi, Sahra Ghalebikesabi, Peter Kairouz, Marco Gruteser, Sewoong Oh, Borja Balle, and Daniel Ramage. AirGapAgent: Protecting Privacy-Conscious Conversational Agents. In ACM SIGSAC Conference on Computer and Communications Security (CCS), 2024.

AuthorsOpenReview ↗

Note to Reviewer eYVH

Dear Reviewer eYVH, we have uploaded a second revision, with the changes of this round shown in green. In addition to the replay of real revocations and the filter with automatically extracted constraints that you asked for, it extends the forwarding-notice condition to all eight models and evaluates the output-side filter in this setting (please see our summary of the second revision). We would be grateful if you could let us know whether our responses have addressed your concerns. If they have, we kindly ask you to consider updating your score, and we are happy to clarify anything further.

Reviewer_8BmCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

When an LLM is asked to remove or replace some sensitive information, it may unintentionally reveal the removed information in comments, allowing one to recover it. To study this, the paper introduces RevLeakBench, a benchmark that consists of 100 instances across five scenarios, with 20 instances per scenario. Each task asks a model to remove or replace some information and checks whether the revision traces remain and whether the removed information can be recovered. The evaluation is conducted over six LLMs, including two closed-source and four open-weight models under conversation and agent settings. While the paper tackles a timely problem, I have concerns about the practical significance of the revision traces, and the limited size of the benchmark.

Critical strengths

1. The paper tackles a timely problem and the preliminary experiments on real-world benchmarks provide its presence.

The paper investigates a practically sensitive problem that some important information can be disclosed without careful review of AI-modified contents. The analysis on real-world conversation dataset showed that the revision traces occur in practice.

Critical weaknesses

1. The practical significance of revision traces is unclear, and the low recovery rates may limit the usefulness of the benchmark.

I believe the paper should more clearly distinguish the revision trace and the recovery of removed information. While revision traces disclose that some information is modified or removed, they do not necessarily lead to information disclosure. Traces that merely indicate the change may have a lower privacy risk. In contrast, it becomes more serious when someone can recover sensitive information from the traces. While the abstract and introduction highlight this potential risk, the experiments mainly focus on the revision traces instead. Indeed, the actual recovery rates under the revocation settings are 13.7% and 13.4% for the conversation and agent tracks, respectively, as provided in Table 4.

2. The small size of the benchmark may weaken the generalizability of the empirical results.

RevLeakBench contains only 100 instances, with 20 instances per scenario. 20 instances per scenario might be insufficient to capture the differences in the model behavior across scenarios. Furthermore, while the benchmark is constructed from existing benchmarks, the paper does not provide a detailed justification for how representative the chosen evaluation scope is.

3. The controlled revision settings and limited prompt variations may not well represent the real-world scenario.

The benchmark consists of three types of revisions: model-added, revocation, and replacement. The experiments for revocation and replacement use relatively predefined dialogue structures. However, I am concerned that the fixed conversation may bias the observed behavior of the revision traces. For example, explicitly adding and removing an item through a fixed dialogue may encourage the model to disclose the revision. Furthermore, the evaluation within defenses could benefit from examining how different instruction tones and wording affect revision trace generation.

Questions for authors

Questions

Could the authors provide the results with more diverse revision instructions or natural multi-turn conversation in the model-added settings? For example, it would be useful to include the results using prompts that explicitly define what revision trace is and instruct a model not to include them.

Could the authors further clarify the practical significance of revision traces? I believe both revision traces and recovery are problematic only when the LLM user shares the generated outputs without carefully reviewing them. However, the impact of recovery seems much more serious than that of revision traces. In other words, given that the recovery rates are about 13%, could the authors clarify the practical value of the benchmark designed to study revision traces, even in cases where removed information cannot be recovered?

If my concerns are adequately addressed, I would be willing to update my scores accordingly.

Suggestions

While the paper reports the experience information of annotators and labeling agreements, it does not report how those annotators are paid, or how they are recruited. If the annotators are the authors of this paper themselves, then it would be helpful to clarify this information in the paper.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8BmC (1/2)

Thank you for reviewing our work and recognizing that it tackles a timely problem and that our in-the-wild analysis shows revision traces occur in practice. We note that your main concerns relate to the practical significance of revision traces relative to recovery, the size of the benchmark, and the fixed dialogue structure and prompt wording. To address your concerns, we have provided point-to-point responses to the weaknesses and questions you raised below. Besides, we have carefully revised our manuscript and highlighted the changes in blue. We have also polished the main text for readability.

W1&Q2: Practical Significance of Revision Traces and Low Recovery Rates

Thank you for your insightful comment. We agree that revision traces and recovery should be distinguished. Revision traces are the channel through which recovery happens: deleting only the trace sentence reduces recovery from 303 of 1,239 deliverables to 24, while deleting an unrelated sentence of comparable length leaves 293. Identifying and descriptive traces account for 74.3% and 11.2% of all recoveries.

The 13% average also understates the risk where it matters most. For credentials, personal information, and organizational information, recovery is 15.8% on average and 30.5% in the worst case. Following CIMemories [1], we also measure how recovery accumulates in our benchmark: after five revocations by the same user, at least one deliverable is recovered with probability 42.7% in the conversation track and 39.9% in the agent track.

Regarding Q2, we agree that both matter when users share outputs without careful review, which is also the premise of Non-Adversarial Reproduction [2]. Even when the model is told that its entire reply will be forwarded, the reader still recovers the withdrawn item from 5.6% of deliverables. A trace without recovery can still disclose something: after the user withdraws a credit counseling call from a status update to their manager, GLM-5.3 writes "I left it out of the draft since it's personal", which tells the manager that a personal matter was withheld. This is also why the benchmark measures traces: because the trace sentence carries most recoveries, removing it is an effective defense, and our output-side filter reduces recovery to 1.3% and 3.1% with little loss of required content.

Table 2 now reports recovery next to the trace rates, and we have added these results to Sections 5.2 and 5.4, a paragraph on users and model providers to Section 6, and the full analyses to Appendices K and L of the revision.

W2: Small Benchmark Size and Representativeness

Thank you for your helpful comment. Regarding size, each task runs with two withdrawn items, six models, and two tracks, giving 240 revocation deliverables per scenario and track and more than 20,000 deliverables in total. Since these deliverables share tasks, all our intervals use tasks as bootstrap clusters. To test whether 20 tasks per scenario suffice, we re-estimate the main paired effects with only k tasks per scenario, over 300 random subsets for each k, and record the share of subsets whose 95% interval excludes zero. With 10 tasks, every main effect remains significant in 98% to 100% of the subsets, while the unrelated-deletion control is significant in only 14%. For reference, ConfAIde [3] uses 20 meeting transcripts in its Tier 4, SORRY-Bench [4] uses 10 instructions per category, and AgentHarm [5] notes that domain-specific agent benchmarks typically contain 10 to 30 tasks.

Paired effect (percentage points)All 20 tasksk = 5k = 10k = 15
Trace deletion, recovery−23.0100%100%100%
Revocation vs. static exclusion, traces+16.1100%100%100%
Revocation vs. static exclusion, recovery+8.686%100%100%
Revocation vs. late exclusion, recovery (four models)+8.859%98%100%
Unrelated deletion, recovery (control)−0.911%14%11%

Regarding representativeness, writing is the most common work use of ChatGPT, at 40% of work-related messages [6], and each of our scenarios corresponds to categories in existing writing and work-task taxonomies (Appendix B). The withdrawn and required items come from the item-level annotations of the source benchmarks or from the same source material.

We have revised the Overview paragraph of Section 4 accordingly and added this analysis and the task-level intervals to Appendix L of the revision.

AuthorsOpenReview ↗

Response to Reviewer 8BmC (2/2)

Suggestion: Annotator Recruitment and Compensation

Thank you for pointing this out. Both annotators are authors of this paper, so no external annotators were recruited or paid. This follows prior benchmark work in which the authors perform the human annotation, such as Beyond Memorization [7] and SORRY-Bench [4]. We have stated this in Appendix G of the revision.

W3&Q1: Fixed Dialogue Structure and Defense Wording

Thank you for your constructive suggestion. Regarding the fixed dialogue, we replay 291 real revocations from the public corpora, in which users withdraw an item from the assistant's earlier draft. With each history kept verbatim, our six models leave a trace in 40.4% of their replies, close to the 46.9% of the original replies. Following FormatSpread [8], we also replace our fixed withdrawal with the most common named withdrawals in the public corpora, "Remove X.", "Delete X.", and "Can you remove X?". Full-reply traces range from 50.2% to 59.5% and recovery from 10.5% to 15.8%, with no significant paired change in recovery. In the model-added setting, replacing our instruction with "Remove X." changes traces and recovery by at most 0.8 percentage points.

Regarding defense wording, following Non-Adversarial Reproduction [2], we add a prompt from a deployed agent, the instruction that Ruff gives coding agents [9], and the prompt you suggested, which defines a revision trace and asks the model not to include one. Both lower recovery, but they also reduce utility by 4.4 to 6.7 percentage points, consistent with the trade-off that CIMemories [1] reports for privacy-conscious prompts. Our output-side filter keeps the highest end-to-end rate in both tracks.

Defense (six models)Conv. recoveredConv. utilityConv. end-to-endAgent recoveredAgent utilityAgent end-to-end
No defense13.7%86.5%74.5%13.4%84.4%74.5%
Ruff instruction1.7%79.8%78.8%5.3%78.7%74.9%
Trace definition1.8%79.9%78.6%3.5%80.0%77.3%
Output-side filter (ours)1.3%87.7%84.8%3.1%86.3%81.3%

We have added the two prompts to Section 5.4 and their exact text to Appendix E, and the withdrawal wordings to Section 6 and Appendix M of the revision.

We hope our responses have adequately resolved your concerns regarding 1) the practical significance of revision traces, 2) the size of the benchmark, and 3) the fixed dialogue and prompt wording. Since you offered to update your score if these concerns are addressed, we kindly request you to consider adjusting it to reflect the improvements and clarifications made in response to your input. We would be glad to answer any further questions.

AuthorsOpenReview ↗

References

References

[1] Niloofar Mireshghallah, Neal Mangaokar, Narine Kokhlikyan, Arman Zharmagambetov, Manzil Zaheer, Saeed Mahloujifar, and Kamalika Chaudhuri. CIMemories: A Compositional Benchmark for Contextual Integrity in LLMs. In International Conference on Learning Representations (ICLR), 2026.

[2] Michael Aerni, Javier Rando, Edoardo Debenedetti, Nicholas Carlini, Daphne Ippolito, and Florian Tramèr. Measuring Non-Adversarial Reproduction of Training Data in Large Language Models. In International Conference on Learning Representations (ICLR), 2025.

[3] Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory. In International Conference on Learning Representations (ICLR), 2024.

[4] Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal. In International Conference on Learning Representations (ICLR), 2025.

[5] Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, and Xander Davies. AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents. In International Conference on Learning Representations (ICLR), 2025.

[6] Aaron Chatterji, Thomas Cunningham, David J. Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. How People Use ChatGPT. NBER Working Paper 34255, 2025.

[7] Robin Staab, Mark Vero, Mislav Balunović, and Martin Vechev. Beyond Memorization: Violating Privacy via Inference with Large Language Models. In International Conference on Learning Representations (ICLR), 2024.

[8] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying about Prompt Formatting. In International Conference on Learning Representations (ICLR), 2024.

[9] Astral. Ruff: AGENTS.md. https://github.com/astral-sh/ruff/blob/main/AGENTS.md, 2026.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their time and constructive feedback. We address the concerns of each reviewer individually under each review. We further uploaded a revision, with the changes highlighted in blue and summarized as follows.

  • Real conversations: we replay 291 revocations sampled from the public corpora with our six models, which leave a trace in 40.4% of their replies, close to the 46.9% of the original replies (Section 3 and Appendix P).
  • Withdrawal wordings: across our fixed withdrawal and the three most common named withdrawals in the public corpora, no paired change in recovery is significant (Section 6 and Appendix M).
  • Prompt defenses: the instruction that Ruff gives coding agents and a prompt that defines revision traces lower recovery to 1.7–5.3% but reduce utility by 4.4 to 6.7 percentage points, while our output-side filter keeps the highest end-to-end rate in both tracks (Section 5.4).
  • Extracted constraints: with constraints that an LLM extracts from the conversation alone, every metric of the output-side filter stays within 0.4 percentage points of the annotated version (Section 5.4 and Appendix O).
  • Models: MiniMax-M3.1-Flash and Qwen3.8-Flash-Next leave a full-reply trace in 94.0% and 92.0% of their agent-track deliverables, with body traces confirmed by two annotators (Section 5.2).
  • Number of tasks: with 10 tasks per scenario, the main paired effects remain significant in 98% to 100% of random subsets (Appendix L).
  • Disclosure: deleting only the trace sentence reduces recovery from 303 of 1,239 deliverables to 24, and we report recovery by disclosure level, by information type, and over repeated revocations (Section 5.2 and Appendices K and L).
  • In-the-wild traces: on 331 sampled in-the-wild traces, two annotators confirm 85.9% of the revocation traces, and 65.0% of the revocation traces name the withdrawn item (Section 3 and Appendix A).
  • Presentation: we define revision traces and their disclosure levels before their first use, redraw the overview in Figure 2, show a complete task in Figure 4, report the per-model results in a table in Section 5.2, and enlarge the figures.
AuthorsOpenReview ↗

Summary of the Second Revision

We thank all reviewers again for their time. Following the discussion with Reviewer raDr, we have uploaded a second revision, with the changes of this round shown in green:

  • Disclosure under a forwarding notice: the paper's claims about disclosure to the recipient now rest on the forwarding notice, where the first turn states that the entire reply will be forwarded as is. We extended this condition from four models to all eight. Under the notice, six of the eight models still allow the reader to recover the withdrawn item, from 2.5% to 7.8% of their replies and from up to 12.5% in the agent track, while GPT-5.6 and GPT-5.5 stop almost entirely.
  • Traces and recovery: the trace rates without the notice are now presented as how often a model states the edit, and the abstract gives recovery under the forwarding notice.
  • Defense: under the notice, the output-side filter reduces recovery from 4.4% to 1.0% with a paired utility change of -0.2 percentage points.

The rates under the notice that our earlier responses quoted for four models are replaced by these eight-model rates, and the analysis of repeated revocations is in Appendix K.

We would be grateful for any further feedback before the discussion period closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.