acceptodds
Under review as a conference paper at ICLR 2027

TAPER: Recoverable Observation Control for Privacy–Utility Balance in LLM Agents

Abstract

Tool-using LLM agents face an observation-level privacy–utility tradeoff: tool observations may contain task-critical evidence together with sensitive information unnecessary for the current task. Once exposed to the task agent, such information can affect later reasoning and actions, and downstream output checks cannot undo this exposure. Existing one-shot minimization may either retain sensitive content or remove required evidence; discarded evidence also cannot support later repair. We propose TAPER (Task-Aware Protected Evidence Routing), which derives a unified information boundary from the task, routes each observation into a task-agent-visible Public Task View and a default-hidden Protected State, and applies model-based privacy and task-sufficiency checks before exposure, repairing detected issues when necessary. When later operations require precise evidence, TAPER releases only the task-necessary protected content. Across 336 tasks from two agent privacy benchmarks, three task-agent backbones, and two evaluators, TAPER consistently combines strong task helpfulness with low final leakage and task-unnecessary visible exposure, achieving the strongest joint privacy–utility performance across all six model–evaluator settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Agent Privacy, Privacy–Utility Balance, Observation Control, Privacy–Utility Verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_M9KMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TAPER control tool observations before they reach an LLM agent. It derive task information boundary, separates visible information from protected evidence, checks and repairs the visible view for privacy and utility and selectively release protected information when needed. Experiments on 336336 adapted AgentDAM, and AgentLeak tasks, using three backbones and two evaluators, report the highest joint privacy utility score across all 66 settings.

Critical strengths
  1. Treatment of the privacy utility tradeoff. Preserving protect evidence for later access addresses a practical limitation of irreversible minimization. Measuring unnecessary agent visible exposure alongside final output leakage also capture a important aspect of privacy that output-only evaluation miss.

  2. Broad empirical support in the evaluated setting. Results span 33 backbones and 22 evaluators with component ablations, evaluator-agreement analysis and paired bootstrap comparisons. The analys provide meaningful support for the reported gains.

Critical weaknesses
  1. Task utility is insufficiently grounded in factual correctness and execution success.

The helpfulness evaluator receives the instruction, sensitive-item list and final action but no source records or reference result Appendix H.3. It cann't reliably distinguish a plausible response from one that correctly uses retrieved evidence. Moreover, the original benchmarks are converted into local ReAct tasks rather than evaluated in their original environments. Because Help directly contributes to HPM, this limitation affects the central privacy utility conclusion. Agreement between two LLM judges don't resolve the missing correctness evidence.


  1. The distinctive contribution is not fully isolated from adaptation and additional processing.

Several baselines undergo substantial changes including replacing MINIM’s trained scorer with prompted scoring and representing abstraction as [REDACTED]. The ablations support the usefulness of TAPER’s components, but there is no comparison against a simpler task-aware minimizer with comparable verification and inference resources. Thus, it is unclear how much improvement specifically requires protected-state recovery and selective release.

Questions for authors
  1. Could you validate task success against source record, reference output and executable check and report whether the Help and HPM ranking remain consistent?

  2. How does TAPER compares with a task-aware minimizer given comparable verification, and inference resources? Does this comparison preserve the advantage attributed to recoverability?

  3. Since repaired views are forwarded without another check, what fraction still fails the privacy check, utility check? Figure 4 reports changes in flagged dimensions but don't distinguish these remaining failure types or their downstream consequences.

  4. How does TAPER’s observation-level control relate to the reasoning based and RL contextual-integrity mitigations in [1], particularly in balancing privacy protection and task utility (tradeoff)?

[1] Contextual Integrity in LLMs via Reasoning and Reinforcement Learning. NeurIPS 2025.

2 comments

Thank you for recognizing our work and providing constructive comments. We provide additional responses to your questions below:

  1. Regarding the reliability of task utility evaluation

Neither AgentDAM nor AgentLeak, the two datasets used in our study, provides standardized reference answers or expected actual execution outcomes for the tasks. We follow their evaluation methods and adopt LLM-as-a-judge to evaluate the outputs. Therefore, the Help evaluator indeed does not receive the original source documents, reference answers, or actual execution outcomes. We agree with your point that, without access to the original source documents or reference answers, the original Help evaluator may not be able to fully verify the factual correctness of the final outputs. To address this issue, we plan to combine LLM-based annotation with human review to establish a source-grounded reference answer for each task, specifying the key information that the final output should contain or the correct operation parameters. The evaluator will then assess task completion against the original evidence and fixed reference criteria, while checking for major factual errors or hallucinated content, to improve the credibility and reliability of task utility evaluation.

  1. Regarding the comparison with task-aware minimization and recoverability

As described in Appendix A.3 and Table 3, the single-view ablation in Table 2 is a task-aware minimization method that retains the Task Boundary, Privacy and Utility Verifiers, and Repair. It uses the same budget as full TAPER, and we believe that the performance difference between this method and TAPER can demonstrate the contribution of recoverability.

Specifically, in the ablation experiment on 140 tasks, full TAPER achieves higher Help (2.19 vs. 2.16) and HPM (71.12 vs. 68.24) than Single-View Minimization. To further validate this comparison, we conducted an additional paired analysis on the full set of 336 tasks, performing 10,000 paired bootstrap resamples of the metric differences on the same tasks and calculating two-sided 95% confidence intervals.

Under the DeepSeek evaluator, compared with Single-View Minimization, full TAPER reduces the final privacy leakage rate (LR) from 2.38% to 0.60%, a relative reduction of approximately 75%, with a paired difference of −1.79 percentage points (95% CI: [−3.27, −0.60]). The paired differences in Help and HPM are −0.134 (95% CI: [−0.280, +0.012]) and −3.85 (95% CI: [−8.81, +1.11]), respectively. This indicates that full TAPER has an advantage over task-aware minimization alone in reducing final privacy leakage.

Regarding the effectiveness of recoverability that you raised, we further analyzed the execution logs of selective recovery. Among the 336 tasks, 24.40% initiated recovery requests, and 14.29% received non-empty releases. Of the 55 exact-release entries, 76.36% subsequently appeared in tool arguments or final outputs. These results indicate that the recovery mechanism can provide the evidence needed for subsequent task execution and that full TAPER has an advantage in reducing final privacy leakage. We will further verify the independent benefits of recoverability through experiments with strictly matched inference budgets.

  1. Regarding the privacy and utility effects of Repair

Following your suggestion, we further analyzed the 429 repaired views in Fig. 4, separately examining the judgments of the Privacy and Utility Verifiers before and after Repair.

MetricPrivacyUtility
Failure rate before Repair44.52%86.25%
Failure rate after Repair41.49%50.58%
Percentage of initially failing views that pass after Repair51.31%44.05%
Percentage of initially passing views that fail upon re-evaluation35.71%16.95%

The results show that Repair enables 51.31% of the views that initially failed privacy verification to pass re-evaluation and reduces the overall utility issue flagging rate by 35.66 percentage points. This indicates that Repair can correct some residual privacy exposure issues and improve the sufficiency of information required for the task. Regarding the specific impact of the remaining issues on subsequent task execution, we have not yet conducted a separate statistical analysis, and more targeted experiments may be needed in the future to investigate this impact.

  1. Similarities and differences with Contextual Integrity in LLMs via Reasoning and Reinforcement Learning

Both [1] and TAPER are inspired by contextual integrity (CI) theory and aim to balance privacy protection and task utility during task execution. Our Task Information Boundary similarly incorporates contextual factors such as the purpose of the task and information recipients as the basis for decisions about information access and disclosure. Similar to CI-CoT in [1], TAPER uses prompts to guide general-purpose LLMs in making privacy decisions without requiring additional training.

Regarding the privacy–utility trade-off, [1] primarily improves the model's information disclosure decisions, enabling it to make full use of task-required information while protecting sensitive information. TAPER, in contrast, focuses more on decision errors during information minimization and their recoverability. It corrects insufficient and excessive filtering through privacy and utility verification and allows the agent to recover protected information when it is genuinely needed later, thereby mitigating the loss of task utility caused by upfront privacy protection. The two methods improve the privacy–utility trade-off at different levels and are complementary.

Reviewer_K5ZdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TAPER filters tool observations through a task-specific information boundary, separates public content from protected evidence, checks privacy and utility, and selectively releases hidden information. On 336 tasks adapted from AgentDAM and AgentLeak, it reports the highest joint privacy–utility score across three backbones and two evaluators.

Critical strengths

The distinction between final-output leakage and exposure within the task agent is useful. Evaluating both makes the privacy assessment more informative. The paired comparisons against AirGapAgent-Adapted support an improvement in the reported joint metric in five of six settings.

Critical weaknesses
  1. The evaluation does not establish factual task completion. The Help evaluator receives the instruction, sensitive items, and final action, but no source documents, reference answer, or execution outcome (Appendices C.1 and H.3). It cannot reliably distinguish a plausible response from one that uses the correct evidence or completes the requested operation. This directly undermines the utility claim and HPM, which incorporates Help. The privacy component also depends on necessity labels generated by DeepSeek without a reported independent annotation audit (Appendix C.2). Both evaluators reuse these labels, so their agreement does not validate whether the information was actually unnecessary.

  2. The distinctive benefit of recoverability remains insufficiently established. Repair already receives the raw observation (Section 4.3), so repairing a minimized view does not itself require a persistent Protected State. The relevant Single-View ablation retains verification and repair and achieves HPM 68.24 versus 71.12 and Help 2.16 versus 2.19 (Table 2). This comparison covers one backbone and 140 tasks, with no paired uncertainty estimates. Disabling release while retaining a writer that hides required values does not isolate the benefit over verified single-view minimization. The paper also provides no aggregate breakdown of later recovery events and the tasks they rescue. The evidence therefore leaves the contribution of protected storage and delayed release unresolved.

  3. The verification stage does not check the repaired content before exposure. Section 4.3 forwards the repaired view directly to the agent. The post-hoc analysis finds unchanged issue counts in 41.7% of repaired views and increased counts in 13.3% (Figure 4). These combined privacy/utility counts leave the rate of residual privacy violations unresolved. Since exposure is irreversible, the missing privacy-specific analysis prevents assessing whether repair reliably corrects the violations that triggered it.

  4. Compositional privacy checking does not account for the agent's accumulated context. Section 4.3 claims to check compositional exposure, but its verifier inputs omit prior public views and approved releases. The Release Controller likewise receives the request and requested records without the full visible history (Appendix A.2). Individually acceptable disclosures can jointly reveal a sensitive fact across steps. No targeted evaluation establishes protection against this failure mode, leaving a gap in the proposed sequential observation control.

Questions for authors
  1. Did the Help evaluator receive any source evidence or execution results omitted from the published prompt? What source-grounded checks validate task completion, and what independent checks validate the necessity labels?
  2. How often is protected evidence recovered at a later step, and does full TAPER outperform Single-View Minimization on those tasks under comparable inference budgets? Please provide paired confidence intervals for the full-versus-Single-View comparison.
  3. What fraction of repaired views still fails the Privacy Verifier, and how often does repair introduce a new privacy violation? Please separate these outcomes from utility failures.
  4. Do the Privacy Verifier and Release Controller receive earlier public views and approved releases beyond the documented inputs? What results specifically test sensitive facts inferable by combining disclosures across steps?
2 comments

Thank you for raising specific questions about the validity of the evaluation and the roles of the individual modules. We provide our responses below.

  1. Reliability of task utility evaluation and necessity labels

Neither of the two datasets we use, AgentDAM and AgentLeak, provides standardized reference answers for task execution or expected execution outcomes. We follow their evaluation methods and use an LLM as a judge to evaluate the output content. Therefore, the Help evaluator indeed does not receive original source documents, reference answers, or actual execution outcomes. We agree with your point that this evaluation approach may not adequately verify the factual correctness of the final output or task completion and may therefore be unreliable. To address this issue, we plan to provide standardized reference outcomes for each task through LLM-based annotation combined with human verification or supplementation, giving the judge model clearer scoring criteria rather than relying on speculative assessments based solely on whether the final output appears reasonable.

Second, regarding the model-generated information necessity labels, we conducted an additional independent human audit. Of the 3,821 sensitive information items across the 336 evaluation tasks, we sampled 200 items stratified by dataset and domain (100 necessary and 100 unnecessary items). The first annotator reviewed all 200 items, while the second independently annotated 100 of them. Neither annotator had access to the original model-generated labels.

The results show that the first annotator achieved 90% agreement with DeepSeek (91% for necessary items and 89% for unnecessary items), the second annotator achieved 84% agreement with DeepSeek, and the agreement between the two human annotators was 85%. These results provide further empirical support for the reliability of the model-generated necessity labels. We will include the above analysis in the revised manuscript.

  1. Protected State and Selective Release

We first examined how often protected evidence was retrieved in subsequent steps. Among the 336 evaluation tasks, 24.40% initiated recovery requests and 14.29% received non-empty releases. Across all tasks, there were 55 exact-release entries, 76.36% of which subsequently appeared in tool arguments or final outputs. This indicates that the selective release mechanism can provide evidence for subsequent task execution while maintaining a low proportion of information released.

Regarding the single-view ablation, as described in Appendix A.3 and Table 3, Single-View Minimization also retains the Task Information Boundary, Privacy and Utility Verifiers, and Repair. It therefore operates under the same budget as Full TAPER, and we believe that its performance difference relative to TAPER can demonstrate the effect of recoverability.

Following your suggestion, we supplemented our evaluation with a paired analysis across all 336 evaluation tasks. We aligned the existing experimental results of Full TAPER and Single-View by unique task identifiers and verified the consistency of user instructions, sensitive information items, and necessity labels. We used task-level paired bootstrap resampling, resampling the metric differences on the same tasks with replacement 10,000 times and calculating two-sided 95% confidence intervals.

Under the DeepSeek evaluator, compared with Single-View Minimization, Full TAPER reduces the final-output leakage rate (LR) from 2.38% to 0.60%, a relative reduction of approximately 75%, with a paired difference of −1.79 percentage points (95% CI: [−3.27, −0.60]). The paired differences in Help and HPM are −0.134 (95% CI: [−0.280, +0.012]) and −3.85 (95% CI: [−8.81, +1.11]), respectively. These results indicate that Full TAPER has a significant advantage over Single-View in reducing final-output privacy leakage.

  1. Privacy and utility effects of Repair

We further analyzed the repaired views reported in Fig. 4, separately examining the judgments of the Privacy and Utility Verifiers before and after Repair. The results are shown in the following table:

MetricPrivacyUtility
Failure rate before Repair44.52%86.25%
Failure rate after Repair41.49%50.58%
Proportion of initially failed views that pass after Repair51.31%44.05%
Proportion of initially passed views that fail upon re-evaluation35.71%16.95%

The results show that Repair enables 51.31% of the views that initially failed privacy verification to pass re-evaluation and reduces the overall rate of flagged utility issues by 35.66 percentage points. This indicates that Repair can correct some residual privacy exposure issues and significantly improve the sufficiency of task-required information, demonstrating the practical value of joint privacy–utility verification and repair.

  1. Cross-step compositional privacy leakage

We may not have explained this point clearly in the paper. By compositional exposure checking in Section 4.3, we intended to refer to checking whether information within a single Public Task View could be combined to infer sensitive information, rather than checking whether multiple Public Task Views across steps could jointly lead to compositional leakage. Therefore, the mechanism does not need to receive information from previous steps, and we did not conduct targeted experiments on this issue. We will clarify this point in future revisions.

Reviewer_dKE8OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TAPER, a privacy-preserving framework for tool-using LLM agents that minimizes unnecessary exposure of sensitive information while maintaining task utility. TAPER first derives a task-specific information boundary from the user instruction. It then separates each tool observation into a public view accessible to the task agent and a protected state containing hidden information. Two LLM-based verifiers assess privacy exposure and task sufficiency, followed by a repair mechanism to correct identified issues. The agent can subsequently request protected information when necessary for task execution. Evaluation on 336 tasks from AgentDAM and AgentLeak shows that TAPER achieves the highest joint privacy–utility score across three agent backbones and two evaluators.

Critical strengths
  • The problem is interesting, timely, and important.
  • Recognizing the privacy and utility trade-off is important
  • The multishot angle is great.
Critical weaknesses
  • Lack of rigorous definition of privacy, the work appears to be relying on LM on everything, and there is no rigorous definition or guarantee on the privacy protection.
  • Related, but maybe just a nit pick, is that the paper measures exp
  • The area has been studied by numerous prior work, and its qualitative and quantitative improvement over the existing state of art is not articulated clearly.
  • The overhead is non-trivial, Table 13 reports a median runtime of 87.7 seconds per task for TAPER, compared with 9.5 seconds without protection. This represents approximately a ninefold increase. A break down will be helpful.
  • The current writing leaves all the design decision outside of the main text, it is not even clear how the system works besides piecing together several LMs together.
  • The comparison may not be fair, it looks like the adapted ones are replacing components with prompted zero-shot scoring. According to Appendix C.1 and the evaluation prompt in Appendix H.3, the final evaluator receives the user instruction, sensitive items, and final action, but not the original source documents or reference answers. Consequently, it is unclear how the evaluator can verify factual correctness for tasks requiring information retrieved from those documents.
Questions for authors

see weakness

2 comments

Thank you for raising these questions about our work, which will help us improve the definitions and presentation of the results in our paper. Below are our responses to some of the questions you raised:

  • First, regarding your concerns about the reliance on LLMs in the defense process and the insufficiently clear definition of privacy, our focus is on agents disclosing private information to individuals who should not have access to it. For example, when a user asks an agent to send a project progress update to a client based on internal meeting records, the agent should retrieve the user's local emails, documents, and other relevant materials to provide progress information, but should not disclose internal minimum acceptable pricing or company policies contained in those records to the client. Under the assumptions of this paper, official LLM service providers are trusted data processors and do not actively steal or retain our personal information. The current threat model in the paper does not sufficiently clarify this distinction. In the revision, we will more clearly define what constitutes an "unauthorized downstream recipient" and clarify the scope of our study.
  • Second, regarding improvements over existing work, our work emphasizes how to address errors in information filtering, namely the possibility that one-shot filtering may overlook sensitive information or remove too much information. TAPER checks these two types of errors separately and retains hidden evidence for correction and subsequent access. By introducing a separate storage module, TAPER provides an opportunity to correct initial information selection errors.

We supplemented our evaluation with task-level comparisons against all seven baselines. Under the condition that Help does not decrease and neither of the two privacy outcomes worsens, we count the number of tasks with reduced final-output leakage or unnecessary exposure, and report the number of tasks favoring TAPER and the corresponding baseline (before and after the slash, respectively). The results show that across all six model–evaluator settings, TAPER outperforms each baseline on more tasks than the baseline outperforms TAPER.

BaselineDS/DSDS/GPTGPT/DSGPT/GPTClaude/DSClaude/GPT
No Defense267/8251/7173/11168/41292/2267/4
Side Tool262/8260/10168/10182/34282/2252/5
CDI-Guided272/4265/6187/5199/27287/2257/3
AirGap-A.135/65138/63116/52125/64111/44110/93
SlotGuard-A.242/14239/17183/17181/38281/3265/4
MINIM-A.115/70130/73131/39141/67192/21169/36
Preempt-A.244/17235/16176/19176/26262/12238/14

We further analyzed the execution logs of selective recovery. Among the 336 evaluation tasks, 24.40% initiated recovery requests, and 14.29% received non-empty releases. Of the 55 exact-release entries, 76.36% subsequently appeared in tool arguments or final outputs, including 13 entries that had not previously been matched in the agent-visible history. These results indicate that the selective recovery mechanism can provide additional evidence for subsequent task execution. We will include these statistics and mechanism analyses in the revised manuscript to further clarify the practical contribution of recoverability.

  • Regarding runtime overhead, TAPER primarily focuses on minimizing tool observations before they enter the task agent's context. Therefore, it inevitably needs to process every observation, and its runtime is naturally higher than that of No Defense, CDI-Guided, which provides privacy guidance only at the beginning, or Side Tool, which checks the final output. Accordingly, AirGap-A., SlotGuard-A., MINIM-A., and Preempt-A. in Table 13, which also perform per-observation processing, should serve as more direct cost references. Compared with these methods, TAPER incurs relatively modest additional processing time.

We further analyzed the runtime of each stage in TAPER, as shown in the table below. View construction and repair together account for 62.1% of the total runtime, constituting the main cost of TAPER, while the remaining computational effort is primarily devoted to explicit verification and subsequent access control:

Method and Processing StageAverage Runtime per Task (s)Percentage of the Method's Total Runtime
AirGap-A.: Per-observation reasoning and filtering91.2285.20%
TAPER: Task Information Boundary generation4.643.20%
TAPER: View construction55.6738.30%
TAPER: Privacy and utility verification (parallel)22.8115.70%
TAPER: Repair34.4823.70%
TAPER: Access requests and release decisions9.336.40%
TAPER: Task agent reasoning and response generation18.3412.60%

These processing stages yield practical benefits: in the ablation experiment, while retaining the initial view construction process, introducing verification and repair reduces unnecessary exposure by 71.3% in relative terms without decreasing Help. Thus, the higher overhead reflects the computational effort required to correct filtering errors before information enters the agent's context. We will include this runtime breakdown in the revised manuscript to clarify the relationship between costs and benefits.

  • Regarding the presentation of design decisions: due to space limitations, we placed the complete prompts and some implementation details in the appendix. However, the main text does not sufficiently explain the inputs, outputs, decision criteria, and triggering conditions of each module, making the system difficult to understand. We will adjust the division of content between the main text and the appendix, clearly explaining these key design decisions in the main text while retaining the complete prompts and implementation details in the appendix.
  • Regarding baseline adaptation and evaluation fairness: due to the constraints imposed by the input formats and execution environments of the original methods, we had to adapt them and made every effort to preserve their core ideas and key mechanisms. Regarding the final evaluation, the tasks used in our study do not have standardized natural-language reference answers, so direct reference-answer matching is not feasible. Nevertheless, we provide the evaluator with the same types of information across all methods and use identical prompts and scoring criteria, minimizing the possibility of providing additional information to the evaluator for different methods. Moreover, we primarily use the evaluator to assess task completion and sensitive information disclosure, so the provided user instructions, sensitive information items, and final actions should be sufficient. In future revisions, we will further provide the evaluator with original source records and task success criteria and introduce outcome checks for executable tasks to strengthen the verification of factual correctness and actual task completion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.