acceptodds
Under review as a conference paper at ICLR 2027

From Local Leakage to Global Privacy Risk in Sequential Multi-Agent Systems

Abstract

Sequential multi-agent large language model (LLM) systems are increasingly deployed in sensitive domains such as healthcare, finance, and enterprise decision-making, where multiple specialized agents collaboratively process a single user request. Although individual agents may satisfy local privacy constraints, sensitive information can still be inferred through sequential composition and intermediate representations. In this work, we study compositional privacy leakage in sequential LLM agent pipelines. We formalize leakage using mutual information and derive a compositional upper bound relating agent-local leakage to final-output privacy risk under sequential execution. Motivated by this analysis, we propose a privacy-regularized training framework that directly constrains information flow between agent outputs and agent-local sensitive variables. We evaluate our approach across sequential agent pipelines of varying depth on three benchmark datasets, demonstrating stable optimization dynamics and consistent, interpretable privacy-utility trade-offs. Our results show that privacy in agentic LLM systems cannot be guaranteed by local constraints alone and must instead be treated as a system-level property during both training and deployment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-Agent LLM Systems, Sequential Agent Pipelines, Compositional Privacy, Privacy Leakage, Mutual Information, Agent Communication

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_RpvZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies privacy leakage in sequential LLM-agent pipelines. It defines global leakage through mutual information, derives a composition bound, and proposes MINE-based privacy regularization. Experiments on MedQA, FinQA, and PrivacyLens examine privacy, task utility, pipeline depth, and regularization strategies.

Critical strengths

The final-output attack evaluation directly addresses the stated threat model, using frozen agents and a disjoint attacker-training split. The reported reduction in sensitive-attribute recovery, with much of the benign task performance retained, provides useful evidence of a practical benefit from regularization.

Critical weaknesses

The central position-dependent interpretation is unsupported. Under the conditional data-processing and independence steps used in Appendix A.1, applying the chain rule directly to cumulative leakage yields a tighter additive bound without position-dependent weights. The exponential coefficients therefore reflect looseness in the derivation rather than an established positional effect. Proposition 1 restates these coefficients, leaving the claimed theoretical basis for prioritizing earlier agents unsubstantiated.

The baseline results are inconsistent across tables and metric definitions. Table 7 is labeled MedQA, but its LLaMA-3B baseline and proposed-method rows reproduce the corresponding FinQA results and disagree with the main MedQA table. The displayed PARI differences also do not reconcile with the stated metric formula and component values. These discrepancies leave the dataset provenance and comparative conclusions unresolved.

The protected information and attack task are insufficiently specified. Appendix A.3.4 lists private-prompt embeddings, routing states, and policy indicators without explaining their construction, sampling, or assignment to agents. The final-output evaluation introduces a four-class attack without defining the labels, class frequencies, attacker architecture, or split sizes. These omissions prevent reproduction of the central privacy evaluation and assessment of what sensitive information the reported protection covers.

The claimed privacy-utility advantage lacks a controlled comparison. The alternative interventions lack specified strengths and tuning protocols; the privacy-regularization baseline also lacks an explicit objective. Table 7 reports isolated operating points, while the trade-off sweep covers only MINE-Reg. Equal training budgets do not establish comparable privacy-utility operating points. Without baseline sweeps or comparisons at matched privacy or utility, the reported advantage could reflect regularization strength rather than the proposed mechanism.

Questions for authors
  1. What prevents a tighter additive bound under the assumptions used in Appendix A.1, and what result establishes a positional effect beyond that bound?
  2. Which dataset produced each comparison in Table 7, and do corrected per-seed PARI calculations preserve the reported ranking?
  3. What are the sensitive classes, how are they sampled and assigned to agents, and what attacker architecture, class frequencies, and split sizes underlie the final-output evaluation?
  4. Does MINE-Reg retain its advantage when the alternative interventions are tuned on the same validation split and compared at matched benign success or attack success?
2 comments

We thank the reviewer for the detailed assessment and for recognizing the relevance of our final-output adversarial evaluation. We clarify the theoretical interpretation, dataset labeling, attack protocol, and privacy–utility comparisons below.

W1 / Q1: Composition Bound and Position-Dependent Leakage

Theorem 1 concerns the relationship between agent-local leakage constraints and final-output privacy risk. The central distinction is between marginal leakage I(Oi;Si)I(O_i;S_i) and joint leakage I(ON;S1:N)I(O_N;S_{1:N}), which need not be characterized by the same information quantities.

As explained in our response to Reviewer ZwPM (W2/Q2), the direct additive recurrence requires additional conditional-independence conditions. Since each agent processes (Oi−1,Di,Si)(O_{i-1},D_i,S_i), conditional data processing gives

I(Oi;S<i∣Si)≤I(Oi−1,Di;S<i∣Si),I(O_i;S_{\lt i}\mid S_i) \leq I(O_{i-1},D_i;S_{\lt i}\mid S_i),

which cannot generally be reduced to an expression involving Oi−1O_{i-1} alone.

More precisely, defining Gi=I(Oi;S1:i)G_i=I(O_i;S_{1:i}), we obtain

Gi≤τi+Gi−1+δi+κi,G_i\leq\tau_i+G_{i-1}+\delta_i+\kappa_i,

where

δi=I(S<i;Di∣Oi−1,Si),κi=I(S<i;Si∣Oi−1).\delta_i=I(S_{\lt i};D_i\mid O_{i-1},S_i), \qquad \kappa_i=I(S_{\lt i};S_i\mid O_{i-1}).

These terms capture dependencies involving stage-specific inputs and upstream representations. Under the sufficient independence conditions detailed in Reviewer ZwPM (W2/Q2), both terms vanish, yielding the tighter additive bound.

Under those conditions, the exponentially weighted expression in Theorem 1 remains a valid conservative upper bound. Its coefficients arise from the recursive bounding procedure and are not claimed to be tight lower bounds on positional leakage. Accordingly, Proposition 1 describes sensitivity of this upper bound to individual leakage budgets, rather than proving that earlier agents necessarily leak more information.

The empirical motivation remains distinct: intermediate representations can retain dependencies involving multiple agent-local secrets, and their final-output recoverability must be evaluated at the system level.

W2 / Q2: Table 7 Dataset Labeling and PARI Consistency

Dataset clarification. We thank the reviewer for identifying the labeling error. Appendix Table 7 reports alternative privacy-intervention comparisons on FinQA, not MedQA. The table caption and corresponding dataset references in Appendix A.5.4 were incorrectly labeled. Thus, the LLaMA-3B Base and MINE-Reg results in Table 7 correctly match FinQA Table 2 for both two-agent and three-agent settings. We will correct these labels in the revised PDF; the numerical entries in Tables 2 and 7 remain unchanged.

PARI Consistency. We thank the reviewer for raising this concern. The apparent discrepancy arises from reconstructing a nonlinear composite metric using component values reported to only two decimal places. Appendix A.4.6 defines

PARI=0.3,BS+0.1,OT+0.5(1−MIavgMIbaseline)+0.1,RT,\mathrm{PARI} = 0.3\\,\mathrm{BS} + 0.1\\,\mathrm{OT} + 0.5\left(1- \frac{\mathrm{MI}_{\mathrm{avg}}} {\mathrm{MI}_{\mathrm{baseline}}}\right) + 0.1\\,\mathrm{RT},

where RT=1\mathrm{RT}=1.

Consider the FinQA/LLaMA-3B three-agent MINE-Reg configuration, for which Table 2 reports MI =0.26=0.26, baseline MI =0.87=0.87, BS =0.75=0.75, and PARI =0.781=0.781. Substituting these rounded values and setting OT =1=1 produces PARI ≈0.7756\approx0.7756. However, the displayed precision does not uniquely determine the underlying component values.

Specifically, the reported entries correspond to the rounding intervals

MIavg∈[0.255,0.265),MIbaseline∈[0.865,0.875),BS∈[0.745,0.755).\begin{aligned} \mathrm{MI}_{\mathrm{avg}}&\in[0.255,0.265),\\\\ \mathrm{MI}_{\mathrm{baseline}}&\in[0.865,0.875),\\\\ \mathrm{BS}&\in[0.745,0.755). \end{aligned}

These intervals admit PARI values up to approximately 0.78080.7808 when OT =1=1, which round to 0.7810.781. Thus, the reported PARI is mathematically compatible with the displayed component precision; substituting rounded table entries is not a valid test of exact numerical equality.

The distinction is particularly relevant because privacy integrity contains a ratio, so its value must be evaluated using the corresponding underlying MI quantities. Furthermore, averaging per-run ratios is generally not equivalent to taking the ratio of averaged MI values.

PARI serves as a composite summary, while the primary privacy–utility conclusions are independently supported by the reported MI, sensitive-blocking, and benign-success metrics. In particular, the comparisons in Appendix Table 7 can be evaluated directly through these primitive metrics without relying on the composite index.

W3 / Q3: Sensitive Variables and Final-Output Adversarial Evaluation

Sensitive information. Our framework protects agent-local sensitive variables SiS_i rather than assuming that all sensitive information corresponds to a single shared secret. Appendix A.3.4 specifies private-prompt embeddings, routing states, and policy indicators for MedQA/FinQA, while PrivacyLens uses protected seed attributes such as data subject, recipient, and transmission principle.

The attack objective is to predict agent-local sensitive attributes from the information exposed by the pipeline. In the final-output setting, the attacker observes only ONO_N and predicts the protected attributes; it does not access the intermediate agent representations OiO_i.

Independent adversarial evaluation. Section 6.5 (Table 3) evaluates this threat model on MedQA/Qwen-4B. Agent parameters are frozen, and the attacker is trained independently using a disjoint attack-training split.

Table 1: Final-output adversarial inference on MedQA/Qwen-4B (Table 3).

AgentsMethodMacro Acc. ↓\downarrowMacro F1 ↓\downarrowBS ↑\uparrow
3Base0.720.700.92
3MINE-Reg0.300.290.85
5Base0.780.760.88
5MINE-Reg0.320.310.81

MINE-Reg reduces attacker macro accuracy by 0.420.42 and 0.460.46 in the three-agent and five-agent settings, corresponding to relative reductions of 58.358.3\\% and 59.059.0\\%, respectively. Benign success remains 0.850.85 and 0.810.81.

This evaluation is independent of the MINE critics optimized during training and directly tests sensitive-attribute recoverability from the final output.

The evaluation uses a four-class classification setting with a uniform-guessing reference of 0.250.25. The precise class mapping, class frequencies, attacker configuration, and split sizes are necessary to reproduce the attack protocol and assess class imbalance. We distinguish the stated uniform-guessing reference from a class-prior baseline and do not interpret reduced attacker accuracy as a certified MI upper bound.

W4 / Q4: Fairness of Privacy–Utility Comparisons

The alternative interventions in Appendix A.5.4 include Gaussian perturbation of intermediate representations (DP-style Noise) and an agent-local distribution-flattening privacy regularizer (PR). They use the same backbone and nominal training-budget settings as MINE-Reg.

The reviewer asks whether the advantage persists at comparable privacy or utility. The reported LLaMA-3B FinQA results provide informative operating-point comparisons:

Table 2: Privacy–utility comparison on FinQA using LLaMA-3B (Appendix Table 7; corrected dataset label).

AgentsMethodMI$_{\mathrm{avg}}\downarrow$SB ↑\uparrowBS ↑\uparrow
2Base0.690.310.85
DP0.180.680.52
PR0.360.570.74
MINE-Reg0.210.690.77
3Base0.870.260.82
DP0.220.640.48
PR0.480.520.72
MINE-Reg0.260.640.75

Comparison with DP-style Noise. With two agents, MINE-Reg achieves SB 0.690.69 versus 0.680.68 for DP-style Noise, while preserving substantially higher benign success (0.770.77 versus 0.520.52). With three agents, both methods achieve SB 0.640.64, but MINE-Reg retains BS 0.750.75 compared with 0.480.48 for DP-style Noise. These correspond to 2525- and 2727-percentage-point improvements in benign success at the reported equal-SB operating points.

Comparison with PR. For two agents, MINE-Reg achieves higher sensitive blocking (0.690.69 versus 0.570.57) and benign success (0.770.77 versus 0.740.74), with lower estimated MI (0.210.21 versus 0.360.36). The same pattern holds for three agents: SB improves from 0.520.52 to 0.640.64, BS from 0.720.72 to 0.750.75, and estimated MI decreases from 0.480.48 to 0.260.26.

Thus, the reported results show a favorable privacy–utility balance that is not attributable solely to the composite PARI metric. In particular, MINE-Reg achieves stronger utility than DP-style Noise at equal reported sensitive blocking, and improves both blocking and utility relative to PR at the evaluated operating points. These comparisons do not establish dominance over independently optimized privacy–utility frontiers.

Reviewer_c1HHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how local secret information leakage can lead to global privacy risk in sequential multi-agent settings. The paper identifies a problem that even though individual agent may satisfy the local constraints, some sensitive information may still be passed through the sequential composition. The paper formalizes the leakage amount and gives an upper bound for it. Motivated by this idea, the paper further presents a training framework to directly constraints information flow between agents and the results from various model and various tasks support the creation of framework

Critical strengths

The problem is practically relevant. The paper studies compositional privacy leakage in sequential multi-agent LLM systems, which is a relevant problem as agent-based systems become more widely deployed. The distinction between agent-local leakage and global privacy risk is clear and worth investigating.

The experiment results show a good trade-off. The authors evaluate multiple model families and agent depths across 3 main benchmarks. The reported results show substantial reductions in estimated privacy leakage while preserving most of the performance

Critical weaknesses

The exponential coefficients in Theorem 1 appear to result from a relatively loose sequence of upper bounds. In addition, the proof in Appendix A.1 uses conditional independence relations that do not seem to follow directly from the stated assumptions. I’m therefore questioning the connection between the proposed bound and the claimed position-dependent privacy risk.

The implementation of the sequential multi-agent pipeline is unclear. The proposed method is a multi-agent system, but the task is a QA benchmark. How the QA benchmark could be turned into input text split among multiple agents, and how to use these benchmarks to create secret information for each individual agent, is unclear.

Questions for authors

Could the authors clarify how the LoRA adapters of different agents are trained? I believed all agents are jointly optimized through end-to-end backprop. How are gradients propagated through the intermediate agent outputs? I am also curious about the actual training cost. Could the authors report the total training time and GPU resources required?

Could the authors provide some additional discussion on the role of Theorem 1? Like the conclusion is that the first agent will contribute more to the global leakage, but did the algorithm actually regulate the first agent more? The regulation parameter now seems more based on validation tuning. Do the authors want to create a formal upper bound, or is it more like a motivation for regulation for an individual agent?

I noticed that the LLaMA-3B results in Table 7 (MedQA) appear to be identical to those in Table 2 (FinQA) for both the 2-agent and 3-agent settings, including MIavg, SB, BS, and PARI. Is it a reporting error?

3 comments

We thank the reviewer for the constructive feedback and appreciate the recognition of the practical relevance and privacy-utility improvements of our framework. We clarify the theoretical contribution, sequential-agent implementation, and experimental details below.

W1 / Q2: Composition Bound and Its Role in MINE-Reg

Theorem 1 characterizes how agent-local information constraints relate to final-output leakage in a sequential system. Its significance lies in distinguishing local leakage I(Oi;Si)I(O_i;S_i) from global leakage I(ON;S1:N)I(O_N;S_{1:N}), rather than asserting that realized privacy loss necessarily grows exponentially.

As discussed in our response to Reviewer ZwPM (W2/Q2), the direct additive recurrence requires additional conditional-independence conditions involving the upstream representation and stage-specific inputs DiD_i. In particular, letting Gi=I(Oi;S1:i)G_i=I(O_i;S_{1:i}), the general stage transition gives

Gi≤τi+Gi−1+δi+κi,G_i\leq\tau_i+G_{i-1}+\delta_i+\kappa_i,

where

δi=I(S<i;Di∣Oi−1,Si),κi=I(S<i;Si∣Oi−1).\delta_i=I(S_{\lt i};D_i\mid O_{i-1},S_i), \quad \kappa_i=I(S_{\lt i};S_i\mid O_{i-1}).

These terms capture additional information introduced through stage inputs and conditional dependencies. Under the sufficient independence conditions specified in Reviewer ZwPM (W2/Q2), they vanish, yielding the additive bound; the exponentially weighted expression in Theorem 1 remains a valid conservative upper bound under those same conditions.

Connection to the algorithm. The theorem motivates controlling information at communication boundaries, but its coefficients are not prescribed as training weights. MINE-Reg instead optimizes

Ltotal=Lutility+∑i=1NβiI^MINE(Oi;Si),\mathcal L_{\mathrm{total}} = \mathcal L_{\mathrm{utility}} + \sum_{i=1}^{N} \beta_i\widehat I_{\mathrm{MINE}}(O_i;S_i),

where βi\beta_i balances the privacy and utility objectives for each agent.

These coefficients are selected through validation or adaptive dual updates (Sec. 4.1; Appendix A.7.1), rather than fixed to 2N−i2^{N-i}. This is appropriate because a worst-case upper-bound coefficient need not correspond to the optimal regularization strength for a particular agent's task and representation.

Importantly, Appendix A.7.2 shows that regularizing early agents reduces estimated leakage relative to no regularization, whereas regularizing all agents yields stronger suppression. This supports information-flow control at multiple boundaries without assuming that the theoretical weights define an optimal regularization schedule.

W2: Sequential-Agent Construction for QA Benchmarks

The QA benchmarks are processed through a sequential workflow; the original question is not divided into unrelated tasks. Each agent receives the upstream representation, task-relevant context, and its locally accessible sensitive information:

Oi=Ai(Oi−1,Di,Si).O_i=A_i(O_{i-1},D_i,S_i).

Appendix A.3.2 defines five roles: input interpretation, task reasoning, contextual refinement, privacy-aware rewriting, and compliance checking. Pipelines with N∈2,3,4,5N\in\\{2,3,4,5\\} use the corresponding first NN roles.

For example, in MedQA, the input interpreter processes the clinical question, the task reasoner produces a candidate interpretation or answer, and subsequent agents refine the reasoning or apply privacy-aware checks. In FinQA, the same architecture operates on financial questions and supporting numerical/textual evidence, with the final agent responsible for the task prediction.

Agent-local sensitive variables. The protected context SiS_i is distinct from the public task label yy. Appendix A.3.4 considers private-prompt embeddings, internal routing states, and policy indicators in MedQA/FinQA. PrivacyLens uses privacy-sensitive seed attributes, including data subject, recipient, and transmission principle.

Thus, the task objective preserves information needed to answer the user query, while the privacy objective discourages the propagation of agent-local sensitive information through intermediate representations.

The precise construction of the MedQA/FinQA sensitive-label spaces and their assignment to individual agents depends on the experimental preprocessing protocol; the conceptual agent roles alone do not determine those assignments.

Q1: LoRA Optimization, Gradient Propagation, and Training Cost

Joint optimization. Each agent has an independent rank-88 LoRA adapter (scaling factor 1616), while the MINE critics are two-layer MLPs with hidden dimension 256256 (Appendix A.3). Algorithm 1 alternates between optimizing the critics and updating the agent adapters using the combined task and privacy objective.

For a differentiable intermediate representation, the gradient with respect to an upstream adapter follows

∇θiLtotal=∇θiLutility+∑j=1Nβj∇θiI^MINE(Oj;Sj).\nabla_{\theta_i}\mathcal L_{\mathrm{total}} = \nabla_{\theta_i}\mathcal L_{\mathrm{utility}} + \sum_{j=1}^{N} \beta_j\nabla_{\theta_i} \widehat I_{\mathrm{MINE}}(O_j;S_j).

Consequently, downstream losses can influence upstream adapters through the sequential computational graph whenever the relevant communication operations retain differentiability.

This formulation does not require different agents to share parameters. It requires a differentiable communication path for any gradient propagated across agent boundaries; if the exchanged messages are generated discrete tokens, an additional gradient-estimation mechanism is necessary. The gradient expression alone does not establish which communication implementation was used.

Training configuration and overhead. Training uses AdamW (2×10−52\times10^{-5}), batch size 1616, ten epochs, bfloat16 precision, and NVIDIA H100 GPUs. The critics use Adam (10−410^{-4}) with exponential-moving-average stabilization (Appendix A.3.6).

Appendix A.6 (Table 6) reports the following per-iteration overhead:

Table 1: Relative training cost of MINE-Reg on MedQA (Table 6).

ModelAgentsBaselineMINE-Reg
LLaMA-7B21.00×1.00\times1.15×1.15\times
LLaMA-7B31.00×1.00\times1.18×1.18\times
LLaMA-13B21.00×1.00\times1.22×1.22\times
LLaMA-13B31.00×1.00\times1.25×1.25\times

The additional critic optimization introduces 1515–2525\\% per-iteration overhead in these configurations. These values quantify relative computation; total GPU-hours additionally depend on the number of iterations, sequence lengths, and GPU allocation.

Q3: Table 7 Dataset Labeling Error

We thank the reviewer for identifying this error. Appendix Table 7 reports the alternative privacy-intervention comparisons on FinQA, not MedQA. The table caption and corresponding dataset references were incorrectly labeled as MedQA. Therefore, the LLaMA-3B Base and MINE-Reg results in Table 7 correctly match those in FinQA Table 2 for both the two-agent and three-agent settings.

This is a reporting error in the dataset labeling, not an inconsistency in the experimental results. We will correct the Table 7 caption and the corresponding references in Appendix A.5.4 in the updated PDF. All numerical results in Tables 2 and 7 remain unchanged.

Thank you for your response and clarifications. These explanations resolve some of my concerns. Though even with the tighter additive bound, I’m still unsure what Theorem 1 tells us about the role of agent position. I maintain my score.

Reviewer_N9CWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates information leakage within sequential Large Language Model (LLM) agent pipelines—an architecture where each stage not only receives representations from upstream components but also gains access to its own specific, sensitive context. We define "global leakage" as the mutual information between the final output and the sensitive variables local to each agent. Under independence and Markov assumptions, we derive an upper bound on this leakage, characterized by an exponentially weighted sum based on the position of each agent in the sequence. Furthermore, we propose the MINE-Reg method, which fine-tunes agents by combining task-specific loss with neural estimates of local mutual information, employing either fixed or adaptively updated regularization weights. Experimental results on the MedQA, FinQA, and PrivacyLens datasets demonstrate that this approach reduces both the estimated information leakage and the success rate of adversarial inference; however, it also leads to a decline in performance on primary tasks and incurs additional training overhead.

Critical strengths
  1. This paper addresses not only the issue of models memorizing their training data but also the sensitive contextual information introduced during actual deployment. In domains such as healthcare, finance, and enterprise-level workflows, tracking the flow of information across the boundaries of different agents is of critical importance.

  2. By distinguishing between "local quantities" and "leaks in the final output," this paper clarifies the distinction between "representation diagnostics" and "threat models involving external observers." It explicitly notes that the average MINE (Mutual Information Neural Estimation) value does not serve as a strict upper bound for the total amount of information leaked.

  3. The evaluation encompasses three benchmarks, a range of model scales, and workflows involving between two and five agents. The appendix further explores various intervention strategies, selective regularization, adaptive weighting, experiments based on three different random seeds, conditional leakage diagnostics, and computational overhead.

Critical weaknesses
  1. The objective provided is a variational lower bound; a decrease in the estimate obtained via a finite-capacity critic does not necessarily imply a reduction in the true mutual information (MI). Even if marginal local mutual information could be computed precisely, it would not account for all conditional or joint leakage terms. While the authors acknowledge the limitations of the estimation process and introduce "probes"—which is a constructive step—it remains to be demonstrated that the observed defensive gains hold up against independently trained, more powerful adversaries, and that the final reported metrics align with the threat model defined in the paper. Consequently, the claim that the "optimization process enforces information constraints" should be stated with greater caution.

  2. In Appendix A.3.4, sensitive variables are described as "private-prompt embeddings," "routing states," or "policy indicators"; however, the paper lacks actionable construction methods, label spaces, and sampling distributions, nor does it provide concrete examples sufficient to replicate the task. The definition of the communication interface is also ambiguous: if agents exchange generated text, the paper must explain how end-to-end gradients are propagated through the generation process; if they exchange continuous states, the deployment assumptions and the implementation of cross-model interfaces must be clarified. Furthermore, the paper should explicitly detail the training/test set splits, the nature of the attacker's inputs, the attacker's capabilities, and the relevant chance baselines. Additionally, the probes OiO_i targeting SiS_i (as mentioned in Appendices A.4 and A.7) contradict the paper’s definition of an external adversary, who is stated to have access only to ONO_N.

  3. Increasing the depth introduces distinct agent roles and additional local context; therefore, this does not constitute a "pure" intervention targeting sequence depth alone. Moreover, the "selective regularization" applied to early stages lacks a clear comparison against a control condition where regularization is applied exclusively during later stages. The trends observed in the experiments may stem from variations in factors such as the content of the secret, the roles of the agents, model capabilities, or the difficulty of the training process. Before attributing the observed effects to "position" or "iterative transformation" per se, more rigorous controlled experiments are required.

  4. In Appendix A.1, the authors define upstream information using conditional data processing and the assumption of independence. Following the same logic, if we let Gi=I(Oi;S1,…,Si)G_i = I(O_i; S_1, \dots, S_i), then Gi=I(Oi;Si)+I(Oi;S<i∣Si)≤τi+Gi−1G_i = I(O_i; S_i) + I(O_i; S_{<i} | S_i) \le \tau_i + G_{i-1}, which implies GN≤∑iτiG_N \le \sum_i \tau_i. The paper should explain why this straightforward recursive derivation is insufficient, or clarify what specific advantage the proposed exponential bound offers over this alternative. A large coefficient in a loose upper bound does not, in itself, prove that leakage from an "early agent" has a more significant intrinsic impact. This critique concerns the interpretation and implications of the bound, rather than asserting that the inequality itself is mathematically incorrect.

  5. In Appendix A.1, the authors utilize the mutual independence of sensitive variables to eliminate the conditional dependence on SiS_i within terms involving Oi−1O_{i-1}. This implicitly requires that upstream representations do not depend on SiS_i—whether through common inputs, task-specific inputs, or other pathways. Given that DiD_i is explicitly included in the model, the dependencies between these components must be clearly defined. Furthermore, when subsequent agents acquire new "side information," the conditions required for the "conditional data processing" step become more stringent than those implied by a simple "sequential update rule." While the authors posit an assumption of "no additional access paths," they should precisely define which joint distributions are excluded by this assumption and clarify whether the experimental setup satisfies this condition.

Questions for authors
  1. Given the conditional data processing and independence steps employed in Appendix A.1, why is the direct recurrence relation Gi≤τi+Gi−1G_i \le \tau_i + G_{i-1} inapplicable? Are there counterexamples that satisfy the stated assumptions, or is there a "tightness" result that necessitates the introduction of exponential coefficients?

  2. What forms of dependency are permitted between the public input DiD_i and the sensitive variables? Under what additional conditions can the conditional dependence on SiS_i be removed from the upstream representations? Do the sensitive contexts examined in the experiments satisfy these conditions?

  3. What specific information is exchanged between the agents, and how are gradients propagated across this interface? Please provide a comprehensive example of the workflow, explicitly detailing the inputs, local sensitive variables, transmitted outputs, and the target of the attack.

  4. Among the reported adversarial results, which ones rely solely on the final output ONO_N? Were the attackers used for evaluation trained independently on held-out data using a fixed (frozen) model? Furthermore, what are the class distributions for these attackers, and what is their "chance accuracy" (accuracy of random guessing)?

  5. Please clarify the calculation method for the BS, SB, and OS values ​​presented in Table 5—including the denominators and the logic for conditional filtering—to avoid ambiguity. Are there any comparative results between applying regularization "only at a later stage" versus "only at an early stage"? Such results would allow for an isolated assessment of the impact of the regularization placement, while keeping the roles and the content of the sensitive information constant.

2 comments

We thank the reviewer for the detailed and constructive feedback, particularly for recognizing the distinction between representation-level leakage and final-output privacy risk. We address the concerns below.

W1 / Q4: MI Estimation and Independent Adversarial Evaluation

MINE-Reg employs the Donsker–Varadhan variational objective as a differentiable regularizer, rather than treating its estimate as a certified upper bound on true mutual information. Importantly, our privacy evaluation is not limited to the MINE critic optimized during training.

Independent final-output attacks. Section 6.5 (Table 3) evaluates an independently trained adversary observing only the final output ONO_N, with frozen agent parameters and a disjoint attack-training split. On MedQA/Qwen-4B, attacker macro accuracy decreases from 0.720.72 to 0.300.30 for three agents and from 0.780.78 to 0.320.32 for five agents, corresponding to relative reductions of 58.358.3\\% and 59.059.0\\%, while benign success remains 0.850.85 and 0.810.81.

These improvements are measured using an external attacker rather than the training-time MI critic, providing independent evidence of reduced sensitive-attribute recoverability. The four-class uniform-guessing reference is 0.250.25; the observed attack accuracies therefore indicate substantial protection with residual leakage, not complete elimination.

The intermediate-output probes in Appendix A.7.4 serve a different purpose: they diagnose information recoverability at communication boundaries OiO_i. They are not presented as external attackers, whose observations are restricted to ONO_N. Thus, the two evaluations address complementary aspects of the same information-flow problem.

W2 / Q3: Sequential Workflow, Sensitive Variables, and Gradient Propagation

Sequential-agent construction. Each agent processes the upstream representation, task context, and locally accessible sensitive information:

Oi=Ai(Oi−1,Di,Si).O_i=A_i(O_{i-1},D_i,S_i).

Appendix A.3.2 specifies five roles: input interpretation, task reasoning, contextual refinement, privacy-aware rewriting, and compliance checking. MedQA and FinQA examples are processed through these specialized stages, with the final agent producing the task prediction.

For MedQA/FinQA, the protected context includes private-prompt embeddings, routing states, and agent-specific policy indicators (Appendix A.3.4). PrivacyLens uses sensitive seed attributes, including data subject, recipient, and transmission principle. These variables represent agent-local information that should not be disclosed merely because it is available during task execution.

Optimization. Each agent uses independent rank-88 LoRA adapters, while the MINE critics are two-layer MLPs with hidden dimension 256256. Algorithm 1 alternates between maximizing the MI estimates with respect to critic parameters and minimizing the combined utility–privacy objective with respect to agent parameters:

Ltotal=Lutility+∑i=1NβiI^MINE(Oi;Si).\mathcal L_{\mathrm{total}} = \mathcal L_{\mathrm{utility}} + \sum_{i=1}^{N} \beta_i\widehat I_{\mathrm{MINE}}(O_i;S_i).

For differentiable intermediate representations, downstream losses propagate through the sequential computational graph to upstream adapters. If an interface uses discrete token generation, such cross-agent gradients require an additional estimator or differentiable surrogate; the mathematical objective does not itself imply backpropagation through discrete sampling.

Appendix A.3.6 further specifies AdamW (2×10−52\times10^{-5}), batch size 1616, ten epochs, bfloat16 training, and three random seeds.

W3 / Q5: Depth, Selective Regularization, and Evaluation Metrics

Sequential depth and intervention placement. The depth experiments compare MINE-Reg with the corresponding unregularized pipeline at each fixed model, benchmark, and agent count. For Qwen-4B on MedQA, MIavg\mathrm{MI}_{\mathrm{avg}} decreases from 0.540.54 to 0.060.06 at two agents and from 1.021.02 to 0.130.13 at five agents (Table 1).

Appendix A.7.2 additionally compares no regularization, early-stage-only regularization, and all-stage regularization under a common total privacy budget. Early-stage regularization reduces estimated leakage, while all-stage regularization produces stronger suppression. This supports controlling multiple agent boundaries, although it does not isolate the effect of position independently of agent roles and sensitive contexts.

Metric definitions. Appendix A.4 defines BS over benign evaluation examples and SB=1−LA\mathrm{SB}=1-\mathrm{LA} over sensitive examples. Overall Success (OS) instead requires simultaneous benign correctness and absence of leakage within each paired evaluation scenario:

OS=1Npairs∑jI[BenignCorrectj=1∧Leakagej=0].\mathrm{OS} = \frac{1}{N_{\mathrm{pairs}}} \sum_j \mathbb I[ \mathrm{BenignCorrect}_j=1 \land \mathrm{Leakage}_j=0].

OS is therefore a joint-outcome metric, not generally the product BS×SB\mathrm{BS}\times\mathrm{SB}. For PrivacyLens, leakage accuracy is mapped to the adjusted leakage rate LRh\mathrm{LR}_h, while BS requires helpfulness ≥2\geq2 on the benchmark's 00–33 rubric (Appendix A.4.5).

W4 / Q1: Additive Versus Exponential Composition Bound

The reviewer's additive recurrence identifies a meaningful special case, but its validity depends on conditional-independence assumptions beyond marginal independence of the sensitive variables.

As detailed in our response to Reviewer ZwPM (W2/Q2), defining Gi=I(Oi;S1:i)G_i=I(O_i;S_{1:i}) gives

Gi≤τi+Gi−1+δi+κi,G_i\leq\tau_i+G_{i-1}+\delta_i+\kappa_i,

where

δi=I(S<i;Di∣Oi−1,Si),κi=I(S<i;Si∣Oi−1).\delta_i=I(S_{\lt i};D_i\mid O_{i-1},S_i), \qquad \kappa_i=I(S_{\lt i};S_i\mid O_{i-1}).

The additive recurrence follows when these conditional-dependence terms vanish. Thus, the direct additive argument is not implied by mutual independence of S1,…,SNS_1,\ldots,S_N and the general transition equation alone.

Under the sufficient information-flow assumptions discussed in Reviewer ZwPM (W2/Q2), the exponentially weighted bound in Theorem 1 remains a valid conservative guarantee, although the additive expression is tighter. The exponential coefficients characterize the recursive upper-bound construction rather than a proved intrinsic amplification factor. The key contribution is the analysis of local versus joint system-level information disclosure, which remains relevant independently of the tightness of the particular upper bound.

W5 / Q2: Public Inputs and Conditional Independence

The distinction between the upstream representation Oi−1O_{i-1} and stage-specific public input DiD_i is important. Conditional data processing applies to the complete stage input:

I(Oi;S<i∣Si)≤I(Oi−1,Di;S<i∣Si).I(O_i;S_{\lt i}\mid S_i) \leq I(O_{i-1},D_i;S_{\lt i}\mid S_i).

For Oi−1O_{i-1} to be the sole conduit of upstream-secret information, a sufficient condition is

I(S<i;Di∣Oi−1,Si)=0.I(S_{\lt i};D_i\mid O_{i-1},S_i)=0.

Additionally, removing the conditioning on SiS_i requires an appropriate exogeneity condition, such as Si⊥(Oi−1,S<i)S_i\perp(O_{i-1},S_{\lt i}).

These conditions specify the scope of the formal composition guarantee. The empirical experiments separately assess privacy on the observed benchmark distributions through representation-level estimates and independent final-output adversarial inference; their measured improvements do not require interpreting those estimates as certified global MI bounds.

Reviewer_ZwPMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies privacy leakage in sequential multi-agent LLM systems, where individual agents have access to local sensitive information and pass intermediate representations to downstream agents. The authors formalize local leakage and final-output privacy risk using mutual information (MI) and derive an exponentially position-weighted upper bound relating per-agent leakage budgets to global leakage under Markov and independence assumptions.

Motivated by this analysis, the paper proposes MINE-Reg, a training framework that regularizes the mutual information between each agent's output and its local sensitive variables. The method is evaluated on MedQA, FinQA, and PrivacyLens using LLaMA and Qwen models with different pipeline depths. The experiments report substantial reductions in estimated leakage and final-output adversarial inference accuracy, with moderate degradation in benign task performance.

Critical strengths

Privacy leakage across interacting LLM agents is an important problem. The paper considers a well-defined sequential architecture and attempts to connect local information leakage with system-level privacy risk. The focus on information propagation across agent boundaries is practically relevant.

The experiments cover multiple model sizes, agent depths, and application domains. The reported reductions in leakage are substantial, while benign task performance remains relatively high. The additional ablations on privacy weights, selective regularization, and computational overhead are also useful.

Critical weaknesses

It remains unclear what fundamentally new insight this work provides. Privacy-aware multi-stage LLM processing has already been studied by Wenkai Li et al. (LLMSEC 2025, "1-2-3 Check") and Siyan et al. (NAACL 2025, "PAPILLON"). Fan et al. (IJCNN 2025) discuss the limitations of local privacy protection in sequential decision-making and propose information-theoretic approaches to collaborative privacy, while Cundy et al. (AISTATS 2024) establish MI-based privacy regularization. Although these works do not address precisely the same setting, much of the motivation and methodology is already established. The authors do not sufficiently clarify what novel privacy phenomenon or scientific insight arises specifically from sequential LLM-agent composition.

The main sequential-specific contribution is an exponentially position-weighted upper bound on global leakage. However, its tightness and significance are not established. Under natural feed-forward independence assumptions, a tighter additive bound appears possible. Moreover, the proof seems to rely on independence assumptions that are not fully justified. It is therefore unclear whether the exponential weights capture an intrinsic property of sequential leakage or merely reflect looseness in the analysis.

The proposed method applies established MI regularization independently across agent stages. It neither directly minimizes global leakage nor systematically exploits the position-dependent weights derived in the theorem. Consequently, the method appears largely independent of the paper's main theoretical contribution.

The experiments demonstrate the effectiveness of MI regularization but do not isolate the effects of sequential composition. Increasing pipeline depth also changes agent roles and system capacity, while the primary MI metric measures average local rather than global leakage. The experiments therefore do not convincingly validate the claimed position-dependent leakage behavior.

Questions for authors

What is the fundamentally new insight about privacy leakage that follows specifically from the sequential multi-agent setting?

Why is the exponential position-weighted bound necessary or informative?

What evidence shows that the proposed method specifically benefits from the new sequential composition analysis?

3 comments

We thank the reviewer for the detailed assessment. We clarify the requested concerns below.

W1 / Q1: Novelty and Sequential-Specific Privacy Leakage

We thank the reviewer for highlighting these related works. We agree that multi-agent privacy protection and MI-based regularization have been studied; however, our work addresses a different question: how privacy risks introduced at separate agent boundaries interact when their representations are repeatedly transformed and reused downstream.

Distinction from prior work. Li et al. (1-2-3 Check) and Siyan et al. (PAPILLON) primarily design privacy-aware workflows to prevent inappropriate disclosures. Their privacy objectives concern the content released through specific communication steps. Cundy et al. and Fan et al. address information-theoretic privacy in sequential decision-making, where privacy constraints are imposed on policies, actions, or collaborative decisions. Our setting instead involves multiple LLM agents, each introducing a distinct sensitive variable SiS_i, with the final observer potentially recovering information about the joint sensitive context from ONO_N. This requires distinguishing privacy at individual agent interfaces from privacy of the composed system.

Sequential-specific insight. The central issue is not merely that leakage accumulates with more agents. A downstream representation can preserve or transform information about upstream secrets while incorporating new sensitive information, so marginal leakage measurements need not characterize joint disclosure. For example, with independent binary secrets and O1=S1O_1=S_1, O2=O1⊕S2O_2=O_1\oplus S_2, the final output satisfies I(O2;S1)=I(O2;S2)=0I(O_2;S_1)=I(O_2;S_2)=0, yet I(O2;S1,S2)=1I(O_2;S_1,S_2)=1 bit. This illustrates why assessing individual-secret recoverability alone can miss compositional disclosure.

Our contribution is to make this distinction explicit in a sequential LLM-agent model, analyze the relationship between local leakage constraints and final-output risk, and connect intermediate representation regularization to an independently evaluated final-output adversary. Thus, rather than proposing another privacy-aware agent architecture, we study what agent-local privacy control does and does not imply for end-to-end disclosure, and how to mitigate that risk during training.

W2 / Q2: Composition Bound, Assumptions, and Position Dependence

The reviewer raises an important distinction between the validity, tightness, and interpretation of the bound in Theorem 1. We adressess those concerns below.

The direct recurrence requires conditional independence. Let Gi=I(Oi;S1:i)G_i=I(O_i;S_{1:i}). By the MI chain rule,

Gi=I(Oi;Si)+I(Oi;S<i∣Si).G_i= I(O_i;S_i)+I(O_i;S_{\lt i}\mid S_i).

The actual stage transition is Oi=Ai(Oi−1,Di,Si)O_i=A_i(O_{i-1},D_i,S_i); consequently, conditional data processing applies to the complete input:

I(Oi;S<i∣Si)≤I(Oi−1,Di;S<i∣Si).I(O_i;S_{\lt i}\mid S_i) \leq I(O_{i-1},D_i;S_{\lt i}\mid S_i).

The right-hand side cannot generally be replaced by I(Oi−1;S<i)I(O_{i-1};S_{\lt i}) merely because the sensitive variables are mutually independent.

To make the dependence explicit, define

δi=I(S<i;Di∣Oi−1,Si),\delta_i = I(S_{\lt i};D_i\mid O_{i-1},S_i),

κi=I(S<i;Si∣Oi−1).\kappa_i = I(S_{\lt i};S_i\mid O_{i-1}).

Using mutual independence of the sensitive variables and the MI chain rule gives

Gi≤τi+Gi−1+δi+κi.\boxed{ G_i\leq\tau_i+G_{i-1}+\delta_i+\kappa_i. }

Here, δi\delta_i measures additional information about upstream secrets introduced through stage-specific inputs, while κi\kappa_i measures dependence between the current and upstream secrets induced by conditioning on the upstream representation.

Thus, the proposed additive recurrence does not follow from the sequential update rule and marginal secret independence alone. It additionally requires controlling the above conditional-dependence terms.

Relationship to Theorem 1. Under the sufficient no-additional-access and exogeneity conditions

S<i⊥Di∣(Oi−1,Si),Si⊥(Oi−1,S<i),S_{\lt i}\perp D_i\mid(O_{i-1},S_i), \qquad S_i\perp(O_{i-1},S_{\lt i}),

both correction terms vanish. The direct recurrence then gives

GN≤∑i=1Nτi.G_N\leq\sum_{i=1}^{N}\tau_i.

Under these same conditions, the submitted expression

GN≤∑i=1N2N−iτiG_N\leq\sum_{i=1}^{N}2^{N-i}\tau_i

is also a valid, conservative upper bound, although the additive inequality is tighter.

The exponential coefficients arise from the recursive conditional-leakage accounting in Appendix A.1 (Eqs. 17–27). Their presence describes the structure of that particular worst-case upper bound; it does not establish a tight amplification factor or imply that observed leakage necessarily grows exponentially with depth.

This distinction does not negate the compositional privacy problem: even a linear sum of local leakage budgets can grow with pipeline depth, and marginal measurements can miss dependencies revealed jointly at the final output, as demonstrated above.

We therefore distinguish the information-theoretic validity of a conservative composition guarantee from claims of intrinsic position-dependent amplification. For pipelines where stage-specific inputs carry additional secret-correlated information, the δi\delta_i and κi\kappa_i terms make the required assumptions explicit; the original budget-only bound is not asserted for arbitrary side-information dependencies.

W3 / Q3: Connection Between Composition Analysis and MINE-Reg

The theoretical analysis identifies communication boundaries as the locations where sensitive dependence is introduced, retained, and subsequently transformed. MINE-Reg operationalizes this observation by penalizing agent-local dependence during training:

Ltotal=Lutility+∑i=1NβiI^MINE(Oi;Si)\mathcal L_{\mathrm{total}} = \mathcal L_{\mathrm{utility}} + \sum_{i=1}^{N} \beta_i\widehat I_{\mathrm{MINE}}(O_i;S_i)

(Sec. 4.1, Eqs. 7–9).

Why regularize intermediate representations? A final-output privacy mechanism acts only after upstream representations have already been processed by subsequent agents. MINE-Reg instead penalizes sensitive dependence at each stage where it enters the communicated representation. Under the feed-forward information-flow assumptions, downstream propagation of an upstream secret is constrained by the information retained in the representation passed across that boundary.

This is the practical connection between the composition analysis and the objective: intermediate representations are relevant to final-output privacy because they constitute the communication paths through which earlier sensitive information can propagate.

Role of the privacy weights. The coefficients βi\beta_i control the utility–privacy tradeoff and are selected through validation or adaptive dual optimization (Sec. 4.1; Appendix A.7.1). They are not set proportional to 2N−i2^{N-i}. This distinction is intentional: coefficients in a conservative upper bound do not generally determine optimal Lagrange multipliers, which depend on the achievable privacy–utility tradeoff at each stage.

Selective-regularization evidence. Appendix A.7.2 compares unregularized, early-stage-only, and all-stage regularization under a common total privacy budget. Early-stage regularization reduces estimated leakage relative to no regularization; all-stage regularization provides stronger suppression, consistent with sensitive dependence being distributed across multiple communication boundaries.

Conditional-leakage diagnostics. Appendix A.7 (Table 9) also reports stagewise conditional-MI diagnostics on a five-agent Qwen-4B/MedQA pipeline:

Table 1: Conditional leakage estimates on Qwen-4B/MedQA (submitted Appendix Table 9). Values are variational MINE estimates, not certified MI upper bounds.

Methodi=1i=1i=2i=2i=3i=3i=4i=4i=5i=5Sum
Baseline0.610.720.830.941.064.16
MINE-Reg0.040.050.070.080.100.34

The sum of estimated conditional leakage decreases from 4.164.16 to 0.340.34 (91.891.8\\%), with reductions at every stage. These measurements provide a composition-oriented diagnostic complementary to marginal MIavg\mathrm{MI}_{\mathrm{avg}}. Because the critic provides variational estimates, the reported sum is not treated as a certified numerical upper bound on global leakage.

Together, these results establish a concrete empirical connection between stagewise privacy regularization and reduced information leakage across the evaluated pipeline. They do not require assuming that the exponential coefficients prescribe the optimal regularization schedule.

W4: Empirical Evaluation of Sequential Composition

We distinguish variation in pipeline depth from the effect of privacy regularization within a fixed pipeline configuration.

Depth-dependent diagnostics. On MedQA/Qwen-4B, unregularized MIavg\mathrm{MI}_{\mathrm{avg}} increases from 0.540.54 at two agents to 1.021.02 at five agents, while MINE-Reg reduces the corresponding estimates to 0.060.06 and 0.130.13 (Table 1). These are average agent-local MI estimates, not measurements of I(ON;S1:N)I(O_N;S_{1:N}).

Increasing pipeline depth also introduces additional agent roles and sensitive contexts; accordingly, this comparison characterizes the evaluated pipeline configurations rather than isolating an intrinsic mathematical position effect.

Independent final-output validation. The more direct test of the claimed privacy benefit is Sec. 6.5 (Table 3), which evaluates an independently trained adversary observing only ONO_N. The agents are frozen and attacker training uses a disjoint split.

Table 2: Final-output adversarial inference on MedQA/Qwen-4B . Lower attacker accuracy and F1 indicate less recoverable sensitive information.

AgentsMethodMacro Acc. ↓\downarrowMacro F1 ↓\downarrowBS ↑\uparrow
3Baseline0.720.700.92
3MINE-Reg0.300.290.85
5Baseline0.780.760.88
5MINE-Reg0.320.310.81

At three agents, MINE-Reg reduces final-output attacker accuracy from 0.720.72 to 0.300.30 (58.358.3\\% relative reduction); at five agents, accuracy decreases from 0.780.78 to 0.320.32 (59.059.0\\% relative reduction). Benign success remains 0.850.85 and 0.810.81, respectively.

These evaluations are particularly relevant because the attacker observes only the final system output, rather than the intermediate states used by the MINE critics. The results therefore provide privacy evidence independent of the optimization surrogate and directly address the external-observer threat model.

The observed reductions demonstrate substantially lower sensitive-attribute recoverability under the evaluated attack, rather than certified elimination of mutual information. They also do not depend on interpreting the exponentially weighted theorem as a prediction of measured leakage growth.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.