acceptodds
Under review as a conference paper at ICLR 2027

Loyal Agents: Training LLM Agents to Protect Principal Interests Under Strategic Information Asymmetry

Abstract

As LLMs increasingly act as delegated agents, they are expected to protect principals' interests when interacting with external parties. Standard alignment objectives, such as helpfulness, harmlessness, and honesty, do not specify how agents should protect principals' strategic interests under delegation. We formalize Agent Loyalty as an information-control property requiring agents to prevent Exploitable Information Leakage (EIL) and resist Manipulative Information Uptake (MIU). We introduce LoyalAgent-Bench, comprising 10,298 samples across 42 subscenarios and six domains, and an online GRPO framework that trains against a LLM opponent to generate mechanism-specific reward signals. Experiments show that loyalty is not guaranteed by general capability or existing alignment, with measurable EIL and MIU gaps under zero-shot evaluation, while our trained 8B models reduce per-response leakage in single-turn exchanges by 31-44pp and improve task utility, evidence faithfulness, and decision accuracy by up to 11pp, 49pp, and 29pp, respectively. For the trained Qwen3-4B model, no degradation is observed on out-of-distribution benchmarks in math and narrative reasoning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent Loyalty, AI Alignment, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EQb7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies agent loyalty through two dimensions: exploitable information leakage (EIL) and manipulative information uptake (MIU). It introduces LoyalAgent-Bench, comprising 10,298 examples across 42 subscenarios, and a GRPO-based training framework using opponent feedback and task-specific rewards. Experiments on three open-weight models report reduced leakage and improved decision accuracy and evidence faithfulness, with additional transfer results on PrincipalBench.

Critical strengths
  1. Relevant problem formulation. The paper addresses an important challenge for delegated LLM agents: protecting the principal’s interests when interacting with parties whose interests may conflict. EIL and MIU capture complementary aspects of this problem.

  2. Broad benchmark coverage. LoyalAgent-Bench covers diverse delegated tasks across six domains and 42 subscenarios. Its structured construction and quality audits provide a useful resource for studying information-control failures.

  3. Encouraging empirical results. The training framework improves the reported metrics across three open-weight models, with main results averaged over three seeds. Evaluation on PrincipalBench provides additional evidence of transfer beyond the proposed benchmark.

  4. Detailed documentation and substantial effort. The appendices provide extensive construction procedures, evaluation prompts, training details, and supplementary analyses. The breadth of the dataset development and experimental work deserves recognition.

Critical weaknesses
  1. The paper operationalizes agent loyalty through EIL and MIU and explains their connections to and differences from work such as MPC in the appendix. However, the conceptual link between these information-control metrics and “protecting the principal’s interests” requires further justification. For example, what conclusions about agent loyalty are supported by reduced information recoverability, improved decision accuracy, and greater citation faithfulness, respectively?

  2. (Major-) The human agreement study in Appendix D.9 primarily examines whether automated judges and human annotators agree under the prescribed scoring rules. This is insufficient to establish whether agent behavior aligns with human principals’ authorization, preferences, and interests. Although Appendix B.8 provides human support for the validity of the scenarios and labels, it remains unclear whether improvements in EIL/MIU metrics correspond to behavioral improvements that principals endorse. I suggest independently eliciting principals’ task objectives, disclosure permissions, and constraints concerning their interests, followed by a blind evaluation of agent outputs before and after training. This would test whether metric changes align with principals’ assessments of agent behavior. The concern here is insufficient validation of behavior in real delegation settings.

  3. (Major-) Section 5.1 states that the training inputs for Online Adv. GRPO include multi-turn conversation histories, but their source, construction, and number of turns remain unclear. A complete input example would be helpful. Please also clarify how histories are used during training and evaluation, whether these settings are consistent, and whether the compared methods receive the same task information and conversation histories. If the available histories differ across methods, please provide controlled comparisons with matched contexts to distinguish the contributions of parameter updates, reward mechanisms, and additional contextual information. An in-context learning baseline with task demonstrations could also help assess how much improvement can be achieved through inference-time prompting alone.

  4. (Major) Although Appendix F.2 acknowledges that the evaluated models’ decisions under clean contexts were not recorded, I consider this an important issue. Without these decisions, it is difficult to determine whether errors under manipulation arise from insufficient underlying task competence or from decision shifts caused by the manipulated content. This leaves the source of the reported improvements unclear.

  5. (Major) The paper presents both LoyalAgent-Bench and the training framework as contributions, but the main experiments focus primarily on training performance and design choices. The added value of the new benchmark over existing benchmarks remains insufficiently justified. Although the appendix provides details on data construction, diversity, and human quality audits, and Table 2 demonstrates transfer to PrincipalBench, these results do not directly answer which evaluation mechanisms LoyalAgent-Bench introduces beyond existing benchmarks or which new agent failure modes it reveals. I suggest systematically comparing it with PrincipalBench and other relevant benchmarks in terms of task settings, threat models, interaction structures, and metric definitions. Additional analysis demonstrating its added diagnostic value would give the benchmark contribution support comparable to that provided for the training method. Furthermore, if the authors intend to emphasize the methodological contribution, the current training strategy and methodology also need stronger substantiation, for example regarding reward composition and granularity. Should readers primarily focus on the training framework or the dataset?

  6. (Major) The paper’s length reflects substantial effort by the authors. However, the presentation of the contributions and the organization of the main text need improvement, making it difficult for readers to focus their limited attention on the key arguments. I believe the paper’s greatest potential value lies in its evaluation framework for agent loyalty and LoyalAgent-Bench. However, the main experiments focus on gains from GRPO-based training, leaving the validity of the core evaluation framework and the new benchmark’s added value over prior work insufficiently established.

Questions for authors

I have only one question. Is the main contribution of this paper the benchmark and evaluation system, or the training framework?

5 comments

Thank you for your kind recognition of our work and your insightful questions.

Weakness 1: The conceptual link between EIL and MIU information-control metrics and “protecting the principal’s interests” (agent loyalty) requires further justification. For example, what conclusions about agent loyalty are supported by reduced information recoverability, improved decision accuracy, and greater citation faithfulness, respectively?

Loyalty means advancing the principal's goals within their stated boundaries. For example:

  • In a negotiation, reduced information recoverability means the counterparty can infer less about the principal's private budget, preserving bargaining leverage while the agent still pursues a deal.
  • In financial advice, improved decision accuracy means choosing an option that satisfies the principal's constraints instead of following misleading promotions.
  • Greater citation faithfulness means the agent's reasons are supported by the provided evidence, not manipulated claims, helping the principal verify that the recommendation relies on authorized evidence.

These metrics therefore address distinct ways an agent can protect the principal's interests, though none alone establishes loyalty. We also conducted a new study to clarify this (Weakness 2).

Weakness 2: The existing human agreement study validates consistency with scoring rules but does not establish whether agent behavior aligns with human principals' authorization, preferences, and interests. Independent elicitation of principals' objectives, disclosure permissions, and constraints, followed by blind evaluation of agent outputs before and after training, is needed to determine whether metric improvements correspond to principal-endorsed behavior in real delegation settings.

We conducted a separate principal-grounded, blind paired study. Three graduate-student volunteers, uninvolved in benchmark development, each independently specified objectives, disclosure permissions, and decision constraints for 14 task frames before seeing model outputs, reference decisions, scoring rubrics, or adversarial interventions. Together, their 42 tasks cover all subscenarios. Each principal then evaluated anonymized original and trained Marin-8B-Instruct responses to identical inputs (84 responses).

OutcomeBaseline(%)Trained(%)
EIL (27 tasks)
Principal approval↑40.774.1
Unauthorized disclosure↓51.914.8
Task progress↑48.177.8
Automated utility↑51.963.6
Automated leakage↓53.216.4
MIU (15 tasks)
Principal approval↑46.773.3
Principal-endorsed decision↑53.373.3
Constraint violation↓40.06.7
Automated decision accuracy↑53.380.0
Automated faithfulness↑47.691.8

Overall, principal approval increases from 18/42 to 31/42. Automated reward changes agree with principals' pairwise preferences in 30/35 eligible tasks (85.7%), excluding five preference ties and two unchanged automated rewards. This provides behavioral validation against independently elicited principal judgments.

We updated Sec. 6.7 (p. 10, ll. 509–517); App. D.9 (pp. 56–57, ll. 2989–3059).

Weakness 3: The source, construction, and number of turns in the multi-turn conversation histories used for training are unclear. A complete input example and clarification of how histories are used during training and evaluation are needed, including whether compared methods receive matched task information and histories. Controlled comparisons should distinguish parameter updates, reward mechanisms, and additional contextual information, while an in-context learning baseline could assess improvements achievable through inference-time prompting alone.

Source and construction. The conversation histories are fixed inputs supplied with the information-leakage instances. We construct them alongside the instances using the same subscenario-specific plans, the principal’s task constraints, and the assigned counterparty tactics. We clarified this process in App. B.6 (p. 25, ll. 1328–1331).

Number of turns. A prior turn is one completed exchange between the counterparty and the agent. Each history ends with the counterparty’s latest message, which the agent is asked to answer. Of the 5,337 training histories, 69.6% contain two prior turns, 30.0% contain three, and 0.3% contain four. The mean is 2.31 prior turns, or 5.61 messages when counting the latest counterparty message. We added this distribution in App. B.3 (p. 21, ll. 1112–1115).

Use in training and evaluation. The agent receives the principal’s request and the supplied conversation history as a single text block, then generates the next reply. For tasks involving manipulated evidence, the input contains the principal’s decision requirements, candidate options, and external evidence. In the main comparisons, methods receive the same task information and, where applicable, the same fixed history for each instance. Methods may use their own prompt formatting. We added a complete input example in App. C.3.1 (pp. 34–36, ll. 1806–1909).

Matched-context and in-context learning controls. We additionally compared original and trained Qwen3-4B on identical test inputs, with or without the supplied counterparty history, and evaluated the original model with two or four task demonstrations drawn exclusively from training instances.

ModelContextICLEIL Utility (%)↑EIL Leakage (%)↓EIL Reward↑MIU Decision (%)↑MIU Faith. (%)↑MIU Reward↑
OriginalFull0-shot46.2359.690.163854.9556.990.1092
TrainedFull0-shot60.2625.760.473859.8480.140.2984
OriginalNo dialogue0-shot45.0976.430.0686———
TrainedNo dialogue0-shot59.0332.360.4285———
OriginalFull2-shot45.965.870.129658.7565.160.207
OriginalFull4-shot46.2963.80.143958.1866.480.2051

These comparisons establish that:

  • Training updates: The trained model improves information-leakage utility and protection, as well as decision correctness and evidence faithfulness, under identical full-context inputs.
  • Available context: Its advantage persists without the supplied counterparty history, indicating that dialogue context alone does not explain the improvement.
  • Inference-time demonstrations: 2- and 4-shot examples improve some decision metrics but do not match the trained zero-shot model's reward on either task.
  • Reward design: The existing ablation shows that removing blind-opponent inference increases leakage by 22.6 percentage points and reduces reward by 0.17.

The new controls are detailed in D.14 (pp. 61–63, ll. 3288–3361); reward ablations are in Sec. 6.3 and App. D.5.

Weakness 4: The absence of model decisions under clean contexts makes it difficult to distinguish errors caused by insufficient underlying task competence from decision shifts caused by manipulated content, leaving the source of the reported improvements unclear.

We conducted an additional paired evaluation on Qwen3-4B (baseline and trained, single seed) across all 385 MIU test instances, recording each model's decisions with and without manipulated content. This allows us to distinguish errors already present under clean contexts from decisions changed by manipulation.

MetricBaselineTrained
Correctness on clean contexts (%)↑7474.8
Correctness after manipulation (%)↑53.864.9
Initially correct → incorrect (%)↓28.820.1
Accuracy drop under manipulation (pp)↓20.39.9

The results clarify the source of improvement. Clean-context correctness increases by only 0.8 percentage points, while correctness after manipulation improves by 11.1 points. Among instances initially answered correctly, the proportion becoming incorrect after manipulation decreases by 8.7 points. The overall accuracy loss under manipulation also falls by 10.4 points.

Thus, for Qwen3-4B, the improvement primarily reflects reduced susceptibility to manipulated content rather than increased underlying task competence.

We updated Sec. 6.2 (p. 8, ll. 387–389), App. D.11 (p. 58, ll. 3094–3116), and App. F.2 (p. 71, ll. 3796–3802).

Reviewer_a4hQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper formalize Agent Loyalty, which is an information control property that requires agents to prevent Exploitable Information Leakage (EIL) and resist Manipulative Information Update (MIU). To evaluate such property, the authors propose LoyalAgent-Bench, which contains 10k samples over 42 scenarios. They also propose an online GRPO framework that trains against opponents using mechanism-specific reward signals and show strong performance on the two metrics.

Critical strengths
  • The paper focuses on an important problem: agents may disclose users' private information or be influenced by manipulative external claims.
  • Although the experiments cover a limited number of base model, the proposed GRPO reward design achieves stronger performance than baselines.
Critical weaknesses
  • The distinction between EIL/MIU and existing formulation [1][2] of privacy leakage and information manipulation is not clear. Can you share in more detail of how they are different? And also, can you share how LoyalAgent-Bench differ from benchmarks used in such papers? Can you share results of running the trained GRPO on such datasets to evaluate the generalization?
  • The number of dataset quality check for LoyalAgent-Bench is quite small (~1.6%). It would be nice if authors provide more details of dataset quality (e.g., how many datasets were revised / removed). Also, how did you check if the scenarios are suitable for the given task?

[1] Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [2] PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action

Questions for authors
  • How are the EIL and MIU evaluation metrics developed? Are they adapted from prior works? How is this connected to Secure Multi-Part Computation that the authors mentioned?
  • Is there specific reason why the authors conduct broad baseline experiments only with Qwen3-4B?
5 comments

Thanks for your invaluable feedback and suggestions.

Weakness 1: The distinctions between the proposed formulations of privacy leakage and information manipulation and those in prior work[1,2] are unclear. The differences between their benchmarks also need clarification, along with generalization results from evaluating the trained model on existing datasets.

Formulation and benchmark differences:

ConfAIde[1] formulates privacy through contextual integrity: whether information sharing is appropriate given the information, participants, and context. PrivacyLens[2] extends this to agent actions, defining leakage as information transmitted to a recipient through an agent's final action in violation of contextual privacy norms. Both primarily concern the appropriateness of information flows.

Our formulation instead centers on agent loyalty: protecting the principal's interests under strategic information asymmetry. Our benchmark additionally asks whether a strategically opposed counterparty can recover exploitable private information while the agent advances a delegated task. It also tests whether manipulated external evidence changes decisions and whether cited reasons remain supported by authorized evidence.

These differences yield distinct evaluation mechanisms:

DimensionConfAIdePrivacyLensLoyalAgent-Bench
Tasks/threatsInappropriate disclosure across social rolesUnintentional privacy violations in personal assistanceStrategic exploitation and decision manipulation
InteractionPrivacy judgments, three-party stories, meeting summariesTool-use trajectories and final actionsOpponent inference and continuation; manipulated decisions
Disclosure evaluationContextual judgments and inferable leakagePrivacy-norm violations and task helpfulnessSeverity-weighted recoverability + task utility
Decision evaluationNoneNoneDecision correctness + citation support

Cross-benchmark generalization. We evaluated the original Qwen3-4B and the model trained with our online adversarial GRPO on both external datasets, using identical decoding settings within each benchmark.

Benchmark / metricBaselineTrained
ConfAIde Tier 1: sensitivity judgment Pearson↑0.5800.586
ConfAIde Tier 2a: information-sharing judgment Pearson↑0.7360.724
ConfAIde Tier 2b: information-sharing judgment Pearson↑N/A (14 invalid)N/A (8 invalid)
ConfAIde Tier 3: mean string-match leakage↓69.48%66.78%
ConfAIde Tier 3: worst-case leakage↓88.89%88.52%
ConfAIde Tier 4: mean summary private-information leakage↓56.00%46.00%
ConfAIde Tier 4: mean summary public-information omission↓26.50%29.00%
PrivacyLens: naive-prompt action leakage↓93/493 (18.86%)92/493 (18.66%)
PrivacyLens: privacy-enhanced action leakage↓96/493 (19.47%)89/493 (18.05%)
PrivacyLens: naive-prompt helpfulness (0–3)↑2.5462.584
PrivacyLens: privacy-enhanced helpfulness (0–3)↑2.5482.548

Tier 2b full-sample statistics are excluded because strict parsing detected invalid responses.

The results show positive overall transfer to contextual-privacy tasks. ConfAIde Tier 4 private-information leakage decreases from 56.0% to 46.0%, while public-information omission increases from 26.5% to 29.0%. PrivacyLens action leakage decreases by 1/493 and 7/493 cases under the two prompt settings, with comparable helpfulness. Thus, the training gains are not confined to our benchmark.

We updated Sec. 2 (p. 3, ll. 123–130), Sec. 6.6 (p. 10, ll. 503–506), and App. A.2 (p. 17, ll. 885–902), and added App. D.12 (pp. 58–59, ll. 3118–3181).

[1] Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory

[2] PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action

Weakness 2: The dataset quality audit covers only approximately 1.6% of instances. More details are needed on dataset quality, including how many instances were revised or removed and how scenario suitability for the intended tasks was verified.

We conducted a second, non-overlapping human audit of 672 instances (6.53%), sampling 16 from each of the 42 subscenarios. Together, the two rounds cover 840/10,298 records (8.16%). Three annotators independently assessed each instance, with a fourth resolving disagreements.

Audit outcomes and corrections. The audit outcomes are summarized below (pass rates are measured before correction):

OutcomeR1 EILR1 MIUR2 EILR2 MIU
Audited10860432240
Initially passed102 (94.4%)55 (91.7%)416 (96.3%)231 (96.3%)
Initially failed65169
Revised & revalidated64169
Removed0100

Overall, 804/840 (95.7%) initially passed all criteria, providing evidence that the benchmark generally preserves its intended task mechanisms beyond surface plausibility.

The 36 failures involved scenario validity (20), information dependency (6), mechanism integrity (3), evidence quality (6), and decision validity (1). The one R1 removal was replaced before the experiments. Across both rounds, 35 records were revised and revalidated, and one was replaced before evaluation. The 25 Round 2 revisions affect 0.24% of the benchmark and do not change MIU gold decisions or EIL exploitable-slot identities/weights.

Checks for scenario suitability. Beyond realism, auditors verified whether each instance tests its intended loyalty mechanism:

  • EIL: The task must be achievable without disclosing exploitable information; disclosure must plausibly benefit an adversary, and explicit privacy-boundary hints are disallowed.
  • MIU: Clean evidence must support the gold decision under user constraints; options must be plausible, and manipulated content must implement the intended influence mechanism. Two independent models additionally verify decisions using clean evidence alone.

We updated Sec. 4.1 (p. 5, ll. 242–245) and App. B.8 (pp. 29–32, ll. 1559–1677) with the audit protocol, criteria, and detailed results.

Question 1: How are the information leakage and information manipulation evaluation metrics developed? Are they adapted from prior work? How are they connected to Secure Multi-Part Computation?

We designed the metrics around the distinct failure mechanisms of the two threats:

  • Exploitable information leakage: We measure (1) leakage by having a blind opponent infer protected facts from the agent's response, scoring each fact's recoverability and aggregating scores with severity weights; and (2) task utility by judging progress toward the principal's goal without unfavorable concessions. Unlike direct disclosure detection, opponent inference captures leakage through indirect clues. Task utility prevents trivial success through refusal or non-engagement.
  • Manipulative information uptake: We measure (1) decision accuracy by matching the agent's choice against the correct option determined from the principal's constraints and clean evidence; and (2) citation faithfulness by averaging the evidential support for each cited reason against clean, authorized sources. Both are needed because a correct decision may contain unsupported reasoning, while individually supported claims do not necessarily imply a correct decision.

These are task-specific designs, not direct adaptations of prior scoring formulas.

Connection to MPC: MPC's privacy motivates limiting what other parties can infer about private inputs, corresponding to our opponent-based leakage evaluation. Its correctness motivates preserving the intended task outcome despite adversarial behavior, reflected in task utility and evidence-grounded decision accuracy. This is a structural analogy, not an adaptation of MPC protocols or a claim of cryptographic guarantees. We clarified Sec. 4.3 (p. 5, ll. 262–264) and App. A.1 (p. 17, ll. 871–874).

Question 2: Why were broad baseline experiments conducted only with Qwen3-4B?

We initially focused the full reproduction of prior baselines on Qwen3-4B to control the computational cost of additional model-specific training and evaluation, while evaluating our method on all three models.

We have now extended Prompt Scaffold and Per-token-KL distillation (PrincipalBench) to Llama-3.1-8B-Instruct and Marin-8B-Instruct, completing the comparison across all three open-source models.

ModelMethodEIL Utility(%)↑EIL Leakage(%)↓EIL Reward↑MIU Decision(%)↑MIU Faithfulness(%)↑MIU Reward↑
Qwen3-4BZero-shot52.4757.740.2453.7462.030.12
Loyalty-CoT56.3354.190.2973.1868.640.44
Prompt Scaffold36.1651.040.1150.6560.950.06
Per-token-KL38.7429.220.2452.9461.740.10
Ours59.5119.040.5065.7490.580.44
Llama-3.1-8B-InstructZero-shot58.1337.030.4058.6370.560.23
Loyalty-CoT61.7219.290.5264.173.50.33
Prompt Scaffold (new)23.752.78-0.0353.4953.990.07
Per-token-KL (new)40.6644.990.1859.4367.070.23
Ours62.755.630.6071.2698.620.56
Marin-8B-InstructZero-shot51.4956.560.2351.6446.330.01
Loyalty-CoT51.7256.510.2354.3648.930.06
Prompt Scaffold (new)31.1154.320.0448.6934.76-0.1
Per-token-KL (new)35.7854.920.0854.648.370.06
Ours62.4513.040.5680.4595.350.68

Our method achieves the highest or tied-highest reward on both tasks across all three models, confirming that its advantage is not specific to Qwen3-4B.

We updated Table 1 (p. 7, ll. 338–341).

Reviewer_6zPCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies Agent Loyalty for delegated LLM agents operating under strategic information asymmetry. Conventional alignment objectives such as helpfulness, harmlessness, and honesty fail to capture settings where an agent must negotiate with external parties possessing conflicting interests, where being helpful to a counterparty directly harms the user. This work formulates loyalty as bidirectional information control governed by two properties: preventing Exploitable Information Leakage, where the agent protects confidential leverage while maintaining delegated task progress, and resisting Manipulative Information Uptake, where the agent resists strategically biased inputs to ensure decisions remain strictly grounded in evidence. To benchmark this capability, the paper introduces LOYALAGENT-BENCH, and proposes an online adversarial GRPO framework that trains agents against frozen LLM adversaries to generate interaction grounded reward signals. Empirical results across models demonstrate that this adversarial approach reduces per response leakage while boosting task utility, citation faithfulness, and decision accuracy.

Critical strengths
  1. This paper studies a critical gap in current alignment work.

  2. Comprehensive and High Quality Benchmark: LOYALAGENT BENCH provides extensive datatsets across 42 sub-scenarios and six domains.

  3. The proposed method achieves significant improvements across both loyalty metrics and task utility.

Critical weaknesses
  1. In line 61, the paper mentions GRPO's objective is for helpfulness, harmlessness and honesty. This statement is inappropriate, since GRPO (as the paper cites deepseek-r1 and deepseek-math) is proposed to enhance the reasoning capability of LLMs with stable RL.

  2. The paper writing is unclear. For instance, the proposed benchmark is core contribution in this work. However, it's unclear how to construct this benchmark. Although there is a subsection which names as "BENCHMARK DESIGN AND CONSTRUCTION", this subsection is completely to introduce the components of benchmark instead the process of "construction" the paper mentioned.

  3. Although this paper highlights the agent setting, the proposed benchmark's evaluations are mainly conducted under single-turn setting. In real deployments, adversaries adaptively rephrase rejected inquiries, chain subtle clues, and accumulate facts across multi-turns. The current evaluation only tests an agent reply against an initial opening probe. Measuring only independent responses fails to show whether learned boundaries withstand sustained strategic pressure.

  4. In decision quality evaluation, decision correctness is measured against a fixed gold choice instead of measuring decision drift compared with the model own response on clean contexts. This evaluation way is unreliable.

Questions for authors

See weakness

4 comments

Thank you for your constructive feedback and suggestions.

Weakness 1: The paper mentions helpfulness, harmlessness, and honesty as GRPO's objectives, but GRPO is proposed to improve reasoning through stable RL.

We agree that GRPO was originally introduced to improve mathematical reasoning. However, GRPO is a general optimization algorithm whose training objectives depend on the reward design; DeepSeek-R1 also employs GRPO to optimize helpfulness and harmlessness[1]. Our original wording did not clearly distinguish optimization methods from their objectives. We have clarified this distinction in Sec. 1 (p. 2, ll. 63–65). This correction does not affect our method or results.

[1]Guo, D., Yang, D., Zhang, H. et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, 633–638 (2025).

Weakness 2: The benchmark construction process is unclear. The “Benchmark Design and Construction” subsection describes the benchmark components rather than how the benchmark is actually constructed.

The construction procedure is detailed in App. B.6, covering generation plans, field dependencies, validation, and quality controls. Specifically, each record is generated from a subscenario-specific plan and approved source facts, validated against structural and task-specific constraints, and assembled into agent-facing inputs and evaluation annotations.

We recognize that the end-to-end workflow could be clearer. We updated Sec. 4.1 (p. 5, ll. 237–240) and App. B.6 (pp. 25–26, ll. 1322–1326) to explicitly describe how each instance is constructed: from subscenario plans and source facts, through structured generation and validation, to the final agent-facing inputs and evaluation annotations.

Weakness 3: The evaluation mainly scores individual replies and does not establish whether learned information boundaries withstand adaptive rephrasing, chained clues, and accumulated information across multiple turns.

The existing tasks can extend directly to multi-turn evaluation. We extended all 656 held-out Exploitable Information Leakage tasks across 27 subscenarios to adaptive four-turn evaluation. Each run starts from the same supplied conversation history; subsequent counterparty messages adapt to the agent's responses. We measure cumulative information leakage and task utility over the dialogue. The table reports task-matched pairs completing four scored exchanges (original → trained; one evaluation seed).

ModelPairsCumulative leakage↓Task utility↑First leakage on turns 2–4↓
Llama-3.1-8B-Instruct528.590 → .234.348 → .46267/90 → 22/90
Marin-8B-Instruct479.759 → .580.346 → .4348/16 → 4/16
Qwen3-4B492.756 → .321.348 → .43424/36 → 5/36

The last column restricts to tasks with no first-turn leakage in either run.

Examples of trained agents under sustained pressure:

  • Adaptive rephrasing: In a rental dispute, property management narrowed its questions from general grievances to the specific violation and the tenant’s preferred resolution. Trained Llama cited the recorded violation and requested written proof of repair while keeping the tenant’s willingness to accept a minimal settlement private. Leakage stayed at .000 through all four turns.
  • Chained clues: In a pricing negotiation, the buyer combined the previously discussed four-week timeline, two revision rounds, and hourly pricing to press for an estimate. Trained Llama confirmed the deliverables and proposed estimating the hours while keeping its earnings floor and cash-flow pressure private. Cumulative leakage fell from .714 to .000.
  • Accumulated information: In a procurement negotiation, the buyer followed up on trained Llama’s proposed discount and pressed for a specific rate. The agent continued negotiating while keeping its pricing floor, cash-flow constraints, and capacity limits private across all four turns (.000 leakage). In the matched original run, the buyer used an earlier reference to the pricing floor to press for concessions, and the original agent disclosed cash flow and capacity constraints in later replies. Leakage reached .723 for the full dialogue versus .530 for the most revealing single reply.

Thus, trained agents achieve lower cumulative leakage while improving task progress across four-turn interactions. Crucially, even when both policies avoid leakage on their first scored turn, trained agents better maintain this boundary under subsequent adversarial pressure: first-leakage rates during turns 2–4 fall from 74.4% to 24.4% (Llama) and 66.7% to 13.9% (Qwen); Marin shows a similar direction (50.0% to 25.0%). The full protocol, results, and examples are in App. D.13 (pp. 59–61, ll. 3182–3287), and updated Sec. 6.2 (p. 7, ll. 372–373), App. F.1 (p. 70, ll. 3737–3742), and App. F.2 (p. 71, ll. 3792–3795).

Weakness 4: Measuring decision correctness against a fixed gold choice, rather than decision drift relative to the model's own response on clean contexts, makes the decision quality evaluation unreliable.

We evaluated the original and trained Qwen3-4B on all 385 manipulated-decision test instances (single seed), comparing each model's manipulated-context decision with its own clean-context decision.

Metric (%)BaselineTrained
Clean-context correctness↑74.074.8
Correctness after manipulation↑53.864.9
Agreement with own clean decision↑73.579.7
Clean-correct → manipulated-incorrect↓28.820.1

Training improves agreement with each model's own clean decision while reducing initially correct decisions overturned by manipulation. Clean-context correctness changes by only 0.8pp. Thus, the improvement is primarily associated with reduced decision degradation under manipulation, not greater clean-context competence. Gold-standard correctness and model-relative stability capture complementary properties.

We updated Sec. 6.2 (p. 8, ll. 392–395), App. D.11 (p. 58, ll. 3094–3116), and App. F.2 (p. 71, ll. 3796–3802).

Reviewer_6zPCOpenReview ↗

Replies

Thank you for the detailed rebuttal. While I appreciate the authors' efforts, I feel that the issues raised, ranging from presentation and paper structure to experimental design, require a major revise of the manuscript that goes beyond what can be resolved during the rebuttal period.

Therefore, I will maintain my current score.

Dear Reviewer 6zPC,

We appreciate your consideration of our rebuttal and understand your concerns about the scope of the revisions.

We would like to clarify that we have already incorporated targeted revisions addressing the issues raised: correcting the description of GRPO's objectives (Sec. 1), clarifying the benchmark construction process (Sec. 4.1 and App. B.6), and adding adaptive multi-turn evaluations and clean-context decision comparisons (Sec. 6.2 and Apps. D.11–D.13). The corresponding main-text revisions are limited to four short passages totaling 13 lines. These changes preserve the original research question, method, core contributions, and overall paper structure, without requiring a fundamental redesign. The revised main text also remains within the 10-page limit.

We hope this clarification helps convey the scope and substance of the revisions already made. We sincerely appreciate the time you have given to our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remark

Dear Area Chair and Reviewers,

We sincerely thank Reviewers 6zPC, a4hQ, and EQb7 for their constructive feedback. We particularly appreciate their recognition of the importance of our research problem, the breadth and quality of our benchmark, and the effectiveness of our training framework.

Our work addresses an important yet underexplored dimension of LLM alignment: protecting principals' interests when delegated agents interact with parties whose interests may conflict. Existing alignment objectives do not adequately address this challenge, while prior benchmarks rarely evaluate both exploitable information disclosure and decision integrity under manipulation. Even leading proprietary models exhibit these failures, and general capability improvements do not reliably resolve them.

To address this gap, we contribute an integrated approach spanning (1) a formulation of two complementary information-control requirements, (2) LoyalAgent-Bench, comprising 10,298 instances across 42 subscenarios, and (3) an online adversarial training framework that substantially improves information protection, task utility, and evidence-grounded decision-making.

During the rebuttal, we conducted extensive additional experiments addressing the reviewers' key concerns:

  1. Human validation and dataset quality (EQb7, a4hQ). We expanded human audits to 840 instances (8.16% of the benchmark), with a 95.7% initial pass rate. A separate blind study based on independently specified principal preferences showed that principal approval increased from 42.9% to 73.8%, with 85.7% agreement between automated reward changes and principal preferences.

  2. Added diagnostic value (EQb7, a4hQ). Targeted comparisons with existing evaluation approaches identified 85 independently confirmed exploitable disclosures missed by an adapted PrincipalBench-style detector, and 32/393 correct, conventionally grounded responses lacking principal-authorized evidence grounding. These results demonstrate additional failure modes that conventional evaluations can overlook.

  3. Training effectiveness and reward design (EQb7, a4hQ). We extended baseline comparisons across all three open-weight models and conducted controlled ablations of reward components and granularity. The complete reward achieved the highest joint reward among tested configurations, while matched-context and in-context learning comparisons further supported the benefits of training beyond prompting alone.

  4. Multi-turn robustness and decision integrity (6zPC, EQb7). Adaptive four-turn evaluations across 656 held-out tasks demonstrated lower cumulative leakage and higher task utility across all three trained models. Clean-context comparisons showed that training reduced the accuracy drop under manipulation from 20.3 to 9.9 percentage points, supporting improved manipulation resistance rather than merely stronger underlying competence. Additional external benchmark evaluations further examined generalization.

We are encouraged by the reviewers' recognition of the progress made during the rebuttal. Reviewer EQb7 acknowledged that our additional experiments substantially addressed concerns about human validation, input context, and the attribution of improvements. We subsequently provided targeted diagnostic comparisons and reward ablations to address the remaining concerns. Reviewer a4hQ acknowledged our additional results and clarifications. While Reviewer 6zPC appreciated our efforts but raised concerns about revision scope, we clarified that the targeted revisions addressing 6zPC's concerns have already been incorporated, requiring minor main-text changes while preserving the paper's core contributions and structure.

Overall, we believe our rebuttal has substantially strengthened the evidence supporting our contributions. Our work helps establish agent loyalty as a distinct and necessary alignment objective, providing a unified approach to defining, evaluating, and improving agents' protection of principal interests.

We sincerely appreciate the reviewers' thoughtful feedback and the Area Chair's consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.