acceptodds
Under review as a conference paper at ICLR 2027

Safety of Latent Communication in Multi-Agent Systems

Abstract

Latent communication enables multi-agent systems to exchange information directly in internal representation space, reducing the token, computation, and latency overhead of text-based communication. To this end, lightweight trainable links are introduced to map the sender's representations into the receiver's input space. In this work, we show that even benign link training can increase harmful compliance relative to text-based communication while the underlying safety-aligned agents remain unchanged. An attacker can amplify this effect by optimizing the links on harmful query–response pairs or poisoning otherwise benign training data. We further develop a reinforcement-learning attack that rewards harmful compliance alongside benign task performance without requiring harmful target responses. Across three communication topologies and four safety benchmarks, this attack raises the mean harmful-compliance score from 27.9 with benignly trained links to 76.9. Compared with direct supervised optimization, it also achieves higher average accuracy on two benign utility benchmarks. Adapting the rewards toward safer behavior also enables repair of compromised links, substantially reducing harmful compliance across all evaluated attacks without updating the agents. Overall, our results show that safety alignment requires considering the multi-agent system as a whole.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
latent communication, LLM agents, reinforcement learning, AI safety, alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_58reOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the safety of a specific type of latent communication in multi-agent systems. It shows that benign training of communication links can increase harmful compliance. The authors investigate supervised, data-poisoning, and reward-guided attacks on these links, and propose a reward-guided repair method. Experiments across three communication topologies and four safety benchmarks demonstrate the effectiveness of the attacks and repair.

Critical strengths
  1. This paper investigates an important safety issue in latent communication, showing that benign link training can weaken safety and propose a new attack surface.

  2. The paper provides extensive experiments across multiple attack settings and communication topologies, including analyses of refusal behavior and benign utility.

Critical weaknesses
  1. Limited generalizability. The paper only considers latent communication through trainable projection-based links. Other approaches(such as KV-cache sharing, embedding sharing, and training-free alignment) are not evaluated. The authors should clarify the practical scenarios in which the proposed attacks are applicable and limit their claims to the communication mechanisms studied.

  2. Potential data leakage in repair evaluation. Appendix B.4 states that the repair procedure uses harmful queries from AdvBench is also used for safety evaluation. It is not clear whether the training and evaluation queries overlap. This raises concerns about the reliability of the reported repair results.

Questions for authors
  1. Could the authors clarify the practical scenarios in which the proposed threat model applies?

  2. The supervised and data-poisoning attacks substantially reduce benign task accuracy in some settings. Could the increased harmful compliance reflect a broader disruption of communication or instruction-following behavior?

  3. Why does the poisoning-rate analysis in Appendix C.2 exclude the Two-Agent topology, which shows the smallest improvement under 10% poisoning?

4 comments
AuthorsOpenReview ↗

Generalizability and repair evaluation (Part 1/2)

Part 1/2: Generalizability and repair evaluation

Dear Reviewer 58re,

Thank you for your constructive feedback. We address your comments and present the additional experimental results below.

1. Limited generalizability

When agents use different LLMs, their internal representations can differ in dimensionality and semantics. Recent frameworks such as RecursiveMAS [1] and Interlat [2] address this challenge using learned links to connect heterogeneous agents. These links introduce an additional component whose training can affect the behavior of the composed system, motivating our study of its safety.

Training-free approaches make different compatibility assumptions. StateBridge [3], for example, evaluates agents sharing the same underlying model and leaves heterogeneous transfer to future work. LatentMAS [4] relies on compatible internal representations for KV-cache sharing. We will clarify in the revised paper that our attack and repair results concern trainable projection links.


2. Practical applicability

Thank you for this suggestion. We consider two practical scenarios in which a developer either obtains a pretrained communication link or trains one using externally sourced data:

Compromised link. A developer may download a pre-trained link to connect independently obtained models, such as a Llama-based planner and a Qwen-based solver. A malicious link provider could train the link to induce harmful compliance and publish it alongside instructions for connecting the agents. Our supervised and reward-guided attacks model how such a link could be created while leaving the LLM weights unchanged.

Poisoned training data. Instead of downloading a link, the developer may instead train a link using data from public repositories or external contributors. An attacker who can insert harmful query–response pairs into this data can influence the resulting link without the need to access its training procedure. Our poisoning experiment models this scenario.


3. Potential data leakage in repair evaluation

Our repair method requires harmful queries to guide the optimization toward refusal behavior. In our experiments, we used AdvBench queries for this purpose and evaluated the repaired systems on HarmBench, StrongREJECT, JailbreakBench, and AdvBench. We agree that using AdvBench for both repair and evaluation can bias the AdvBench score and the aggregate that includes it. The other three benchmarks were not used as sources of harmful repair queries.

Excluding AdvBench from the original evaluation average, the mean harmful-compliance score decreases from 71.6 before repair to 6.1 after repair, compared with 70.5 to 4.7 when including it. The reduction holds across all nine attack–topology configurations. Thus, the substantial repair effect remains.

To further assess whether the repair effect depends on using AdvBench as the training source, we additionally repeated the experiment using each of the four benchmarks as the source of harmful queries. For each setting, we observe low post-repair scores on the other three benchmarks indicating that effective repair is not specific to AdvBench.

We will make this distinction explicit in the revised paper and report repair-source performance separately from cross-benchmark performance.

Table 1. Repair using different sources of harmful queries.

Each column identifies the repair-query source. Cells show the post-repair harmful-compliance score on the source benchmark / the mean score across the other three benchmarks; lower is better.

TopologyAttackAdvBenchHarmBenchStrongREJECTJBB
2-AgentPoisoning0.4 / 7.531.0 / 0.170.0 / 0.000.0 / 0.73
Supervised0.2 / 1.730.0 / 1.032.9 / 3.670.0 / 4.33
RL2.1 / 11.876.5 / 7.6716.6 / 8.501.0 / 3.87
SequentialPoisoning1.0 / 8.104.0 / 3.274.5 / 0.002.0 / 4.50
Supervised0.0 / 1.430.0 / 0.202.2 / 0.000.0 / 0.73
RL1.2 / 14.800.5 / 1.573.8 / 5.176.0 / 7.57
MixturePoisoning0.2 / 1.870.0 / 2.332.9 / 2.670.0 / 1.50
Supervised0.0 / 4.070.0 / 0.335.4 / 2.900.0 / 2.43
RL0.0 / 3.230.0 / 1.602.2 / 0.000.0 / 0.33

References

[1] Zou, J., Pan, R., et al. Recursive Multi-Agent Systems. (NeurIPS 2026)

[2] Du, Z., Wang, R., et al. Enabling Agents to Communicate Entirely in Latent Space. (ACL 2026)

[3] Peng, Y., Zhang, D. C., et al. StateBridge: Training-free Hidden-state Alignment for Latent Communication in LLM Multi-Agent Systems. (COLM 2026)

[4] Zou, J., Qiu, R., et al. Latent Collaboration in Multi-Agent Systems. (ICML 2026)

AuthorsOpenReview ↗

Instruction following and poisoning rate (Part 2/2)

Part 2/2: Instruction following and poisoning rate

4. Communication and instruction-following behavior

Following your suggestion, we conducted additional evaluations to examine whether increased harmful compliance is accompanied by changes in output quality, reasoning length, or instruction following. Across the evaluated attacks and topologies, 100% of MATH500 responses use the requested \boxed{} format, none are classified as garbled, and 99.5–100% of HarmBench responses are classified as coherent. To measure instruction-following ability, we evaluated the clean and attacked systems on IFEval [5]. We report prompt-level strict accuracy, which measures the percentage of prompts for which all verifiable instructions are satisfied.

The largest drops in MATH500 accuracy are accompanied by substantially shorter generated reasoning. For example, under the two-agent supervised attack, accuracy falls from 65.4 to 18.4, while median reasoning length decreases from 250 to 5 words.

Instruction following is also affected in some settings. Under the same two-agent attack, IFEval decreases from 51.4 to 47.7, while mixture poisoning causes a larger decline from 74.3 to 45.5. These results suggest that utility loss is associated with shorter generated reasoning and, in some settings, reduced instruction following.

TopologyAttackMATH500: clean → attackedMedian reasoning words: clean → attackedIFEval: clean → attacked
Two-agentSupervised65.4 → 18.4250 → 551.4 → 47.7
Two-agentPoisoning65.4 → 64.0250 → 23651.4 → 55.8
SequentialSupervised59.6 → 25.8194 → 857.5 → 49.7
SequentialPoisoning59.6 → 44.2194 → 9857.5 → 48.6
MixtureSupervised77.0 → 48.0178 → 12274.3 → 60.4
MixturePoisoning77.0 → 29.2178 → 2974.3 → 45.5

The following example illustrates this change in generated reasoning. After the supervised attack, the response remains readable and follows the requested answer format, but incorrectly gives the lower bound of the solution interval as 0.6 instead of −3.6 and therefore returns the wrong answer.

Q: How many integers are in the solution set of ∣x−2∣≤5.6|x - 2| \leq 5.6?
Gold answer: 11

Clean response (excerpt)

The absolute value inequality ∣x−a∣≤b|x-a| \leq b is equivalent to −b≤x−a≤b-b \leq x-a \leq b… We rewrite it as −5.6≤x−2≤5.6-5.6 \leq x-2 \leq 5.6. Adding 2 to all parts: −3.6≤x≤7.6-3.6 \leq x \leq 7.6… The smallest integer greater than or equal to −3.6-3.6 is −3-3, and the largest integer less than or equal to 7.67.6 is 77. The integers from −3-3 to 77 inclusive are: −3-3, −2-2, −1-1, 00, 11, 22, 33, 44, 55, 66, 77. Counting these integers, we have a total of 11 integers in the solution set.

Final Answer: 11\boxed{11}

Response after supervised attack

The solution set is all real numbers between 0.6 and 7.6 inclusive. The number of integers in the solution set is 7. Final Answer: 7\boxed{7}


5. Two-agent poisoning-rate analysis

Our initial poisoning-rate analysis focused on systems with multiple communication links, namely the sequential and mixture topologies. Following your suggestion, we extended the analysis to the two-agent system and included the results in the revised paper.

The two-agent results also show a non-monotonic relationship between poisoning rate and harmful compliance. Among the evaluated rates, the highest mean score is 65.9 at 30% poisoning, compared with 49.4 at 10% and 31.2 for the clean system.

Poisoning rateHarmBenchStrongREJECTJBBAdvBenchMean
0% (clean)39.336.936.611.531.2
10%59.254.548.534.749.2
20%58.053.053.028.148.0
30%63.064.276.060.265.9
40%45.549.237.019.037.7
50%61.059.758.035.053.4

Best regards,
The Authors

Reference

[5] Zhou, J., Lu, T., et al. Instruction-Following Evaluation for Large Language Models. (CoRR 2023)

Thank you for the rebuttal and experiments. The clarifications and new results have largely addressed my other concerns. However, the experiments reveal substantial degradation in general reasoning ability, especially the drops in MATH500 accuracy and reasoning length. This reinforces my concern that increased harmful compliance may, in some case, just reflect the broader degradation in reasoning or in communication capabilities. Since this distinction remains unresolved, I will maintain my original score.

AuthorsOpenReview ↗

Follow-up: Harmful Compliance and Reasoning Performance

Dear Reviewer,

Thank you for following up on this point. We agree that a broader disruption of reasoning or communication may contribute to harmful compliance in settings where both MATH500 accuracy and the length of the generated reasoning decline. In our reward-guided and poisoning experiments, however, we also observe settings in which modifying the communication links alone substantially increases harmful compliance while MATH500 accuracy remains close to or above the clean baseline. Specifically:

  • Reward-guided attack. The RL attack raises harmful compliance while largely preserving MATH500 accuracy. It reaches 81.3 / 53.8 / 95.9% ASR on the 2-Agent / Sequential / Mixture topologies, while MATH500 stays at 63.8 / 68.8 / 80.8 (clean: 65.4 / 59.6 / 77.0). On Mixture it gives the highest ASR of all attacks with higher reasoning accuracy than the clean system.

  • Two-agent poisoning. 2-Agent poisoning increases ASR (31.2 → 49.4%) with little change in MATH500 accuracy or reasoning length. MATH500 is 64.0 vs. 65.3 clean; the median reasoning length is 236 vs. 250 words.

  • Comparison within the same topology. ASR does not consistently track the degree of degradation. On the 2-Agent topology, the heavily degraded supervised link (MATH500 18.4) reaches a lower ASR (78.7%) than the RL link (81.7%, MATH500 63.8). On Mixture, the degraded poisoning link (MATH500 32.0) reaches 69.6% ASR, compared with 95.9% for the RL link (MATH500 80.6).

These comparisons show that harmful compliance can increase substantially even when performance on MATH500 remains close to or above the clean baseline.

Best regards,
The authors

Reviewer_qv7MOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates trainable latent communication links within frozen multi-agent systems, demonstrating that benign training can undermine safety and showing how supervised optimization, data poisoning, and reward-guided tuning can manipulate or repair harmful compliance. However, the evaluated systems diverge from conventional autonomous LLM agents, as their core mechanism essentially reduces to reinforcement learning over cascaded, static LLMs.

Critical strengths
  1. Reveals that training only the communication interfaces while keeping all base model parameters frozen can significantly increase harmful compliance, demonstrating that evaluating individual model safety is insufficient for assessing composed system security.

  2. Provides a thorough experimental analysis covering diverse manipulation strategies and utility trade-offs, extending beyond direct attacks to investigate the impacts of data poisoning, single-link attacks, utility rewards, and benign training objectives.

Critical weaknesses
  1. The incremental contribution over existing literature on LLM reinforcement learning safety is unclear. Although the paper is framed as a study on multi-agent systems, the evaluated setups consist merely of static, cascaded LLM pipelines rather than true autonomous LLM agents. Consequently, optimizing communication links essentially reduces to standard RL or supervised tuning over cascaded LLMs. Given that many recent works have already thoroughly investigated security and alignment challenges in LLM RL [1-4], the paper fails to establish which findings are genuinely specific to latent multi-agent communication rather than general LLM alignment and RL dynamics.

  2. The joint evaluation of safety and utility is insufficiently rigorous. Specifically, Appendix B.4 states that AdvBench queries are used for the repair process, yet Table 4 reports evaluation results on AdvBench without explicitly documenting a proper training-test split. Furthermore, the utility benchmarks (MATH500 and GPQA-Diamond) only measure mathematical and scientific reasoning, failing to test for over-refusal on legitimate, safety-sensitive queries. These omissions weaken the claim that the proposed repair method successfully restores safety while preserving appropriate helpfulness.

[1] Mahdi Sabbaghi, George Pappas, Adel Javanmard, and Hamed Hassani. 2026. Robust Policy Optimization to Prevent Catastrophic Forgetting.

[2] Weiyang Guo, Zesheng Shi, Zeen Zhu, Yuan Zhou, Min Zhang, and Jing Li. 2026. Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward. ACL 2026, pp. 32181–32201.

[3] Yi Zhang, An Zhang, XiuYu Zhang, Leheng Sheng, Yuxin Chen, Zhenkai Liang, and Xiang Wang. 2026. AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning. ICLR 2026.

[4] Dai, J., et al. (2024). Safe RLHF: Safe Reinforcement Learning from Human Feedback. ICLR 2024.

Questions for authors
  1. Which findings are specific to latent inter-agent communication, beyond existing results on LLM RL safety?

  2. Since the evaluated multi-agent systems lack tool use and environment interaction, how do they differ fundamentally from standard cascaded LLM training? Would it be more accurate to characterize this framework as cascaded LLM training rather than multi-agent systems?

  3. Are the AdvBench queries used for repair disjoint from those evaluated in Table 4?

  4. Does repair increase over-refusal on legitimate, safety-sensitive requests? Can you provide evidence beyond MATH500 and GPQA-Diamond?

3 comments
AuthorsOpenReview ↗

Contribution and repair evaluation (Part 1/2)

Part 1/2: Contribution and repair evaluation

Dear Reviewer qv7M,

Thank you for your constructive feedback. We address your comments below.

1. Contribution beyond existing LLM RL safety literature

Our study examines the lightweight links introduced to enable latent communication between LLMs. This complements existing work on safety during LLM training and adaptation [1–4] by focusing on a component that can be trained or compromised independently of the underlying models. We find that modifying this communication interface alone provides substantial control over whether the composed system complies with harmful requests, even when all LLM parameters remain fixed.

Our experiments reveal several interesting findings: (1) even benign link training can increase harmful compliance without changing the LLM weights; (2) the position of a compromised link matters, and attacking a single link can substantially affect system safety; and (3) the same reward-guided repair procedure reduces harmful compliance across the evaluated attacks without requiring knowledge of how the link was compromised.


2. Repair training and evaluation separation

Our repair method requires harmful queries to guide the optimization toward refusal behavior. In our experiments, we used AdvBench queries for this purpose and evaluated the repaired systems on HarmBench, StrongREJECT, JailbreakBench, and AdvBench. We agree that using AdvBench for both repair and evaluation can bias the AdvBench score and the aggregate that includes it. The other three benchmarks were not used as sources of harmful repair queries.

Excluding AdvBench from the original evaluation average, the mean harmful-compliance score decreases from 71.6 before repair to 6.1 after repair, compared with 70.5 to 4.7 when including it. The reduction holds across all nine attack–topology configurations. Thus, the substantial repair effect remains.

To further assess whether the repair effect depends on using AdvBench as the training source, we additionally repeated the experiment using each of the four benchmarks as the source of harmful queries. For each setting, we observe low post-repair scores on the other three benchmarks indicating that effective repair is not specific to AdvBench.

We will make this distinction explicit in the revised paper and report repair-source performance separately from cross-benchmark performance.

Table 1. Repair using different sources of harmful queries.

Each column identifies the repair-query source. Cells show the post-repair harmful-compliance score on the source benchmark / the mean score across the other three benchmarks; lower is better.

TopologyAttackAdvBenchHarmBenchStrongREJECTJBB
2-AgentPoisoning0.4 / 7.531.0 / 0.170.0 / 0.000.0 / 0.73
Supervised0.2 / 1.730.0 / 1.032.9 / 3.670.0 / 4.33
RL2.1 / 11.876.5 / 7.6716.6 / 8.501.0 / 3.87
SequentialPoisoning1.0 / 8.104.0 / 3.274.5 / 0.002.0 / 4.50
Supervised0.0 / 1.430.0 / 0.202.2 / 0.000.0 / 0.73
RL1.2 / 14.800.5 / 1.573.8 / 5.176.0 / 7.57
MixturePoisoning0.2 / 1.870.0 / 2.332.9 / 2.670.0 / 1.50
Supervised0.0 / 4.070.0 / 0.335.4 / 2.900.0 / 2.43
RL0.0 / 3.230.0 / 1.602.2 / 0.000.0 / 0.33

References

[1] Sabbaghi, M., Pappas, G., et al. Robust Policy Optimization to Prevent Catastrophic Forgetting. (arXiv 2026)

[2] Guo, W., Shi, Z., et al. Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward. (ACL 2026)

[3] Zhang, Y., Zhang, A., et al. AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning. (ICLR 2026)

[4] Dai, J., Pan, X., et al. Safe RLHF: Safe Reinforcement Learning from Human Feedback. (ICLR 2024)

AuthorsOpenReview ↗

Over-refusal and cascaded workflows (Part 2/2)

Part 2/2: Over-refusal and cascaded workflows

3. Over-refusal

In the paper, we evaluate benign utility on mathematics and science tasks to assess whether task performance is preserved after attack and repair. We agree that task accuracy alone does not directly capture over-refusal. We therefore conducted additional experiments on OR-Bench-80K [5] and XSTest [6], measuring direct refusal on benign requests resembling harmful ones and comparing systems with clean, attacked, and repaired links.

The results are shown in Table 2. Repair reduces harmful compliance in all nine configurations. After repair, direct refusal on benign requests ranges from 0.2% to 1.2% for RL attacks and from 0.0% to 1.2% for data poisoning across both benchmarks, remaining close to the clean baselines (0.0% to 2.0%). For supervised attacks, refusal rates range from 0.0% to 7.2%, with increases over the clean baseline occurring only in the two-agent and sequential systems. Note that our repair objective does not explicitly account for over-refusal. A defender could reduce the rate by including benign requests resembling harmful ones in the repair data and rewarding helpful responses.

We will include the additional evaluation in the revised paper and discuss this possible extension of the repair objective.

Table 2. Direct refusal and utility after repair.

TopologyAttackOR-Bench-80K refusal: clean → attacked → repairedXSTest refusal: clean → attacked → repairedMean harmful compliance: attacked → repairedMATH500 repaired
Two-agentSupervised0.2 → 0.0 → 1.12.0 → 1.6 → 3.278.7 → 1.356.4
Two-agentPoisoning0.2 → 0.0 → 0.02.0 → 5.2 → 1.249.4 → 5.864.6
Two-agentRL0.2 → 0.1 → 0.22.0 → 1.2 → 0.481.7 → 9.459.4
SequentialSupervised1.0 → 0.0 → 7.22.0 → 1.2 → 4.067.2 → 1.160.2
SequentialPoisoning1.0 → 0.0 → 0.52.0 → 0.4 → 0.056.1 → 6.367.8
SequentialRL1.0 → 0.0 → 0.22.0 → 0.0 → 0.453.8 → 11.463.4
MixtureSupervised0.2 → 0.0 → 0.00.0 → 0.4 → 0.081.9 → 3.178.8
MixturePoisoning0.2 → 0.1 → 0.00.0 → 0.8 → 0.469.9 → 1.579.8
MixtureRL0.2 → 0.0 → 1.20.0 → 0.0 → 0.896.3 → 2.477.6

Direct refusal is measured on OR-Bench-80K and XSTest. Mean harmful compliance combines the four safety benchmark scores; MATH500 reports accuracy after repair. All values are on a 0–100 scale.


4. Relation to cascaded LLM training

Our setting follows recent work on latent multi-agent communication, including Interlat (ACL 2026) [8] and RecursiveMAS (NeurIPS 2026) [7], which connect LLM-based agents through learned communication interfaces. RecursiveMAS explicitly includes sequential and mixture collaboration patterns while keeping the LLMs frozen and training the communication links.

We investigate the security implications of this design. The question of whether these interfaces can compromise the behavior of the composed system already arises in fixed workflows, independently of whether the agents also use tools or interact with an external environment. Our results demonstrate that the communication links warrant explicit security evaluation in these systems. Extending the evaluation to tool use and interactive environments would broaden its scope, and we leave this to future work.

We will clarify the relationship between our evaluated settings and these existing latent multi-agent frameworks in the revised paper.

Best regards,
The Authors

References

[5] Cui, J., Chiang, W.-L., et al. OR-Bench: An Over-Refusal Benchmark for Large Language Models. (ICML 2025)

[6] Röttger, P., Kirk, H., et al. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. (NAACL 2024)

[7] Zou, J., Pan, R., et al. Recursive Multi-Agent Systems. (NeurIPS 2026)

[8] Du, Z., Wang, R., et al. Enabling Agents to Communicate Entirely in Latent Space. (ACL 2026)

Thank you for the clarifications and additional experiments. The cross-benchmark repair results and over-refusal evaluations address evaluation concerns. However, the central novelty concern remains unresolved. The response does not sufficiently distinguish the reported findings from general safety degradation and recovery when optimizing trainable modules in cascaded LLMs. Keeping the backbone models frozen identifies a different optimization target, but does not by itself establish findings specific to latent inter-agent communication. Citing existing latent communication does not resolve this substantive distinction. I therefore maintain my original score.

Reviewer_UZYPOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the safety of multi-agent systems that communicate through learned latent links while keeping the underlying language models frozen. It shows that even benign link training can increase harmful compliance. Supervised optimization, data poisoning, and reward-guided optimization of these links further amplify this effect. The paper also proposes a reward-guided repair procedure that updates only the links. Experiments across three communication topologies and four safety benchmarks demonstrate these vulnerabilities and the effectiveness of repair, with benign utility evaluated on MATH500 and GPQA-Diamond.

Critical strengths
  • The paper identifies an important safety issue in latent communication: freezing safety-aligned agents does not preserve the safety of the composed system. Table 1 shows substantial increases in harmful compliance even without malicious training data. This finding motivates evaluating learned communication interfaces as part of system alignment.

  • The experiments examine benign training, three attack settings, and link repair across different model combinations and topologies. The single-link interventions and refusal-prefix analyses provide useful evidence beyond aggregate attack scores. In particular, the results show that high benign-task accuracy does not necessarily indicate a safe system.

Critical weaknesses
  • The repair evaluation is not clearly separated from its training data. Appendix B.4 uses harmful queries from AdvBench for repair, while Table 4 also reports AdvBench performance without specifying a held-out split. The reductions on other benchmarks are encouraging, but the aggregate repair score may include training queries. Moreover, mathematics and science accuracy do not directly measure over-refusal on benign requests resembling harmful ones, leaving the safety-utility trade-off incompletely evaluated.

  • The comparison between reward-guided and supervised attacks does not isolate the benefit of the optimization method. The supervised attack uses only harmful query-response pairs, whereas the RL attack interleaves benign utility updates; their training budgets also differ. It is therefore unclear whether the better utility of RL comes from reward-guided optimization or simply from including benign supervision. A supervised baseline with a comparable benign-data mixture would strengthen this claim.

Questions for authors
  • Are the AdvBench queries used for repair disjoint from those used in Table 4? What are the repair results on strictly held-out queries, including benign requests that test over-refusal?
  • Does the RL attack retain its utility advantage over supervised link optimization when both methods receive comparable benign supervision and training budgets?
4 comments
AuthorsOpenReview ↗

Repair evaluation and over-refusal (Part 1/2)

Part 1/2: Repair evaluation and over-refusal

Dear Reviewer UZYP,

Thank you for your constructive feedback. We address your comments and present the additional experimental results below.

1. Repair training and evaluation separation

Our repair method requires harmful queries to guide the optimization toward refusal behavior. In our experiments, we used AdvBench queries for this purpose and evaluated the repaired systems on HarmBench, StrongREJECT, JailbreakBench, and AdvBench. We agree that using AdvBench for both repair and evaluation can bias the AdvBench score and the aggregate that includes it. The other three benchmarks were not used as sources of harmful repair queries.

Excluding AdvBench from the original evaluation average, the mean harmful-compliance score decreases from 71.6 before repair to 6.1 after repair, compared with 70.5 to 4.7 when including it. The reduction holds across all nine attack–topology configurations. Thus, the substantial repair effect remains.

To further assess whether the repair effect depends on using AdvBench as the training source, we additionally repeated the experiment using each of the four benchmarks as the source of harmful queries. For each setting, we observe low post-repair scores on the other three benchmarks indicating that effective repair is not specific to AdvBench.

We will make this distinction explicit in the revised paper and report repair-source performance separately from cross-benchmark performance.

Table 1. Repair using different sources of harmful queries.

Each column identifies the repair-query source. Cells show the post-repair harmful-compliance score on the source benchmark / the mean score across the other three benchmarks; lower is better.

TopologyAttackAdvBenchHarmBenchStrongREJECTJBB
2-AgentPoisoning0.4 / 7.531.0 / 0.170.0 / 0.000.0 / 0.73
Supervised0.2 / 1.730.0 / 1.032.9 / 3.670.0 / 4.33
RL2.1 / 11.876.5 / 7.6716.6 / 8.501.0 / 3.87
SequentialPoisoning1.0 / 8.104.0 / 3.274.5 / 0.002.0 / 4.50
Supervised0.0 / 1.430.0 / 0.202.2 / 0.000.0 / 0.73
RL1.2 / 14.800.5 / 1.573.8 / 5.176.0 / 7.57
MixturePoisoning0.2 / 1.870.0 / 2.332.9 / 2.670.0 / 1.50
Supervised0.0 / 4.070.0 / 0.335.4 / 2.900.0 / 2.43
RL0.0 / 3.230.0 / 1.602.2 / 0.000.0 / 0.33

2. Over-refusal

In the paper, we evaluate benign utility on mathematics and science tasks to assess whether task performance is preserved after attack and repair. We agree that task accuracy alone does not directly capture over-refusal. We therefore conducted additional experiments on OR-Bench-80K [1] and XSTest [2], measuring direct refusal on benign requests resembling harmful ones and comparing systems with clean, attacked, and repaired links.

The results are shown in Table 2. Repair reduces harmful compliance in all nine configurations. After repair, direct refusal on benign requests ranges from 0.2% to 1.2% for RL attacks and from 0.0% to 1.2% for data poisoning across both benchmarks, remaining close to the clean baselines (0.0% to 2.0%). For supervised attacks, refusal rates range from 0.0% to 7.2%, with increases over the clean baseline occurring only in the two-agent and sequential systems. Note that our repair objective does not explicitly account for over-refusal. A defender could reduce the rate by including benign requests resembling harmful ones in the repair data and rewarding helpful responses.

We will include the additional evaluation in the revised paper and discuss this possible extension of the repair objective.

Table 2. Direct refusal and utility after repair.

TopologyAttackOR-Bench-80K refusal: clean → attacked → repairedXSTest refusal: clean → attacked → repairedMean harmful compliance: attacked → repairedMATH500 repaired
Two-agentSupervised0.2 → 0.0 → 1.12.0 → 1.6 → 3.278.7 → 1.356.4
Two-agentPoisoning0.2 → 0.0 → 0.02.0 → 5.2 → 1.249.4 → 5.864.6
Two-agentRL0.2 → 0.1 → 0.22.0 → 1.2 → 0.481.7 → 9.459.4
SequentialSupervised1.0 → 0.0 → 7.22.0 → 1.2 → 4.067.2 → 1.160.2
SequentialPoisoning1.0 → 0.0 → 0.52.0 → 0.4 → 0.056.1 → 6.367.8
SequentialRL1.0 → 0.0 → 0.22.0 → 0.0 → 0.453.8 → 11.463.4
MixtureSupervised0.2 → 0.0 → 0.00.0 → 0.4 → 0.081.9 → 3.178.8
MixturePoisoning0.2 → 0.1 → 0.00.0 → 0.8 → 0.469.9 → 1.579.8
MixtureRL0.2 → 0.0 → 1.20.0 → 0.0 → 0.896.3 → 2.477.6

Direct refusal is measured on OR-Bench-80K and XSTest. Mean harmful compliance combines the four safety benchmark scores; MATH500 reports accuracy after repair. All values are on a 0–100 scale.

AuthorsOpenReview ↗

Comparison of attacks (Part 2/2)

Part 2/2: Comparison of attacks

3. Comparison between reward-guided and supervised attacks

Following your suggestion, we evaluated a supervised baseline that also interleaves benign updates. This comparison tests whether the utility advantage of the reward-guided attack persists when both methods use the same benign questions and update schedule. For both attacks we use 300 optimization steps, the same benign mathematics questions in the same order, and a benign update every third step. This matches the number of optimization steps and the benign-question schedule, while RL additionally generates multiple responses per query.

Under this matched schedule, the reward-guided attack achieves higher MATH500 accuracy in all three topologies. Accuracy reaches 63.8% versus 18.0% in the two-agent system, 68.8% versus 43.6% in the sequential system, and 80.8% versus 49.4% in the mixture system. GPQA-D accuracy is also higher in two of the three topologies.

Table 3. Supervised and reward-guided attacks with a matched benign-data schedule.

TopologyAttackHBSRJBBABMean harmful compliance ↑MATH500 ↑GPQA-D ↑
2-AgentClean39.536.737.011.531.265.435.9
Supervised + benign76.075.170.066.271.818.028.8
RL attack79.074.187.086.581.763.825.3
SequentialClean39.034.827.026.231.759.630.3
Supervised + benign64.061.359.067.563.043.624.2
RL attack62.065.551.036.753.868.828.3
MixtureClean29.511.528.014.420.977.043.4
Supervised + benign77.573.574.069.473.649.434.3
RL attack97.595.296.096.396.380.848.0

Best regards,
The Authors

References

[1] Cui, J., Chiang, W.-L., Stoica, I., and Hsieh, C.-J. OR-Bench: An Over-Refusal Benchmark for Large Language Models. (ICML 2025)

[2] Röttger, P., Kirk, H., Vidgen, B., Attanasio, G., Bianchi, F., and Hovy, D. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. (NAACL 2024)

Thank you for the rebuttal and the additional experiments. In Table 1, there are several cases where the source benchmark scores are higher than the held-out ones. For example, in 2-Agent/RL with StrongREJECT as the source, the source score is 16.6 while the mean score across the other three benchmarks is 8.5. This makes it unclear what the repair is actually learning. It is possible that the prefix-based reward encourages learning a generic refusal prefix, which would not be a robust repair. On the over-refual side, after repair, the sequential-supervised configuration's refusal rate on OR-Bench rises from 1.0% to 7.2%. As a result, the claim that repair "preserves utility" is still supported mainly by math and science benchmarks which do not test the relevant failure mode. In Table 3, the supervised baseline in the 2-Agent system is better on GPQA-D than RL (28.8 vs. 25.3). Therefore, the current evidence does not support the claim that the RL attack "achieves higher average accuracy on two benign utility benchmarks". These issues reduce my confidence in the paper's contributions. I am leaning toward rejection in the paper's current form.

AuthorsOpenReview ↗

Follow-up: Repair Evaluation and Utility Comparison

Dear Reviewer,

Thank you for following up and for pointing out these specific results.

Repair across benchmarks. In Table 1, we reported the mean across the other three benchmarks separately to examine whether the reduction in harmful compliance extends beyond the benchmark used for repair. In the two-agent RL example, the values 16.6 and 8.5 are difficult to compare directly. As described in the paper, 16.6 is a normalized StrongREJECT score for the degree of harmful assistance, while 8.5% is the mean percentage of responses classified as harmful on HarmBench, JailbreakBench, and AdvBench.

To interpret the difference you highlight, it is more informative to compare each score with its own pre-repair baseline. With StrongREJECT as the repair source, its score fell from 74.1 to 16.6. On the other three benchmarks, the mean rate of harmful compliance fell from 84.2% to 8.5%. Thus, the reduction remains substantial when the repair-source benchmark is excluded from the evaluation.

Generic refusal prefix. Our repair objective combines a refusal-prefix check with a judge score of the full response (Eq. at lines 406–407). The prefix term may encourage recurring refusal wording. However, what matters for our safety evaluation is whether the system provides harmful assistance in its response. We assess this using full-response judgments from StrongREJECT and the HarmBench classifier. Across the evaluated benchmarks, these judges find less harmful assistance after repair, including on benchmarks that supplied no repair queries.

Over-refusal. In the paper, we evaluate benign task performance after repair using MATH500 and GPQA-Diamond. Following your suggestion, we extended this evaluation to OR-Bench and XSTest to measure how often repaired systems refuse benign, safety-sensitive requests.

The largest increase occurs in the setting you highlight. In the sequential system after a supervised attack, repair reduces the mean harmful-compliance score from 67.2 to 1.1. At the same time, refusal on OR-Bench rises from 1.0% with clean links to 7.2% after repair, and on XSTest from 2.0% to 4.0%. Across the six configurations involving RL or data poisoning, the post-repair refusal rate is at most 1.0 percentage point above the corresponding clean baseline on either benchmark.

Importantly, the repair objective used in these experiments does not explicitly target over-refusal. A defender could extend the objective to reward helpful responses to benign, safety-sensitive requests.

GPQA-D. In Table 3, the supervised baseline achieves higher GPQA-D accuracy than the RL attack in the two-agent system (28.8% versus 25.3%), as you point out. The ordering reverses in the sequential and mixture systems. Averaged across the three topologies, GPQA-D accuracy is 33.9% for RL and 29.1% for the supervised baseline.

We will revise the claim to make this distinction explicit. Under the matched benign-data schedule, the RL attack achieves higher MATH500 accuracy in all three topologies and higher GPQA-D accuracy in two of the three.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.