- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RoleSplit, a configuration-poisoning attack that distributes complementary malicious roles across heterogeneous components of an LLM agent. The attack separates an additional-operation requirement from a false consequence description . In the skill–tool construction, a poisoned workflow makes an additional operation appear procedurally required, while poisoned tool metadata falsely describes its public effect as private. In the memory–tool construction, a policy-bearing memory annotation makes the additional operation task-dependent and binds its arguments to task-relevant records, while tool metadata again misrepresents the operation's effects.
Critical strengths
The controlled placement comparison is well designed. The main experiments preserve identical instantiated attack-role text across the joint and complete conditions, making it possible to isolate placement more cleanly than a comparison between separately engineered attack prompts.
I also appreciate that the authors anticipate a natural concern about the complete baselines being poorly structured. The supplementary experiment introduces an explicit coherent single-component procedure that specifies the full private-then-audit workflow; joint placement remains substantially more effective on Qwen 3.7. This strengthens the central empirical claim.
The use of environment-state scoring is another strength. Attack success requires an actual public disclosure while maintaining correct private task completion, rather than merely detecting malicious language in the model's output.
The paired statistical analysis is appropriate. Comparisons use matched content instances with exact McNemar tests and Holm correction, and the paper reports joint-only versus comparator-only outcomes. This provides substantially more information than aggregate ASR alone.
The task-applicability intervention in M+T is particularly convincing. Changing only the malicious annotation's applicable-policy reference sharply reduces attack success while preserving the relevant task facts and correct private answer. This supports the interpretation that the attack exploits the relationship between poisoned guidance and task-relevant state rather than simply the presence of malicious text.
Finally, the trusted-information study provides useful defensive insight: trusted task boundaries and tool-effect information can substantially reduce attack success while preserving performance on the evaluated legitimate tasks.
Critical weaknesses
My first concern is novelty. Distributed or conjunctive malicious content across multiple sources has already been explored in prior work, including attacks that distribute sub-payloads across skills or tool metadata. The novel distinction here appears to be the allocation of semantically complementary roles across heterogeneous components according to component function. I find this distinction meaningful, but currently the paper does not fully isolate whether the observed gains come specifically from role-to-function alignment versus more generic effects of source separation, prompt position, or context structure.
A direct role-swapping control would strengthen this claim substantially. For example, one could retain two-component heterogeneous placement but place in the workflow/memory and in tool metadata. If the proposed function-aligned placement substantially outperforms this misaligned distributed condition, the claimed mechanism would be much clearer.
Second, the evaluation is quite narrow despite the large number of trajectories. The two main studies are synthetic variants of private-notification tasks with unauthorized public disclosure as the adversarial objective. This controlled setting is appropriate for a causal study, but it is unclear whether the phenomenon generalizes to qualitatively different agent actions such as file modification, code execution, external API calls, purchases, or credential access.
Third, the threat model assumes that the attacker can modify multiple heterogeneous configuration components. In many realistic systems, skills/memory and tool metadata may belong to different trust domains or providers. The paper would benefit from a clearer discussion of practical deployment scenarios in which one adversary controls both sources, and how the increased attack effectiveness should be weighed against the stronger compromise assumption.
Fourth, each content-condition pair is evaluated with a single rollout. The paired analysis controls content variation well, but it does not quantify within-instance stochastic generation variance. Repeated rollouts for at least a subset of models and conditions would increase confidence that the observed placement effects are stable.
Finally, the two constructions should not be compared by absolute ASR: S+T and M+T use different tasks and different requested thinking modes. The paper largely treats them separately, but I think this limitation should be made more explicit.
Questions for authors
-
Can the authors evaluate a heterogeneous but role-misaligned placement, e.g. in skill/memory and in tool metadata, to distinguish functional alignment from generic payload distribution?
-
Does RoleSplit improve attack success for qualitatively different unauthorized operations beyond private-to-public information disclosure?
-
What realistic system or supply-chain scenarios allow a single attacker to modify both the workflow/memory and tool metadata? Are these assumed to originate from the same third-party provider?
-
Can the authors repeat a subset of the main experiments with multiple rollouts per content to estimate stochastic variability?
-
Are the S+T and M+T success rates intended only for within-construction comparison? If any cross-construction conclusion is intended, how do the authors account for the different task structures and thinking modes?
3 comments
Thank you for the additional experiments and clarifications. The role-reversal control, second unauthorized action, and repeated runs address several of my concerns.
One question remains about the role-to-function alignment claim: for S+T disclosure, the Joint-only and Reversed-only outcomes are relatively close (21 vs. 15). Could you clarify how strongly this result supports functional alignment, as opposed to a more general benefit from distributing the payload across components?
Thank you for this important distinction. We agree that S+T disclosure alone provides limited evidence for a specific role-to-function alignment mechanism. Joint succeeds on 34/56 contents versus 28/56 for Reversed, but the paired discordant counts (21 Joint-only versus 15 Reversed-only) do not establish a strong difference. Both distributed placements substantially exceed the complete single-component placements (Complete S: 0/56; Complete T: 6/56) in this follow-up, so a more general benefit from distributing the roles is consistent with the S+T disclosure result.
Our evidence for allocation sensitivity beyond distribution alone comes from the different outcomes under the same two-component delivery structure when the roles are exchanged:
| Matched Qwen 3.7 setting | Joint | Reversed | Joint-only / Reversed-only |
|---|---|---|---|
| S+T, public disclosure | 34/56 | 28/56 | 21 / 15 |
| M+T, public disclosure | 47/56 | 27/56 | 24 / 4 |
| S+T, shared-record modification | 35/56 | 0/56 | 35 / 0 |
| M+T, shared-record modification | 56/56 | 56/56 | 0 / 0 |
These comparisons (Section 5.3; Appendix F.2, Table 13) show that which component carries which role can matter, but the effect is not uniform: it is pronounced in M+T disclosure and S+T record modification, modest in S+T disclosure, and unidentifiable at the 56/56 ceiling for M+T record modification. They do not by themselves separate semantic component function completely from presentation context or source authority. Accordingly, our conclusion is an allocation effect conditional on the construction and requested operation, not universal superiority of the function-aligned placement. This is the intended interpretation in Section 6.
Thank you for the detailed review and for recognizing the matched placement design. We evaluated role reversal, a second unauthorized action, and repeated calls on Qwen 3.7.
Reversed placement. In the same matched Qwen 3.7 follow-up, Joint and Reversed achieve 34/56 and 28/56 for S+T disclosure, and 47/56 and 27/56 for M+T disclosure. The paired outcomes include 21 Joint-only and 15 Reversed-only successes in S+T, and 24 and 4, respectively, in M+T. Reversal holds two-component delivery fixed while changing which component carries each role and the context in which it is presented. Together with the shared-record modification results below, these findings indicate an allocation effect that depends on the construction and requested operation (Section 5.3, Table 5, and Appendix F.2).
Other unauthorized actions. We evaluated an unauthorized shared-record status change while keeping the legitimate private notification. For S+T, Joint achieves 35/56, compared with 0/56 for Reversed, 0/56 for Complete S, and 1/56 for Complete T. For M+T, Joint, Reversed, and Complete T all reach 56/56, and Complete M reaches 47/56. The S+T benefit therefore extends to this shared-state action, while the M+T result shows a ceiling among three placements (Table 5 and Appendix F.1).
Dual-component access. Our threat model assumes that the attacker controls a Skill or Memory entry and has prepared a tool with poisoned registration metadata. When the compromised Skill or Memory entry is invoked, it explicitly instructs the agent to install or enable this specific attacker-prepared tool. Once the tool is registered, its poisoned description becomes part of the agent's configuration. In the main Joint allocation, R resides in the Skill or Memory entry, while D resides in the registered tool metadata. This establishes the two-component configuration studied in RoleSplit without requiring the attacker to directly modify the description of an already-registered tool.
Repeatability. We added two repeat executions for each of the three main-study placements on the same 56 Qwen 3.7 contents per construction. Including the original run, Joint succeeds on 30/56, 30/56, and 30/56 S+T contents and 49/56, 49/56, and 48/56 M+T contents, exceeding both Complete placements in every round. The runs retain each construction's generation settings and state-based endpoints (Appendix F.3, Table 14).
Within-construction comparisons. Yes. We compare placements on the same contents, with the same outcome definition and generation settings within each construction. S+T and M+T use different tasks, success criteria, and requested thinking modes. Their absolute success rates describe their respective settings (Section 4.1, Table 1, and Figure 2).