- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates structural trajectory backdoors in tool-calling LLM agents. The authors propose TrajBD, which uses ordered tool-role and observation-derived state patterns to trigger unauthorized tool calls without explicit lexical markers. They further introduce TrajBD-NC, which incorporates confirmation supervision on neighboring trajectories to improve trigger specificity. Experiments across four LLM configurations demonstrate high attack activation on exact-trigger histories and relatively low violation rates on constructed structural neighbors. However, the results also reveal substantial leakage on some natural non-matching histories, raising questions about the generalizability of the proposed approach.
Critical strengths
- The paper investigates structural backdoors in tool-calling agents, where malicious behavior is triggered by interaction histories rather than explicit textual markers.
- The paper highlights that high attack success rates do not necessarily imply precise trigger behavior. Evaluating activation and specificity separately provides a useful perspective for understanding backdoor behavior in agent systems.
- The proposed TrajBD-NC combines paired data construction, neighbor-based supervision, and auxiliary training objectives. The paper provides detailed descriptions of the trigger construction, training procedure, and evaluation metrics, which helps with understanding and reproducing the approach.
Critical weaknesses
- The main components of TrajBD-NC, including structured triggers, negative examples for non-trigger conditions, and auxiliary supervision, are closely related to existing backdoor techniques. Although applying them to completed tool-interaction trajectories is interesting, the paper could better explain what new technical challenges arise in this setting and why the proposed training approach is fundamentally different from existing methods.
- The strongest specificity results are obtained on controlled neighbors constructed by repeating selected tool-call blocks. These examples cover only a small subset of possible structural variations. More importantly, Figure 2 shows substantial leakage on natural exchange neighbors, with violations on 2–3 out of 4 examples across seeds.
- The controlled neighbors are created by inserting repeated read-only tool interactions, which changes not only the role sequence but also the input length, token positions, and local repetition patterns. Consequently, it is unclear whether the model learns the intended role-state condition or relies on simpler surface-level differences between exact triggers and neighboring histories. More diverse structural perturbations would help establish what the model actually learns.
- The authors explicitly mention that development feedback informed recipe selection. However, most reported results, including the main comparisons and ablations, are evaluated on these development histories. Without an independent held-out test set, it is difficult to determine whether the reported improvements generalize beyond the data used during method development.
- The experiments primarily evaluate next-response generation on fixed histories, while the native-tool replay executes previously generated calls in isolated environments. These evaluations demonstrate potential consequences but do not show how often the backdoor would activate during autonomous agent interactions.
Questions for authors
- The controlled-neighbor experiments show low violation rates, while the natural exchange neighbors exhibit substantial leakage. Could the authors explain this discrepancy and provide additional evaluations using more diverse, naturally occurring structural variations?
- Could the authors provide an independent held-out evaluation and more closely matched baseline comparisons?