- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TRIAD: a theoretical framework for detecting adversarial attacks on LLMs in multi-turn, multi-modal environments. To detect attacks, the TRIAD framework: (1) defines a distance metric on the latent space of the model, (2) integrates it into a survival function, and (3) uses a hazard function to trigger an early stop if a threshold corresponding to safety-failure risk is exceeded. The paper presents two theoretical arguments that claim mathematical guarantees on the performance of the framework, which are accompanied by a proposed simulation protocol; however, no experimental results are reported.
Strengths:
- Addresses an important gap in LLM safety
- Claims mathematical guarantees on the performance of the framework
- Integrates behavioral signals, which could potentially help to distinguish safe from unsafe usage
Weaknesses:
- Deployment limitations are acknowledged but not sufficiently addressed. It is not established that the proposed mitigations resolve the limitations or preserve the claimed guarantees.
- Numerous grammatical errors make the manuscript difficult to follow at times.
- Mathematical contributions are not self-contained. Assumptions are not always clearly stated or motivated. Mathematical arguments are at times difficult to follow due to variables seemingly being re-used for different meanings.
Critical strengths
The main substantive strength is in addressing attacks that accumulate across multiple multimodal conversation turns. Prior work in this domain has demonstrated that intelligent attackers are capable of planning and executing their attack over multiple sequential turns. I believe that this form of attack will become more standardized, and therefore it is important to address methods of defense.
Critical weaknesses
The critical weaknesses are:
Line 134: "To statistically separate the mechanical characteristics of an automated attacker from the cognitive reflection of a normal user...": The behavioral distinction (mechanical characteristics of automated attacker from cognitive reflection of normal user) is insufficiently justified and does not address benign automation, human attackers, or automation that mimics human behavior.
Proof of Theorems 1 and 2: Regarding Theorem 1, the authors claim that a lower bound on one term of the distance formula applies as a lower bound on the entire squared-distance increment. It is both (a) not clear that this bound is correctly applied, and (b) not clear that the divergence test is applied correctly, and so it seems the proof does not adequately establish the claimed guarantee. Regarding Theorem 2, the authors appear to rely on a minimum speed assumption that is not justified. Theorem 2 appears to claim that if speed is above that minimum threshold, that this must imply strictly positive acceleration, but it is not clear that is true. For example, consider constant positive speed directly away from the reference. This would result in linearly increasing Mahalanobis distance and zero trajectory acceleration.
The gaps in the proof reduce my confidence in the paper's central theoretical contributions.
Questions for authors
Line 134: "To statistically separate the mechanical characteristics of an automated attacker from the cognitive reflection of a normal user...": What justifies separating malicious attackers from benign users based on automation-related behavior? Couldn't a malicious attacker be human (and therefore exhibit a human behavioral signature), or a benign user be automation (and therefore exhibit an automation behavioral signature)?
Re: Theorem 1: Would it be possible to elaborate on how the squared distance increment secures the given lower bound? I can see how the given bound derives from the last term in Eq. (11), but it seems like the cross term could be negative. I am also not clear how the divergence test applies here. Showing that a strictly positive exponential increment is added to the covariate is not the same as showing that terms fail to approach zero or that the sum approaches infinity.
Re: Theorem 2: How is this minimum speed assumption justified? Also, why does a positive ||V'(t)|| >= c > 0 for constant c imply that acceleration never converges to 0? Couldn't V'(t) be positive, constant, and directed away from \mu such that D_M increases linearly but trajectory acceleration is zero?
Re: the manifold heterogeneity section: Is it possible that an adversarial conversation could cause a change in the selected GMM cluster, reducing the measured distance and therefore artificially reducing acceleration under Eq. (7) to negative or zero despite the attack continuing? Could this possibly violate the claim in L258 that adversarial conversations exhibit "acceleration that strictly diverges in the positive direction"?