- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TMT for runtime backdoor detection in vision-language-action policies, with an emphasis on unseen tasks. A token-manifold branch models benign input representations, while a latent-transition branch predicts changes between adjacent backbone layers. Either branch can trigger an alarm. Before subsequent monitoring, an incident-discovery procedure uses a suspicious rollout to select the monitored transition. The paper also explores policy purification through self-distillation on paired benign and triggered observations. Experiments cover three attacks and three victim-policy families across the reported settings, ten adapted baselines, and real-world robot execution. After incident-driven transition selection, the unseen-task evaluation on OpenVLA-OFT reports TDR/FRR of 100.00%/0.57% for BadVLA, 83.43%/8.57% for GoBA, and 73.60%/0.00% for DropVLA.
Critical strengths
- Important problem, practical design, and broad evaluation. VLA backdoors can cause unsafe physical behavior, making this a timely problem. Combining benign representation modeling with layer-transition prediction on a frozen policy is a reasonable, implementable design. The evaluation spans several attack and victim settings, ten baseline adaptations, robot experiments, and a purification extension. The baseline adaptation effort is substantial, and the robot study provides preliminary evidence of deployment feasibility.
- Candid and informative ablations. The authors explicitly report that the token-manifold branch reaches only 0.57% and 15.20% TDR on unseen GoBA and DropVLA tasks, respectively, and acknowledge that unseen-task detection mainly relies on latent transitions. This transparency is valuable and helps identify the actual source of the gains.
Critical weaknesses
- Incident-driven adaptation leaves the threat model and attack dependence unclear. Section 3 excludes triggered training examples, but Section 4.3 and Appendix F use a suspicious incident to select the monitored transition. Even if predictor fitting uses only benign data, transition selection remains part of detector adaptation. Not reading malicious labels is not equivalent to avoiding attack-related observations. The discovery data source, its relationship to the evaluated attack, and the fallback when no incident is accepted are insufficiently specified. It is also unclear whether selection is repeated for each attack/checkpoint; stability across incidents and seeds and transfer across attacks are not analyzed. If selection requires an incident from the same attack, the results establish cross-task detection after attack-related adaptation. This limits the scope of the central generalization claim.
- Branch complementarity is limited in the main unseen-task setting. On unseen GoBA tasks, adding the token-manifold branch leaves TDR unchanged at 83.43% and latency unchanged at 0.28 queries, while increasing FRR from 7.43% to 8.57%. On unseen DropVLA tasks, the combination likewise leaves TDR unchanged at 73.60%, although latency improves from 6.25 to 5.68 queries. These results support the latent-transition branch, but provide limited evidence for complementary detection coverage during unseen-task monitoring. The manifold branch may still help discover the initial incident; that role should be distinguished from its incremental value during subsequent monitoring.
Questions for authors
- Discovery and benign-only evaluation. Where do discovery rollouts come from, and what happens if no incident is accepted? What performance remains when fitting, selection, and calibration all exclude triggered observations?
- Stability and attack dependence. Is selection repeated for each attack/checkpoint? How stable is the selected transition across incidents and seeds, and can it transfer to another attack without reselection?
- Paper and code correspondence. Please clarify how the released SAFE-ID implementation maps to TMT, particularly the two-query window and discovery-score aggregation, and whether Equation (8) uses the raw or calibrated transition score.
Rebuttal by Authors
We thank the reviewer for the careful review and constructive feedback. We provide additional clarifications and experimental evidence below.
Data source. Both predictors are trained using only benign inference-time rollouts from the frozen, potentially backdoored VLA policy. The tasks corresponding to these rollout samples refer to the seen tasks for TMT.
We clarify that our unseen-task evaluation measures cross-task generalization after incident-driven transition selection and threshold calibration, rather than detection without transition selection.
No-incident setting. The token-manifold branch alone achieves 95.41–100% TDR on seen tasks (seen setting). For unseen tasks, latent-transition monitoring provides broader detection coverage. We therefore additionally explored benign-anomaly-based transition selection as a potential fallback without using triggered observations for selection.
Transition-selection ablation. We compared the original, second-ranked, and benign-anomaly-selected transitions.
Notably, even when a benign rollout is used for transition selection, TMT still achieves the highest TDR among all baselines in Table 1. We have included these additional results in the Appendix D.2.
The results in Table 1 are averaged over three independent random seen/unseen-task splits. Across all three splits, TMT consistently selects the same transition for each attack: layer 30 for BadVLA, layer 17 for GoBA, and layer 20 for DropVLA. This demonstrates stable transition selection across different task splits.
Our additional ablation further shows that even when a transition is selected wrong, TMT retains competitive detection performance, achieving the highest or joint-highest TDR among the baselines in Table 1.
The three attacks use distinct triggers, and we observed 0% backdoor activation when applying one attack's trigger to another attack's backdoored policy. Accordingly, TMT independently models the benign token manifold and latent transitions of each potentially backdoored policy, with transition selection performed separately for each model. Cross-attack transfer without reselection is therefore not part of our policy-specific detection setting.
We agree that the two branches play different roles during detection. The token-manifold branch provides strong detection on seen tasks and is responsible for discovering suspicious incidents, while the latent-transition branch provides the main detection coverage on unseen tasks.
However, in real-world deployment, robots may repeatedly execute previously seen tasks while also encountering unseen tasks. Therefore, we recommend keeping both branches active during runtime monitoring to maintain detection capability across both settings, rather than relying solely on the latent-transition branch. Their complementarity lies in supporting different task settings, rather than necessarily improving TDR within the unseen-task setting alone.
Thank you for requesting this clarification. SAFE-ID was the original name of the method now presented as TMT. The two-query window is an implementation detail of online detection in the latent-transition branch, while discovery-score aggregation belongs to the incident-discovery procedure for transition selection.
We evaluated all compared methods within a shared codebase developed for VLA backdoor benchmarking. Due to the supplementary material size limit, we extracted the relevant detector and transition-selection modules for submission. We plan to release the complete benchmarking framework after standardizing the implementation details, hoping to facilitate future research on VLA backdoor attacks and defenses.
Equation (8). Section 4.4 introduces the calibrated transition score s~D, whereas Equation (8) uses sD. We have corrected this notation (line 274).
We hope our clarifications and additional experiments have adequately addressed your concerns, and we would be very grateful if you might consider raising your score in light of these updates.