- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PAC-CTCN, a runtime adversarial detector for VLA models that compares current visual features with predictions based on the previous observation and executed action. It provides a conditional detection bound and evaluates detection and stop-based containment across three VLA models on LIBERO benchmarks.
Critical strengths
1.Practical detector interface that requires no VLA modification.
2.Evaluation across three VLA architectures, including simulated stop-based containment.
Critical weaknesses
1.Limited guarantee. The theory assumes sufficiently large displacement in the detector’s features, but does not establish that attacks harming the VLA must cause such displacement.
2.Insufficient adaptive evaluation. The stated attack mainly optimizes detector evasion rather than jointly optimizing evasion and task damage. Table 24 reports evasion dropping from 93.6% at to 12.2% at . For a fixed detector and evaluation set, smaller-budget solutions remain feasible at larger budgets, raising concerns about attack optimization.
3.Inconclusive ablation. Zeroing actions at inference introduces distribution shift. A separately trained action-free model is needed to establish the necessity of action conditioning.
4.Numerical inconsistencies. Table 16’s per-sample latency and throughput disagree by approximately 64× at batch size 64. Figure 2’s plotted statistics also differ from its caption, implying different calibration thresholds.
Questions for authors
1.Can joint attacks cause task failure while keeping the detector score above threshold?
2.Does large-budget detection remain effective when attacks retain successful smaller-budget perturbations?
3.Can the authors reconcile the latency–throughput mismatch in Table 16 and the conflicting calibration statistics in Figure 2?
2 comments
Response to Reviewer gbtA (2/2)
W3 — Interpreting the action ablation
The existing ablation measures sensitivity to removing an input from the trained detector. Its reported AUROCs are:
| Policy | Full detector | Action zeroed at inference |
|---|---|---|
| OpenVLA | 0.999 | 0.48 |
| Octo | 0.994 | 0.47 |
| π₀ | 0.992 | 0.49 |
The reviewer is right that the input mismatch prevents this comparison from isolating the benefit of action conditioning. Revised Tables 12 and 16 label this as inference-time action removal, and the necessity interpretation is removed. A separately trained action-free model, with matched capacity, data, training budget, selection, and evaluation, remains needed. Its performance must be measured rather than inferred from the bound-ratio/AUROC trend.
W4/Q3 — Correcting throughput (submission Table 16; revised Table 19)
The throughput column effectively counts the batch factor twice. Since latency is reported per sample, throughput = 1,000 ÷ latency in milliseconds, giving:
| Batch size | Per-sample latency (ms) | Corrected samples/s, rounded |
|---|---|---|
| 1 | 0.88 | 1,140 |
| 4 | 0.31 | 3,230 |
| 16 | 0.12 | 8,330 |
| 64 | 0.058 | 17,200 |
At batch 64, the correction is 1.10 million → approximately 17,200 samples/s. These values are recomputed from the reported latencies, not new timing measurements. The revised table corrects the throughput column, and the super-linear scaling claim is removed. The reported batch-one latency of 0.88 ms is unchanged by this arithmetic correction.
W4/Q3 — Reconciling Figure 2 and calibration
The per-target adaptive table uses the combined score (α = 0.5) for both Adv mean and Evasion %, even when the attack targets the predictive branch. Its clean mean and spread are 0.876 and 0.188, giving τ = 0.312. The predictive-only threshold 0.988 belongs to a different score. The revised table headers and caption now identify this mapping.
| Score | Mean μ | Spread σ | Threshold μ − 3σ |
|---|---|---|---|
| Predictive: Figure 2 | 0.9960 | 0.0039 | 0.9843 |
| Predictive: deployment text | 0.997 | 0.003 | 0.988 |
| Combined: adaptive diagnostics, α = 0.5 | ≈0.876 | ≈0.188 | ≈0.312 |
The Pred-target attacked means 0.884 / 0.884 / 0.880 are combined-score measurements, close to the clean combined mean and above 0.312. The reported 100% evasion is the measured fraction of individual samples above that threshold; it is not inferred from their mean. The cross-branch columns report AUROCs, not branch-score means.
The predictive-only threshold 0.988 is used for the closed-loop and small-budget evaluations. A separate figure issue remains: the existing graphic prints 0.9960 / 0.0039, implying 0.9843. Its caption now matches that graphic, while its difference from the stated predictive calibration remains explicit. AUROC is threshold-free; TPR at 1% or 5% FPR uses its corresponding ROC operating point.
The frame-level false-alarm rate must also be measured from held-out clean scores. Neither a Gaussian fit nor the original paper's 3.1% rollout false-stop rate determines it, because temporal dependence and the two-frame debounce affect rollout stops. The revision distinguishes these two metrics; a separate held-out frame false-alarm rate is not reported.
Summary. The supported contribution remains a practical, conditional detector alongside unchanged VLAs. The revision narrows the guarantee and ablation claims, reports the measured detector-aware failure, corrects throughput, and identifies the adaptive score/threshold mapping. The separate Figure 2 calibration discrepancy remains explicit.
Response to Reviewer gbtA (1/2)
We thank the reviewer for the careful review and for recognising the practical detector interface and cross-architecture evaluation.
W1 — The scope of the detection guarantee
We agree that the proposition establishes a predictive-score margin when displacement in the detector's features is sufficiently large. It does not establish that every attack harming the policy must produce that displacement. A4's measured relation between detector and VLA feature displacements is an empirical check on the evaluated attacks, not a proof of that implication.
The revision replaces “any effective attack is forced to reveal itself” with the precise conditional statement. The three-policy, three-regime gray-box results support the evaluated detector, while leaving its behaviour against other attacks to measurement.
The approximately 0.32 displacement cap for evading predictive scores is likewise conditional on the clean error bound. It supplies no corresponding action-damage or task-failure bound, and substituting a mean prediction error does not certify individual samples.
W2/Q1 — Joint attacks and the stop mechanism
In the additional detector-aware closed-loop evaluation, we measured 0% task success and 0% episodes halted. This demonstrates harmful evasion of the stop rule.
With the two-frame debounce, zero halts establishes evasion of the stop rule, not that every frame stayed above threshold. Revised Table 3 reports this measured outcome. The claims that evasion necessarily costs the attacker its damage and that joint optimisation “does not cleanly succeed” have been removed.
Revised Tables 26–27 (submission Tables 23–24) are labelled as detector-only tests. Their 100-step, 10-restart PGD settings do not specify a joint attack. The joint objective, update direction, and candidate-selection rule remain required protocol details.
W2/Q2 — Retaining smaller-budget solutions
The reviewer is correct that successful 4/255 perturbations remain feasible at larger budgets. The original measured evasion rates were:
Retaining the measured 4/255 solutions would preserve at least 93.6% evasion at 8/255 and 16/255 on the same samples and threshold. This is a feasibility implication, not a new experimental result. The measured 12.2% therefore cannot establish large-budget recovery.
The revision removes that interpretation and its training-budget explanation. A corrected evaluation must retain feasible smaller-budget candidates and report damage, evasion, and their intersection under fixed criteria. The existing table does not establish performance under that protocol.