- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes V-TAP, a representation-guided candidate selection method for red-teaming tool-using LLM agents under limited execution budgets. Built on Tree of Attacks with Pruning (TAP), V-TAP trains a linear probe on the target model's hidden states using previously observed agent task-completion outcomes. The resulting Concept Activation Vector (CAV) ranks candidate attacks before expensive agent rollouts, while actual execution outcomes determine attack success.
The authors evaluate V-TAP primarily on AgentHarm with Qwen3.5-9B. In the online evaluation, V-TAP achieves 34/116 successful attacks compared with 32/116 for random selection. A separate reproducible selector improves Success@3 on 48 held-out candidate pools from 16.67% to 18.75% relative to random expectation. Additional analyses examine representation quality, supervision choices, training sensitivity, and reproducibility. The authors acknowledge that the observed improvements are statistically inconclusive and do not consistently generalize across models.
Critical strengths
1. Execution-level supervision is a well-motivated choice for agent red-teaming.
The paper correctly distinguishes initial model compliance from successful completion of a multi-step harmful objective. Using actual agent execution outcomes as supervision is conceptually appropriate, particularly because a model may begin executing tools without completing the requested task. The integration with TAP is also straightforward: candidate generation and execution feedback are preserved, while the learned selector only determines which candidates receive expensive rollouts. This makes the proposed component relatively easy to isolate experimentally.
2. Careful reporting of negative evidence and experimental uncertainty.
The authors provide unusually explicit diagnostics regarding the limitations of their method, including paired online outcomes, ranking uncertainty, training-data sensitivity, and differences between historical and reproducible selectors. The analysis separating refusal, tool engagement, and strict task completion is particularly informative. These results help identify where representation-based preselection may fail, rather than relying solely on aggregate attack success rates.
Critical weaknesses
1. The central claim of improved evaluation efficiency is not convincingly supported.
The primary online experiment reports 34 successful tasks for V-TAP versus 32 for random selection across 116 tasks. This corresponds to only four V-TAP-only successes and two random-only successes, with an exact McNemar p-value of 0.6875. The difference is therefore too small to establish a reliable improvement.
The offline evidence is similarly limited. At K=3, V-TAP succeeds on 9/48 pools, while random selection has an expected success count of approximately eight. Moreover, 21 of the 48 pools contain only two candidates, meaning that the deployed three-candidate budget already evaluates the entire pool. The reported improvement is consequently driven by very few additional successful selections.
More importantly, the method introduces extra target-model forward passes for scoring every candidate, while the experiments match only the number of expensive rollouts. No wall-clock or total-compute comparison establishes that this additional overhead is compensated by the observed increase in successful attacks. Since computational efficiency is the primary motivation, this missing evidence directly affects the paper's significance.
2. The learned representation does not convincingly capture the claimed execution-success signal.
The probe's discriminative performance raises a more fundamental concern than the small ASR improvement. Appendix A.6 reports an AUC of 0.542 for strict success versus all failures, but only 0.498 for strict success versus tool-engaged failures. The strongest observed distinction is instead tool engagement versus refusal (AUC 0.589). This suggests that the representation primarily captures a weak willingness-to-act signal rather than the likelihood of completing the requested objective.
The supervision ablation reinforces this concern. Training on tool-engagement or non-refusal labels actually produces better held-out ranking performance than training on strict completion labels. Furthermore, the full-context representation provides no improvement over candidate-only features in Success@1 or Success@2.
These findings substantially weaken the proposed methodological distinction between an execution-grounded CAV and a conventional refusal- or engagement-related probe. The training collection contains only 14 successful examples across 19 tasks, making it difficult to establish that the learned direction encodes a stable completion-related property.
3. The principal online and offline findings rely on different selector artifacts.
The online results use the historical selector H, whereas the reproducible offline analyses use a separately fitted selector R. Despite high global score correlation, their top-ranked candidates disagree in 7/48 pools, and their Success@1 values differ substantially (14.58% versus 10.42%).
The authors acknowledge that the historical environment cannot be reconstructed exactly. Consequently, the offline reproducibility checks do not validate the selector responsible for the reported online improvement. This issue is particularly consequential when the improvement itself is only two tasks.
The generalization evidence is also narrow: the main results are confined to one benchmark and one target model, while the supplementary Gemma evaluation does not reproduce the improvement. The paper therefore provides limited evidence that the proposed selection strategy is reliable outside the specific experimental configuration.
Questions for authors
1. Can the authors reproduce the online comparison using the current, fully reproducible selector R?
Given the disagreement between H and R on held-out candidate rankings, an online evaluation using R would be necessary to connect the reproducible methodology with the claimed improvement in end-to-end attack success. Reporting paired uncertainty across multiple independent search runs would also help establish whether the effect survives search stochasticity.
2. What evidence supports the existence of an execution-specific representation beyond refusal or tool engagement?
Since the completion-trained probe cannot distinguish successful executions from engaged failures above chance, and alternative supervision labels perform better, can the authors demonstrate an advantage over simpler refusal or engagement predictors under matched training conditions? This would clarify what information the proposed execution CAV actually contributes.
3. Does V-TAP improve the number of successful attacks per unit of total computation?
A comparison incorporating candidate feature-extraction costs, target rollout costs, and elapsed execution time would directly address the paper's efficiency motivation. It would also be useful to evaluate larger candidate pools, where ranking can meaningfully reduce the number of executed candidates.
Overall, the paper addresses a meaningful practical problem: allocating expensive agent rollouts using inexpensive pre-execution signals. The integration with TAP is technically reasonable, and the authors deserve credit for carefully documenting uncertainty and negative results.
However, I do not find sufficient evidence that the proposed method achieves its central objective. The online improvement over random selection is statistically inconclusive and amounts to only two additional successful tasks. The offline improvement is similarly small and occurs in candidate pools where the selection budget often covers most or all available candidates. More fundamentally, the learned probe appears substantially better at distinguishing engagement from refusal than at predicting successful execution, undermining the motivation for completion-supervised representation learning.
Selector Reproducibility, Completion Signals, and Computational Cost
Thank you for recognizing the motivation for execution-level supervision and the value of the negative evidence. We agree that the small positive estimates do not establish the intended efficiency benefit. Your three questions distinguish the method's intended objective from what the experiments actually demonstrate; we address each below.
1. Reproducing the online comparison with current selector R.
The existing evidence does not include this experiment. Historical H produces the reported 34/116 successes versus 32/116 for random selection. Current R is a separately fitted artifact whose features, parameters, logits, and rankings repeat exactly on fixed inputs in the audited environment. That check establishes numerical repeatability, not reproduction of H's online outcome.
The discrepancy is consequential: the exported candidate table confirms first-choice disagreement in 7/48 pools and Success@1 of 14.58% for H versus 10.42% for R. High global score correlation cannot establish within-pool selection equivalence. H's historical result and R's offline analysis therefore cannot be combined into evidence for one reproduced end-to-end selector. A locked-R online comparison across independent search repetitions remains necessary.
For H, four tasks succeed only under H and two only under random selection, giving exact McNemar p=0.6875. A task-pair bootstrap of the reported aggregate table gives a 95% percentile interval of [-2.59, +6.03] percentage points for H minus random (20,000 draws). This resamples task entries, assumes their independence, and does not capture repeated-search variability. It reinforces the inconclusive interpretation.
2. Evidence for a completion-specific representation.
Completion labels are conceptually aligned with the evaluation objective because initiating tools is insufficient for task completion. But this motivates the supervision target; it does not prove that the fitted direction encodes a distinct completion property. The direct implementation requires strict final grading even though the stored label field is called
subgoal_success.Your strongest diagnostic is the success-versus-engaged-failure AUC of 0.498. Engagement/non-refusal supervision also performs better on held-out ranking summaries, and full-context features give no Success@1 or Success@2 advantage over candidate-only features. Different positive-example counts prevent a controlled comparison of supervision meanings, but provide no basis for claiming completion supervision is better.
There is a narrower distinction between weak pooled discrimination and absent within-pool ranking information. In the exported table, the completion probe correctly orders 15/21 within-pool success/failure pairs. This reproduces the reported 71.4% pairwise statistic, whereas pooled AUC compares candidates across different tasks and contexts. These metrics answer different questions. The small, dependent set of pairs and the near-chance engaged-failure comparison prevent a strong inference from the pairwise result. Thus, modest observed coverage gains do not establish a completion-specific representation or explain their mechanism.
3. Successful attacks per unit total computation and larger pools.
The reported comparison fixes maximum full-rollout allocations, while adding feature-extraction passes for V-TAP. Realized rollout counts are approximately unchanged. Without a complete accounting of scoring, collection, fitting, rollout computation, and elapsed time, successful attacks per unit total computation remains unestablished. The efficiency motivation should be distinguished from an efficiency result.
Pool sizes also limit the evidence: 21/48 pools contain two candidates, so K=3 already includes the entire pool. Selection can affect K=3 coverage only in the other 27 pools. On that size-defined subset, R obtains 18.52% Success@3 versus 14.81% for exact random expectation. This removes the pools where exclusion is unnecessary, but remains a small analysis of the same recorded candidate sets. It does not answer the request for larger newly generated pools. Across all pools, R covers nine success-containing pools versus a random expectation of eight; reaching this observed ceiling should not be interpreted as general optimality.
The study connects pre-response target representations to rollout allocation, evaluated with strict outcomes and failure diagnostics. We agree that the present implementation has not demonstrated reliable efficiency or cross-model gains. These distinctions preserve the diagnostic contribution without overstating the positive estimates.