acceptodds
Under review as a conference paper at ICLR 2027

V-TAP: Execution-Grounded Vector Guidance for Tool-Using Agent Red-Teaming

Abstract

Red-teaming tool-using language-model agents is expensive: testing a single candidate prompt can require multiple model responses and tool interactions. An important practical question is therefore how to uncover more safety failures within a limited evaluation budget. We introduce V-TAP, which uses the target model's internal response to a candidate prompt to decide whether that prompt merits a full agent execution. A scoring model learned from previous execution outcomes prioritizes candidates within Tree of Attacks with Pruning, while actual executions determine success. On 116 AgentHarm tasks with Qwen3.5-9B, V-TAP increases observed attack success from 27.59% to 29.31% under the same maximum rollout budget, a 6.25% relative increase in successfully attacked tasks. It succeeds on four tasks missed by random selection, while missing two tasks that random selection solves. Separately, a reproducibly fitted selector increases three-candidate coverage on 48 held-out pools from 16.67% to 18.75%, a 12.5% relative increase, reaching the successful-candidate coverage available in those pools. The online difference remains statistically inconclusive, and a 20-task Gemma-3-27B-IT evaluation does not reproduce the advantage. These findings motivate target-informed candidate selection as a way to improve the yield of costly agent evaluations, while identifying the evidence still needed to establish reliable gains across models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Automated Red-Teaming, Representation-Based Candidate Selection

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_378LOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes V-TAP, a representation-guided candidate selection method for red-teaming tool-using LLM agents under limited execution budgets. Built on Tree of Attacks with Pruning (TAP), V-TAP trains a linear probe on the target model's hidden states using previously observed agent task-completion outcomes. The resulting Concept Activation Vector (CAV) ranks candidate attacks before expensive agent rollouts, while actual execution outcomes determine attack success.

The authors evaluate V-TAP primarily on AgentHarm with Qwen3.5-9B. In the online evaluation, V-TAP achieves 34/116 successful attacks compared with 32/116 for random selection. A separate reproducible selector improves Success@3 on 48 held-out candidate pools from 16.67% to 18.75% relative to random expectation. Additional analyses examine representation quality, supervision choices, training sensitivity, and reproducibility. The authors acknowledge that the observed improvements are statistically inconclusive and do not consistently generalize across models.

Critical strengths

1. Execution-level supervision is a well-motivated choice for agent red-teaming.

The paper correctly distinguishes initial model compliance from successful completion of a multi-step harmful objective. Using actual agent execution outcomes as supervision is conceptually appropriate, particularly because a model may begin executing tools without completing the requested task. The integration with TAP is also straightforward: candidate generation and execution feedback are preserved, while the learned selector only determines which candidates receive expensive rollouts. This makes the proposed component relatively easy to isolate experimentally.

2. Careful reporting of negative evidence and experimental uncertainty.

The authors provide unusually explicit diagnostics regarding the limitations of their method, including paired online outcomes, ranking uncertainty, training-data sensitivity, and differences between historical and reproducible selectors. The analysis separating refusal, tool engagement, and strict task completion is particularly informative. These results help identify where representation-based preselection may fail, rather than relying solely on aggregate attack success rates.

Critical weaknesses

1. The central claim of improved evaluation efficiency is not convincingly supported.

The primary online experiment reports 34 successful tasks for V-TAP versus 32 for random selection across 116 tasks. This corresponds to only four V-TAP-only successes and two random-only successes, with an exact McNemar p-value of 0.6875. The difference is therefore too small to establish a reliable improvement.

The offline evidence is similarly limited. At K=3, V-TAP succeeds on 9/48 pools, while random selection has an expected success count of approximately eight. Moreover, 21 of the 48 pools contain only two candidates, meaning that the deployed three-candidate budget already evaluates the entire pool. The reported improvement is consequently driven by very few additional successful selections.

More importantly, the method introduces extra target-model forward passes for scoring every candidate, while the experiments match only the number of expensive rollouts. No wall-clock or total-compute comparison establishes that this additional overhead is compensated by the observed increase in successful attacks. Since computational efficiency is the primary motivation, this missing evidence directly affects the paper's significance.

2. The learned representation does not convincingly capture the claimed execution-success signal.

The probe's discriminative performance raises a more fundamental concern than the small ASR improvement. Appendix A.6 reports an AUC of 0.542 for strict success versus all failures, but only 0.498 for strict success versus tool-engaged failures. The strongest observed distinction is instead tool engagement versus refusal (AUC 0.589). This suggests that the representation primarily captures a weak willingness-to-act signal rather than the likelihood of completing the requested objective.

The supervision ablation reinforces this concern. Training on tool-engagement or non-refusal labels actually produces better held-out ranking performance than training on strict completion labels. Furthermore, the full-context representation provides no improvement over candidate-only features in Success@1 or Success@2.

These findings substantially weaken the proposed methodological distinction between an execution-grounded CAV and a conventional refusal- or engagement-related probe. The training collection contains only 14 successful examples across 19 tasks, making it difficult to establish that the learned direction encodes a stable completion-related property.

3. The principal online and offline findings rely on different selector artifacts.

The online results use the historical selector H, whereas the reproducible offline analyses use a separately fitted selector R. Despite high global score correlation, their top-ranked candidates disagree in 7/48 pools, and their Success@1 values differ substantially (14.58% versus 10.42%).

The authors acknowledge that the historical environment cannot be reconstructed exactly. Consequently, the offline reproducibility checks do not validate the selector responsible for the reported online improvement. This issue is particularly consequential when the improvement itself is only two tasks.

The generalization evidence is also narrow: the main results are confined to one benchmark and one target model, while the supplementary Gemma evaluation does not reproduce the improvement. The paper therefore provides limited evidence that the proposed selection strategy is reliable outside the specific experimental configuration.

Questions for authors

1. Can the authors reproduce the online comparison using the current, fully reproducible selector R?

Given the disagreement between H and R on held-out candidate rankings, an online evaluation using R would be necessary to connect the reproducible methodology with the claimed improvement in end-to-end attack success. Reporting paired uncertainty across multiple independent search runs would also help establish whether the effect survives search stochasticity.

2. What evidence supports the existence of an execution-specific representation beyond refusal or tool engagement?

Since the completion-trained probe cannot distinguish successful executions from engaged failures above chance, and alternative supervision labels perform better, can the authors demonstrate an advantage over simpler refusal or engagement predictors under matched training conditions? This would clarify what information the proposed execution CAV actually contributes.

3. Does V-TAP improve the number of successful attacks per unit of total computation?

A comparison incorporating candidate feature-extraction costs, target rollout costs, and elapsed execution time would directly address the paper's efficiency motivation. It would also be useful to evaluate larger candidate pools, where ranking can meaningfully reduce the number of executed candidates.

Overall, the paper addresses a meaningful practical problem: allocating expensive agent rollouts using inexpensive pre-execution signals. The integration with TAP is technically reasonable, and the authors deserve credit for carefully documenting uncertainty and negative results.

However, I do not find sufficient evidence that the proposed method achieves its central objective. The online improvement over random selection is statistically inconclusive and amounts to only two additional successful tasks. The offline improvement is similarly small and occurs in candidate pools where the selection budget often covers most or all available candidates. More fundamentally, the learned probe appears substantially better at distinguishing engagement from refusal than at predicting successful execution, undermining the motivation for completion-supervised representation learning.

1 comment
AuthorsOpenReview ↗

Selector Reproducibility, Completion Signals, and Computational Cost

Thank you for recognizing the motivation for execution-level supervision and the value of the negative evidence. We agree that the small positive estimates do not establish the intended efficiency benefit. Your three questions distinguish the method's intended objective from what the experiments actually demonstrate; we address each below.

1. Reproducing the online comparison with current selector R.

The existing evidence does not include this experiment. Historical H produces the reported 34/116 successes versus 32/116 for random selection. Current R is a separately fitted artifact whose features, parameters, logits, and rankings repeat exactly on fixed inputs in the audited environment. That check establishes numerical repeatability, not reproduction of H's online outcome.

The discrepancy is consequential: the exported candidate table confirms first-choice disagreement in 7/48 pools and Success@1 of 14.58% for H versus 10.42% for R. High global score correlation cannot establish within-pool selection equivalence. H's historical result and R's offline analysis therefore cannot be combined into evidence for one reproduced end-to-end selector. A locked-R online comparison across independent search repetitions remains necessary.

For H, four tasks succeed only under H and two only under random selection, giving exact McNemar p=0.6875. A task-pair bootstrap of the reported aggregate table gives a 95% percentile interval of [-2.59, +6.03] percentage points for H minus random (20,000 draws). This resamples task entries, assumes their independence, and does not capture repeated-search variability. It reinforces the inconclusive interpretation.

2. Evidence for a completion-specific representation.

Completion labels are conceptually aligned with the evaluation objective because initiating tools is insufficient for task completion. But this motivates the supervision target; it does not prove that the fitted direction encodes a distinct completion property. The direct implementation requires strict final grading even though the stored label field is called subgoal_success.

Your strongest diagnostic is the success-versus-engaged-failure AUC of 0.498. Engagement/non-refusal supervision also performs better on held-out ranking summaries, and full-context features give no Success@1 or Success@2 advantage over candidate-only features. Different positive-example counts prevent a controlled comparison of supervision meanings, but provide no basis for claiming completion supervision is better.

There is a narrower distinction between weak pooled discrimination and absent within-pool ranking information. In the exported table, the completion probe correctly orders 15/21 within-pool success/failure pairs. This reproduces the reported 71.4% pairwise statistic, whereas pooled AUC compares candidates across different tasks and contexts. These metrics answer different questions. The small, dependent set of pairs and the near-chance engaged-failure comparison prevent a strong inference from the pairwise result. Thus, modest observed coverage gains do not establish a completion-specific representation or explain their mechanism.

3. Successful attacks per unit total computation and larger pools.

The reported comparison fixes maximum full-rollout allocations, while adding feature-extraction passes for V-TAP. Realized rollout counts are approximately unchanged. Without a complete accounting of scoring, collection, fitting, rollout computation, and elapsed time, successful attacks per unit total computation remains unestablished. The efficiency motivation should be distinguished from an efficiency result.

Pool sizes also limit the evidence: 21/48 pools contain two candidates, so K=3 already includes the entire pool. Selection can affect K=3 coverage only in the other 27 pools. On that size-defined subset, R obtains 18.52% Success@3 versus 14.81% for exact random expectation. This removes the pools where exclusion is unnecessary, but remains a small analysis of the same recorded candidate sets. It does not answer the request for larger newly generated pools. Across all pools, R covers nine success-containing pools versus a random expectation of eight; reaching this observed ceiling should not be interpreted as general optimality.

The study connects pre-response target representations to rollout allocation, evaluated with strict outcomes and failure diagnostics. We agree that the present implementation has not demonstrated reliable efficiency or cross-model gains. These distinctions preserve the diagnostic contribution without overstating the positive estimates.

Reviewer_xNMhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes V-TAP, a candidate-selection extension of Tree of Attacks with Pruning for red-teaming tool-using agents. The method trains a linear probe on the target model’s pre-response hidden states using recorded task-completion outcomes, then uses its scores to prioritize candidates for execution. Evaluation includes online searches and frozen candidate pools on AgentHarm. The problem is relevant, but the reported improvements are small and statistically inconclusive, and the experiments do not establish the benefit of the method’s central design choices.

Critical strengths
  • The paper addresses a meaningful practical problem: allocating expensive agent rollouts among candidate attacks.

  • The matched random-selection control and fully observed candidate pools provide useful ways to isolate candidate selection from other search components.

  • The authors clearly disclose negative results, uncertainty, and limitations rather than presenting the small gains as conclusive.

  • The failure-category analysis and supervision ablations are informative, even though their findings do not strongly support the proposed design.

Critical weaknesses
  • The main experiments do not establish a reliable advantage over random selection. The online evaluation yields 34 successful tasks out of 116 for V-TAP versus 32 for random selection, an absolute improvement of 1.72 percentage points. Only six paired outcomes differ, and the exact McNemar test gives (p=0.6875). This does not demonstrate a reliable improvement. The supplementary Gemma evaluation also provides no supporting evidence, with three successes for V-TAP versus four for random selection on 20 tasks.

  • The frozen-pool results are similarly limited. At (K=3), coverage improves from a random expectation of 16.67% to 18.75%, corresponding to nine covered pools versus an expectation of eight. Only nine of the 48 pools contain successful candidates, leaving little evidence from which to assess a general ranking advantage.

  • The ablations do not establish the value of execution-completion supervision or full agent context. Appendix C.4 shows that tool-engagement and non-refusal supervision outperform completion supervision on the reported ranking summaries. Both alternatives achieve 14.58% Success@1, compared with 10.42% for completion supervision. Although these comparisons are confounded by differences in positive-example availability, they do not demonstrate an advantage for the proposed execution-grounded labels.

Similarly, full-context and candidate-only features achieve identical Success@1 and Success@2, with only a small MRR difference. Thus, the empirical benefit of including agent context remains unclear.

The learned score has weak discrimination for the intended outcome. The current selector obtains an AUC of 0.542 for strict success versus all failures and 0.498 for strict success versus tool-engaged failure. These results raise a substantive concern about whether the representation captures successful task completion beyond a weak association with initiating tool use. The small training collection—19 tasks with only 14 completion-positive trajectories—further limits the conclusions.

  • The online result and reproducibility analysis concern different selectors. The historical selector used online cannot be reconstructed exactly from the available environment and rendering information. The current reproducible selector changes the top-ranked candidate in seven of the 48 frozen pools. Exact repeatability of the current pipeline therefore does not reproduce the selector responsible for the principal online result. A fresh online evaluation of the reproducible implementation is needed.

  • Practical computational benefit and generalization remain unestablished. Matching maximum rollout counts does not match total computation because V-TAP adds feature-extraction forward passes. Realized rollout counts are approximately unchanged, and the paper does not demonstrate reduced total computation or latency.

  • Evaluation is also restricted to one benchmark and a narrow search configuration. The TF-IDF baseline sees candidate text only, whereas V-TAP receives additional agent context, so that comparison cannot isolate the value of target-model representations.

Questions for authors

Can the authors evaluate the current reproducible selector online, with additional independent search runs and paired confidence intervals for its difference from random selection?

Does completion supervision outperform non-refusal or tool-engagement supervision when training-set size and positive-example availability are controlled? What evidence supports learning a completion-specific signal?

Can the authors report total inference computation and wall-clock latency, including feature extraction, and compare attack yield under a matched computational budget?

How does V-TAP compare with context-matched text-representation selectors or simpler refusal-oriented scores trained on the same data?

Do the findings persist with larger candidate pools, additional target models, and evaluation splits that separate underlying behavior families rather than only task entries?

1 comment
AuthorsOpenReview ↗

Paired Uncertainty, Within-Pool Ranking, and Limits of the Evidence

Thank you for the careful assessment and for recognizing the practical allocation problem, the matched selection control, and the value of the diagnostics. We agree that the evidence does not establish a reliable end-to-end improvement. We address your questions directly below, including an additional offline uncertainty calculation from the saved candidate-level table.

1. Online evaluation of R and paired uncertainty.

The reported online experiment uses historical selector H, not current selector R. There is no completed online-R comparison in the available evidence. Exact repeatability of R on fixed inputs cannot answer the request for independent online search repetitions.

For the existing H experiment, the paired table contains 30 joint successes, four H-only successes, two random-only successes, and 80 joint failures. Reanalyzing this reported table with 20,000 task-pair bootstrap draws gives a 95% percentile interval of [-2.59, +6.03] percentage points for the observed +1.72-point ASR difference. This assumes independent task entries and does not account for behavior-family dependence or replace repeated searches. Together with exact McNemar p=0.6875, it leaves the online benefit unresolved.

We also computed paired task-cluster bootstrap intervals for R minus exact random expectation on the exported 48-pool table, resampling the 20 tasks jointly for both selectors (20,000 draws):

BudgetCoverage difference (percentage points)95% percentile interval
K=1+1.22[-4.08, +6.88]
K=2+1.74[-1.14, +5.43]
K=3+2.08[0.00, +4.72]

These quantify paired differences, unlike the manuscript's individual-selector intervals. The K=3 boundary at zero reflects R reaching every success-containing pool in this small archive; it is not evidence of a reliable advantage on new pools. This reanalysis involves no new agent executions.

2. Completion supervision and what the score represents.

The existing ablation does not control positive-example availability: the source collection has 14 completion positives, 75 engagement positives, and 71 non-refusal positives. Engagement/non-refusal each achieve 14.58% Success@1 versus 10.42% for completion. We cannot conclude that completion supervision is superior, or explain away this result using class imbalance.

One distinction helps interpret the diagnostics. The pooled success-versus-all-failures AUC of 0.542 compares scores across tasks and pools; candidate selection depends on ordering within each pool. The exported table reproduces 15 correctly ordered pairs out of 21 within-pool success/failure pairs, matching the reported 71.4% pairwise statistic. These are dependent comparisons from a small sample, not 21 independent trials. They explain why modest within-pool coverage gains can coexist with a weak pooled AUC; they do not establish a completion-specific signal. In particular, success-versus-engaged-failure AUC is 0.498. Full and candidate-only inputs also tie at Success@1 and Success@2, so context superiority remains unsupported.

3. Computation and context-matched alternatives.

The experiment controls maximum full-rollout allocation. It does not establish total-compute or latency savings: feature extraction adds target-model forward passes, realized rollout counts are approximately unchanged, and collection/fitting costs must also be accounted for. A matched-compute comparison remains necessary for the practical efficiency claim.

The TF-IDF selector is a useful learned baseline but sees candidate text only. It cannot isolate target representations from the additional context supplied to the CAV. A context-matched text/embedding selector and data-controlled refusal/engagement predictors remain missing comparisons. The current alternative-label results make those predictors substantive alternatives, not merely sanity checks.

4. Larger pools and generalization.

The N>3 analysis uses 27 existing pools where the cap excludes candidates; it is not a new larger-pool experiment. Only nine of all 48 pools contain successes, and R's K=3 result covers nine versus a random expectation of eight. Task partitions are disjoint by entry, without verified behavior-family separation. The Gemma result is unfavorable, at 3/20 versus 4/20.

Accordingly, the contribution supported here is a specified agent-search selection interface and an empirical account of its promise and failure modes. The paired reanalysis sharpens the uncertainty assessment; it does not supply the missing online-R, controlled-supervision, matched-compute, or broader-generalization evidence.

Reviewer_CG8nOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes an improved next-candidate recommendation algorithm for agentic red-teaming using the Tree of Attack Prompts (TAP). The original goal is to identify more successful attacks under a limited execution budget. However, the results presented in the paper are inconclusive, and there are several missing details throughout the text.

The algorithm leverages a concept vector to represent a notion of future success for agentic trajectories. This concept vector guides the selection and ranking of the next candidate to be chosen to perform a complete rollout.

Critical strengths

I did not find any strength to address the plethora of concerns I have, which are mentioned below. There are no positive contributions, and the literature survey omits a major body of relevant work on identifying better candidates for a tree search problem.

Critical weaknesses

Several weaknesses:

  1. Setup and Experiments are not explained properly: The text is tough to follow, no notion of where the historical candidates are coming from, and how the historical evidence is morphed to be applicable for training the concept vector. Some bare-minimum equations have been provided but they do not present a clear account of the problem's scope or the solution.
  2. Why is TAP being used for agentic red-teaming over multiple turns? Why is candidate selection a problem in the first place with TAP, did the authors find an inefficiency in the working of TAP that they set out to address? Why is CAV the chosen way to capture execution relevant information? The paper needs a section on how the data looks and how the data is prepared to predict the success of a multi-step objective. What is the influence of the turns?
  3. Results are inconclusive. Gemma reproduction does not even preserve the direction of improvement. 3 on V-TAP vs 4 with random selection. There seems to be no selection advantage from using CAV.
  4. Full context features and candidate only features seem to achieve an identical success@1 and 2. Completion supervision is somehow worse than with-tool runs and non-refusal supervision.
  5. Active search experiment uses the historical selector H, but reproducibility uses selector R. R does not reproduce the fitted model responsible for the headline online result.
  6. No account of cost in terms of time or money spent in collecting ground truth traces and trajectories.
Questions for authors

Several issues to address above:

  1. Why were easier baselines not tried: re-ranker, embedding model, or any other inexpensive scoring method would have performed equally well for this task? Further, you have provided CAV with additional context and tool schemas. So it's not clean target-model representations either.
  2. How were the training samples collected?
  3. What changes were made to TAP to adapt it to agentic tasks?
2 comments
AuthorsOpenReview ↗

Clarifying the Training Protocol, Agent Search, and Evaluation Scope

Thank you for the detailed review. V-TAP asks whether a pre-rollout score can help allocate agent executions among generated candidates. We clarify the data flow, TAP adaptation, and limits of the comparison below.

1. Training data and prediction timing.

The collection configuration runs Direct TAP without a CAV and enables evaluation of all generated candidates. A training example consists of a task ID, the candidate's pre-response hidden state, and the strict-completion outcome of its subsequent rollout. The source archive contains 176 usable examples from 19 tasks in the 20-entry collection block, including 14 positives. External fixed-pool and online task entries are excluded from this collection. This is entry-level separation; we have not verified separation of all underlying behavior families.

At scoring time, the model sees the candidate and target-visible input, including available tool schemas, but no future responses, tool results, or completion labels. Although the stored outcome field is named subgoal_success, the direct adapter sets objective_scope=full_task, and the orchestrator requires final grader success. Phase progress alone cannot make a direct example positive.

For current selector R, fitting partitions tasks deterministically by a SHA256-based split with training fraction 0.7. Normalization statistics are estimated on training rows only. A class-weighted, regularized logistic probe is then fitted, with at most 24 examples per task/label group. These details describe R, not a reconstruction of H's historical pipeline.

2. Adapting TAP to agent execution.

Each candidate is a single attacker user message addressing the complete objective. Evaluating that message permits the target to conduct a tool loop of up to ten iterations. TAP depth indexes candidate-search rounds, not successive attacker interventions during that rollout. The direct experiment does not use STING task decomposition.

The adaptation uses isolated agent rollouts, strict task grading, and execution feedback for subsequent search. V-TAP additionally ranks candidates before rollout allocation, with width 3, branching factor 2, depth 3, and at most three rollouts per depth. Its score does not modify activations or replace execution grading. The random control isolates selection within our agent-adapted TAP; it does not establish superiority over other search algorithms.

3. Why a linear score, and is there useful selection information?

A linear probe provides an inspectable test of target-side ranking; linear probing itself is not the novelty. Selection can matter when eligible pools exceed the rollout cap. Here that pressure is limited: 21/48 frozen pools contain two candidates, while 27 contain four or six. On the size-defined N>3 subset, R's Success@1 is 11.11% versus 7.10% for random expectation; Success@3 is 18.52% versus 14.81%. These are descriptive comparisons on the existing archive, not evidence from newly generated larger pools.

The ablations constrain the interpretation. Full context and candidate-only features tie at Success@1 and Success@2. Engagement and non-refusal labels outperform completion labels on the reported ranking summaries. Their greater positive-example availability is a confound, not evidence that completion labels would win under controlled conditions. “Execution CAV” describes the training outcome, not identification of a completion-specific mechanism.

4. Baselines, cost, and fairness of reporting.

The TF-IDF baseline tests one learned lexical selector; it does not rule out stronger embedding models or rerankers. Its input lacks the context available to the CAV, so the comparison cannot attribute the difference solely to target internals. The related-work framing should also connect the method more explicitly to learned candidate-ranking heuristics.

The online comparison matches maximum rollout allocations, not total computation. Collection, fitting, and feature extraction add costs; their total wall-clock and monetary cost has not been established. Thus, net computational savings cannot be claimed. The online outcome is 34/116 for H versus 32/116 for random selection (exact McNemar p=0.6875); Gemma gives 3/20 versus 4/20. These do not establish reliable or cross-model improvement.

Regarding the fairness concern, H and R must remain separately attributed: H supplies the historical online result, while R supplies current offline diagnostics and repeatability. Their first choices differ in 7/48 pools, so R's repeatability does not reproduce H's online experiment. The contribution we defend is the pre-execution selection formulation and its controlled diagnostic evaluation, with the artifact discrepancy, unmatched costs/context, and negative results explicit.

Thank you for the response. It answers my factual questions about data collection and the TAP adaptation, but my main concerns remain.

No benefit over random. The online result is 34/116 vs 32/116 (p = 0.6875) and the paired interval includes zero. Gemma reverses the direction, and selection barely matters when 21/48 pools are fully covered at K = 3. Design choices unsupported. Candidate-only features tie full context, and engagement and non-refusal labels beat completion labels. Missing baselines and cost. Cheaper selectors were not run, and total compute and time were not measured, even though efficiency is the paper’s motivation. Headline result not reproducible. The online number comes from selector H, which cannot be reconstructed. R has not been evaluated online.

There are no new experiments or revisions, so my assessment is unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.