- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates reward design strategies for multi-turn red-teaming agents within a rigorous four-field data-exfiltration benchmark. Specifically, a Qwen3.5-4B attacker interacts with a frozen Qwen3.5-9B victim for up to five turns, where programmatic verification mandates that all four target values appear within a single outbound API call across 24 held-out goals.
The empirical findings reveal that dense process rewards elevate the held-out process score from 0.603 to 0.653. However, they paradoxically degrade the full-attack success rate, or ASR, by 14.0 percentage points at iteration 16, dropping from 46.0% to 32.0% across six seeds. Control experiments with relaxed targets exhibit similar performance degradation, while a constructed counterexample and the theoretical analysis in Appendix B.1 demonstrate that the dense reward arm replaces rather than augments the terminal reward objective.
To address this limitation, the authors propose Outcome-Faithful Consolidation, or OFC, an online success-filtered self-training paradigm. Under OFC, trajectories that successfully pass the strict verifier are retained, and the attacker is optimized via cross-entropy loss over these successful traces, omitting any Kullback–Leibler penalty and skipping updates on empty success batches. OFC improves held-out ASR by 4.6 percentage points at iteration 16, increasing from 43.7% to 48.3%. Nevertheless, this performance gap narrows and reverses to -3.18 points by iteration 32, and the improvements are not statistically significant under a larger 9B/27B model pairing.
Critical strengths
S1. The study provides a compelling, well-controlled comparison where the dense and sparse reward arms differ exclusively in their reward formulations. The divergence between process score improvements and ASR degradation emerges as early as iteration 2 both in-sample and on held-out data, supported by convergent evidence from relaxed-target controls.
S2. The evaluation protocol employs fixed readouts, paired seeds, full per-seed reporting, an anonymized codebase, unfiltered cross-entropy and positive-only reinforcement learning controls, transparent negative results, and a candid disclosure of the analysis history.
S3. The counterexample effectively decouples objective mismatch from optimization noise. Furthermore, Equation 4 and the shaping comparison offer a principled framework for learning from verified successes, elegantly formulated as an analytical identity rather than an over-claiming convergence guarantee.
Critical weaknesses
W1.The evaluation lacks an intermediate baseline that preserves the terminal reward while incorporating a dense auxiliary signal, such as potential-based reward shaping or RSPO-style updates. Although analyzed theoretically in Appendix B.1, such an arm is never empirically implemented. Additionally, the process score remains destination-agnostic, crediting even without actual data exfiltration. Consequently, the results demonstrate that this specific proxy replacement is detrimental, rather than establishing that dense feedback is inherently harmful.
W2.OFC functions essentially as a variant of success-filtered self-training, a lineage duly acknowledged by the authors. However, its distinguishing feature—supervising zero-advantage responses within successful trajectories—is not fully isolated: the positive-only control also drops the KL penalty, and the unfiltered cross-entropy ablation is limited to a single complete run without uncertainty estimation.
W3.The +4.6-point gain rests upon a 43.7% baseline with a paired half-width of 2.37. The initial eight seeds yield p = 0.064, the performance advantage reverses to -3.18 at iteration 32 with confidence intervals crossing zero, and the effect lacks statistical significance under the 9B/27B model pairing or Slack transfer tasks. The empirical evidence is restricted to a single task family and model scale.
W4.The manuscript overlooks relevant prior studies that integrate process rewards into multi-turn attackers with contrasting outcomes, train hierarchical red-teaming agents via token-level rewards, or demonstrate that learned process rewards induce reward hacking. Incorporating these discussions would substantially sharpen the paper's claimed novelty.
Questions for authors
- Could the authors implement an RSPO-style or potential-based shaping baseline under the same 4B/9B configuration to decouple the negative effects of dense feedback from this specific proxy replacement?
- Would the observed performance trend persist if Dense were re-evaluated using a destination-aware process score that credits unique target fields rather than generic chain steps?
- Does the OFC performance contrast survive multiplicity correction across the seven arms in Table 1A, given that the initial eight seeds already yield ?
- How robust are these performance contrasts under greedy decoding, expanded rollout budgets, and goal-level confidence intervals? Do bootstrap intervals align with the paired t-tests on the four-to-seven-seed cohorts?
- Given that the OFC–Sparse performance gap reverses by iteration 32, could the authors report trajectory diversity metrics over the course of training? Does annealing OFC or interleaving Sparse updates mitigate this crossover?
- Since OFC performs no parameter updates on batches lacking successful trajectories, as observed with the 27B victim, under what success-rate regimes does OFC provide meaningful gradient signals, and how should it be coupled with an explorer capable of discovering initial successes?
- Given that the unfiltered cross-entropy arm currently comprises only a single complete run, can the authors complete 4 to 6 seed runs to rigorously attribute performance gains to outcome-based selection rather than cross-entropy training alone?
2 comments
Thanks for the clarification. My main concern regarding the missing terminal-reward-preserving comparison and the success-rate regimes remain unsolved. Therefore, I will keep my score.
Response to Reviewer 7Pif: Shaping Equivalence, Re-Run, and OFC Evidence
Thank you for the review! You raise two points: whether OFC is new, and which factor explains its performance, given that Positive-only RL does about as well. We have thoroughly conducted another round of experiment and hopefully they can resolve your concerns! Fuller answers, with Figures R3 and R5, are in Section D of the supplementary file (supplementary_responses.pdf), and its Section F holds new Tables S1 to S7. Other numbers refer to the submitted PDF.
Novelty\ The paper does not propose a new loss mechanism, and it says so: OFC is success-filtered self-training in the line of RFT and ReST-EM (Section 2 and Appendix A.2; the final version will add MART), a lineage that Reviewer 7Pif notes we acknowledge. Its contribution is a result about what a multi-turn attacker should learn from. You call that result "an important finding to reward design in agent training", and the other two reviews credit the same things. Reviewer 7Pif finds the comparison compelling and well controlled and describes Eq. 4 with the shaping comparison as a principled framework for learning from verified successes. Reviewer PvuX lists the dense-reward finding and the paired held-out improvement among the strengths.
A controlled comparison of reward designs\ With attacker, victim, estimator and seeds held fixed, replacing the terminal reward with a dense process score raised that score and lowered complete attack success by 14.0 points in the original campaign (0 of 6 seeds ahead). In the re-run, declared in advance, it gave no gain either: 1.7 points below outcome-only RL (95% CI [-4.5, +1.1]; Table S7). Denser feedback improved complete attack success in neither.
A principle with an explanation\ Updates should follow the verified outcome. Eq. 4 shows that likelihood on verified successes has the gradient of the strict objective under on-policy sampling, while unfiltered self-training has zero expected gradient toward success (Appendix B). The experiments bear out both statements. OFC is ahead of outcome-only RL at iteration 8 in four separate cohorts, by 6.0 to 10.7 points and in 38 of 41 seeds. At iteration 16 (the fixed read-out) it is ahead by 4.6 points in the original campaign (13 of 16 seeds, p = 0.0044 after correction over seven arms) and by 4.2 points in the re-run declared in advance (8 of 8 seeds, corrected p = 0.017). The unfiltered variant stays between 9% and 13% at every read-out (Table S6).
A boundary\ As you might have noticed, the gain is largest at an early budget: in the two cohorts between the original campaign and the re-run, outcome-only RL has caught up by iteration 16 and by iteration 24 (Section 4.4, and Section A of the supplementary file). The success-filtered methods we cite work on single-turn outputs, and none of them is compared with an RL baseline. What this paper adds is the comparison against outcome-only RL across training budgets, on multi-turn tool use.
Which factor explains OFC's performance, and Positive-only RL\ Positive-only RL is our own control, built to answer this question, and the fact that it also gains is evidence for the paper's claim. OFC differs from Sparse RL in the three ways you list, and the existing arms vary them one at a time (table in Section D).
Sample selection explains the gain. Unfiltered CE has OFC's loss and no KL term. Without the outcome filter it reaches 11.3% at iteration 8 on three seeds, against 39.0% for OFC on the same seeds: 27.7 points lower, in every seed (p = 0.003; Table S6).
The optimization objective does not explain it. Positive-only RL keeps the policy-gradient loss, selects the responses with positive advantage, and gains +9.0 points over matched Sparse seeds at iteration 8, against +7.1 for OFC. Compared directly on the six seeds they share, the two are not separated at any read-out (third block of Table S2).
KL regularization does not explain it either. Dropping the KL term is not enough, since unfiltered CE has none and collapses, and the gain appears with the KL term still applied to every response Positive-only RL trains on. Sizing the term from the logged loss ratio puts its penalty at 0.2 points of held-out ASR at iteration 8, against an OFC gain of 7 to 11 points (Figure R3).
Two different losses gain alike when they select on the outcome, and the loss that does not select fails. Together these arms provide the factor-by-factor ablation you ask for, and their answer is the principle the method is named for: outcome-faithful selection. Between the two selection rules OFC is the simpler: it needs the verifier and a cross-entropy loss, with no advantage estimator, baseline or KL term.
We hope this answers both questions, and we would be grateful if you would reconsider your rating. Thanks so much!