acceptodds
Under review as a conference paper at ICLR 2027

When Progress Is Not Success in Multi-Turn LLM Agent Red-Teaming

Abstract

Cybersecurity evaluation of tool-using LLM agents requires testing attacks that unfold over multiple turns. Training red-teaming agents for these attacks is challenging because complete successes are rare, leaving strict terminal rewards with little signal for learning. A natural response is to provide denser rewards for intermediate progress. But these rewards can steer learning toward the wrong behavior: an agent may make more progress without becoming more likely to complete the attack. We therefore study how to improve learning while keeping the terminal security objective fixed. We introduce Outcome-Faithful Consolidation (OFC), which trains directly on trajectories that achieve verified attack success. In a strict four-field data-exfiltration task with a 4B attacker and a frozen 9B victim, dense-reward training raises held-out process scores but lowers complete attack success by 14.0 percentage points relative to outcome-only RL. OFC also improves success rate by 4.6 percentage points at a fixed training budget across 16 paired seeds. Together, these results suggest that the key challenge is not simply obtaining more feedback, but learning from feedback that remains faithful to the terminal cybersecurity outcome.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, reward shaping, credit assignment, reward hacking, LLM agents, multi-turn, red-teaming, prompt injection, rejection sampling fine-tuning, process reward models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7PifOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates reward design strategies for multi-turn red-teaming agents within a rigorous four-field data-exfiltration benchmark. Specifically, a Qwen3.5-4B attacker interacts with a frozen Qwen3.5-9B victim for up to five turns, where programmatic verification mandates that all four target values appear within a single outbound API call across 24 held-out goals.

The empirical findings reveal that dense process rewards elevate the held-out process score from 0.603 to 0.653. However, they paradoxically degrade the full-attack success rate, or ASR, by 14.0 percentage points at iteration 16, dropping from 46.0% to 32.0% across six seeds. Control experiments with relaxed targets exhibit similar performance degradation, while a constructed counterexample and the theoretical analysis in Appendix B.1 demonstrate that the dense reward arm replaces rather than augments the terminal reward objective.

To address this limitation, the authors propose Outcome-Faithful Consolidation, or OFC, an online success-filtered self-training paradigm. Under OFC, trajectories that successfully pass the strict verifier are retained, and the attacker is optimized via cross-entropy loss over these successful traces, omitting any Kullback–Leibler penalty and skipping updates on empty success batches. OFC improves held-out ASR by 4.6 percentage points at iteration 16, increasing from 43.7% to 48.3%. Nevertheless, this performance gap narrows and reverses to -3.18 points by iteration 32, and the improvements are not statistically significant under a larger 9B/27B model pairing.

Critical strengths

S1. The study provides a compelling, well-controlled comparison where the dense and sparse reward arms differ exclusively in their reward formulations. The divergence between process score improvements and ASR degradation emerges as early as iteration 2 both in-sample and on held-out data, supported by convergent evidence from relaxed-target controls.

S2. The evaluation protocol employs fixed readouts, paired seeds, full per-seed reporting, an anonymized codebase, unfiltered cross-entropy and positive-only reinforcement learning controls, transparent negative results, and a candid disclosure of the analysis history.

S3. The counterexample effectively decouples objective mismatch from optimization noise. Furthermore, Equation 4 and the shaping comparison offer a principled framework for learning from verified successes, elegantly formulated as an analytical identity rather than an over-claiming convergence guarantee.

Critical weaknesses

W1.The evaluation lacks an intermediate baseline that preserves the terminal reward while incorporating a dense auxiliary signal, such as potential-based reward shaping or RSPO-style updates. Although analyzed theoretically in Appendix B.1, such an arm is never empirically implemented. Additionally, the process score remains destination-agnostic, crediting Φ=1\Phi=1 even without actual data exfiltration. Consequently, the results demonstrate that this specific proxy replacement is detrimental, rather than establishing that dense feedback is inherently harmful.

W2.OFC functions essentially as a variant of success-filtered self-training, a lineage duly acknowledged by the authors. However, its distinguishing feature—supervising zero-advantage responses within successful trajectories—is not fully isolated: the positive-only control also drops the KL penalty, and the unfiltered cross-entropy ablation is limited to a single complete run without uncertainty estimation.

W3.The +4.6-point gain rests upon a 43.7% baseline with a paired half-width of 2.37. The initial eight seeds yield p = 0.064, the performance advantage reverses to -3.18 at iteration 32 with confidence intervals crossing zero, and the effect lacks statistical significance under the 9B/27B model pairing or Slack transfer tasks. The empirical evidence is restricted to a single task family and model scale.

W4.The manuscript overlooks relevant prior studies that integrate process rewards into multi-turn attackers with contrasting outcomes, train hierarchical red-teaming agents via token-level rewards, or demonstrate that learned process rewards induce reward hacking. Incorporating these discussions would substantially sharpen the paper's claimed novelty.

Questions for authors
  1. Could the authors implement an RSPO-style or potential-based shaping baseline under the same 4B/9B configuration to decouple the negative effects of dense feedback from this specific proxy replacement?
  2. Would the observed performance trend persist if Dense were re-evaluated using a destination-aware process score that credits unique target fields rather than generic chain steps?
  3. Does the OFC performance contrast survive multiplicity correction across the seven arms in Table 1A, given that the initial eight seeds already yield p=0.064p = 0.064?
  4. How robust are these performance contrasts under greedy decoding, expanded rollout budgets, and goal-level confidence intervals? Do bootstrap intervals align with the paired t-tests on the four-to-seven-seed cohorts?
  5. Given that the OFC–Sparse performance gap reverses by iteration 32, could the authors report trajectory diversity metrics over the course of training? Does annealing OFC or interleaving Sparse updates mitigate this crossover?
  6. Since OFC performs no parameter updates on batches lacking successful trajectories, as observed with the 27B victim, under what success-rate regimes does OFC provide meaningful gradient signals, and how should it be coupled with an explorer capable of discovering initial successes?
  7. Given that the unfiltered cross-entropy arm currently comprises only a single complete run, can the authors complete 4 to 6 seed runs to rigorously attribute performance gains to outcome-based selection rather than cross-entropy training alone?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 7Pif: Shaping Equivalence, Re-Run, and OFC Evidence

Thank you for the review! You raise two points: whether OFC is new, and which factor explains its performance, given that Positive-only RL does about as well. We have thoroughly conducted another round of experiment and hopefully they can resolve your concerns! Fuller answers, with Figures R3 and R5, are in Section D of the supplementary file (supplementary_responses.pdf), and its Section F holds new Tables S1 to S7. Other numbers refer to the submitted PDF.

Novelty\ The paper does not propose a new loss mechanism, and it says so: OFC is success-filtered self-training in the line of RFT and ReST-EM (Section 2 and Appendix A.2; the final version will add MART), a lineage that Reviewer 7Pif notes we acknowledge. Its contribution is a result about what a multi-turn attacker should learn from. You call that result "an important finding to reward design in agent training", and the other two reviews credit the same things. Reviewer 7Pif finds the comparison compelling and well controlled and describes Eq. 4 with the shaping comparison as a principled framework for learning from verified successes. Reviewer PvuX lists the dense-reward finding and the paired held-out improvement among the strengths.

A controlled comparison of reward designs\ With attacker, victim, estimator and seeds held fixed, replacing the terminal reward with a dense process score raised that score and lowered complete attack success by 14.0 points in the original campaign (0 of 6 seeds ahead). In the re-run, declared in advance, it gave no gain either: 1.7 points below outcome-only RL (95% CI [-4.5, +1.1]; Table S7). Denser feedback improved complete attack success in neither.

A principle with an explanation\ Updates should follow the verified outcome. Eq. 4 shows that likelihood on verified successes has the gradient of the strict objective under on-policy sampling, while unfiltered self-training has zero expected gradient toward success (Appendix B). The experiments bear out both statements. OFC is ahead of outcome-only RL at iteration 8 in four separate cohorts, by 6.0 to 10.7 points and in 38 of 41 seeds. At iteration 16 (the fixed read-out) it is ahead by 4.6 points in the original campaign (13 of 16 seeds, p = 0.0044 after correction over seven arms) and by 4.2 points in the re-run declared in advance (8 of 8 seeds, corrected p = 0.017). The unfiltered variant stays between 9% and 13% at every read-out (Table S6).

A boundary\ As you might have noticed, the gain is largest at an early budget: in the two cohorts between the original campaign and the re-run, outcome-only RL has caught up by iteration 16 and by iteration 24 (Section 4.4, and Section A of the supplementary file). The success-filtered methods we cite work on single-turn outputs, and none of them is compared with an RL baseline. What this paper adds is the comparison against outcome-only RL across training budgets, on multi-turn tool use.

Which factor explains OFC's performance, and Positive-only RL\ Positive-only RL is our own control, built to answer this question, and the fact that it also gains is evidence for the paper's claim. OFC differs from Sparse RL in the three ways you list, and the existing arms vary them one at a time (table in Section D).

Sample selection explains the gain. Unfiltered CE has OFC's loss and no KL term. Without the outcome filter it reaches 11.3% at iteration 8 on three seeds, against 39.0% for OFC on the same seeds: 27.7 points lower, in every seed (p = 0.003; Table S6).

The optimization objective does not explain it. Positive-only RL keeps the policy-gradient loss, selects the responses with positive advantage, and gains +9.0 points over matched Sparse seeds at iteration 8, against +7.1 for OFC. Compared directly on the six seeds they share, the two are not separated at any read-out (third block of Table S2).

KL regularization does not explain it either. Dropping the KL term is not enough, since unfiltered CE has none and collapses, and the gain appears with the KL term still applied to every response Positive-only RL trains on. Sizing the term from the logged loss ratio puts its penalty at 0.2 points of held-out ASR at iteration 8, against an OFC gain of 7 to 11 points (Figure R3).

Two different losses gain alike when they select on the outcome, and the loss that does not select fails. Together these arms provide the factor-by-factor ablation you ask for, and their answer is the principle the method is named for: outcome-faithful selection. Between the two selection rules OFC is the simpler: it needs the verifier and a cross-entropy loss, with no advantage estimator, baseline or KL term.

We hope this answers both questions, and we would be grateful if you would reconsider your rating. Thanks so much!

Thanks for the clarification. My main concern regarding the missing terminal-reward-preserving comparison and the success-rate regimes remain unsolved. Therefore, I will keep my score.

Reviewer_PvuXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a study on multi-turn LLM agent red-teaming. Specifically, the aurthors have pointed out that denser rewards for intermediate progress can steer learning toward the wrong behavior. To address this point, the authors have proposed Outcome-Faithful Consolidation (OFC), which trains directly on trajectories that achieve verified attack success. The experimental results indicate that OFC can improve the attack success rate.

Critical strengths
  1. The authors have shown that denser process feedback can improve process scores while reducing complete attack success.
  2. The authors have proposed proposed Outcome-Faithful Consolidation (OFC), which preserves strict outcome verification and learns from all attacker responses in every verified successful trajectory.
  3. The authors have established an improvement in held-out attack success under a fixed training budget through paired experiments.
Critical weaknesses
  1. The organization of this paper needs major improvements, e.g., Figure 1 and Figure 2 are not referred in the main text at all, whereas Figure 6 is referred in Section 3.2 but is located in the appendix.
  2. Section 4 lacks ablation studies on Outcome-Faithful Consolidation.
  3. In the experimental setup, Qwen3.5-4B is set as an attacker, whereas a frozen Qwen3.5-9B model is used as a victim. However, it is unclear why Qwen models are selected to act as an attacker and a victim, i.e., it is worth to show experimentally how well other large language models perform in the setting of OFC, in order to better justify the selection of Qwen models.
Questions for authors

Please address the points mentioned in the list of weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer PvuX: Organization Fixes, OFC Ablations, and Model Choice

Thank you for the review!
We address all three weaknesses: the organization is fixed in the updated paper, the ablations of OFC are made explicit here and extended, and the choice of models is justified. We hope the ablations and model justification resolve your soundness concerns; please tell us if any remain. Fuller answers, with Figures R3 to R5, are in Section C of the supplementary file (supplementary_responses.pdf), and its Section F holds new Tables S1 to S7. Other numbers refer to the submitted PDF.

W1. Organization
The final version fixes both. Figure 1 will be cited in the Introduction where the text first contrasts progress with complete success, and Figure 2 where OFC is introduced. The counterexample that Section 3.2 discusses (Figure 6, in the appendix) will move into Section 3.2. While checking, we found Tables 3, 5 and 6 uncited; they will be cited, so that every figure and table is referenced.

W2. Ablations of OFC
Section 4 does contain ablations of OFC. The submission did not present them as such, which is our problem:( The final version will label them, and Table S2 adds one. There are five ablation controls: two vary OFC's update rule (Section 4.3), and three test whether OFC simply benefits from more or better experience (Table 1(A)).

Selection on the verified outcome is what matters. Without the filter, OFC's own loss reaches 11.3% held-out ASR at iteration 8 on three seeds, against 39.0% for OFC on the same seeds (Table S6). With selection kept and the loss changed to policy gradient, Positive-only RL gains about as much as OFC over matched Sparse seeds (+9.0 against +7.1 points), and the new direct comparison does not separate the two (Table S2). The gain comes from selecting on outcomes and does not depend on the loss form.

More or better experience does not explain it either. Doubling the rollouts gives +4.1 points at twice the sampling cost and does not survive the multiplicity correction of Table S1. OFC gives +4.6 points at the same sampling cost as Sparse and does survive (adjusted p = 0.0044).

The one factor the arms do not separate is the KL term, and no run isolates it. Sizing it from the logged loss ratio puts the Sparse penalty at 0.2 points of held-out ASR at iteration 8, against OFC's gain of 7 to 11 points (Figure R3). On more seeds of unfiltered CE: the three measured seeds give a 95% prediction interval of 14.7 to 40.6 points below OFC for a further seed (Figure R5).

The main comparison was also re-run on eight seeds with the test declared in advance: OFC is ahead of Sparse in every seed (+4.2 points, 95% CI [+1.5, +7.0], corrected p = 0.017; Table S7).

W3. Why Qwen models, and other models
The updated paper gives the reason in Section 4.1. The Qwen line offers open weights at 4B, 9B and 27B with one tool-call format, and full-parameter training of the 4B attacker beside a served 9B victim fits one 80 GB GPU. Also, the same model line-up eliminates the variations from different model structure, which can help us to focus on the training mechanism itself. We need open weights on both sides: the attacker is trained with full-parameter updates, and the victim is served locally at a fixed revision and precision so that every arm faces the same victim.

The conclusions do not lean on the strength of any one model. Every comparison is made with the attacker and the victim held fixed, on matched seeds, so what is measured is the effect of the training signal. A different model family would change the absolute success rates. The paper's claims are about the ordering of training rules for a fixed pair.

The paper also includes a second pairing: a 9B attacker against a 27B victim (Section 4.5). For this paring, OFC is ahead in 4 of 4 seeds at iteration 8 (+3.9 points, p = 0.058). Across the five Qwen cohorts we ran, OFC is ahead at iteration 8 in all five, with a mean gain of +7.1 points (95% CI [+3.5, +10.7]) and a 95% prediction interval of -1.5 to +15.7 points for a further setting of the same kind (Figure R4). Our code is in the anonymized repository, and a final OFC attacker of the re-run has been added to it (https://anonymous.4open.science/r/when-progress-is-not-success-0F02), so both the training rule and the trained attacker can be tried against other victims. You are more than welcomed to check it out:)

We hope these answers resolve the three points! We would be grateful if you would reconsider your rating, in particular the soundness score. Thank you so much!

Thanks for your response. You clarification helps address some of my concerns so I have adjusted my score accodingly.

Reviewer_7mLAOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reward design problem in multi-turn LLM agent red-teaming. The paper points out that even a dense process reward can reduce terminal reward sparsity; intermediate progress may not necessarily be consistent with attack success. Hence, optimize process reward may reduce the actual attack rate. To solve this problem, this paper proposes outcome-faithful consolidation. By strict outcome verification, it select the complete attack chain and use SFT to learn all attcker reponses from those chains.

Critical strengths

By controlled evaluation, the paper shows that the dense process reward event increases the process score. But it leads to a decrease in the final attack success rate. These findings reveal the difference between intermediate progress and terminal success. This is an important finding to reward design in agent training.

Critical weaknesses

The novelty is limited. The facts of this paper are that it applies rejection sampling fine-tuning with success-filtered self-training to multi-turn read teaming. The core mechanism is to filter the success chains and perform SFT. In the paper, the positive-only RL also achieves similar improvements, while the current ablation does not prove that OFC has its uniqueness.

Questions for authors

Can the author explain what facts leads the OFC good performance? i.e., a detailed ablation. As compared to Sparse RL, the OFC change Sample Selection、Optimization Objective and KL Regularization. Also, Positive-only RL seems to achieve similar good performance.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7mLA: Reward Design Contribution and Factor Ablation

Thank you for the review! You raise two points: whether OFC is new, and which factor explains its performance, given that Positive-only RL does about as well. A direct test answers the second, and the answer supports the paper's claim. Fuller answers, with Figures R3 and R5, are in Section D of the supplementary file (supplementary_responses.pdf), and its Section F holds new Tables S1 to S7. Other numbers refer to the submitted PDF.

Novelty
The paper does not propose a new loss, and it says so: OFC is success-filtered self-training in the line of RFT and ReST-EM (Section 2 and Appendix A.2; the final version will add MART), a lineage that Reviewer 7Pif notes we acknowledge. Its contribution is a result about what a multi-turn attacker should learn from. You call that result "an important finding to reward design in agent training", and the other two reviews credit the same things. Reviewer 7Pif finds the comparison compelling and well controlled, and describes Eq. 4 with the shaping comparison as a principled framework for learning from verified successes. Reviewer PvuX lists the dense-reward finding and the paired held-out improvement among the strengths.

A controlled comparison of reward designs. With attacker, victim, estimator and seeds held fixed, replacing the terminal reward with a dense process score raised that score and lowered complete attack success by 14.0 points in the original campaign (0 of 6 seeds ahead). In the re-run, declared in advance, it gave no gain either: 1.7 points below outcome-only RL (95% CI [-4.5, +1.1]; Table S7). Denser feedback improved complete attack success in neither.

A principle with an explanation
Updates should follow the verified outcome. Eq. 4 shows that likelihood on verified successes has the gradient of the strict objective under on-policy sampling, while unfiltered self-training has zero expected gradient toward success (Appendix B). The experiments bear out both statements. OFC is ahead of outcome-only RL at iteration 8 in four separate cohorts, by 6.0 to 10.7 points and in 38 of 41 seeds. At iteration 16 (the fixed read-out) it is ahead by 4.6 points in the original campaign (13 of 16 seeds, p = 0.0044 after correction over seven arms) and by 4.2 points in the re-run declared in advance (8 of 8 seeds, corrected p = 0.017). The unfiltered variant stays between 9% and 13% at every read-out (Table S6).

A boundary
The gain is largest at an early budget: in the two cohorts between the original campaign and the re-run, outcome-only RL has caught up by iteration 16 and by iteration 24 (Section 4.4, and Section A of the supplementary file). The success-filtered methods we cite work on single-turn outputs, and none of them is compared with an RL baseline. What this paper adds is the comparison against outcome-only RL across training budgets, on multi-turn tool use.

Which factor explains OFC's performance, and Positive-only RL?
Positive-only RL is our own control, built to answer this question, and the fact that it also gains is evidence for the paper's claim. OFC differs from Sparse RL in the three ways you list, and the existing arms vary them one at a time (table in Section D).

Sample selection explains the gain. Unfiltered CE has OFC's loss and no KL term. Without the outcome filter it reaches 11.3% at iteration 8 on three seeds, against 39.0% for OFC on the same seeds: 27.7 points lower, in every seed (p = 0.003; Table S6).

The optimization objective does not explain it. Positive-only RL keeps the policy-gradient loss, selects the responses with positive advantage, and gains +9.0 points over matched Sparse seeds at iteration 8, against +7.1 for OFC. Compared directly on the six seeds they share, the two are not separated at any read-out (third block of Table S2).

KL regularization does not explain it either. Dropping the KL term is not enough, since unfiltered CE has none and collapses, and the gain appears with the KL term still applied to every response Positive-only RL trains on. Sizing the term from the logged loss ratio puts its penalty at 0.2 points of held-out ASR at iteration 8, against an OFC gain of 7 to 11 points (Figure R3).

Two different losses gain alike when they select on the outcome, and the loss that does not select fails. Together these arms provide the factor-by-factor ablation you ask for, and their answer is the principle the method is named for: outcome-faithful selection. Between the two selection rules OFC is the simpler: it needs the verifier and a cross-entropy loss, with no advantage estimator, baseline or KL term.

We hope this answers both questions! We would be grateful if you would reconsider your rating. Thank you for your feedback!

Dear Authors, I do really appreciate your feedback. However, as I still don't understand what this paper is about and your feedback (it's on my end, as I am not in that field), I may need to keep my score. I do apologize for that. I also noticed ACs before regarding this issue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.