acceptodds
Under review as a conference paper at ICLR 2027

From Expected Harmfulness to Likelihood: A Probabilistic Reformulation of Jailbreaking LLM Agents

Abstract

When the harmfulness of an LLM agent's output can be quantified, a natural jailbreaking objective is to maximize expected harmfulness over admissible input modifications. An alternative approach constructs or selects harmful target outputs and modifies the input to increase their likelihood. We establish a precise connection between these two approaches through a probabilistic reformulation. Specifically, we show that the gradient of the logarithm of expected harmfulness with respect to the input equals the expected input gradient of the model's log-likelihood under a harmfulness reweighted output distribution. This identity provides a unified interpretation of expected harmfulness and target likelihood optimization. Building on this connection, we propose **OPUR**, a sampling distribution designed to generate highly harmful target outputs and use the resulting samples to guide likelihood-based input optimization. Experiments demonstrate the effectiveness of the resulting method in jailbreaking LLM agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Jailbreaking, Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5RjuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies adversarial-suffix jailbreaks against LLM agents. It connects two existing ways of framing the attack objective, the first being the REINFORCE style objective over possible inputs, and the second being the GCG style optimization objective of maximizing the log likelihood of the harmful response by combining them to sample from a distribution in which responses have high weight if they are likely to be sampled by the model and also harmful. The authors develop a practical algorithm for sampling from this distirbution called OPUR.This outperforms the prior baseline by 13.07% on AgentHarm and similar gains on WebShop and InjecAgent.

Critical strengths
  1. The link between the two kinds of objectives as effectively two different paradigms for LLM jailbreaking is very interesting and the theoretical derivation is nice. It also explains a known failure of REINFORCE-style attacks as early in optimization nearly all samples are benign so the gradient is near 0 and reweighting toward likely and harmful responses addresses exactly that.

  2. Agent rollouts are stochastic, so a deterministic attack that places targets using a single rollout, as UDora does, can overfit to one trajectory. OPUR instead aggregates several rollouts and samples insertion positions, which introduces stochasticity into the objective to prevent overfitting to any one kind of objective.

Critical weaknesses
  1. I'm not sure how relevant the theory is to the actual method. The theory uses fairly standard techniques from reward weighted regression and expectation maximization, but the algorithm itself doesn't really use the method at all. The paper says its a practical realization of sampling targets from that distribution of both likely and harmful responses, but OPUR never defines the harmful distribution, or attempts to somehow actually estimate this likely and harmful response distribution. Instead it uses the splicing technique to insert jailbreaks and then uses UDora's loss function, which raises the question of the actual novelty of the approach beyond the theory.

  2. There are no seed values to justify the statistical significance of the results.

Questions for authors
  1. What is the per-step and total compute (forward/backward passes, wall-clock) of OPUR versus UDora and other baselines?
  2. How does softmax position sampling relate with greedy top scoring or random selection?
  3. Is there any actual link between the theory and the algorithm? Where does OPUR actually sample from this likely and harmful response distribution, and how is it actually constructed practically.
3 comments

Thank you for your review and the helpful comments on the theory–algorithm connection, variability across random seeds, computational cost, and position-selection mechanism. We address each point below.

  1. Connection between π\pi and OPUR

    Thank you for pointing out that this connection needed clarification. Theorem 2 connects the gradient of log expected harmfulness to likelihood gradients under π\pi. OPUR provides a practical surrogate for this target-selection role: bootstrap-style rollouts supply contexts grounded in current model behavior, target segments encode the attack objective, and probabilistic position selection and probe screening prioritize locally compatible targets. The retained contexts guide likelihood-based suffix optimization, while the added component ablations provide empirical support for this construction.

  2. Evaluation across multiple random seeds

    Thank you for highlighting the need to account for randomness. We have repeated the evaluation of all five methods using three random seeds on both models and all three benchmarks. We report average ASR (%) as mean ± standard deviation. The complete results are summarized below.

    Results on Llama-3.1-8B-Instruct:

    MethodAgentHarmInjecAgentWebShop
    GCG35.04 ± 2.1530.00 ± 1.001.11 ± 0.96
    REINFORCE-GCG52.65 ± 2.2947.00 ± 2.007.22 ± 0.96
    UDora Sequential61.93 ± 1.7142.33 ± 2.0818.33 ± 1.67
    UDora Joint63.07 ± 2.0545.33 ± 0.5816.11 ± 0.96
    OPUR76.70 ± 1.7148.67 ± 2.5227.22 ± 0.96

    Results on Ministral-8B-Instruct-2410:

    MethodAgentHarmInjecAgentWebShop
    GCG88.07 ± 1.1430.00 ± 1.005.56 ± 1.93
    REINFORCE-GCG97.92 ± 0.6638.00 ± 1.006.67 ± 0.00
    UDora Sequential99.24 ± 0.3325.33 ± 1.5325.00 ± 2.89
    UDora Joint98.49 ± 0.8728.67 ± 1.1518.33 ± 2.89
    OPUR99.62 ± 0.3341.67 ± 1.5338.33 ± 1.67

    OPUR achieves the highest mean average ASR in all six model–benchmark combinations. These repeated-run results provide further evidence for its mean-performance advantage across the evaluated settings.

  3. Runtime Computation

    Thank you for requesting clearer computational-cost accounting. Each OPUR iteration generates KoptK_{\mathrm{opt}} rollouts, scores candidate positions, and evaluates candidate contexts using forward-only probe computations. Only retained contexts contribute to the target-span gradients used for suffix-candidate generation; candidate evaluation is also forward-only. Under the four-to-two configuration, up to 4Kopt4K_{\mathrm{opt}} contexts are probed and 2Kopt2K_{\mathrm{opt}} contribute to gradient computation. Compared to GCG, OPUR adds rollout context construction. Compared to UDora, it adds sampled-context exploration and screening. We have also added total-runtime comparisons and rollout-budget runtime ablations in the Runtime Analysis in Appendix D , documenting the time cost associated with the ASR improvements.

  4. Relationship between Softmax, greedy, and random selection

    Thank you for asking us to clarify the selection mechanism. Greedy and Softmax selection use the same insertion-position scores: greedy selection chooses the highest-scoring feasible positions, whereas Softmax converts these scores into sampling probabilities proportional to exp⁡(rj/T)\exp(r_j/T). Uniform random selection ignores the scores. Lower temperatures concentrate probability on higher-scoring positions, while higher temperatures approach uniform selection over eligible positions. These probabilities guide surrogate-context construction; suffix gradients are subsequently computed from the retained contexts. We have added comparisons of all three strategies and temperature-sensitivity curves in Appendix C.

  5. Clarifying the relationship in the manuscript

    Thank you for emphasizing the need to explain this relationship explicitly in the paper. We have added a discussion following Eq. (8) and clarified the motivation and algorithm description in Section 4. The revised text explains the target-selection role of π\pi, how OPUR’s components implement the corresponding design considerations, and how the component experiments empirically support the surrogate construction.

Reviewer_5RjuOpenReview ↗

Official Comment by Reviewer 5Rju

Thank you for the response and the new experiments. My main concern remains the link between the theory and the algorithm. The revised text now describes OPUR as a practical surrogate for which the theory provides only a design rationale, with its effectiveness supported by attack performance. This is fine, but I think that the paper needs to be structured differently because a large majority of the paper is deriving this.

The rollout comparison is nice, and does show that more compute does lead to better ASR. Ideally I would have liked to see a matched compute vs ASR plot for all the methods, but the results show that there is still a gain at roughly the same compute, so I think this is fine. If the authors restructure the first part of the paper to talk less about the theory and more about the fundamental way the algorithm works, I will increase my score on this condition.

Thank you for your constructive suggestion regarding the structure of the paper. We have revised the manuscript accordingly, expanding the algorithm section to explain OPUR’s workflow and implementation details more clearly, including surrogate target construction, probabilistic position selection, probe-loss screening and suffix optimization. We have also condensed the theoretical exposition while preserving the core results, allocating more space to the algorithmic mechanisms. We hope these revisions address your concern and make the method easier to follow. Please refer to the revised PDF for the updated structure and presentation.

Reviewer_4SwMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates LLM agent jailbreaking through a probabilistic lens. By introducing a harmfulness-reweighted response distribution, the authors establish a gradient-level equivalence between maximizing expected harmfulness and optimizing target likelihood. Building on this theoretical formulation, they propose OPUR, an optimization-based jailbreak method that constructs surrogate optimization contexts via stochastic agent rollouts, probabilistic position sampling, and probe-based screening. These contexts are then leveraged to guide adversarial suffix optimization. Evaluated on AgentHarm, InjecAgent, and WebShop, OPUR consistently achieves competitive or superior attack success rates (ASRs) compared to existing optimization-based baselines.

Critical strengths
  1. This paper offers an interesting perspective on jailbreak optimization by establishing a gradient-level equivalence between maximizing expected harmfulness and optimizing target likelihood.

  2. The authors evaluate OPUR on three representative benchmarks, covering both malicious-instruction and malicious-environment attack scenarios. The results demonstrate improvements over several established optimization-based baselines.

Critical weaknesses
  1. Limited evaluation on realistic agent frameworks. The authors evaluate OPUR exclusively on agents built with ReAct prompting, which may not adequately represent modern agentic systems such as Claude Code, Codex, and Gemini CLI. These systems involve more complex tool interactions and execution workflows, potentially introducing different attack surfaces and challenges. The authors should consider evaluating OPUR on more realistic agent frameworks or discussing its applicability and limitations in such settings.

  2. Add more relevant baselines. The authors mainly consider comparing with classical jailbreaking methods, but some recent studies[1-3] have been proposed to specifically target LLM Agents.

[1] Yueh-Han, Chen, et al. "Monitoring decomposition attacks in llms with lightweight sequential monitors." arXiv preprint arXiv:2506.10949 (2025).

[2] Li, Xu, et al. "Unsafer in many turns: Benchmarking and defending multi-turn safety risks in tool-using agents." arXiv preprint arXiv:2602.13379 (2026).

[3] Talokar, Nivya, et al. "Helpful to a fault: Measuring illicit assistance in multi-turn, multilingual llm agents." arXiv preprint arXiv:2602.16346 (2026).

  1. The details of the experimental settings are unclear. For example, in Section 5.2, the authors only provide the mathematical definition of ASR without specifying how attack success is determined in practice. In particular, it is unclear whether success is evaluated based on the generation of target tool calls, the actual execution of harmful actions, or benchmark-specific success criteria. The authors should clarify the evaluation procedures and success criteria for each benchmark to improve reproducibility and ensure a fair comparison with existing baselines.
Questions for authors

Please see the weaknesses above and the following concerns:

  1. The author should add more discussion about attack efficiency. OPUR introduces additional computational overhead through multiple stochastic rollouts, candidate context sampling, and probe-based screening. However, the authors only report ASR without comparing computational costs.

  2. The author should discuss the sensitivity of sampling hyperparameters. For example, how sensitive is OPUR to the Softmax temperature τ\tau? A sensitivity analysis would help assess the robustness of the proposed method.

4 comments

Thank you for your review and the helpful suggestions on evaluation scope, baseline coverage, experimental clarity, and computational cost. We have expanded the experiments and revised the manuscript accordingly.

  1. Evaluation on different agent frameworks

    Thank you for highlighting the importance of evaluation beyond ReAct-prompted agents. Our current optimization requires gradients with respect to the adversarial input representation, so access to an agent framework’s source code or API alone does not provide the necessary access to its underlying model. We have added a Discussion and Limitations section acknowledging that our current experiments do not establish effectiveness in more complex deployed agent workflows. We also discuss extensions using compatible open-weight backends and transfer-based or gradient-free adaptations for settings without gradient access.

  2. Comparisons with recent methods

    Thank you for recommending these recent works. We have added comparisons with the attack-construction procedures from Chen et al. (2025) and Li et al. (2026), adapted to our three benchmarks. The details of ASR are summarized below:

    Agentharm

    methodDetailed + HintDetailed − HintBrief + HintBrief − HintAvg. ASR
    Monitoring81.82%54.55%63.64%72.73%68.18%
    ToolShield36.36%31.82%29.55%29.55%31.82%
    OPUR72.73%75.00%72.73%79.55%75.00%

    Injecagent

    methoddirect harmdata stealingAvg. ASR
    Monitoring40%48%44%
    ToolShield26%42%34%
    OPUR48%50%49%

    webshop

    methodprice dismatchattribute mismatchcategory dismatchall dismatchAvg. ASR
    Monitoring40.00%40.00%20.00%20.00%30.00%
    ToolShield0.00%6.67%6.67%0.00%3.33%
    OPUR13.33%40.00%33.33%20.00%26.67%

    OPUR achieves the highest average ASR on AgentHarm and InjecAgent and outperforms the attack construction from Li et al. on all three benchmarks. On WebShop, the method from Chen et al. achieves a higher average ASR than OPUR. These comparisons broaden the empirical evaluation and reveal differences in effectiveness across benchmarks.

  3. Clarifying experimental settings and success criteria

    Thank you for pointing out that the success criteria were insufficiently specified. We have expanded the Metrics section in 5.2 to state the benchmark-specific definitions following UDora’s published protocol. On AgentHarm, it requires non-refusal and a call to the designated target function. On InjecAgent, success corresponds to generating the designated adversarial tool action. On WebShop, it corresponds to generating a click action on the attacker-designated product, identified by its ASIN. The revised text makes explicit the targeted action measured on each benchmark.

  4. Computational-cost analysis

    Thank you for suggesting a computational-cost analysis. We have added a Runtime Analysis in Appendix D comparing the overall runtime of all five methods on Llama and Ministral under the Kopt=1K_{\mathrm{opt}}=1 setting. We also report runtime of ablation studies for UDora Sequential, UDora Joint, and OPUR with Kopt∈{1,2,3}K_{\mathrm{opt}}\in\{1,2,3\}. Together with the ASR ablations, these results show where OPUR’s additional runtime yields improved attack success.

  5. Softmax-temperature sensitivity

    Thank you for suggesting this analysis. We have added temperature-sensitivity curves on all three benchmarks using Llama-3.1-8B-Instruct, reporting average ASR for T∈{0.1,0.2,0.5,0.7,1.0,1.5}T\in\{0.1,0.2,0.5,0.7,1.0,1.5\}, with greedy selection denoted by T=0T=0 as a reference. The curves show that the effect of temperature is non-monotonic and varies across benchmarks. The experimental settings and results are provided in Appendix C.3 .

Thank you for your thoughtful feedback and the time you have devoted to reviewing our work. We have provided our responses to your comments and revised the manuscript accordingly. As the discussion deadline approaches, we would appreciate any follow-up feedback. If you have any remaining questions or concerns, we would be happy to address them and provide further clarification. Thank you again for your time and consideration.

Sorry for my late reply. Although there are some minor concerns (e.g., presentation, analysis), I think my major concerns have been addressed. I will increase my score.

Thank you for your careful review and constructive feedback. We are glad that your major concerns have been addresse. And we will continue to refine the presentation and strengthen the analysis based on your comments. We sincerely appreciate your time and suggestions, which have helped us improve the clarity and quality of our work.

Reviewer_wM8BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified probabilistic framework for jailbreak optimization.It proves that the gradient of log expected harmfulness equals the expected model log-likelihood gradient under a response distribution proportional to model likelihood times response harmfulness. Locally, maximizing expected harmfulness can therefore be interpreted as increasing the likelihood of adaptively selected harmful targets.

To approximate this distribution when harmful outputs are difficult to sample directly, the paper proposes OPUR. It gathers stochastic agent rollouts, scores candidate insertion positions using target-token matches and probabilities, samples position sets with a temperature-controlled softmax, screens surrogate contexts with a probe loss, and aggregates gradients across rollouts to update a discrete adversarial suffix.

Experiments on AgentHarm, InjecAgent, and WebShop demonstrate the effectiveness of OPUR based on Llama-3.1-8B-Instruct and Ministral-8B-Instruct-2410.

Critical strengths

The paper provides a clear probabilistic connection between expected harmfulness and likelihood-based optimization.

The proposed OPUR addresses stochasticity in agent trajectories through multi-rollout position sampling and context screening.

Critical weaknesses

Section 3 introduces the harmfulness-reweighted response distribution π\pi, but lacks further analysis of how π\pi is approximated in proposed OPUR.

Lack of comparison between some strong baselines and some mainstream llm.

Lack of experiments on other-scale models and new open-source models.

Questions for authors

Can you report ablations for probabilistic position sampling such as softmax sampling, always choosing the highest-scoring position, random sampling? Can you report ablations to demonstrate the effectiveness of screen sampled context?

2 comments

Thank you for your review and the helpful comments on the connection between our formulation and algorithm, the breadth of model evaluation, and the component ablations. We address these points below.

  1. Clarifying the connection between π\pi and OPUR

Thank you for pointing out that this connection needed clarification. We have revised the discussion following Eq. (8) and the presentation in Section 4. The distribution π\pi motivates jointly considering model compatibility and the harmful objective when selecting likelihood targets. OPUR provides a practical surrogate for this target-selection role: bootstrap-style rollouts supply contexts reflecting current model behavior, target construction encodes the attack objective, and probabilistic position selection and probe screening assess local target compatibility. The revised text explains this design rationale, while the added component experiments provide empirical support for the surrogate construction.

  1. Evaluation on an additional model

Thank you for suggesting evaluation on an additional model. We have completed experiments on Qwen3-8B across AgentHarm, InjecAgent, and WebShop. OPUR achieves the highest average ASR on all three benchmarks

Agentharm

methodDetailed + HintDetailed − HintBrief + HintBrief − HintAvg. ASR
GCG86.36%79.55%81.82%59.09%76.70%
REINFORCE-GCG100.00%81.82%90.91%68.18%85.23%
UDora Sequential90.91%72.73%86.36%72.73%80.68%
UDora Joint93.18%79.55%86.36%68.18%81.82%
OPUR100%90.91%81.82%81.82%88.64%

Injecagent

methoddirect harmdata stealingAvg. ASR
GCG4%6%5%
REINFORCE-GCG6%10%8%
UDora Sequential2%4%3%
UDora Joint2%8%5%
OPUR8%14%11%

Webshop

methodprice dismatchattribute mismatchcategory dismatchall dismatchAvg. ASR
GCG0.00%6.67%13.33%0.00%5.00%
REINFORCE-GCG13.33%13.33%6.67%0.00%8.33%
UDora Sequential6.67%20%0.00%0.00%6.67%
UDora Joint6.67%13.33%6.67%0.00%6.67%
OPUR6.67%20.00%13.33%6.67%11.67%
  1. Strong Baseline Comparison Thank you for recommending comparisons with stronger baselines. Please refer to our response to Reviewer 4SwM, Point 2.

  2. Ablation of the sampling stage Thank you for suggesting this ablation. We have added comparisons of greedy, random and Softmax position selection on all three benchmarks using Llama-3.1-8B-Instruct. Details of data are included in Appendix C.

BenchmarkGreedyRandomSoftmax
AgentHarm50.00%43.18%56.82%
InjecAgent44.00%40.00%46.00%
WebShop5.00%3.33%5.00%
  1. Probe-loss screening and the four-to-two retention scheme

Thank you for asking us to examine this component more closely. We have added three context-retention strategies with a candidate budget of four per rollout: Keep All retains all valid candidates; Random Keep uniformly retains two without replacement; and Probe Loss retains the two candidates with the lowest probe loss under the current suffix. The detailed results are provided in Appendix C.

BenchmarkKeep AllRandom KeepProbe Loss
AgentHarm45.45%50.00%56.82%
InjecAgent18.00%17.00%46.00%
WebShop3.33%1.67%5.00%

6 . Ethicla implications

Thank you for highlighting the ethical implications of this work. We acknowledge that improving jailbreak and prompt-injection attacks against LLM agents introduces dual-use risks. We have expanded the Ethics Statement to explicitly discuss potential misuse and clarify our defensive research motivation: identifying agent vulnerabilities through controlled security evaluation and informing the development of stronger safeguards.

Thank you for your thoughtful feedback and the time you have devoted to reviewing our work. We have provided our responses to your comments and revised the manuscript accordingly. As the discussion deadline approaches, we would appreciate any follow-up feedback. If you have any remaining questions or concerns, we would be happy to address them and provide further clarification. Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

We thank the reviewers for their constructive feedback.

  1. A probabilistic connection between expected harmfulness and target likelihood. Theorem 2 establishes an exact local gradient identity. At a given input, the gradient of log expected harmfulness equals the expected log-likelihood gradient under the harmfulness-reweighted response distribution π\pi. The revised discussion after Eq. (8) explains what this identity means for target selection. A target should reflect both the harmful objective and what the model is likely to generate. Direct sampling from π\pi is difficult when harmful responses are rare, which motivates a practical surrogate.

  2. OPUR as a practical realization of the framework. Section 4 now explains how OPUR’s rollout-based surrogate accounts for both considerations. Section 5.2 specifies which agent actions count as success on each benchmark. Section 5.4 and Appendix C extend the rollout-budget study with component ablations and a temperature-sensitivity analysis. Probabilistic sampling achieved the highest average attack success rate (ASR) on AgentHarm and InjecAgent and tied greedy selection on WebShop. Probe-loss screening achieved the highest average ASR in its ablation on all three benchmarks. Appendix D adds runtime comparisons and quantifies how computational cost changes with the rollout budget.

In our point-by-point responses, OPUR achieved the highest average ASR among the methods evaluated on Qwen3-8B, reaching 88.64% on AgentHarm, 11.00% on InjecAgent, and 11.67% on WebShop. Across three random seeds on Llama and Ministral, OPUR had the highest mean ASR in all six model–benchmark pairs. In comparisons with attack-construction procedures adapted from Chen et al. (2025) and Li et al. (2026), OPUR led on AgentHarm and InjecAgent. The adapted Chen et al. procedure led on WebShop, with 30.00% ASR compared with 26.67% for OPUR.

The new Discussion and Limitations section states that the experiments cover ReAct agents built on open-weight models and explains the current implementation’s requirement for gradient access. The Ethics Statement addresses dual-use risks and explains how controlled benchmark evaluation can inform stronger safeguards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.