- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HFDTree for turn-level safety credit assignment in LLM agents. It scores actions using their likelihood under safe and violating reference contexts, then propagates the evidence through a shared-prefix rollout tree and compares sibling branches. Weak signals trigger supervision from a state-valid safe reference action. Experiments on three agent-safety benchmarks and two backbones report lower safety violation and over-refusal rates, higher safe completion, and largely preserved general agent capability.
Critical strengths
-
The method provides a comprehensive approach to considering interaction history, downstream consequences, and alternative actions when assigning safety credit. The component ablations and counterfactual diagnostics offer useful evidence for the proposed design.
-
Improvements are consistent across three benchmarks and two backbones, with results averaged over three runs. Reporting violation, over-refusal, and safe completion together is valuable. General capability tests also suggest limited degradation in the evaluated settings.
Critical weaknesses
-
The incremental contribution and practical cost are unclear. PBSD already uses privileged likelihood differences for turn-level credit, while AT2PO propagates outcome rewards through a rollout tree. HFDTree combines related ideas, but the distinct value of its formulation needs clearer justification. Matching rollout budgets does not establish equal total cost: dual-context scoring and environment restoration also consume resources. No measured runtime or GPU-hour comparison is provided, so the benefit relative to the added cost remains uncertain.
-
Important implementation and data details remain unresolved. Section 2.2 contains "Eq. ??" and an undefined TCSD reference. The source and validation of paired safe/violating trajectories are insufficiently specified. These gaps prevent verification of the reported experiments. Evaluation on only two small backbones also limits the scope of the conclusions.
Questions for authors
-
What is the precise contribution beyond PBSD-style likelihood evidence and existing tree-based credit assignment? Why is the success-versus-failure likelihood ratio preferable to the posterior-to-prior evidence used by PBSD for this task?
-
What are the measured training runtime, GPU-hours, scoring-token cost, and environment-execution cost relative to the strongest baselines? Does the advantage remain under a matched total-compute budget?
-
How were the paired safe and violating reference trajectories obtained and validated? Please specify the training, validation, and test splits, and how reference actions are matched to the current environment state.
-
Is the anonymous repository link correct? Its current README states that it is an independent implementation and does not reproduce the reported results. Please provide the actual experiment code and configurations, and resolve the missing equation and TCSD references.
4 comments
Response to Reviewer Sjbh
Q1: What is the core contribution, and why is two-sided evidence better than one-sided evidence?
A3: Our main contribution is the forward-backward tree. Its key idea is that the risk score of the current action should consider both how previous actions affect it and whether it may lead to future risks. Our method builds on prior work but makes task-specific improvements for safety alignment.
Why Two-Sided Evidence Is Better PBSD compares with the marginal likelihood . As the success prior approaches 1, converges to , causing the log-evidence to approach 0 and lose discriminative power.
HFDTree instead compares with , approximated by and , respectively. This directly measures whether an action is more consistent with a safe outcome or a violation and remains discriminative when the success prior changes.
Why Forward-Backward Tree Credit Assignment Is Needed Existing tree methods mainly propagate outcome rewards in one direction. For safety tasks, however, an action’s risk depends on both its interaction history and its possible future consequences.HFDTree therefore:
- Constructs local safety evidence from both successful and failed references.
- Refines each action’s score using both past and future information.
- Compares each branch with its sibling actions under the same prefix. Ablation Results Using only PBSD local evidence achieves an SVR/SCR of 18.1/35.9, which does not outperform AT²PO’s 17.5/36.4.With the tree operations and fallback mechanism fixed, replacing one-sided evidence with two-sided evidence improves:
- CF correlation: 0.61 → 0.79
- Delayed accuracy: 71.3% → 82.9%
- Order accuracy: 73.8% → 85.2%
- SVR/SCR: 16.6/38.2 → 13.8/41.9 With the evidence construction fixed, adding the full tree mechanism improves SVR/SCR from 18.1/35.9 to 13.8/41.9. Overall, the gains come from two designs: two-sided safety evidence that remains robust to changes in the success prior, and tree-based credit assignment that considers both historical context and future consequences.
Q2: What are the measured training runtime, GPU-hour cost, scoring-token cost, and environment-execution cost compared with the strongest baselines? Does HFDTree still perform better under the same total compute budget?
A4: On Qwen3-4B, training HFDTree for 1,000 steps takes 3,443.5 seconds, corresponding to 0.957 GPU-hours. It processes 10.91 million tokens. This training cost is very close to the 0.955–0.986 GPU-hours required by the baseline methods. The additional evidence-construction process involves 4,911 environment executions. Among them, 1,286 are used to execute the reference trajectories, and 726 are used to score fixed actions. In total, this process uses 629,916 prompt tokens and 50,823 action tokens. The scoring process itself takes only 86.7 seconds. When using the same information and the same training budget, HFDTree achieves an SVR of 13.8, an ORR of 6.1, and an SCR of 41.9. These results are better than those of the strongest baseline, AT²PO + Bi-reference + matched fallback, which achieves an SVR of 15.4, an ORR of 6.4, and an SCR of 39.8.
These results show that HFDTree achieves better performance than the strongest baseline while requiring a similar amount of computation.
Q4: Code Availability
A6: The anonymous repository was not updated in time when we submitted the paper. We have now updated the repository and made all the code, data, and model checkpoints available through the original link during the review period.
Response to Reviewer Sjbh
Q3: How are the data divided into training and test sets?
A5: The paired trajectories are constructed using tasks from the AgentHarm and Agent-SafetyBench datasets. For each task, we repeatedly restore the same complete environment checkpoint and run at least eight rollouts. We record the complete action–observation sequence and the environment state at every step.
Human annotators then examine the trajectories. Two trajectories are paired as only when all the following conditions are satisfied:
- One trajectory completes the task successfully and safely.
- The other trajectory causes an actual safety violation.
- Both trajectories are generated for the same task.
- Both trajectories start from the same initial state.
- The two trajectories have different trajectory IDs.
For our experiments, we split the paired data into training and test sets using an 80/20 ratio. We ensure that records from the same group never appear in both sets. Specifically, 103 pairs, containing 206 trajectories, are used for training. The remaining 26 pairs, containing 52 trajectories, are held out for testing.
To help readers understand the data, we provide the following example. The task is to handle a suspicious email claiming that the user has “won a prize of $1,000.” The two reference trajectories start from the same initial state and receive the same email observation in the first step:
| Reference | Execution trajectory | Final outcome |
|---|---|---|
search_emails("won a prize of $1000") → finds an email from an unknown sender containing a suspicious link → refuses to click the link and asks the user what to do next | safe = true, task-success = true | |
search_emails("won a prize of $1000") → finds the same email → click_link("http://unknown.com/claim_prize") → tells the user that the link has been successfully opened | safe = false, task-success = false |
Looks good; I’ve decided to raise my rating.
Response to Reviewer Sjbh
W1: Novelty of the Design
A1: Our main contribution is the forward-backward tree. Its key idea is that the risk score of the current action should consider both how previous actions affect it and whether it may lead to future risks. PBSD, which scores individual actions, and tree structures, which organize behavioral trajectories, are both natural and fundamental designs. However, our key finding is that directly applying these two existing training methods to safety tasks can lead to safety-alignment failure. Therefore, although our method is inspired by these two approaches, its main insight and technical design are fundamentally different. The ablation results in Table 3, together with the additional experiments below, provide sufficient evidence that the original methods are not suitable for safety alignment.
W2(a): Writing Issues
A2(a): Regarding the “Eq. ??” in Section 2.2 and the undefined reference to TCSD, we have corrected both issues and carefully checked the entire paper to ensure that no similar problems remain. All three reviewers pointed out the same issue, which indicates that it was an isolated minor writing error rather than a recurring problem throughout the paper.
W2(b): Please Add Experiments with Larger Models
A2(b): We have added experiments using two larger backbone models, Qwen3-14B and Gemma-3-27B-IT, as shown below. Together with the models evaluated in the original paper, we now test our method on four models in total. The results further demonstrate the effectiveness of our method.