- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HarmonyGuard combines immediate action correction with persistent, policy-indexed violation examples. A Utility Agent checks adjacent steps and suggests revisions, while a Policy Agent maintains structured policies and bounded memory for later executions.
Critical strengths
- Reusing violation evidence for both correction and future judgments is a useful idea.
- Separating fixed policies from bounded example memory makes the design easy to understand.
- The paper reports promising completion-under-policy results, with backbone, memory-budget, and runtime analyses.
Critical weaknesses
- The evidence for cross-task generalization is limited. The multi-round experiment repeats the same 84 WASP tasks in a fixed order. Held-out tasks, shuffled orders, and a memory-reset control would better establish the benefit of retained cases.
- Memory entries come from the guard’s own judgments, without clear independent validation. Incorrect judgments or adversarial content could therefore influence later decisions even when the policy itself remains unchanged.
- The paper would benefit from discussing Wen et al., Learning Efficient Guardrails for Compliance, which introduces PolicyGuard-4B and PolicyGuardBench. It is directly relevant to the policy-compliance detection component. A comparison on compatible settings would help clarify HarmonyGuard’s benefits in accuracy and efficiency. I would consider raising my score if the additional experiments provide convincing evidence addressing these concerns.
Questions for authors
- Do memory gains persist on unseen tasks and with shuffled execution orders?
- How are incorrect or adversarially influenced boundary cases prevented from entering memory?
- What happens if an action remains unsafe after all three assessments?
7 comments
W2: Memory entries come from the guard’s own judgments, without clear independent validation. Incorrect judgments or adversarial content could therefore influence later decisions even when the policy itself remains unchanged.
Q2: How are incorrect or adversarially influenced boundary cases prevented from entering memory?
R2: We clarify HarmonyGuard's safeguards for Boundary-Case Memory through four aspects: (i) Policy-grounded Case Selection, which constrains which cases enter memory; (ii) Policy-constrained Case Usage, which keeps retained cases subordinate to trusted policies; (iii) Bounded Memory Management, which limits redundant and persistent cases; and (iv) Independent Human Audit, which assesses the reliability of retained cases.
Policy-grounded Case Selection. HarmonyGuard separates the trusted Policy Core from dynamically accumulated Boundary Cases. The Policy Core is constructed from trusted policy documents and remains unchanged during execution. The Utility Agent evaluates each adjacent-step transition against the task objective and the trusted Policy Core. A Boundary Case is constructed only when a policy violation is detected and corresponding Policy IDs are returned. Thus, environmental content is evaluated against trusted policies rather than directly accepted as a policy or an independent memory entry.
Policy-constrained Case Usage. Retained Boundary Cases serve as contextual examples for applying their corresponding policies, rather than independent sources of safety rules. They provide concrete historical evidence for subsequent policy judgments but do not override or modify the trusted Policy Core. This separation prevents an erroneous historical judgment from directly changing the rules governing subsequent decisions.
Bounded Memory Management. HarmonyGuard employs near-duplicate filtering with a similarity threshold of 0.85 and maintains independent FIFO queues for each policy, with capacities of 5, 7, and 10 for low-, medium-, and high-risk policies, respectively. These mechanisms limit redundant storage and keep historical experience bounded. Consequently, potentially erroneous cases cannot accumulate indefinitely or remain permanently in memory; their influence is limited by the capacity and retention period of the corresponding policy-specific queue.
Independent Human Audit. To assess the reliability of retained cases, we independently audited 21 Boundary Cases retained after GPT-4o completed 235 tasks on ST-WebAgentBench. The audit confirmed 16 cases (76.2%) as genuine policy violations, and all 16 had correct Policy ID assignments. The remaining five cases raised potential safety concerns but could not be confirmed as violations of specific policies. We report these cases separately rather than treating them as verified violations.
Together, policy-grounded selection, policy-constrained usage, and bounded retention limit how historical cases can influence subsequent decisions. We have added the audit results and clarified these mechanisms in the revised manuscript.
W3: The paper would benefit from discussing Wen et al., Learning Efficient Guardrails for Compliance, which introduces PolicyGuard-4B and PolicyGuardBench. It is directly relevant to the policy-compliance detection component. A comparison on compatible settings would help clarify HarmonyGuard’s benefits in accuracy and efficiency. I would consider raising my score if the additional experiments provide convincing evidence addressing these concerns.
R3: We have added Wen et al.'s work to Related Work and conducted comparisons with PolicyGuard-4B at two levels: (i) Offline Policy Detection: We evaluate HarmonyGuard's detection component on PolicyGuardBench; (ii) Online Agent Execution: We compare PolicyGuard-4B, HarmonyGuard, and their combination on ST-WebAgentBench to examine policy compliance and task completion.
(1) Offline Policy Detection.
We evaluate on the 12,000-sample PolicyGuardBench test set. The results for PolicyGuard-4B and the original Qwen3-4B-Instruct are taken from the original paper, while Qwen3-4B-Instruct-2507 with HarmonyGuard's detection component is evaluated locally. Therefore, the results provide a reference comparison rather than a strictly controlled reproduction under identical implementations.
| Method | Data Source | Accuracy (%) ↑ | F1 (%) ↑ |
|---|---|---|---|
| Qwen3-4B-Instruct | Original paper | 68.97 | 53.48 |
| PolicyGuard-4B | Original paper | 90.14 | 87.59 |
| Qwen3-4B-Instruct-2507 + HarmonyGuard Detection | Our evaluation | 90.08 | 87.74 |
Qwen3-4B-Instruct-2507 with HarmonyGuard's detection component achieves 90.08% accuracy and 87.74% F1, closely matching the reported PolicyGuard-4B results of 90.14% and 87.59%. The differences are only 0.06 percentage points in accuracy and 0.15 percentage points in F1. These results show that HarmonyGuard's detection component achieves competitive policy-compliance detection performance with a small backbone.
(2) Online End-to-End Comparison.
We further evaluate all three configurations on the 235 ST-WebAgentBench tasks. All configurations use GPT-5.6-luna as the execution agent and GPT-6.1-terra as the outcome evaluator. Each configuration is evaluated once (run0). PCR, SR, and CuP are macro-averaged across policy categories.
| Method | PCR (%) ↑ | SR (%) ↑ | CuP (%) ↑ |
|---|---|---|---|
| PolicyGuard-4B | 96.37 | 9.16 | 9.16 |
| Qwen3-4B-Instruct-2507 + HarmonyGuard | 98.40 | 10.38 | 10.38 |
| PolicyGuard-4B + HarmonyGuard | 99.01 | 11.52 | 11.52 |
Compared with PolicyGuard-4B alone, Qwen3-4B-Instruct-2507 with HarmonyGuard improves PCR from 96.37% to 98.40% and both SR and CuP from 9.16% to 10.38%.
Combining PolicyGuard-4B with HarmonyGuard achieves the highest results across all three metrics, reaching 99.01% PCR and 11.52% for both SR and CuP. Compared with PolicyGuard-4B alone, this represents improvements of 2.64 and 2.36 percentage points in PCR and SR, respectively.
These results show that HarmonyGuard improves policy compliance when integrated with different detection backbones. The smaller Qwen3-4B-Instruct-2507 backbone achieves better results across all three metrics than PolicyGuard-4B alone, while combining both guardrails yields the strongest overall policy-compliance performance.
We have incorporated these additional comparisons and related discussions into the revised manuscript, and hope that these revisions adequately address the reviewer's concerns.
Dear Reviewer xJCa,
Thank you for your constructive feedback. We have carefully addressed your comments and extended the revised manuscript and experiments in four aspects : (1) Cross-task Generalization: We introduce a disjoint construction–evaluation split, together with shuffled-order and memory-reset controls, to examine whether retained Boundary Cases transfer to unseen tasks. (2) Memory Reliability: We clarify the policy-grounded case selection and retention mechanisms and provide an independent human audit of retained cases. (3) Comparison with PolicyGuard: We add offline detection and online task-execution comparisons with PolicyGuard-4B and its underlying Qwen backbone. (4) Correction Budget: We clarify the bounded correction mechanism and the handling of unresolved candidates. The corresponding results and revisions are detailed below. We hope these additional experiments and clarifications adequately address your concerns.
R1: To evaluate whether retained Boundary Cases generalize to unseen tasks, we conducted an additional experiment using separate memory-construction and held-out evaluation task sets, with shuffled-order and memory-reset comparisons.
We stratify the 84 WASP tasks across GPI, GUI, RPI, and RUI, assigning 42 tasks to memory construction and 42 separate tasks to held-out evaluation. We compare three settings:
After each construction round, all three settings are evaluated on the same 42 held-out tasks in the same order. No new Boundary Cases are stored during held-out evaluation.
The held-out results are reported below.
The results show that the benefits of retained cases persist on the held-out tasks under both task-order settings. By Round 3, Same-Order Memory achieves 92.9% PCR and 78.6% CuP, compared with 88.1% PCR and 73.8% CuP under No Boundary Memory. Shuffled-Order Memory also reaches 92.9% PCR and 76.2% CuP, exceeding the memory-reset baseline by 4.8 and 2.4 percentage points, respectively.
The gains under shuffled execution orders indicate that memory benefits are not restricted to a fixed task sequence. The comparison with No Boundary Memory further isolates the contribution of retaining cases across construction rounds. We have added this held-out evaluation and the corresponding controlled comparisons to the revised manuscript.