1. The comparison design does not establish the necessity or advantage of domain-specific GAP training.
GAP is trained separately for each evaluation domain through SFT and RL, with supervision from a privileged teacher that observes the user's legitimate intent, gold action, and adversarial context. In contrast, the frontier-model baselines in Table 1 receive the runtime policy and zero-shot instructions without comparable domain adaptation. Therefore, the result that GAP-trained Qwen3-14B outperforms larger models conflates the benefit of the proposed gating mechanism with that of domain-specific training.
Vanilla DAPO and Vanilla OPSD are useful objective-level controls within the GAP pipeline, but they do not establish that the overall training procedure is necessary. In particular, the +prompt condition uses detailed zero-shot instructions but no few-shot demonstrations or retrieval. Since the authors already possess successful training trajectories and request-specific gold plans, a stronger training-free baseline could retrieve relevant plan templates or demonstrations from the training split. The current experiments show that zero-shot instructions are insufficient, but not that in-context adaptation is insufficient or that per-domain SFT and RL are required.
The paper also discusses symbolic guardrails, policy engines, reviewing agents, and pre-write reflection in Section 2.2, but does not evaluate these alternatives. An external verifier or deterministic gate could check the required evidence and explicit confirmation immediately before a write action, potentially avoiding expensive per-domain training. Without comparisons to such alternatives, the practical necessity of learning a self-enforced gate inside the policy remains unclear.
2. The methodological novelty is limited, while transferability and residual failure modes are insufficiently examined.
GAP combines established components, including GRPO, OPSD, role-separated advantage estimation, explicit planning, and checklist-based verification, within a persistent verify-ask-confirm plan and request-state machine. This may be a useful system design, but the contribution can also be viewed as a domain-specific integration of existing techniques rather than a fundamentally new learning principle. Strong evidence of transferability and a detailed account of the resulting behavioral improvements are therefore important.
However, all GAP results use a single Qwen3-14B backbone, and a separate policy is trained for each service domain. The OOD evaluations vary the attack strategy and user simulator, but not the service domain, policy, or tool schema. The paper does not evaluate:
- joint multi-domain training,
- transfer to a held-out service domain,
- adaptation to a new policy or tool schema, or
- application to other model families and model scales.
It is therefore unclear whether every new domain requires a new round of data generation, SFT, and RL, or whether the learned behavior transfers across service settings.
The failure analysis is also limited despite the evaluation containing only 80 underlying test tasks. The paper reports aggregate attack-type metrics, LLM-judged plan scores, and three illustrative cases in which GAP succeeds, but these cases are not paired under identical adversarial interactions. A systematic analysis should distinguish:
- prompt failure and GAP success,
- failure by both methods, and
- prompt success and GAP failure.
The remaining attack successes and legitimate-task failures should also be categorized into planning omissions, verification errors, confirmation mistakes, cross-request state confusion, over-refusal, and execution failures. Such an analysis is important for determining what the learned gate actually fixes, what regressions it introduces, and why it still fails after domain-specific training.
Reviewer bJdi’s Response to the Rebuttal
Thank you for the additional experiments and analysis. I still have substantial concerns about the reproducibility of the evaluation and the novelty of the method.
On reproducibility, the Qwen3-8B results are useful, but the paper does not explain why the smaller model achieves a lower ASR in some settings. Both the 8B and 14B models are from the Qwen family, so it remains unclear whether GAP would provide similar gains on another model family. I would also find results for GAP across different model sizes and families more informative than the Qwen3.5 35B and 122B baseline results. Reducing that baseline table to three or four of the strongest closed models and using the space to examine GAP’s generality would address an important concern.
The new few-shot results also change how I read the main comparison. GAP’s TSR and ASR differences from DeepSeek-V4-Pro + few-shot are not statistically significant (Table 10). Its TSR advantage is concentrated in Hotel, a domain constructed by the authors with only ten test tasks. The paper does not examine why the difference is so much larger there, making it difficult to judge how much weight to place on the overall result. Appendix U says the full few-shot prompts are in the supplementary material, but I could not find the demonstrations in the
supplementary/promptsfiles available to me. Table 18 also does not show which tasks are handled differently by GAP and the new few-shot baseline. The experiments show that the full GAP pipeline works, but they do not establish why training an internal gate is preferable to request-specific retrieval or an external gate applied under comparable training conditions.On novelty, I remain unsure what methodological contribution goes beyond combining existing approaches to planning, distillation, and role-specific learning. My original question about the need for domain-specific training and learned self-gating has therefore not been resolved. The few-shot baseline uses fixed, manually written examples; it does not retrieve relevant plans or successful trajectories from the training data.
A smaller model with strong performance could be a practical benefit. However, the throughput experiment in Appendix H fixes the input and output lengths and excludes the cost of generating and maintaining GAP’s plans. Table 19 shows that GAP produces substantially more output tokens and takes much longer per dialogue than base Qwen, while providing no end-to-end cost comparison with DeepSeek. The current results therefore do not establish that GAP offers comparable performance more efficiently when run locally.
As a minor presentation issue, I cannot tell what Table 15 is intended to show. It appears that something went wrong with the LaTeX table formatting.
For these reasons, I am keeping my rating. I appreciate the additions. However, I do not think they resolve the central concerns. The advantage over the strongest few-shot baseline is not statistically established, much of the observed TSR difference comes from the small Hotel test set, and I cannot verify the construction of the few-shot prompts from the files available to me. It also remains unclear whether domain-specific training and an internal gate are needed relative to the alternatives above, or whether the method transfers beyond the Qwen family.