- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes HAC, a two-stage framework for task-oriented dialogue policy optimization. It first contrasts an original LLM with a model fine-tuned on non-factual data to construct a corrected action prior and a compact candidate set. Reinforcement learning then selects actions using a categorical Q-network trained with Gaussian soft targets and cross-entropy loss. Experiments cover three dialogue domains, human evaluation, factuality benchmarks, and component ablations. The reported results show improved task success and factuality over the evaluated baselines.
Critical strengths
Relevant problem and reasonable architecture. Separating candidate generation from long-term action evaluation is a sensible approach to constrained, multi-turn dialogue.
Consistent improvements within the evaluated setting. HAC improves final success rates over LLMSAP by 8.5, 7.0, and 9.2 percentage points across the three domains. Human evaluation and factuality results provide additional evidence of potential utility
Critical weaknesses
Insufficiently strong baselines leave the central motivation unvalidated. The core dialogue evaluation relies on older backbones and insufficiently specified direct-generation baselines, without competitive contemporary models such as Qwen3.5. Consequently, the authors do not establish whether unreliable action priors remain a significant bottleneck with stronger models, or whether simply upgrading the backbone achieves comparable benefits. Table 3 includes stronger references only on general factuality benchmarks, where every HAC-enhanced model remains below DeepSeek-Chat on the reported factuality metrics. Without competitive task-level comparisons and cost measurements, the practical justification for additional fine-tuning and dual-model decoding remains unclear.
The contribution of hallucination correction is not isolated. The ablation combines LLM guidance, action compression, and correction, preventing attribution of gains specifically to correction. No direct action-violation or valid-action coverage metrics are reported. Moreover, the attention visualizations do not establish causality, and the “corrected” examples in Figure 3 still contain constraint violations. The proposed explanation linking attention drift, action reliability, and policy improvement therefore lacks sufficient evidence.
Questions for authors
What existing evidence shows that action hallucinations remain a substantial bottleneck with contemporary strong models, under comparable task information and inference budgets? Are controlled results available that change only the prior correction while fixing the backbone, candidate count, projection procedure, and RL learner? How do action-violation rates and valid-action coverage change?
3 comments
Questions For Authors:
Q1. What existing evidence shows that action hallucinations remain a substantial bottleneck with contemporary strong models, under comparable task information and inference budgets? Are controlled results available that change only the prior correction while fixing the backbone, candidate count, projection procedure, and RL learner? How do action-violation rates and valid-action coverage change?
(a)The experiments already in the manuscript provide the following evidence. Table 3 shows that the original LLM prior scores poorly on TruthfulQA and FactScore across four backbones, establishing that unreliable action priors are a real phenomenon; the same table shows that HAC improves factuality within each backbone, indicating that the correction reduces unreliable predictions. Table 1 shows that under the corrected prior, success rate improves and dialogue turns decrease across Movie, Restaurant and Taxi, suggesting that a more reliable prior benefits downstream policy learning. App. B (Fig. 4) shows that scaling from 3B to 7B helps but 14B and 32B yield only marginal gains, indicating that the bottleneck is not merely removed by increasing model size.
(b)We acknowledge that these are indirect measurements and that we did not report a direct quantitative analysis of action reliability. We plan to add a controlled experiment in which only the action prior is varied — the original versus the corrected — while the backbone, candidate count , projection procedure, and distributional Q-learner are all fixed. On the three TOD domains we will measure (i) the action violation rate, i.e. how often generated actions violate the user's stated constraints, and (ii) valid action coverage, i.e. whether the ground-truth action is retained in . Comparing the two priors under identical settings will isolate the correction's effect on action reliability.
Thank you for this rebuttal, I will keep my score.
We sincerely thank you for all your valuable suggestions and efforts toward improving our manuscript. We will address your concerns point by point:
Critical Weaknesses:
Q1. Insufficiently strong baselines leave the central motivation unvalidated.
A1. (a)We would like to clarify that our baseline set already spans the three established lines of TOD policy optimization: RL-centric exploration (ε-DQN, ICM DQN, Noisy DQN), direct LLM generation (LLM DA, LLM Word), and LLM-guided RL (LLM PRIOR CQL, LLMSAP). Table 1 shows that HAC outperforms all seven baselines in success rate across Movie / Restaurant / Taxi, so the empirical claim is supported. To strengthen this, we will additionally include a stronger contemporary backbone as both a direct-generation baseline and a HAC backbone.
(b) We would like to clarify that the gap is expected and does not reflect a failure of the correction. DeepSeek-Chat is a commercially aligned model with large-scale instruction and preference training, whereas our backbones (Qwen2.5-7B, Llama2-7B, Mistral-7B, Gemma2-2B) are not aligned to that scale. We report DeepSeek-Chat as a reference point for the current frontier, not as a model HAC must beat from a base checkpoint. The informative comparison is therefore within the same backbone, where HAC improves all four on MC2 and FactScore. To examine whether the bottleneck persists at that scale, we will also report a before-and-after-HAC comparison on a stronger backbone.
Q2. The contribution of hallucination correction is not isolated. The (H) ablation combines LLM guidance, action compression, and correction, preventing attribution of gains specifically to correction. No direct action-violation or valid-action coverage metrics are reported. Moreover, the attention visualizations do not establish causality, and the “corrected” examples in Figure 3 still contain constraint violations. The proposed explanation linking attention drift, action reliability, and policy improvement therefore lacks sufficient evidence.
A2. (a) Regarding the ablation study, we acknowledge that combining hallucination correction and action-space compression into a single component does not sufficiently distinguish their individual contributions. We will separate these two components and conduct additional controlled experiments under identical action-space compression and RL settings, comparing the original and corrected LLM priors to further isolate the contribution of hallucination correction.
(b) Regarding action reliability metrics, we acknowledge that action violation rate and valid action coverage are not reported in the current manuscript. These metrics directly quantify whether the corrected prior produces constraint-consistent actions, and we will add them in the revision, evaluated on the same three TOD domains, providing a more direct measure of action reliability than success rate alone.
(c) Regarding the residual constraint violations in Fig. 3, we clarify that this is expected and consistent with the design of HAC. The correction is not intended to eliminate LLM hallucinations entirely — which is neither achievable nor its objective — but to reduce the probability of unreliable actions so that the compressed set Ck(st) becomes a more reliable search space for the downstream policy. The final action is still selected by the distributional Q-network within the specific dialogue context, so the prior acts as a filter over the action space rather than a final guarantee. Fig. 1 and Fig. 3 are descriptive evidence of this probability shift; we will revise the captions to state this clearly and avoid implying a causal claim beyond what the evidence supports.