acceptodds
Under review as a conference paper at ICLR 2027

Hallucination-Corrected Action Compression via LLM-RL Coordination for Task-Oriented Dialogue policy optimization

Abstract

Task-oriented dialogue policy optimization remains challenging due to the large and highly structured action space. Large language models (LLMs) can provide state-conditioned action priors to compress this space, but their hallucinations may introduce task-irrelevant, constraint-violating, or infeasible actions. Such unreliable actions not only undermine action-space compression but also bias downstream action-value estimation and policy learning. To address this issue, we propose Hallucination-corrected Action Compression (HAC), a two-stage framework that corrects the LLM prior before using it to guide dialogue policy optimization. Stage I performs hallucination-induced correction on the original LLM prior. Specifically, an offset LLM is induced to amplify hallucination tendencies, and its predictions are contrasted with those of the original LLM to suppress unreliable preferences. This yields a factually corrected action prior, from which a compressed action set is constructed—producing a more trustworthy action space than directly using the original LLM. Stage II then performs stable RL-based policy optimization solely over this compressed set. A distributional Q-network estimates return distributions for each action, and a categorical cross-entropy objective aligns predicted and target distributions, enabling stable and discriminative learning within a reduced and reliable action space. Experiments on multiple task-oriented dialogue and factuality benchmarks show that HAC achieves superior task performance, improved factual reliability, and strong generalization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reinforcement Learning, Dialogue Policy, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dKK9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HAC, a two-stage framework for task-oriented dialogue policy optimization. It first contrasts an original LLM with a model fine-tuned on non-factual data to construct a corrected action prior and a compact candidate set. Reinforcement learning then selects actions using a categorical Q-network trained with Gaussian soft targets and cross-entropy loss. Experiments cover three dialogue domains, human evaluation, factuality benchmarks, and component ablations. The reported results show improved task success and factuality over the evaluated baselines.

Critical strengths

Relevant problem and reasonable architecture. Separating candidate generation from long-term action evaluation is a sensible approach to constrained, multi-turn dialogue.

Consistent improvements within the evaluated setting. HAC improves final success rates over LLMSAP by 8.5, 7.0, and 9.2 percentage points across the three domains. Human evaluation and factuality results provide additional evidence of potential utility

Critical weaknesses

Insufficiently strong baselines leave the central motivation unvalidated. The core dialogue evaluation relies on older backbones and insufficiently specified direct-generation baselines, without competitive contemporary models such as Qwen3.5. Consequently, the authors do not establish whether unreliable action priors remain a significant bottleneck with stronger models, or whether simply upgrading the backbone achieves comparable benefits. Table 3 includes stronger references only on general factuality benchmarks, where every HAC-enhanced model remains below DeepSeek-Chat on the reported factuality metrics. Without competitive task-level comparisons and cost measurements, the practical justification for additional fine-tuning and dual-model decoding remains unclear.

The contribution of hallucination correction is not isolated. The HH ablation combines LLM guidance, action compression, and correction, preventing attribution of gains specifically to correction. No direct action-violation or valid-action coverage metrics are reported. Moreover, the attention visualizations do not establish causality, and the “corrected” examples in Figure 3 still contain constraint violations. The proposed explanation linking attention drift, action reliability, and policy improvement therefore lacks sufficient evidence.

Questions for authors

What existing evidence shows that action hallucinations remain a substantial bottleneck with contemporary strong models, under comparable task information and inference budgets? Are controlled results available that change only the prior correction while fixing the backbone, candidate count, projection procedure, and RL learner? How do action-violation rates and valid-action coverage change?

3 comments

We sincerely thank you for all your valuable suggestions and efforts toward improving our manuscript. We will address your concerns point by point:

Critical Weaknesses:

Q1. Insufficiently strong baselines leave the central motivation unvalidated.

A1. (a)We would like to clarify that our baseline set already spans the three established lines of TOD policy optimization: RL-centric exploration (ε-DQN, ICM DQN, Noisy DQN), direct LLM generation (LLM DA, LLM Word), and LLM-guided RL (LLM PRIOR CQL, LLMSAP). Table 1 shows that HAC outperforms all seven baselines in success rate across Movie / Restaurant / Taxi, so the empirical claim is supported. To strengthen this, we will additionally include a stronger contemporary backbone as both a direct-generation baseline and a HAC backbone.

(b) We would like to clarify that the gap is expected and does not reflect a failure of the correction. DeepSeek-Chat is a commercially aligned model with large-scale instruction and preference training, whereas our backbones (Qwen2.5-7B, Llama2-7B, Mistral-7B, Gemma2-2B) are not aligned to that scale. We report DeepSeek-Chat as a reference point for the current frontier, not as a model HAC must beat from a base checkpoint. The informative comparison is therefore within the same backbone, where HAC improves all four on MC2 and FactScore. To examine whether the bottleneck persists at that scale, we will also report a before-and-after-HAC comparison on a stronger backbone.


Q2. The contribution of hallucination correction is not isolated. The (H) ablation combines LLM guidance, action compression, and correction, preventing attribution of gains specifically to correction. No direct action-violation or valid-action coverage metrics are reported. Moreover, the attention visualizations do not establish causality, and the “corrected” examples in Figure 3 still contain constraint violations. The proposed explanation linking attention drift, action reliability, and policy improvement therefore lacks sufficient evidence.

A2. (a) Regarding the ablation study, we acknowledge that combining hallucination correction and action-space compression into a single component does not sufficiently distinguish their individual contributions. We will separate these two components and conduct additional controlled experiments under identical action-space compression and RL settings, comparing the original and corrected LLM priors to further isolate the contribution of hallucination correction.

(b) Regarding action reliability metrics, we acknowledge that action violation rate and valid action coverage are not reported in the current manuscript. These metrics directly quantify whether the corrected prior produces constraint-consistent actions, and we will add them in the revision, evaluated on the same three TOD domains, providing a more direct measure of action reliability than success rate alone.

(c) Regarding the residual constraint violations in Fig. 3, we clarify that this is expected and consistent with the design of HAC. The correction is not intended to eliminate LLM hallucinations entirely — which is neither achievable nor its objective — but to reduce the probability of unreliable actions so that the compressed set Ck(st)C_k(s_t) becomes a more reliable search space for the downstream policy. The final action is still selected by the distributional Q-network within the specific dialogue context, so the prior acts as a filter over the action space rather than a final guarantee. Fig. 1 and Fig. 3 are descriptive evidence of this probability shift; we will revise the captions to state this clearly and avoid implying a causal claim beyond what the evidence supports.

Questions For Authors:

Q1. What existing evidence shows that action hallucinations remain a substantial bottleneck with contemporary strong models, under comparable task information and inference budgets? Are controlled results available that change only the prior correction while fixing the backbone, candidate count, projection procedure, and RL learner? How do action-violation rates and valid-action coverage change?

(a)The experiments already in the manuscript provide the following evidence. Table 3 shows that the original LLM prior scores poorly on TruthfulQA and FactScore across four backbones, establishing that unreliable action priors are a real phenomenon; the same table shows that HAC improves factuality within each backbone, indicating that the correction reduces unreliable predictions. Table 1 shows that under the corrected prior, success rate improves and dialogue turns decrease across Movie, Restaurant and Taxi, suggesting that a more reliable prior benefits downstream policy learning. App. B (Fig. 4) shows that scaling from 3B to 7B helps but 14B and 32B yield only marginal gains, indicating that the bottleneck is not merely removed by increasing model size.

(b)We acknowledge that these are indirect measurements and that we did not report a direct quantitative analysis of action reliability. We plan to add a controlled experiment in which only the action prior is varied — the original pϕp_\phi versus the corrected pcorrp_{\text{corr}} — while the backbone, candidate count kk, projection procedure, and distributional Q-learner are all fixed. On the three TOD domains we will measure (i) the action violation rate, i.e. how often generated actions violate the user's stated constraints, and (ii) valid action coverage, i.e. whether the ground-truth action is retained in Ck(st)C_k(s_t). Comparing the two priors under identical settings will isolate the correction's effect on action reliability.

Thank you for this rebuttal, I will keep my score.

Reviewer_fDmSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HAC, a task-oriented dialogue policy optimization framework that integrates LLM hallucination correction with reinforcement learning. The approach aims to construct a more reliable candidate action space by correcting LLM-generated action priors, while leveraging reinforcement learning to optimize long-term decision-making. The research offers an interesting perspective, particularly on improving decision reliability and exploration efficiency in LLM-guided policy optimization.

Critical strengths
  1. The paper addresses an important and practically relevant problem.The unreliability of LLM-generated action priors can negatively affect exploration and policy optimization in task-oriented dialogue systems. Improving action reliability while reducing the exploration burden is a meaningful research direction.

  2. The proposed framework is well-structured and intuitive. HAC separates hallucination-corrected action-space compression from reinforcement learning-based policy optimization. This two-stage design provides a clear division between candidate action generation and long-term action-value estimation.

  3. The experimental results are encouraging. HAC achieves higher final task success rates than seven baselines across three task-oriented dialogue domains. The paper also includes human evaluation, factuality benchmarks, cross-model analysis, and ablation studies, providing empirical evidence for the effectiveness of the overall framework.

Critical weaknesses

The hallucination-induced correction mechanism closely resembles the previously proposed Induce-then-Contrast Decoding (ICD) method (Zhang et al., NAACL Findings 2025). In particular, the offset-model training objective and the contrastive correction equations appear mathematically identical to those in ICD. However, the paper does not clearly acknowledge or distinguish this prior work. While integrating hallucination-corrected priors with action-space compression and distributional reinforcement learning may constitute a useful contribution, the methodological novelty of the hallucination-correction component itself remains unclear.

1.The paper does not sufficiently establish that fine-tuning on the non-factual dataset produces an offset model that specifically amplifies hallucination-prone preferences, rather than introducing unrelated distributional shifts. This assumption is central to the proposed correction mechanism but lacks direct empirical validation.

2.The ablation study combines hallucination correction and action-space compression into a single module, without independently evaluating their respective contributions to performance improvements. The authors are encouraged to conduct separate ablation experiments to clarify whether the observed improvements arise from hallucination correction, action-space compression, or their combined effects.

3.The proposed framework appears to be primarily an integration of existing techniques, including hallucination-aware contrastive decoding, LLM-guided action-space compression, and distributional Q-learning. While the empirical improvements are encouraging, the paper does not sufficiently demonstrate a distinct methodological innovation beyond combining these established components. Further clarification of the technical novelty and stronger comparisons with existing alternatives would help establish the contribution of the proposed framework.

Questions for authors
  1. What exactly is the non-factual dataset (D_h) used in this paper? What are its data sources, construction procedures, and training samples?

  2. The ablation study combines hallucination correction and action-space compression into a single module. Could the authors clarify how much of the performance improvement is attributable specifically to hallucination correction, under identical action-space compression and RL settings?

2 comments

We sincerely thank you for all your valuable suggestions and efforts toward improving our manuscript. We will address your concerns point by point:

Critical Weaknesses:

Q1. The hallucination-induced correction mechanism closely resembles the previously proposed Induce-then-Contrast Decoding (ICD) method. The methodological novelty of the hallucination-correction component itself remains unclear.

A1. (a) We thank the reviewer for this careful reading, and we apologize for the omission. The manuscript states that we follow an "induce-then-contrast paradigm" (line 201) without attributing it; Eqs. (1)–(4) do correspond to the offset-model training objective and the contrastive decoding formulation of ICD (Zhang et al., 2025). We will cite ICD explicitly in Sec. 3.1.1 and discuss it in Related Work.

(b) We would also like to clarify the scope of our contribution. We do not claim novelty for the contrastive correction rule. What HAC contributes is (i) the placement of correction before compression — the corrected prior, rather than the original prior, is used to construct the executable candidate set — and (ii) policy optimization over that set with distributional Q-learning. ICD stops at the corrected distribution and evaluates free-text factuality; it contains no action-space projection, no candidate construction, and no policy learning. This difference is visible in our results: LLMSAP also performs compression and RL, but compresses the uncorrected prior, whereas HAC achieves higher success across all three domains (e.g., Taxi 0.802 vs. 0.710 at epoch 500). We will state these distinctions explicitly and temper the novelty claim accordingly.


Q2. The paper does not sufficiently establish that fine-tuning on the non-factual dataset produces an offset model that specifically amplifies hallucination-prone preferences, rather than introducing unrelated distributional shifts.

A2. We thank the reviewer and accept this criticism: the manuscript does not validate the offset LLM directly, and this is a genuine gap.

We will add the offset LLM (ϕ+Δϕ\phi+\Delta\phi) as an additional row to Table 3, reporting TruthfulQA (MC1–MC3), %Respond, and FactScore alongside the original LLM pϕp_\phi and HAC+ on the same four backbones. If fine-tuning on the non-factual dataset Dh\mathcal{D}_h amplifies hallucination tendencies, the offset LLM should score below the original LLM while keeping a comparable %Respond; otherwise, the contrast in Eq. (3) would not be suppressing hallucination-prone preferences as intended. Reporting this row therefore makes the assumption behind Eq. (1) directly checkable. We will also state how Dh\mathcal{D}_h is constructed in Appendix.


Q3. The ablation study combines hallucination correction and action-space compression into a single module.

A3. Regarding the ablation study, we acknowledge that combining hallucination correction and action-space compression into a single component does not sufficiently distinguish their individual contributions. We will separate these two components and conduct additional controlled experiments under identical action-space compression and RL settings, comparing the original and corrected LLM priors to further isolate the contribution of hallucination correction.


Q4. The proposed framework appears to be primarily an integration of existing techniques.

A4.TOD policy optimization faces a concrete problem: the action space is large and highly structured, and although the LLM prior can compress it, that prior is itself unreliable — its hallucinations introduce task-irrelevant, constraint-violating, or infeasible actions that then bias action-value estimation. HAC addresses this at its source. We trace the cause to the original prior's token-level preference: it assigns disproportionately high probability to repeated action templates and slot names, while constraint tokens and slot values receive less (Fig. 1, Fig. 3, Table 3). Therefore motivates correcting the action prior before compression. This one decision fixes the role of everything downstream: contrastive decoding yields a factually corrected prior pcorrp_{\text{corr}} that defines Ck(st)C_k(s_t), so unreliable actions never enter the candidate set and need not be removed later by RL; and over this corrected yet still stochastic multi-turn set, distributional Q-learning separates actions with similar means but different risk, which a scalar value cannot. Our contribution is thus a correction-before-compression formulation of the TOD action space. HAC attains higher success rates than LLMSAP on Movie / Restaurant / Taxi (Table 1), with further gains from each stage in Table 5.

Questions For Authors:

Q1. What exactly is the non-factual dataset (D_h) used in this paper? What are its data sources, construction procedures, and training samples?

A1. (D_h) is constructed from TOD data. For each dialogue in Movie, Restaurant and Taxi, we take the system prompt sis_i, the user turn uiu_i, and the action that correctly satisfies the request, and use ChatGPT with few-shot prompts to convert it into a non-factual action by altering the slot values that carry the user's constraints — time, location, and quantity — while preserving fluency and syntactic validity. For instance, a request for two tickets at 7pm on Saturday is paired with an action for three tickets at 9pm on Sunday; a table for four at 8pm on the north side becomes a table for two at 7pm on the south side; and a taxi from the airport to the university before 6pm becomes a taxi from the university to the airport at 6:30pm. The triples {(si,ui,oi)}i=1M\{(s_i,u_i,o_i)\}_{i=1}^{M} follow Eq. (1), with oio_i the rewritten output. In the revised appendix, we will present the original and rewritten samples side by side for all three domains.


Q2. The ablation study combines hallucination correction and action-space compression into a single module. Could the authors clarify how much of the performance improvement is attributable specifically to hallucination correction, under identical action-space compression and RL settings?

A2. Regarding the ablation study, we acknowledge that combining hallucination correction and action-space compression into a single component does not sufficiently distinguish their individual contributions. We will separate these two components and conduct additional controlled experiments under identical action-space compression and RL settings, comparing the original and corrected LLM priors to further isolate the contribution of hallucination correction.

Reviewer_HwiSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Hallucination-Corrected Action Compression (HAC), a two-stage framework for improving task-oriented dialogue policy optimization. The authors argue that hallucinations in LLM-generated dialogue actions arise from token-level attention drift, which causes models to overlook task-critical constraints. To address this issue, HAC first fine-tunes an offset LLM to amplify hallucination tendencies and applies contrastive decoding to obtain corrected action priors. These priors are then used to construct a compressed action space. In the second stage, distributional Q-learning is employed to optimize action selection within the compressed space. Experiments on three task-oriented dialogue benchmarks show improvements in task success rates and dialogue efficiency, while additional factuality and human evaluations further support the effectiveness of the framework.

Critical strengths

The paper addresses a relevant problem and achieves consistent improvements in task success rates across three dialogue domains, supported by additional human evaluations.

Critical weaknesses

1. Unclear problem formulation and unsupported causal claims. The paper broadly labels constraint-violating and infeasible actions as "hallucinations" without a precise definition, and further attributes these errors to "token-level attention drift" without sufficient empirical or causal evidence. Moreover, the proposed correction operates on output token probabilities rather than directly modifying attention weights. The paper does not explain how the "corrected attention" in Figure 3 is computed, or provide evidence that the proposed method actually mitigates attention drift.

2. Limited methodological novelty and insufficient differentiation from prior work. The hallucination-correction mechanism closely follows Induce-then-Contrast Decoding (ICD, NAACL Findings 2025), adopting essentially the same offset-model training objective and contrastive decoding formulation. Meanwhile, LLM-guided action compression has already been explored in LLMSAP (EMNLP Findings 2025). The paper does not adequately differentiate its contributions from these methods. Furthermore, HAC omits ICD's Adaptive Plausibility Constraint without justification, raising concerns about whether the correction may suppress valid tokens alongside hallucinated ones. [1] Zhang, Y., Cui, L., Bi, W., & Shi, S. (2025). Alleviating Hallucinations of Large Language Models through Induced Hallucinations. Findings of NAACL 2025, 8233–8247. https://aclanthology.org/2025.findings-naacl.459/ [2] Zhao, Y., Niu, B., Tan, Y., Wang, S., & Qin, L. (2025). Semantic-Aware Action Space Compression via LLM-DRL Synergy for Efficient Task-oriented Dialogue Policy Exploration. Findings of EMNLP 2025, 17808–17820. https://aclanthology.org/2025.findings-emnlp.968/

3. Insufficient experimental validation of the central claims. Although HAC improves downstream task performance and general factuality scores, the experiments do not directly demonstrate reduced hallucinations in dialogue actions, such as lower constraint violation or invalid action rates. More importantly, the ablation study combines hallucination correction and action compression into a single component, making it impossible to isolate the contribution of the proposed correction mechanism. A controlled comparison using original versus corrected LLM priors under identical compression and RL settings is necessary.

Questions for authors
  1. Why would token-level attention drift cause the model to violate task constraints? Is there any evidence supporting this causal relationship? Can you demonstrate that the corrected LLM generates constraint-satisfying actions with higher probability than the original LLM?
  2. How are the attention scores in Figures 1 and 3 computed? Since your method only modifies output token probabilities, how does this demonstrate that attention drift has been corrected?
  3. How is your correction method different from ICD? The core formulation appears nearly identical. Also, why did you omit ICD's Adaptive Plausibility Constraint, which was designed to prevent the suppression of valid tokens?
  4. What do the equation references in Figure 2 refer to? They do not match the equations in the main text.
5 comments

We sincerely thank you for all your valuable suggestions and efforts toward improving our manuscript. We will address your concerns point by point:

Critical Weaknesses:

Q1. Unclear problem formulation and unsupported causal claims.

A1. Thank you for raising this concern. We would like to clarify two points.

(a)We do not define all constraint-violating or infeasible actions as hallucinations. As explicitly stated in the Abstract (Lines 15–16), “their hallucinations may introduce task-irrelevant, constraint-violating, or infeasible actions.” This statement indicates that hallucinations may lead to these undesirable actions, rather than defining these actions as hallucinations.

Figure 1 (Lines 57–75) and Section 4.4 (Lines 351–360) illustrate how the original LLM tends to focus on repeated action templates and slot names rather than task-critical constraints and slot values, potentially generating syntactically valid but constraint-inconsistent actions. Figure 3 further presents the changes in token-level focus after HAC correction, demonstrating reduced template bias and increased focus on task-critical information. Moreover, the quantitative evaluation on two established factuality benchmarks, TruthfulQA and FACTSCORE (Table 3, Section 4.7), shows consistent factuality improvements across different LLM backbones. These qualitative and quantitative results support the effectiveness of HAC in mitigating unreliable generation preferences and improving factuality.

(b) We do not modify the LLM's internal attention module because HAC uses the LLM primarily to generate candidate actions for constructing the compressed action space, rather than as the final dialogue policy. Therefore, directly adjusting the output token probabilities provides a more efficient way to refine the generated action proposals without additional modifications to the model's internal architecture, reducing computational overhead while meeting the requirements of action-space construction.

Regarding Figures 1 and 3, token probabilities serve as the statistical measure of the model's generation preferences and are mapped to token-level focus scores for heatmap visualization. Higher token probabilities correspond to higher focus scores. We acknowledge that the original manuscript does not explicitly describe this mapping. We will clarify the relationship between token probabilities and the resulting focus scores and provide the corresponding mathematical formulation and computation procedure in the revised manuscript.

Q2. Limited methodological novelty and insufficient differentiation from prior work.

A2. Thank you for raising this concern.

(a) We apologize for the missing citation. Although Line 201 mentions “Following an induce-then-contrast paradigm”, it does not explicitly cite ICD, and Eqs. (1)–(4) indeed follow the offset-model training objective and contrastive decoding formulation proposed by Zhang et al. (2025). We acknowledge this oversight. Unlike ICD, which focuses on mitigating hallucinations in general LLM generation, HAC applies this mechanism to TOD by constructing a corrected action prior, projecting generated proposals into a compact executable action space, and further optimizing dialogue policies through distributional Q-learning. Thus, our contribution lies in integrating hallucination correction with action-space compression and RL-based policy optimization for TOD, rather than proposing a new contrastive decoding mechanism. We will introduce ICD in the Related Work section, explicitly cite it in Section 3.1.1, and clarify the contributions of HAC.

(b) We acknowledge that LLM-guided action compression has been explored in LLMSAP (Zhao et al., 2025); however, the two methods construct and employ the LLM prior differently. LLMSAP prunes implausible actions directly from the original prior, whereas HAC applies hallucination-induced contrastive correction prior to compression, and subsequently optimizes the policy via distributional Q-learning rather than conventional DQN. The advantage of this design is demonstrated empirically: at epoch 500 (Table 1), HAC achieves superior performance to LLMSAP across all three domains in success rate (Movie 0.870 vs. 0.785; Restaurant 0.654 vs. 0.584; Taxi 0.802 vs. 0.710), reward, and dialogue efficiency, and Table 2 exhibits the same advantage under human evaluation. These distinctions will be stated explicitly in the Related Work section.

(c) We acknowledge the role of the Adaptive Plausibility Constraint (APC) in preventing contrastive correction from suppressing valid tokens together with hallucinated ones. However, HAC is designed for TOD, where candidate actions are determined by task-specific slots and slot values rather than individual tokens. As illustrated in Figures 1 and 3 and Section 4.4, our correction primarily reduces the model's excessive focus on repeated action templates and slot names while increasing its focus on task-critical constraints and slot values. After correction, the generated proposals are further projected into the admissible action space to construct the candidate action set (Eqs. 5–6), and distributional Q-learning subsequently distinguishes candidate actions according to their state-dependent return distributions. This action-level evaluation provides a more fine-grained distinction among candidate actions and reduces erroneous filtering based solely on token-level plausibility. Nevertheless, we agree that APC provides a complementary safeguard, and we will further discuss its relationship with HAC and consider incorporating it in future work.


Q3. Insufficient experimental validation of the central claims.

A3. Thank you for raising this concern.

(a) Thank you for this valuable comment. Regarding the first concern, our existing experiments provide evidence supporting the effectiveness of hallucination correction, particularly the analysis in Section 4.4. Specifically, Figures 1 and 3 compare the token-level focus patterns before and after HAC correction, showing that HAC reduces the original LLM's excessive focus on repeated action templates and slot names while increasing its focus on task-critical constraints and slot values. The examples and analysis in Section 4.4 (Lines 351–360) further illustrate how this correction helps mitigate the generation of constraint-inconsistent actions. In addition, the factuality evaluations on TruthfulQA and FACTSCORE (Table 3, Section 4.7), together with the downstream performance improvements in Tables 1 and 2 (Sections 4.5 and 4.6), provide complementary evidence of HAC's effectiveness.

(b) Regarding the ablation study, we acknowledge that combining hallucination correction and action-space compression into a single component does not sufficiently distinguish their individual contributions. We will separate these two components and conduct additional controlled experiments under identical action-space compression and RL settings, comparing the original and corrected LLM priors to further isolate the contribution of hallucination correction.

Questions For Authors:

Q1. Why would token-level attention drift cause the model to violate task constraints? Is there any evidence supporting this causal relationship? Can you demonstrate that the corrected LLM generates constraint-satisfying actions with higher probability than the original LLM?

A1. Thank you for raising this important concern.

(a) Regarding the relationship between token-level attention drift and constraint violations, Section 4.4 illustrates that the original LLM tends to assign greater generation preference to frequent action templates and slot names, while giving relatively less emphasis to task-critical constraints and slot values. Since these constraints and values determine whether a generated action satisfies the user's requirements, such an imbalance may lead to constraint-inconsistent action proposals. Figures 1 and 3 provide qualitative evidence of this phenomenon and the corresponding changes after HAC correction.

(b)To provide more direct evidence, we will further compare the output probabilities assigned to constraint-satisfying and constraint-violating actions by the original and corrected LLMs under identical dialogue states and candidate action sets. This comparison directly evaluates whether HAC increases the probability mass assigned to valid actions while suppressing invalid ones. We will incorporate the corresponding quantitative results.


Q2.How are the attention scores in Figures 1 and 3 computed? Since your method only modifies output token probabilities, how does this demonstrate that attention drift has been corrected?

A2.We thank the reviewer for this question.

(a) We acknowledge that the manuscript does not explain how these scores are obtained. The displayed scores are not extracted self-attention weights. For a generated token yty_t we obtain its conditional probability pt=P(yt∣y<t,x)p_t = P(y_t \mid y_{<t}, x) and map it to a normalized focus score Ft=pt/max⁡jpjF_t = p_t / \max_j p_j for visualization, so that a higher score indicates a stronger generation preference for that token. We will provide the explicit formulation and the sampling settings in the revised version.

(b)Figures 1 and 3 should be read as showing the change in token-level generation preference after correction, rather than as evidence that the internal self-attention mechanism was modified. The corrected model assigns relatively lower probability to repeated action templates and slot names, and relatively higher probability to task-critical constraints and slot values. We will state this distinction explicitly and revise the wording "attention drift" so that it does not imply that HAC alters internal attention weights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.