acceptodds
Under review as a conference paper at ICLR 2027

Proactive Dialogue Policy Optimization via Cognitive-State Transition

Abstract

Proactive dialogue requires agents to continually adapt their policies to user feedback while progressing toward task objectives over multiple turns. To move beyond imitation learning on static datasets, recent approaches use user simulators to collect interactive data for policy optimization. However, many simulators do not explicitly model the evolution of user cognition, limiting the consistency and state dependence of feedback across turns. Moreover, representing each action only by a high-level strategy label overlooks the large utterance space and cannot distinguish alternative realizations of the same strategy. To this end, we jointly design a nitive User ulator and ognitive-tate ransition–Driven olicy ptimization . Cog-Sim maintains the user's cognitive and affective states and generates responses through constrained state transitions across turns, so feedback depends on both the realized utterance and the user's current state. CSTPO organizes each action as a hierarchical strategy–utterance representation: a high-level strategy label constrains utterance sampling, and utterances are optimized within each label. Sparse complete-branch sampling reuses shared dialogue prefixes and estimates separate strategy-level and utterance-level advantages, enabling fine-grained optimization at both levels. Across three tasks, Cog-Sim exhibits monotonic dose–response relationships and is preferred over prompt-based simulators for naturalness. CSTPO improves Qwen3-14B's performance to a level comparable to that of GPT-5.5-based planning methods.

Primary area:
reinforcement learning
Keywords:
Proactive Dialogue;Large Language Models;Reinforcement Learning;Cognitive Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XXiCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Cog-Sim, a user simulator with explicit cognitive and affective states and constrained transitions, and CSTPO, a policy optimization method that assigns separate advantages to strategy selection and utterance generation using shared-prefix branching.

Experiments on emotional support, donation persuasion, and negotiation report improvements over SFT across two model sizes, competitive performance against GPT-5.5-based baselines, and positive human preferences on several task-relevant dimensions.

Critical strengths
  • A coherent approach to strategy and utterance optimization. Separating strategy selection from its linguistic realization addresses a meaningful limitation of strategy-only planners. Recoverable simulator states allow alternative actions to be compared from the same starting context, providing a reasonable basis for the proposed credit assignment.

  • Encouraging results across multiple settings. Improvements over SFT are consistent across three dialogue tasks and two model sizes. The blind human evaluation provides complementary evidence of better emotional identification, comforting, and persuasiveness, while also reporting dimensions where the method performs worse.

Critical weaknesses
  • Simulator realism is not sufficiently validated. Monotonic dose–response behavior shows that Cog-Sim reacts to stronger interventions, but does not establish that these reactions resemble those of real users. The manually specified transition rules also lack calibration or sensitivity analysis.

  • The simulator’s contribution to policy learning is unclear. Both the full model and “w/o Cog-State” use Cog-Sim to generate user replies; the ablation only removes hidden cognitive information from the critic. This shows that additional critic information helps, but does not establish whether training with Cog-Sim produces better policies than training with a simpler simulator.

  • Evaluation remains tied to the training simulator. Policies are trained and evaluated with Cog-Sim, leaving their performance under other user behaviors uncertain. The human evaluation provides useful evidence about dialogue quality, but annotators assess simulated conversations rather than interact with the policies.

  • The main performance comparisons lack uncertainty estimates. Some headline differences are small, including .460 versus .450 on P4G and .491 versus .489 on CB. Without paired confidence intervals or results across independent training runs, it is difficult to assess whether these gains are robust.

  • The hierarchical-critic ablation needs clearer specification. “w/o Cog-Critic” does not sufficiently explain its advantage construction or use of branch data. These details, together with confirmation of matched rollout budgets, are needed to assess whether the gains come specifically from hierarchical credit assignment.

Questions for authors
  • Does training with Cog-Sim produce better policies than training the same optimizer with a simpler simulator, when both are evaluated under the same conditions?

  • How are advantages constructed and branch data used in “w/o Cog-Critic”? Are its training and rollout budgets matched to the full method?

  • How are the Gaussian parameters in Appendix G estimated? What source of variability do the plotted bands represent?

  • Can you provide paired confidence intervals for the main task-return comparisons? How consistent are the gains across independent training runs?

  • What prompts, reasoning settings, and generation budgets are used for the GPT and Qwen baselines? Please clarify which baselines have thinking enabled.

1 comment
AuthorsOpenReview ↗

Follow-up on Rebuttal and Additional Evidence

Dear Reviewer,

Thank you again for your thoughtful review. We hope our rebuttal and additional experimental evidence have addressed your concerns. We would greatly appreciate your feedback on whether any questions remain and would be happy to provide further clarification.

Thank you for your time and consideration.

Reviewer_n2bVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Cog-Sim and CSTPO for proactive dialogue. Cog-Sim models user cognitive states, while CSTPO optimizes hierarchical strategy-utterance actions.

Critical strengths

This work proposes Cog-Sim and CSTPO for proactive dialogue. Cog-Sim explicitly models dynamic user cognitive-affective states with constrained state transitions and supports counterfactual rollouts. CSTPO adopts a hierarchical strategy–utterance representation and a hierarchical critic for fine-grained credit assignment. It achieves competitive performance with a smaller LLM backbone.

Critical weaknesses

Figure 2 lacks textual description in the main body. For CSTPO, the authors should formally define the two policy levels and their respective advantage estimators. Eq.5 appears problematic, and the derivation of the decomposition requires further clarification. The proposed hierarchical advantage decomposition requires stronger theoretical and empirical justification. Figure 3 has lower image quality than Figures 1 and 2. Table 2 lacks in-text citation within the main body. Ablation studies on hierarchical advantage estimation, dual-value critics, and the use of additional branches in actor updates to clarify the contribution of each component.

Questions for authors

No

1 comment
AuthorsOpenReview ↗

Follow-up on Rebuttal and Methodological Clarifications

Dear Reviewer,

Thank you again for your constructive feedback. We hope our revised theoretical derivations and additional ablations have addressed your concerns. We would greatly appreciate any further feedback and would be happy to clarify any remaining questions before the discussion period ends.

Thank you for your time and consideration.

Reviewer_b5CJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses an important problem in proactive dialogue policy optimization and offers a moderately novel integration of cognitive-state modeling and hierarchical reinforcement learning, although the individual techniques are largely based on existing approaches.

Critical strengths

The paper proposes a well-structured framework combining cognitive-state-driven user simulation with hierarchical dialogue policy optimization, supported by experiments and ablation studies across three dialogue tasks. The proposed Cog-Sim and CSTPO framework offers an interesting integration of cognitive-state-driven user simulation and hierarchical reinforcement learning, enabling separate optimization of strategy selection and utterance generation, with component ablation studies, human evaluations, and training dynamics analyses providing empirical support for the approach.

Critical weaknesses
  1. The main task rewards rely on a frozen LLM Evaluator, and the simulator and evaluator also use the same underlying model service. Although the evaluator does not access the hidden cognitive states, there may still be model-related evaluation bias.

  2. CSTPO introduces strategy–utterance hierarchy, a hierarchical Critic, and shared-prefix branch sampling, but it remains unclear which component contributes most to the performance improvements.

Questions for authors
  1. Although the paper validates the contributions of cognitive-state inputs and the hierarchical Critic, why is there no separate ablation study of shared-prefix branch sampling to demonstrate its independent contribution to performance improvements?

  2. Shared-prefix branch sampling uses fixed numbers of strategies, utterance candidates, and branching checkpoints. How sensitive is the model's performance to these hyperparameters?

1 comment
AuthorsOpenReview ↗

Follow-up on Rebuttal and Additional Experiments

Dear Reviewer,

Thank you again for your valuable feedback. As the discussion period approaches its end, we would greatly appreciate it if you could let us know whether our rebuttal and additional experiments have addressed your concerns. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.