- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The proposed method automatically constructs a complete proactive dialogue agent system from domain textbooks. The system incorporates a hierarchical planner, knowledge graph memory, and a user simulator. It generates supervised signals via self-annotation and employs offline reinforcement learning to train long-horizon planning strategies, achieving promising performance on psychological support and persuasive dialogue tasks.
Critical strengths
1.This work proposes a novel paradigm to automatically build proactive dialogue agents from domain textbooks, greatly reducing the manual cost of domain adaptation. 2.It develops an integrated system with hierarchical planner, knowledge-graph memory and user simulator, forming a complete agent architecture for long-horizon goal-oriented dialogue. 3.The self-annotation pipeline generates supervisory signals automatically, alleviating the heavy annotation burden for psychological support and persuasive dialogue. 4.Offline reinforcement learning is adopted for long-horizon policy optimization, avoiding risky online interactions in sensitive dialogue scenarios.
Critical weaknesses
The computational overhead of knowledge graph retrieval, hierarchical planning and reasoning should be reported. Please verify the correctness of Equation (1) and explain how is calculated. Comparisons with other SOTA methods on the same task are recommended. Figure 6 lacks clarity; vector graphics are suggested. It is necessary to verify whether the user simulator tends to act as a cooperative responder, which may inflate the model performance. The reward function design for offline RL training should be elaborated. Ablation experiments are missing to validate individual contributions of each module (knowledge graph memory, hierarchical planner).
Questions for authors
No
2 comments
5. Is the user simulator over-cooperative?
Over-cooperation is precisely the failure mode that motivates the paper [Fig. 3a–b]. We address it at three levels:
-
Training never uses a simulator. All policies are trained offline on annotated expert dialogues, so they cannot learn to exploit a simulator.
-
Evaluation uses three simulators, two of them external (Patient-Ψ and PatientAct for CBT; a prompting simulator and TRIP for P4G). TOPAS is best in all 12 judge × external-simulator cells, so its gains do not depend on our own simulator; on P4G, several baselines even outperform TOPAS under the TOPAS simulator [Tab. 1].
-
New human studies with 17 clinicians (details in the general response). (i) Simulator realism: 8 clinicians, blind to simulator identity, rate the TOPAS simulator as the most realistic (3.79 vs. 3.54 and 3.47), with the clearest advantage on disclosure pacing, the dimension most tied to over-cooperation. (ii) No simulator at all: 9 clinicians role-played the patients themselves and still rank TOPAS highest (3.80).
6. Reward design for offline RL
TOPAS uses two rewards [Sec. 4.3–4.5]:
-
Option-level reward (dense, extracted from textbooks): for each option, TOPAS extracts a three-level rubric: success (2), progress (1), and failure (0). For cognitive restructuring, for example, these are: balanced thought developed / unhelpful thought identified / no unhelpful thought found [new Fig. 5]. The annotators score the rubric at the end of each option interval, with the same consensus as the other labels, and it trains the intra-option policies.
-
Task reward (sparse): the session outcome at the last turn (CTRS for CBT, persuasion outcome for P4G, case outcome for SCOA), which trains the policy over options and the termination functions.
Splitting credit this way turns one 200-turn credit-assignment problem into short option-level ones.
7. Ablations of individual modules
-
Hierarchical planner [Fig. 12]: removing the learned planner, i.e., only prompting the LLM with the extracted structure, lowers SSR to 60% of TOPAS on CBT (2.45 vs. 4.12). Removing macro-state tracking lowers it to 85% (CBT) and 80% (P4G).
-
Knowledge-graph memory [Fig. 11]: memory improves SSR by 14% over no memory (4.03 vs. 3.52). Learned retrieval matches similarity-based KG retrieval (4.03 vs. 4.08) with 3.5× lower variance, >1000× lower latency, and 2.2× fewer tokens, while putting the full graph in context performs worse (3.81) despite 35× more tokens.
-
Textbook structure [Fig. 12]: replacing it with an LLM-generated structure lowers SSR to 89% (CBT) and 87% (P4G), and retrieving raw textbook passages instead of extracting structure lowers it to 37% (CBT) and 59% (P4G). The textbook-extracted structure also matches or exceeds an expert-designed one (91% of TOPAS on CBT).
We hope these additions resolve the reviewer's concerns.
We thank the reviewer for the positive comments about the novelty of building proactive agents from textbooks. We address the concerns below.
1. Computational overhead of retrieval, planning, and reasoning
TOPAS adds no additional model. All learned components (macro-state probe, retrieval policy, termination functions, policy over options, intra-option policies) are lightweight MLP heads (20M parameters in total) that read a single intermediate hidden state ht of the frozen dialogue LLM. The selected option, micro-action, and retrieved memory are appended to the prompt and reuse its KV cache, so generation requires no second pass over the dialogue [new Fig. 13 in the appendix]. Measured costs on CBT [new Tab. 2]:
2. Correctness of Eq. (1) and how p(⋅) is computed
The reviewer is right that the original Eq. (1) was underspecified. The revision (Sec. 4.4) formulates annotation directly as constrained MAP inference over the posterior (L.305) using an exact chain-rule factorization, first over option intervals I and then over turns.
How p(⋅) is computed. The ground-truth posterior is unknown, so we do not evaluate these probabilities numerically. Each conditional factor is approximated by a prompted LLM whose output is restricted to valid elements of PΩ via structured decoding, and the MAP solution is approximated by greedy sequential decoding:
3. Comparisons with other state-of-the-art methods
We compare against every family of long-horizon dialogue planners [Tab. 1, Fig. 2]:
TOPAS achieves the highest SSR on CBT and P4G under every judge and external simulator, wins three of four head-to-head comparisons on the newly added Supreme Court benchmark, and is rated highest by clinicians (3.80 vs. 3.48 for HiPER and 3.39 for SweetRL).
4. Figure 6 (now Fig. 8)
We re-rendered all figures as vector graphics and clarified the win-rate definition in the caption and text: for each pair of methods, raters compare the two extracted structures. A method's win rate is the fraction of comparisons it wins (ties count as half), averaged over its opponents and then over experts or judge runs, so 50% means parity with the average opponent.