acceptodds
Under review as a conference paper at ICLR 2027

From Textbooks to Proactive Agentic Systems: Scaling Long-Horizon Planning via Automated Agentic System Construction

Abstract

Large language models (LLMs) excel at generating locally coherent responses but remain fundamentally , lacking the ability to plan and steer interactions over long horizons. Existing approaches attempt to induce proactivity through prompting or reinforcement learning with verifiable rewards (RLVR), which we show degrade on long-horizon dialogue tasks. A natural solution is to augment LLMs with planners, memory, and user simulators. However, specifying these components and their interactions requires substantial expert design effort, thereby limiting scalability. We address this challenge by introducing (**T**extbooks t**O** **P**roactive **A**gentic **S**ystems), a for automatically constructing executable agentic systems from domain textbooks. Given unstructured instructional textbooks, jointly derives a hierarchical planner over structured decision space (states, actions, and rewards), together with a user simulator and a graph-based memory structure, thereby defining a complete interaction loop for training and evaluation. It further generates structured supervision via uncertainty-aware self-annotation and learns hierarchical policies over the extracted structure using reinforcement learning. Experiments on psychotherapy and persuasive dialogue benchmarks show that achieves state-of-the-art performance across multiple metrics and enables planning over longer horizons, outperforming prompting-based, RLVR and manually designed planning approaches.

Primary area:
reinforcement learning
Keywords:
Proactive Dialogue Systems, Reinforcement Learning, Knowledge Extraction, Long-Horizon Planning, User Simulator

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1ZtJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The proposed method automatically constructs a complete proactive dialogue agent system from domain textbooks. The system incorporates a hierarchical planner, knowledge graph memory, and a user simulator. It generates supervised signals via self-annotation and employs offline reinforcement learning to train long-horizon planning strategies, achieving promising performance on psychological support and persuasive dialogue tasks.

Critical strengths

1.This work proposes a novel paradigm to automatically build proactive dialogue agents from domain textbooks, greatly reducing the manual cost of domain adaptation. 2.It develops an integrated system with hierarchical planner, knowledge-graph memory and user simulator, forming a complete agent architecture for long-horizon goal-oriented dialogue. 3.The self-annotation pipeline generates supervisory signals automatically, alleviating the heavy annotation burden for psychological support and persuasive dialogue. 4.Offline reinforcement learning is adopted for long-horizon policy optimization, avoiding risky online interactions in sensitive dialogue scenarios.

Critical weaknesses

The computational overhead of knowledge graph retrieval, hierarchical planning and reasoning should be reported. Please verify the correctness of Equation (1) and explain how p(⋅)p(\cdot) is calculated. Comparisons with other SOTA methods on the same task are recommended. Figure 6 lacks clarity; vector graphics are suggested. It is necessary to verify whether the user simulator tends to act as a cooperative responder, which may inflate the model performance. The reward function design for offline RL training should be elaborated. Ablation experiments are missing to validate individual contributions of each module (knowledge graph memory, hierarchical planner).

Questions for authors

No

2 comments

We thank the reviewer for the positive comments about the novelty of building proactive agents from textbooks. We address the concerns below.

1. Computational overhead of retrieval, planning, and reasoning

TOPAS adds no additional model. All learned components (macro-state probe, retrieval policy, termination functions, policy over options, intra-option policies) are lightweight MLP heads (20M parameters in total) that read a single intermediate hidden state hth_t of the frozen dialogue LLM. The selected option, micro-action, and retrieved memory are appended to the prompt and reuse its KV cache, so generation requires no second pass over the dialogue [new Fig. 13 in the appendix]. Measured costs on CBT [new Tab. 2]:

ComponentLLM calls / turnTime / turn
Macro-state probe (incl. extracting hth_t)–310 ms
Knowledge-graph retrieval policy–3.5 ms
Hierarchical planner (termination, option, micro-action)–10.2 ms
Memory update14.16 s
Utterance generation11.62 s
Total26.10 s
  • Planning and retrieval account for only ~5% of per-turn latency (0.32 s). The rest is the two LLM calls, one of which (generation) every LLM agent makes. The memory update runs in parallel with the planner, so it is off the critical path.
  • Retrieval is cheap by design. The retriever is a categorical policy over the node combinations observed in expert dialogues and selects one in a single forward pass. Similarity-based KG retrieval takes 4.0 s per turn, i.e., over 1000× slower.
  • Training takes 2 h on one RTX A6000 (15 GB peak memory), versus 6–14 h for RL fine-tuning baselines that update the full 7B LLM, with 99.7% fewer trainable parameters.
  • One-time costs per domain: structure extraction (193 LLM calls, 29 min) and self-annotation (16.9k LLM calls, 48 h).

2. Correctness of Eq. (1) and how p(⋅)p(\cdot) is computed

The reviewer is right that the original Eq. (1) was underspecified. The revision (Sec. 4.4) formulates annotation directly as constrained MAP inference over the posterior (L.305) using an exact chain-rule factorization, first over option intervals I\mathcal{I} and then over turns.

How p(⋅)p(\cdot) is computed. The ground-truth posterior is unknown, so we do not evaluate these probabilities numerically. Each conditional factor is approximated by a prompted LLM whose output is restricted to valid elements of PΩP_\Omega via structured decoding, and the MAP solution is approximated by greedy sequential decoding:

  1. option segmentation (a parsimony instruction replaces the former penalty λK\lambda K);
  2. micro-action labeling, conditioned on the intervals;
  3. state, reward, and memory labeling, conditioned on both;
  4. consensus across N=3N=3 LLM annotators, approximating max-marginal decoding; low-agreement labels are adjudicated by an LLM or masked.

3. Comparisons with other state-of-the-art methods

We compare against every family of long-horizon dialogue planners [Tab. 1, Fig. 2]:

  • Prompting: ProCoT and, newly added, TheraMind (a psychotherapy-specific prompting method).
  • Token-level RL fine-tuning: ArCHer, SweetRL, ARIA.
  • Learned hierarchical RL: newly added HiPER, which generates free-form subgoals before each utterance.
  • Hand-specified planners (PPDPP, DPDP, DialogXpert): these require a hand-designed action space, which does not exist for CBT or SCOA, so we evaluate them on their native benchmark, ESConv [Fig. 3a–b].
  • Newly added textbook-based baselines: the extracted structure given to the LLM by prompting (no planner), and retrieval of raw textbook passages (no structure) [Fig. 12].
  • Newly added memory baselines: no memory, the full knowledge graph in context, and similarity-based KG retrieval [Fig. 11].

TOPAS achieves the highest SSR on CBT and P4G under every judge and external simulator, wins three of four head-to-head comparisons on the newly added Supreme Court benchmark, and is rated highest by clinicians (3.80 vs. 3.48 for HiPER and 3.39 for SweetRL).


4. Figure 6 (now Fig. 8)

We re-rendered all figures as vector graphics and clarified the win-rate definition in the caption and text: for each pair of methods, raters compare the two extracted structures. A method's win rate is the fraction of comparisons it wins (ties count as half), averaged over its opponents and then over experts or judge runs, so 50% means parity with the average opponent.

5. Is the user simulator over-cooperative?

Over-cooperation is precisely the failure mode that motivates the paper [Fig. 3a–b]. We address it at three levels:

  • Training never uses a simulator. All policies are trained offline on annotated expert dialogues, so they cannot learn to exploit a simulator.

  • Evaluation uses three simulators, two of them external (Patient-Ψ and PatientAct for CBT; a prompting simulator and TRIP for P4G). TOPAS is best in all 12 judge × external-simulator cells, so its gains do not depend on our own simulator; on P4G, several baselines even outperform TOPAS under the TOPAS simulator [Tab. 1].

  • New human studies with 17 clinicians (details in the general response). (i) Simulator realism: 8 clinicians, blind to simulator identity, rate the TOPAS simulator as the most realistic (3.79 vs. 3.54 and 3.47), with the clearest advantage on disclosure pacing, the dimension most tied to over-cooperation. (ii) No simulator at all: 9 clinicians role-played the patients themselves and still rank TOPAS highest (3.80).


6. Reward design for offline RL

TOPAS uses two rewards [Sec. 4.3–4.5]:

  • Option-level reward ror^o (dense, extracted from textbooks): for each option, TOPAS extracts a three-level rubric: success (2), progress (1), and failure (0). For cognitive restructuring, for example, these are: balanced thought developed / unhelpful thought identified / no unhelpful thought found [new Fig. 5]. The annotators score the rubric at the end of each option interval, with the same consensus as the other labels, and it trains the intra-option policies.

  • Task reward rr (sparse): the session outcome at the last turn (CTRS for CBT, persuasion outcome for P4G, case outcome for SCOA), which trains the policy over options and the termination functions.

Splitting credit this way turns one 200-turn credit-assignment problem into short option-level ones.


7. Ablations of individual modules

  • Hierarchical planner [Fig. 12]: removing the learned planner, i.e., only prompting the LLM with the extracted structure, lowers SSR to 60% of TOPAS on CBT (2.45 vs. 4.12). Removing macro-state tracking lowers it to 85% (CBT) and 80% (P4G).

  • Knowledge-graph memory [Fig. 11]: memory improves SSR by 14% over no memory (4.03 vs. 3.52). Learned retrieval matches similarity-based KG retrieval (4.03 vs. 4.08) with 3.5× lower variance, >1000× lower latency, and 2.2× fewer tokens, while putting the full graph in context performs worse (3.81) despite 35× more tokens.

  • Textbook structure [Fig. 12]: replacing it with an LLM-generated structure lowers SSR to 89% (CBT) and 87% (P4G), and retrieving raw textbook passages instead of extracting structure lowers it to 37% (CBT) and 59% (P4G). The textbook-extracted structure also matches or exceeds an expert-designed one (91% of TOPAS on CBT).

We hope these additions resolve the reviewer's concerns.

Reviewer_nCjTOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a framework that extracts structured domain knowledge from textbooks to construct proactive dialogue agents. The extracted structures guide dialogue annotation and offline hierarchical policy learning, with evaluations in cognitive behavioral therapy and donation persuasion.

Critical strengths
  • Important problem and interesting application settings;
  • The hierarchical extraction pipeline provides a thoughtful design for incorporating global context from long textbooks;
  • The evaluation is broad in scope, combining expert assessments of extracted structures, downstream comparisons across two application domains, and several informative ablations.
Critical weaknesses
  • The downstream evaluation remains limited in scale and realism;
  • The paper lacks a comparison against a simpler agent provided with the same textbook knowledge through prompting or retrieval.
Questions for authors

N/A

1 comment

We thank the reviewer for the positive comments on TOPAS and address the concerns in the following.

1. Scale and realism of the downstream evaluation.

  • A third, more challenging benchmark. We added U.S. Supreme Court oral advocacy (SCOA): 1,445 cases with ~258 utterances per hearing (details in App. B.3). Two planners argue against each other, so 50% is parity. TOPAS wins against SweetRL, ProAct, and ARIA (53.6–55.8%) and never loses to them as respondent [Tab. 1].
  • Two human evaluations on CBT (details in the general response):
    • Planner evaluation: 9 licensed clinicians role-played 15 patients and rated sessions on the CTRS. They rank TOPAS highest (3.80), ahead of HiPER (3.48) and SweetRL (3.39).
    • Simulator evaluation: 8 clinicians interacted blindly with each simulator and rated TOPAS-sim as the most realistic (3.79 vs. 3.54 and 3.47).

The evaluation now spans three domains with sessions of up to 200 turns, and human experts confirm both the planner ranking and the realism of our simulator, addressing the concern that our gains could stem from LLM judges or over-cooperative simulators.


2. Comparison with simpler agents given the same textbook knowledge.

We added two baselines that use the same textbooks without our learned planner, both on the same dialogue LLM (Qwen2.5-7B-Instruct) [Fig. 12, shown as "w/o RL" and "w/o RL & structure"; full results in Tab. 21]:

  • Structure prompting (no learned planner): the extracted structure (options, micro-actions, states, rubrics) is given to the dialogue LLM in its prompt. It reaches 60% of TOPAS's SSR on CBT, 99% on P4G, and parity on SCOA (50% win rate). Learning when to apply the structure thus matters most over long horizons: on ~200-turn CBT sessions, prompting alone cannot sustain a coherent strategy, whereas on ~10-turn P4G dialogues the structure in the prompt suffices.

  • Textbook retrieval (no structure, no planner): relevant passages from the same textbooks are retrieved and appended to the dialogue LLM's context. It reaches only 37% of TOPAS on CBT and 59% on P4G, well below structure prompting on both, and clinicians rate it below TOPAS (3.09 vs. 3.80).

Together, these comparisons separate the two contributions: extracting structure consistently improves over retrieving raw text, and learning to plan over it is what enables long-horizon performance.

We hope these additions address the reviewer's concerns.

Reviewer_Fk4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TOPAS, Textbooks tO Proactive Agentic Systems, a framework for automatically constructing proactive dialogue agents capable of long-horizon planning from domain-specific textbooks. The authors argue that existing LLM-based dialogue systems are largely reactive and struggle to maintain coherent long-term strategies, while planner-augmented approaches rely heavily on manually designed state spaces, action spaces, and reward functions.

To address these limitations, TOPAS introduces a three-stage pipeline. First, it extracts structured domain knowledge from textbooks to construct a hierarchical Markov Decision Process (MDP), a knowledge-graph-based memory system, and a user simulator. Second, it automatically annotates unlabeled dialogue trajectories with high-level planning options, primitive actions, states, and rewards, using uncertainty-aware refinement to improve annotation reliability. Third, it trains hierarchical planning policies and memory retrieval policies through offline reinforcement learning, particularly Implicit Q-Learning (IQL), while using pretrained LLMs for natural-language generation.

The authors evaluate TOPAS on two dialogue domains, cognitive behavioral therapy (CBT) and persuasive dialogue. Experimental results show that TOPAS outperforms prompting-based and RL-finetuned baselines, achieving up to 23.5% and 15.5% improvements in soft success rate on the CBT and persuasion benchmarks, respectively. Additionally, TOPAS requires 99.7% fewer trainable parameters and achieves 3–7× faster training compared with the RL baselines. Ablation studies further demonstrate the importance of hierarchical policy optimization and conversational state tracking. Overall, the paper presents an approach to automatically constructing domain-grounded agentic systems by combining textbook-derived symbolic structures with data-driven policy learning.

Critical strengths
  1. Novel, scalable construction of planner-augmented agents. TOPAS replaces hand-designed MDPs with structures extracted from textbooks: a hierarchical MDP, KG memory, and user simulator. In expert evaluation (41 experts, double-blind), extractions win 58%–82% of comparisons against baselines. Textbook-derived components (SSR 4.78) nearly match expert-designed ones (4.97), which supports textbook extraction as a viable replacement for manual domain engineering.

  2. Strong long-horizon performance with high efficiency. A lightweight hierarchical planner (~20M parameters, trained with offline RL) is decoupled from a frozen LLM. It outperforms prompting and RL-finetuning baselines (+23.5% SSR on CBT, +15.5% on P4G), and the gains hold up to 200 turns. It also uses 99.7% fewer trainable parameters and trains 3–7× faster.

Critical weaknesses
  1. Evaluation relies entirely on LLM simulators and LLM judges, and the judge saturates. All planner results come from LLM-simulated users scored by LLM judges. There is no human evaluation of the generated dialogues, even for CBT, a clinical domain. This matters because the paper itself (Fig. 2) argues that over-cooperative simulators and brittle success criteria cause reward hacking. In addition, the GPT-4.1 judge gives the maximum CTRS score (6.00 ± 0.00) to almost every ablation variant in Tab. 2, so at least one judge cannot discriminate between systems. This weakens the reported gains and the ablation conclusions. A human or expert evaluation of a subset of sessions would make the claims considerably more convincing.

  2. The evidence for long-horizon behavior and generality is narrow. Only CBT (about 200 turns per session) actually tests long horizons; P4G dialogues average only 10.3 turns. Both domains also have well-established instructional textbooks, so it is unclear how TOPAS performs when textbooks are sparse, conflicting, or only weakly procedural. The authors acknowledge this dependence on textbook coverage in their limitations. Finally, the headline numbers are reported inconsistently, as "up to" in the abstract and intro but as average SSR gains in Sec. 5, and the main planner table (Tab. 10) appears only in the appendix.

Questions for authors
  1. Which user simulator is used to evaluate the planners? Is it TOPAS's own textbook-derived simulator? If so, how do you rule out TOPAS benefiting from being evaluated in the same environment its structure came from? Could you also report a human or expert evaluation on a subset of CBT sessions, given that the GPT-4.1 judge saturates at 6.00 in Tab. 2?
  2. Sec. 4 mentions refining high-uncertainty annotations with expert labels "when available." How many expert labels were used in the reported experiments, and how does performance change without them? This determines how "automated" the pipeline really is.
  3. How sensitive is downstream planning to errors in the extracted MDP, for example missing options or incorrect reward definitions? What happens with less comprehensive textbooks, or with books that conflict with each other?
  4. Please clarify whether the +23.5% / +15.5% gains are maximum or average improvements, and over which baseline and horizon. Please also consider moving the main planner results (Tab. 10) into the main text.
2 comments

We thank the reviewer for the positive comments on the novelty, scalability, and efficiency of TOPAS. We address the concerns below.

Informativeness. We respectfully believe the paper offers several findings that are new to the community, which the revision makes more explicit:

  1. Token-level RL does not recover long-horizon strategy. As sessions grow toward 200 turns, fine-tuned baselines increasingly repeat interventions and diverge from experts in which phases they use, how they execute them, and in what order [Fig. 3c–d]. TOPAS stays repetition-free and closest to expert behavior.
  2. Strategy can be read from textbooks instead of learned from scratch. Three textbooks already recover over 80% of an expert-designed structure [Fig. 9], and the textbook-extracted structure matches or exceeds the expert-designed one downstream (the latter reaches 91% of TOPAS's SSR on CBT) [Fig. 12].
  3. Structure matters more than scale. A 20M-parameter planner over a frozen LLM outperforms 7B-parameter RL fine-tuning, with 99.7% fewer trainable parameters and 3–7× less training time, and clinicians rate it highest (3.80 vs. 3.48 for the strongest baseline).

Together, these results challenge the assumption that long-horizon dialogue strategy must be either hand-designed or learned end-to-end from interaction data: much of it is already documented by domain experts, and learning only needs to decide when and how to apply it.


Human evaluation. We added two human studies on CBT (details in the general response).

  • Planner evaluation: 9 licensed clinicians role-played 15 PatientAct patients and rated each session on the CTRS. They rank TOPAS highest (3.80), ahead of HiPER (3.48) and SweetRL (3.39), confirming the LLM-judge ranking.
  • Simulator evaluation: 8 clinicians interacted blindly with each simulator and rated TOPAS-sim as the most realistic overall (3.79 vs. 3.54 and 3.47).

GPT-4.1 saturation. We agree that GPT-4.1 compresses scores near the top of the CBT scale. This happens mainly under Patient-Ψ, where three variants score ≥5.94/6 [Tab. 26], and GPT-4.1 also agrees least with the other judges (App. E). The ablation conclusions do not depend on it:

  • GPT-4.1 still ranks TOPAS first. Averaged over the three CBT simulators, TOPAS scores 5.28, against 3.58 w/o RL, 4.74 w/o conversation states, 5.08 with the LLM-generated structure, and 4.93 with the expert-designed one. Under PatientAct, where scores do not saturate, it separates TOPAS from every variant (3.89 vs. 2.12–3.27).
  • The other two judges do not saturate. No CBT cell from DeepSeek or GPT-5.1 exceeds 4.29, and both rank TOPAS first. Removing RL lowers SSR from 3.36 to 1.86 (DeepSeek) and from 3.72 to 1.93 (GPT-5.1).
  • Clinicians confirm the trend without any LLM judge, rating TOPAS 3.80 vs. 3.08 for the LLM-generated structure.

Fig. 12 summarizes the ablations across benchmarks; Tab. 26 gives the per-judge, per-simulator breakdown.


Evidence for long-horizon behavior. We agree that P4G dialogues are short. We therefore:

  • added Supreme Court oral advocacy (SCOA; 1,445 hearings, ~258 utterances each), where two planners argue against each other; TOPAS wins against three of four baselines [Tab. 1];
  • added two long-horizon analyses on CBT: baselines increasingly repeat interventions as sessions grow toward 200 turns, while TOPAS stays repetition-free and closest to expert option, micro-action, and transition distributions [Fig. 3c–d];
  • evaluated memory over four consecutive sessions with the same patient, where learned retrieval yields the best and most stable performance [Fig. 11].

Which simulator is used, and is TOPAS favored by its own simulator? Each domain is evaluated with three simulators, two external ones (CBT: Patient-Ψ and PatientAct; P4G: a prompting-based simulator and TRIP) and the TOPAS-derived one, each with 3 runs per profile and 3 LLM judges [Tab. 1]. Two observations rule out a home advantage:

  • TOPAS wins without its own simulator. It is best in all 12 judge × external-simulator cells across CBT and P4G, and clinicians, who play the patient themselves, also rate it highest (3.80).
  • The TOPAS simulator does not favor TOPAS. On P4G, ProCoT, SweetRL, and ARIA all score higher than TOPAS under it (e.g., 1.87 for ProCoT vs. 1.43–1.46 for TOPAS), even though they share none of its structure.

The Avg. column includes all three simulators; excluding the TOPAS simulator increases TOPAS's margin on P4G (0.97 vs. 0.84 for the strongest baseline).

"Up to" gains. Each gain is the improvement in average SSR (over judges, simulators, and horizons) relative to the strongest baseline on that benchmark; "up to" referred to the larger of the per-benchmark gains. Because the Avg. column now also includes the TOPAS simulator, the numbers changed: +24.8% on CBT (4.12 vs. 3.30) and +3.7% on P4G (vs. +15.5% over external simulators only). The introduction now reports both per benchmark, and the main planner table is in the main text (Tab. 1), as suggested.


Expert labels in annotation. No expert labels were used, so performance does not depend on them: the pipeline is fully automated. Majority voting among three LLMs labels 84.5% of CBT therapist turns (vs. 76.7% with a single LLM), rising to 89.4% after LLM adjudication [Fig. 22]. Expert review confirms the labels without changes and identifies the remaining 10.6% as non-therapeutic content (session logistics, transcript artifacts). Masking these turns prevents TOPAS from learning spurious behaviors that token-level RL baselines reproduce, such as apologies and transcript markers [App. A.3].


Dependence on textbooks and sensitivity to extraction errors.

  • Number of textbooks. Extracting from random subsets of k ∈ {1,…,7} CBT textbooks, three textbooks already recover over 80% of the expert-designed options, micro-actions, conversation states, and memory node types; coverage then plateaus and its variance across subsets shrinks, so a small corpus suffices and additional books mainly add robustness to the choice of sources [Fig. 9].

  • Conflicting sources. Textbooks are selected with domain experts around a shared framework, and each extracted element carries its scope (the context in which the textbook prescribes it) and its reward rubric, so strategies from different contexts are not merged. Domains with sparse or conflicting literature remain a limitation (Sec. 6).

We hope this addresses the reviewer's concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their thoughtful comments. Below we summarize our main contributions and the modifications to the paper. We address individual concerns in the per-reviewer responses.

Main contributions

1. State-of-the-art planners fail to plan over long horizons in multi-turn dialogues. The extended Fig. 3 adds two panels:

(c) Baselines increasingly repeat interventions as sessions grow.

(d) Their strategies diverge from those of experts in high-reward dialogues: which phases are used (options, p(o)p(o)), how they are executed (primitive actions, p(a∣o)p(a \mid o)), and in what order (transitions, p(o′∣o)p(o' \mid o)).

We attribute this to the difficulty of recovering a hierarchical policy in token space. In contrast, TOPAS stays repetition-free up to 200 turns and is closest to expert behavior.

2. We extract the planning structure from domain textbooks instead of learning it from scratch. TOPAS uses LLMs to parse established strategies from textbooks into a planning substrate (options, primitive actions, states, reward rubrics, memory ontology, and user-state space). It then trains only lightweight planner and memory-retrieval policies over this structure with offline RL, while the dialogue LLM stays frozen. Across psychotherapy, persuasion, and Supreme Court advocacy, TOPAS improves goal success over state-of-the-art baselines by up to 24.8%, is preferred by clinicians, and wins most head-to-head debates. It sustains its performance over sessions of up to 200 turns with ~99.7% fewer trainable parameters and 3–7× less training time.


Main modifications

1. An additional, more challenging benchmark: Supreme Court oral advocacy (SCOA). 1,445 Oyez arguments from the 1964–2023 terms with recorded case outcomes (~258 utterances per hearing), following Lin et al. (2026). Two planners argue against each other, so 50% is parity. TOPAS beats SweetRL, ProAct, and ARIA (53.6–55.8% win rate) and never loses to them as respondent, the role that requires rebutting arguments as they unfold [Tab. 1].

2. Two human evaluation studies on CBT.

  • Planner evaluation. Nine licensed clinicians role-played 15 PatientAct patients in multi-turn sessions with TOPAS and the baselines, and rated each session on the CTRS. They rate TOPAS highest (3.80), ahead of HiPER (3.48), SweetRL (3.39), and the remaining baselines (1.61–2.11). Human experts thus confirm the LLM-judge ranking. This provides evidence that our gains are not an artifact of the judges or simulators [Tab. 1, Fig. 1c].
  • Simulator evaluation. Eight clinicians played the therapist with each simulator on 11 profiles, blind to the simulator's identity and in random order (Krippendorff's α = 0.43). TOPAS-sim obtains the highest overall rating (3.79, vs. 3.54 for PatientAct and 3.47 for Patient-Ψ) and leads on disclosure pacing, the dimension most directly tied to over-cooperation; PatientAct remains slightly more resistant. LLM-judge scores correlate moderately with clinician ratings (Spearman rs=0.40r_s = 0.40), which supports their use for larger-scale evaluation [Fig. 10].

3. Additional baselines and analyses.

  • Planners: HiPER (Peng et al., ICML 2026), which learns to generate free-form subgoals before each utterance, and TheraMind (Hu et al., 2026), a psychotherapy-specific prompting baseline.
  • Memory: on CBT, over four consecutive sessions with the same patient, we compare TOPAS's learned retrieval with no memory, the full knowledge graph in context (Full KG), and similarity-based retrieval over the same graph (KG RAG) [Fig. 11].
    • Memory improves mean SSR by 14% over no memory (4.03 vs. 3.52).
    • Learned retrieval matches KG RAG (4.03 vs. 4.08) with 3.5× lower variance across simulator backbones, over 1000× lower latency (3.5 ms vs. 4.0 s per turn), and 2.2× fewer context tokens.
    • Injecting the full graph performs worse (3.81) despite 35× more tokens: selective retrieval, not more context, drives the gain.
  • Ablations across benchmarks: removing both RL and the extracted structure lowers SSR to 37% (CBT) and 59% (P4G) of TOPAS; an LLM-generated structure reaches 89% and 87%, and an expert-designed one 91% (CBT); on SCOA, variants match TOPAS (45–50% win rate) [Fig. 12].
  • Sensitivity to the number of textbooks: three textbooks already recover over 80% of an expert-designed structure [Fig. 9].
  • Computational cost: planning (state estimation, retrieval, and policies) takes only ~5% of per-turn latency [Tab. 2].

4. Write-up. We added algorithms for training and inference [Alg. 1, Alg. 2] and a figure positioning TOPAS among long-horizon planning approaches [Fig. 2]. We polished the presentation without changing the paper's message, and condensed parts of the text to make room for the new results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.