- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies reliability failures of LLM-based user simulators — goal omission, constraint violations, and premature termination in long-horizon interactive tasks. It proposes ARGUS, a neuro-symbolic user simulator that decouples state management from response generation: the user's goals are maintained in an explicit LIFO agenda stack manipulated through constrained symbolic operations (PUSH, POP, MODIFY, WAIT) emitted as tool calls by a state controller, while a separate generation LLM produces the user utterance conditioned on the updated agenda. To enable cost-efficient deployment, the authors distill prompt-based ARGUS into compact 4B/8B models via a two-stage pipeline — evaluator-guided SFT (a step-level evaluator scoring Soundness, Faithfulness, and Adherence filters teacher rollouts) followed by online RL with dense step-level rewards (an adapted DAPO variant with balanced Stack/NLG groups). They also introduce the Composite Behavioral Score (CBS), a rubric-based metric that exposes procedural violations missed by outcome-centric metrics. Validation on τ³-Bench (Retail, Airline) and MultiWOZ across six backbones shows substantial CBS gains (e.g., +6.7 on Retail over the best baseline with Qwen3-8B), robustness to controlled input representation and prompting constraints (Table 10), a human agreement study, an alternative-judge analysis, and appendix studies on goal extractors and data filtering. The authors additionally analyze "illusory successes" — task-successful dialogues that still contain simulator violations (Section 6.2).
Critical strengths
- The motivating insight is correct and important: task success alone is not evidence of simulation quality. The paper articulates clearly that a simulator can help an agent succeed by leaking information, relaxing constraints, or omitting goals, so outcome-centric metrics can certify a false evaluation signal. This reframing justifies the whole neuro-symbolic agenda and the CBS metric, and it is consequential because it challenges how the community currently evaluates user simulators — a simulator that "helps too much" is worse than an obviously weak one. The human agreement study and alternative-judge analysis lend credibility to CBS as more than a self-defined metric.
- The empirical case is unusually well-controlled for this literature. The improvements on Retail are large and consistent across six backbones, and — unlike most prompt-strategy comparisons — the authors control for structured inputs, prompting constraints, goal extractors, and data filtering in the appendix, so the gains cannot be dismissed as better prompting. The training pipeline (evaluator-guided SFT + online RL) that distills the architecture into 4B/8B models while advancing the cost-quality Pareto frontier is a practical contribution with clear deployment value. These controls are what make the central claim (that explicit agenda state tracking improves simulator reliability) believable, and they are the main reason I find the paper promising despite my concerns.
Critical weaknesses
- The gains are not attributed to the LIFO stack, which is the paper's core architectural claim. ARGUS bundles several changes at once — two-stage generation, external state, local update operations, and LIFO scheduling — and none of the experiments isolates the stack from the rest. A controller with the same external state and local operations but unrestricted selection among pending tasks might achieve similar results, in which case the contribution lies in constrained state updates rather than stack structure. This also has an applicability dimension: real task-oriented conversations interleave transactions and agents ask about goals not at the top of the stack, so the necessity of strict LIFO is an empirical question the paper never tests. Since the stack is the headline contribution, the absence of a non-LIFO control with other components held fixed leaves the paper's central design choice unsupported by its own evidence.
- The evaluation interpretation overreaches: CBS measures procedural compliance, but is treated as evidence of behavioral realism and of distorted agent evaluation. Section 6.2 labels every successful dialogue with any CBS violation an "illusory success," yet the violations include things like an incorrectly formatted termination marker that plausibly did not help the agent at all — the analysis establishes that successful dialogues can contain violations, not that those violations caused or enabled success. Similarly, the Table 14 example penalizes a user for not restating an expensive-hotel preference before choosing the cheaper option — a script-adherence failure at most, not evidence of unrealistic behavior. Relatedly, RL trades errors rather than uniformly improving reliability: Goal Completion drops from 80.1 (SFT, 20k) to 74.9 (RL) (Appendix K), which deserves prominent discussion given that reducing goal omission is a central motivation. These issues are consequential because the paper's framing stakes its contribution on improving "behavioral fidelity" and agent evaluation quality; without separating violation types by their causal impact on task difficulty, the strength of that claim is not established.
Questions for authors
Prompt 9 instructs the controller to use modify_nearest_main_task, while Prompt 3 (the evaluator) checks modify_top_task and marks modifications to a non-top frame as incorrect. These are not equivalent when a temporary subtask sits on top of its main task. This may be an outdated appendix entry, but it determines which transitions are kept for SFT and rewarded during RL. Which tool definitions and evaluator prompts were actually used in the reported experiments, and could you provide a complete worked example of a temporary subtask with a conditional update to its main task? If this is only a documentation issue, a correction suffices; if not, the training signal itself needs re-examination.
Would a variant of ARGUS with the same external state and local update operations but unrestricted (non-LIFO) selection among pending frames achieve similar results? A single such control would establish whether the contribution lies in the stack structure itself or in constrained state updates more broadly. If the stack is essential, a small set of task-interleaving tests (where the agent asks about a goal that is not on top) would also demonstrate robustness of the strict top-frame priority to realistic dialogue patterns. Answers here directly determine whether the paper's headline architectural claim is supported.
RL improves Faithfulness and Policy Adherence but lowers Goal Completion from 80.1 to 74.9. Could you report premature-termination rates and the proportion of dialogues expressing all required goals for SFT vs. RL from the same checkpoint, with uncertainty estimates? Additionally, UGST-RL trains on pre-sampled interaction states while ARGUS-RL collects states from a live environment — is the reported difference attributable to the architecture or to the state-distribution/training-condition difference?
[1/2] Re to Reviewer ePUE
We sincerely thank the reviewer for the positive assessment of our work and the constructive feedback. We address the two concerns below.
Re to W1: Reliability of CBS
First, we would like to clarify that our training and evaluation do not use the same evaluator or metric. During SFT and RL, we use GPT-5.1 as a step-level evaluator, which assesses individual stack operations and response generation steps along three dimensions: Soundness, Faithfulness, and Adherence. In contrast, the final CBS evaluation uses Qwen3-MoE-235B to assess complete dialogues across 12 behavioral subcriteria. Although both are designed to encourage reliable user behavior, they differ in the judge model, evaluation granularity, and scoring criteria. Therefore, our training does not directly optimize the test-time CBS metric.
Regarding the reliability of CBS, we provide human validation results in Appendix E.2 (Table 7). We evaluated 90 dialogues with five expert annotators and found strong overall human–human agreement (Gwet's AC2 = 0.861) and human–Qwen3-MoE-235B agreement (Gwet's AC1 = 0.812). The overall results suggest that our primary judge aligns well with human annotations. We also tested an alternative judge, DeepSeek-V4.1-Flash, on 24 method–backbone combinations in Retail. The two judges show strong score correlations (Pearson = 0.873, Spearman = 0.886) and agree on 91.67% of method-pair rankings, with an aggregate Kendall's τ of 0.966. This suggests that our main conclusions are not particularly sensitive to the choice of judge.
We agree that a full-scale human evaluation would provide further evidence, but annotating all experimental dialogues would be very costly, especially given the large number of interactive rollouts. Our current human validation and cross-judge analysis provide complementary evidence for the reliability of CBS, although they cannot fully replace a direct human comparison of all methods.
Re to W2: Contribution of the LIFO Agenda Stack
We agree that ARGUS combines several design choices, and it is important to understand where the improvements come from. First, we have already included controlled experiments in Appendix G.2 (Table 10) to examine the effects of structured goal representations and additional prompt constraints. We augment Vanilla and ReAct with the same parsed goals used by ARGUS, matched behavioral instructions, or both. These changes improve the baselines in several settings, but ARGUS still achieves the highest CBS across all four backbone–domain combinations. These results suggest that structured inputs and more detailed prompts alone cannot explain ARGUS's improvements. However, they do not isolate the contribution of the LIFO constraint itself.
To investigate this more directly, we conducted an additional LIFO vs. Non-LIFO experiment on Retail using four Qwen3 backbones. The Non-LIFO variant retains the structured goal representation, two-stage generation, and symbolic state updates, but removes the restriction that operations must target the top task. Instead, the controller can modify or remove tasks at arbitrary positions and must explicitly select the current task focus. We kept the other implementation details, including the prompt structure and examples, as comparable as possible.
The results are shown below (Δ = Non-LIFO − LIFO).
Overall, the two variants achieve comparable performance, with LIFO showing a small average advantage in CBS and TSR. The overall CBS difference is not statistically significant (the bootstrap estimate gives a 95% CI of [−2.15, +0.17] pp). Non-LIFO performs better on some backbones, so we do not view LIFO as universally superior. We believe one advantage of LIFO is that the top frame naturally serves as the current task focus, allowing the model to track task progression without explicitly deciding which pending task to handle at each turn. In contrast, Non-LIFO provides more flexibility but also requires more complex task selection and lifecycle management. In our case analysis, we observed instances where Non-LIFO prematurely removed unfinished tasks or left completed tasks in the agenda, leading to repeated interactions. This may partly explain its lower Goal Completion score (87.72 vs. 89.30) and Format & Protocol score (91.80 vs. 94.30). The difference in Format & Protocol is statistically significant (−2.50 pp, 95% CI [−3.95, −1.05]).