- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
NeedTracer tracks dynamic user needs in multi-turn emotional support by distinguishing within-stage need updates (Keep/Deepen/Correct) from cross-stage transitions, which require sufficient need satisfaction and a consistently supported new focus anchor over two consecutive turns. It maintains global and current user states, generates preference-constrained and free-form responses, and selects them via a Theory-of-Mind-based evaluator. Experiments show gains in support effectiveness, interaction efficiency, and emotional trajectory metrics, supported by human evaluations and ablations.
Critical strengths
-
The paper tackles a real challenge in emotional support: telling a need update apart from a genuine switch to a new need.
-
NeedTracer offers a structured way to track changing user needs and adjust support responses accordingly.
-
The paper tests multiple LLM backbones and benchmarks, and ablation studies plus human evaluations back up the method's effectiveness.
Critical weaknesses
-
The paper doesn't show clearly what the focus anchor adds on its own. The gains could come from extra reasoning steps and longer prompts. A compute-matched comparison without focus anchors would help.
-
Many key decisions depend on LLM judges, which are subjective. We need more on cross-model agreement, false transitions, and missed transitions.
-
Waiting two turns may slow down real transitions. The system may keep addressing an old concern. More analysis of transition delay and different thresholds would help.
-
Better responses don't prove the need tracking is accurate. A dedicated test with annotated stages and transitions, reporting precision, recall, F1, and detection delay, is needed.
-
The compute cost is not well reported. Fewer turns doesn't mean cheaper. LLM calls, tokens, latency, and cost vs. baselines should be given.
-
Results may not generalize. Simulated users and LLM evaluators can bias results, and three seeds don't show significance. Real-user or independent data, paired significance tests, and evaluator-bias analysis would help.
Questions for authors
N/A
2 comments
References
[1] Bohus, D., & Rudnicky, A. I. (2005). Constructing Accurate Beliefs in Spoken Dialog Systems. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU 2005). Paper.
[2] Williams, J., Raux, A., Ramachandran, D., & Black, A. (2013). The Dialog State Tracking Challenge. Proceedings of SIGDIAL 2013, 404–413. Paper.
[3] den Hengst, F., Wolter, R., Altmeyer, P., & Kaygan, A. (2024). Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition. Findings of ACL: NAACL 2024, 2412–2432. Paper.
We thank the reviewer. NeedTracer anchors the highest-priority unresolved concern to distinguish Need Identification Uncertainty from Genuine Need Transition caused by satisfaction or priority change, stabilizing current user need tracking.
Q1: What does focus anchor add beyond extra reasoning?
A1: Independent contribution. Existing emotional support datasets lack comprehensive annotations for evaluating NeedTracer's ability to distinguish Need Identification Uncertainty from Genuine Need Transition and continuously track the current user need through a focus anchor in multi-turn interactions. Therefore, we select 300 dialogues (8–30 turns) from ESConv, ETrajEval, and ExTES, and annotate ground truth (GT) and supporting evidence for focus anchors, current user needs, and need-stage transitions to evaluate the independent contribution of the focus anchor and dynamic need tracking (Table R1).
w/o focus anchor retains stages, current/global states, Keep/Deepen/Correct, and two-turn confirmation, removing anchor/priority constraints. The base model, candidates, ToM reranking, memory, and compute ceiling are fixed.
Table R1. 300 dialogues; mean ± std over 10 seeds (Need Acc./Sat. in %).
Versus w/o anchor: +11.7% Need Acc., +20.8% Need Sat. (absolute); 28.2% fewer transitions.
Q2: How reliable are judges and transition detection?
A2: Human agreement and errors. On 150 cases, Claude Opus 5.5, Gemini 4 Argon, and DeepSeek-V4-Pro are compared with three-annotator majority labels.
Table R2. LLM–human agreement (%); 150 cases.
For transition testing, 200 independent dialogues (8–20 turns) cover 100 Satisfaction-driven (S) and 100 Priority-driven Override (O) cases, with three-annotator labels and expert adjudication.
Table R3. Transition P/R/F1 (%); 200 separate dialogues.
False-discovery (1−P)/miss (1−R) rates: 18.3%/16.6% (S) and 19.7%/17.1% (O).
Q3: Does two-turn confirmation delay transitions?
A3: Stable confirmation and override. Studies support feedback-based belief updates (Bohus & Rudnicky, 2005), multi-turn goal tracking (Williams et al., 2013), and intent clarification (den Hengst et al., 2024). Two consecutive user-feedback turns filter isolated mentions and within-concern Keep/Deepen/Correct changes. Satisfaction-driven Transition follows sufficient support and a new concern; Priority-driven Override updates the anchor when evidence and feedback confirm a higher-priority concern despite unmet needs. Unmet needs and anchors remain in long-term memory for future re-confirmation if priority returns. Revised Algorithm 1 adds Override; O-F1: 74.1%→81.6% (Δ=7.5%). Future work examines W=1/2/3, detection delay, and interrupted-need recovery.
Q4: Is need-stage tracking independently evaluated?
A4: Dedicated transition test. A need stage spans a focus anchor’s valid interval; Keep/Deepen/Correct updates remain within it. On R3’s 200 independent dialogues, cross-stage F1 reaches 82.5% (S)/81.6% (O), +9.7%/+10.4% over w/o focus anchor, independent of response quality.
Q5: What is the compute cost?
A5: Measured overhead. On R1’s 300 dialogues, Full adds 0.2 s RT, 1.1 calls, 28 tokens/turn versus w/o anchor. The +11.7% Need Acc./+20.8% Need Sat. gains justify this modest overhead for better need tracking.
Q6: How robust are findings across data, judges, and seeds?
A6: Multi-source validation. R1: 300 ESConv/ETrajEval/ExTES dialogues, 10 seeds; R2: 150 judge–human agreement cases, three model families; R3: 200 independent human-adjudicated transition dialogues. Across seeds: 78.6%±0.3% Need Acc., 82.6%±1.0% Need Sat., 5.6±0.39 transitions. Future work extends real-user and long-dialogue evaluation, paired tests, and evaluator sensitivity.