- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces τ-Emotion-Bench, a benchmark for evaluating LLM agents under emotion-conditioned user interactions across ten tool-use domains, comprising 580 executable tasks. The benchmark uses five emotion families, 22 fine-grained emotions, and additional behavioral attributes to condition simulated users while preserving task objectives. The authors also propose Tracer, a task-generation and trajectory-selection pipeline that combines executable workflow validation with an interaction-aware success criterion. Experiments evaluate a broad range of frontier and smaller models, showing substantial performance gains after fine-tuning Qwen3 models with Tracer-generated data, as well as reported generalization to unseen domains and external benchmarks. Additional analyses investigate policy violations, tool use versus conversational follow-up, empathy-related behavior, and user-simulator hallucinations.
Critical strengths
- Practical and broad benchmark. τ-Emotion-Bench covers 580 executable tasks across ten domains and introduces emotion-conditioned user interactions, providing a useful testbed for evaluating agent reliability beyond task completion alone.
- Extensive empirical evaluation. The authors evaluate multiple model scales and test generalization to unseen domains and external benchmarks. The reported improvements from Tracer-based fine-tuning suggest practical potential for improving tool-using agents.
Critical weaknesses
- The benchmark does not convincingly isolate the impact of emotions. Although the paper claims to evaluate agent fragility under emotional user interactions, its main results do not clearly quantify the performance difference between the same tasks with and without emotion instructions. Without this controlled comparison, poor performance could result from task difficulty or other user behaviors rather than emotions themselves, leaving the paper's central claim insufficiently supported.
- The experiments do not establish that Tracer's proposed selection mechanism causes the performance gains. The authors compare Tracer-trained models with base models and existing methods, but do not adequately isolate the effect of interaction-aware trajectory filtering from ordinary supervised fine-tuning and training-data composition. Matched-data comparisons and ablations are needed to demonstrate that the proposed mechanism, rather than simply training on additional successful trajectories, explains the improvements.
Questions for authors
-
Emotion effects: Did the authors evaluate the same tasks with and without emotion instructions under otherwise identical conditions? If so, could the paired performance differences be reported? If not, what evidence supports attributing agent failures specifically to emotional conditioning rather than task difficulty or other behavioral factors?
-
Training-data accounting: How many AReaL trajectories were used alongside the 1,498 and 1,958 Tracer trajectories, and were these additional data included when reporting training-data efficiency? Given that AReaL is also included as a baseline, how can the comparison establish Tracer's advantage under comparable data conditions?
-
Tracer ablation: Could the authors provide controlled ablations comparing (i) successful-trajectory fine-tuning without interaction-aware filtering and (ii) the full Tracer pipeline, using matched training data and configurations? This is necessary to establish whether the
###UNSATISFIED###signal provides benefits beyond ordinary supervised fine-tuning and the additional AReaL data.
3 comments
All my concerns have been resolved, and I am raising my rating from 2 to 3.
We sincerely appreciate your acknowledgment and are deeply encouraged by your decision to raise our rating. It is our honor to address your concerns, which have been helpful to our work and will be integral to the improvements in our final version.
Response to Reviewer AnkE
We thank the reviewer for the constructive feedback. We ran the requested paired comparison and ablation, and we clarify the training-data accounting below.
W1, Q1: Isolating the impact of emotion
We evaluated the same test tasks with a neutral user, using the emotion-free variant of each task. Results are Pass^1 / Pass^3 (%):
Removing only the emotion instruction raises Pass^1 by 6.6–11.5 points and Pass^3 by 6.3–16.8 for every model. Since the tasks are identical, the gap cannot come from task difficulty; it isolates the effect of emotion and supports the paper's central claim.
W2, Q3: Isolating the effect of interaction-aware filtering
We regenerated the training data using the same pipeline, removing the user's emotion instruction and the ###UNSATISFIED### termination. In the original pipeline, dissatisfied termination assigns reward −1, and only reward +1 trajectories are retained for training. The ablation therefore evaluates the combined contribution of emotional conditioning and interaction-aware termination against conventional task-success-based training.
Training with the signal outperforms training without it in all 8 comparisons, improving Pass^1 by 1.6–4.5 points and Pass^3 by 2.8–4.6. These results show that the emotion-conditioned interaction signal improves agent performance.
Q2: AReaL data accounting and comparable data conditions
No AReaL data are used to train Tracer models. Tracer models are trained only on Tracer data, and the AReaL baseline only on the public AReaL data. We will correct the ambiguous wording in Appendix C.
Our data-efficiency comparison is therefore unchanged: Tracer-Low uses 1,498 trajectories, compared with 9,000 for EnvScaler and 91,204 for Simia-Tau-SFT.
Comparison with AReaL. AReaL data are generated in the same environments as our in-domain tasks, and neither AReaL nor Tracer-Low is trained on the six OOD domains. With fewer trajectories and samples than AReaL, Tracer-Low clearly outperforms it on these domains (Table 4). Results are Pass^1 / Pass^3 (%):
Comparison with other generation methods. AReaL's data-generation pipeline is not public. We regenerated 2k trajectories each with Simia and EnvScaler using their official repositories and the same teacher model as our pipeline (Qwen3.8-27B), and fine-tuned the same base models with the same configuration. Each baseline receives more data than Tracer-Low (2k vs. 1,498). Results are Pass^1 / Pass^3 (%) on all ten domains:
With more training data than Tracer-Low, the baselines remain far below Tracer at every scale.