acceptodds
Under review as a conference paper at ICLR 2027

-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions

Abstract

In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users. Yet existing benchmarks largely assume cooperative and behaviorally stable users, leaving the effect of emotional variation on agent reliability underexplored. We introduce -Emotion-Bench, a benchmark for evaluating tool-calling agents under emotion-conditioned user interactions across ten domains. To construct the benchmark at scale, we introduce Tracer, a scalable agentic data-generation pipeline that constructs policy-valid tool workflows, grounds them into executable tasks, incorporates user preferences and task-specific emotional context, and verifies task validity against the environment. The resulting tasks are mapped to a structured emotion space and paired with emotion-conditioned user simulators, enabling controlled evaluation while preserving the underlying task semantics. Experiments show that frontier models remain unreliable under these interactions, with substantial variation in task success and consistency across user conditions. Beyond evaluation, we use Tracer-generated data to fine-tune models across scales, resulting in consistent improvements on -Emotion-Bench and strong generalization to unseen domains, external benchmarks, and different user simulators. Overall, -Emotion-Bench and Tracer provide a scalable framework for evaluating agent reliability under diverse user behaviors and improving tool-calling agents through transferable supervision.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Tool-Use Agents, User Simulation, Agent Evaluation, Synthetic Data Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AnkEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces τ-Emotion-Bench, a benchmark for evaluating LLM agents under emotion-conditioned user interactions across ten tool-use domains, comprising 580 executable tasks. The benchmark uses five emotion families, 22 fine-grained emotions, and additional behavioral attributes to condition simulated users while preserving task objectives. The authors also propose Tracer, a task-generation and trajectory-selection pipeline that combines executable workflow validation with an interaction-aware success criterion. Experiments evaluate a broad range of frontier and smaller models, showing substantial performance gains after fine-tuning Qwen3 models with Tracer-generated data, as well as reported generalization to unseen domains and external benchmarks. Additional analyses investigate policy violations, tool use versus conversational follow-up, empathy-related behavior, and user-simulator hallucinations.

Critical strengths
  1. Practical and broad benchmark. τ-Emotion-Bench covers 580 executable tasks across ten domains and introduces emotion-conditioned user interactions, providing a useful testbed for evaluating agent reliability beyond task completion alone.
  2. Extensive empirical evaluation. The authors evaluate multiple model scales and test generalization to unseen domains and external benchmarks. The reported improvements from Tracer-based fine-tuning suggest practical potential for improving tool-using agents.
Critical weaknesses
  1. The benchmark does not convincingly isolate the impact of emotions. Although the paper claims to evaluate agent fragility under emotional user interactions, its main results do not clearly quantify the performance difference between the same tasks with and without emotion instructions. Without this controlled comparison, poor performance could result from task difficulty or other user behaviors rather than emotions themselves, leaving the paper's central claim insufficiently supported.
  2. The experiments do not establish that Tracer's proposed selection mechanism causes the performance gains. The authors compare Tracer-trained models with base models and existing methods, but do not adequately isolate the effect of interaction-aware trajectory filtering from ordinary supervised fine-tuning and training-data composition. Matched-data comparisons and ablations are needed to demonstrate that the proposed mechanism, rather than simply training on additional successful trajectories, explains the improvements.
Questions for authors
  1. Emotion effects: Did the authors evaluate the same tasks with and without emotion instructions under otherwise identical conditions? If so, could the paired performance differences be reported? If not, what evidence supports attributing agent failures specifically to emotional conditioning rather than task difficulty or other behavioral factors?

  2. Training-data accounting: How many AReaL trajectories were used alongside the 1,498 and 1,958 Tracer trajectories, and were these additional data included when reporting training-data efficiency? Given that AReaL is also included as a baseline, how can the comparison establish Tracer's advantage under comparable data conditions?

  3. Tracer ablation: Could the authors provide controlled ablations comparing (i) successful-trajectory fine-tuning without interaction-aware filtering and (ii) the full Tracer pipeline, using matched training data and configurations? This is necessary to establish whether the ###UNSATISFIED### signal provides benefits beyond ordinary supervised fine-tuning and the additional AReaL data.

3 comments
AuthorsOpenReview ↗

Response to Reviewer AnkE

We thank the reviewer for the constructive feedback. We ran the requested paired comparison and ablation, and we clarify the training-data accounting below.


W1, Q1: Isolating the impact of emotion


We evaluated the same test tasks with a neutral user, using the emotion-free variant of each task. Results are Pass^1 / Pass^3 (%):

AgentEmotional userNeutral userΔ (emotional − neutral)
Qwen3.8-27B50.4 / 31.461.9 / 48.2−11.5 / −16.8
Qwen3-235B-A22B-Thinking-250741.8 / 21.952.6 / 35.4−10.8 / −13.5
Qwen3-235B-A22B-Instruct-250741.7 / 20.549.4 / 31.1−7.7 / −10.6
Muse-Glimmer-30B42.9 / 30.349.5 / 37.8−6.6 / −7.5
MiniMax-M2.746.5 / 26.456.8 / 39.6−10.3 / −13.2
Qwen3-8B29.4 / 13.138.1 / 19.4−8.7 / −6.3
Tracer-High-8B49.1 / 29.958.6 / 41.3−9.5 / −11.4
Tracer-Low-8B37.4 / 20.345.0 / 28.5−7.6 / −8.2

Removing only the emotion instruction raises Pass^1 by 6.6–11.5 points and Pass^3 by 6.3–16.8 for every model. Since the tasks are identical, the gap cannot come from task difficulty; it isolates the effect of emotion and supports the paper's central claim.


W2, Q3: Isolating the effect of interaction-aware filtering


We regenerated the training data using the same pipeline, removing the user's emotion instruction and the ###UNSATISFIED### termination. In the original pipeline, dissatisfied termination assigns reward −1, and only reward +1 trajectories are retained for training. The ablation therefore evaluates the combined contribution of emotional conditioning and interaction-aware termination against conventional task-success-based training.

ModelPass^1 with / without signalPass^3 with / without signal
Tracer-Low-1.7B24.1 / 20.011.1 / 6.5
Tracer-Low-4B33.8 / 29.318.0 / 14.0
Tracer-High-1.7B31.0 / 29.415.2 / 12.4
Tracer-High-4B45.2 / 42.623.9 / 21.0

Training with the signal outperforms training without it in all 8 comparisons, improving Pass^1 by 1.6–4.5 points and Pass^3 by 2.8–4.6. These results show that the emotion-conditioned interaction signal improves agent performance.


Q2: AReaL data accounting and comparable data conditions


No AReaL data are used to train Tracer models. Tracer models are trained only on Tracer data, and the AReaL baseline only on the public AReaL data. We will correct the ambiguous wording in Appendix C.

Training setTrajectoriesTraining samples
AReaL (public)2,49933,531
Tracer-High1,95832,740
Tracer-Low1,49821,401

Our data-efficiency comparison is therefore unchanged: Tracer-Low uses 1,498 trajectories, compared with 9,000 for EnvScaler and 91,204 for Simia-Tau-SFT.

Comparison with AReaL. AReaL data are generated in the same environments as our in-domain tasks, and neither AReaL nor Tracer-Low is trained on the six OOD domains. With fewer trajectories and samples than AReaL, Tracer-Low clearly outperforms it on these domains (Table 4). Results are Pass^1 / Pass^3 (%):

ModelBaseAReaLTracer-Low
Qwen3-1.7B11.0 / 1.75.9 / 0.719.1 / 8.3
Qwen3-4B23.9 / 8.315.7 / 4.733.1 / 18.0
Qwen3-8B30.1 / 13.318.0 / 3.339.9 / 20.7

Comparison with other generation methods. AReaL's data-generation pipeline is not public. We regenerated 2k trajectories each with Simia and EnvScaler using their official repositories and the same teacher model as our pipeline (Qwen3.8-27B), and fine-tuned the same base models with the same configuration. Each baseline receives more data than Tracer-Low (2k vs. 1,498). Results are Pass^1 / Pass^3 (%) on all ten domains:

ModelTracer-LowSimiaEnvScaler
Qwen3-1.7B24.1 / 11.15.9 / 0.34.5 / 0.7
Qwen3-4B33.8 / 18.05.6 / 1.09.5 / 2.3
Qwen3-8B37.4 / 20.313.6 / 3.528.7 / 11.1

With more training data than Tracer-Low, the baselines remain far below Tracer at every scale.

All my concerns have been resolved, and I am raising my rating from 2 to 3.

We sincerely appreciate your acknowledgment and are deeply encouraged by your decision to raise our rating. It is our honor to address your concerns, which have been helpful to our work and will be integral to the improvements in our final version.

Reviewer_rv3wOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces τ-Emotion-Bench, 580 tool-calling tasks across ten domains in which the simulated user expresses a conditioned emotion (five families, 22 fine-grained emotions and three behavioral dimensions). The simulator may terminate the interaction as unsatisfied, which yields a -1 reward. The authors also present Tracer, a pipeline that generates tasks (tool dependency graphs, environment grounding, user preferences, task-specific feelings mapped to the emotion space) and training trajectories, retaining only trajectories that succeed without user dissatisfaction. Frontier models reach at most 53.1% average success and 35.2% Pass^3. Fine-tuning Qwen3 1.7B/4B/8B on 1,958 trajectories (all domains) or 1,498 (four domains) improves average success by roughly 20 points, outperforms several agent-training baselines on unseen domains, and transfers to τ-Bench and τ-Trait.

Critical strengths
  1. The pipeline is scalable and the quality controls are reasonable. The hallucination audit (63 confirmed errors among 378 flagged user turns, 0.6% of failed trials) is a valuable reliability analysis.
  2. The practical result is attractive: a small training set (1.5–2k trajectories) has substantial gains for 1.7B–8B models, with transfer to unseen domains, an external benchmark and different simulators. The Tracer-Low/Tracer-High split is a well-designed way to separate in-domain from out-of-domain effects.
Critical weaknesses

Several comparisons may be better for supporting the proposed method. The same 235B simulator generates training data and serves as the evaluation simulator, and in 4 domains training and evaluation tasks share an environment database. Baselines such as EnvScaler and Simia differ in teacher, data scale and setting, so the data-efficiency argument is not a controlled comparison. Evaluation tasks were also filtered using the same 235B model, which may bias the task set. The hallucination audit examines only flagged turns in failed trajectories, so missed errors are unmeasured, and no seeds or confidence intervals are reported for the fine-tuned models

Questions for authors
  1. Do you have results for the same 580 tasks with no emotion instruction, and with a different emotion per task? If not having them, what supports the claim that emotion causes the drop?
  2. Can you do ablation study: Tracer-High with success-only filtering and report the gap, including how many trajectories the unsatisfied rule removes?
  3. For the baselines in Table 4, which agent and user simulator did you use, and did you retrain them on the same teacher data or take published values?
  4. Do the results work with a different user-simulator family for all models, not only in the transfer tests?
  5. In future, pls report seeds or confidence intervals for the fine-tuned models.
4 comments
AuthorsOpenReview ↗

Response to Reviewer rv3w (1/2)

We thank the reviewer for the positive and detailed assessment. We address each point with new experiments and clarifications below.


W1. Q4. Same simulator for training and evaluation


We re-evaluated a subset of the agents with a user simulator from a different model family (GLM-5.2). The remaining models will be reported in the camera-ready version. Results are Pass^1 / Pass^3 (%):

AgentQwen3-235B user (paper)GLM-5.2 user
Qwen3.8-27B50.4 / 31.470.2 / 60.6
Tracer-High-8B49.1 / 29.963.5 / 48.5
MiniMax-M2.746.5 / 26.463.2 / 49.2
Muse-Glimmer-30B42.9 / 30.356.2 / 48.2
Qwen3-235B-A22B-Thinking-250741.8 / 21.958.5 / 41.7
Qwen3-235B-A22B-Instruct-250741.7 / 20.557.3 / 40.3
Tracer-Low-8B37.4 / 20.348.0 / 31.0
Qwen3-8B29.4 / 13.144.4 / 24.8

Agent rankings remain consistent across simulator families (Spearman's ρ = 0.93 for Pass^1, 0.90 for Pass^3), and both Tracer models still outperform their Qwen3-8B base under the GLM-5.2 user.


W1. Shared environment databases


In the four in-domain domains, training and test tasks use the same database file, but no training task shares its ground-truth actions with a test task. The six OOD domains use separate databases, and Tracer-Low, trained only on the four shared-database domains, still generalizes to them (Table 4). Its gains do not depend on the shared databases.


W1. Q3. Baseline comparisons


None of the Table 4 results are published values. We evaluated every baseline ourselves with the suitablef agent harness and the same Qwen3-235B-A22B-Instruct-2507 user simulator. AReaL models were trained on the public AReaL data with our hyperparameters (Appendix C); the other baselines use their officially released checkpoints.

To control for teacher and data scale, we regenerated 2k trajectories each with Simia and EnvScaler using their official repositories and our teacher model (Qwen3.8-27B), and fine-tuned the same base models with the same configuration. Each baseline receives more data than Tracer-Low (2k vs. 1,498). Results are Pass^1 / Pass^3 (%) on all ten domains:

ModelTracer-LowSimiaEnvScaler
Qwen3-1.7B24.1 / 11.15.9 / 0.34.5 / 0.7
Qwen3-4B33.8 / 18.05.6 / 1.09.5 / 2.3
Qwen3-8B37.4 / 20.313.6 / 3.528.7 / 11.1

With the same teacher and more training data, the baselines remain far below Tracer at every scale.


W2. Task-selection bias


Candidate tasks for both the training and test sets are generated and verified by an agentic pipeline built on Qwen3-235B-A22B-Instruct-2507; all 580 test tasks were additionally reviewed manually (Section 3.4). To test the verifier for bias, an annotator blindly re-judged 60 randomly sampled generated tasks, stratified across all 10 domains, three task-length bands and the verifier's accept/reject decisions, using the verifier's four checks (goal orientation, template, solvability and domain violation). Agreement between the verifier and the annotator on the final accept/reject decision is 83% (Cohen's κ = 0.67), with no significant bias:

Possible biasVerifier vs. humanTestp
Direction (rejects valid vs. accepts invalid)6 vs. 4 errorsMcNemar0.75
Task length (short / medium / long)90 / 80 / 80% agreementχ² permutation0.75
Domain (10 domains)67–100% agreementχ² permutation1.00

We observed no evidence that the verifier is stricter or more lenient than the human, or that its errors depend on task length or domain. The test set also does not favor the verifier's backbone model: Qwen3-235B-A22B-Instruct-2507 reaches only 41.7% Avg@3, below 10 of the 18 evaluated models (Figure 1).


W2. Hallucination audit coverage


Three annotators independently re-checked our manual hallucination labels on 50 sampled failed trajectories. They agreed with all confirmed hallucinations and with 80% (73–85) of rejected flags. Even counting every disputed rejection as a hallucination, the rate stays below 2% of failed trials. We audit failed trajectories because a simulator hallucination can only affect the score by causing a false failure. Unflagged turns were not audited, and we will state this as a limitation.

AuthorsOpenReview ↗

Response to Reviewer rv3w (2/2)

W2. Q5. Seeds and confidence intervals


Every model is evaluated over three runs with fixed seeds (361454, 373752, 626729), and Pass^3 reports consistency across them. For training variance, we fine-tuned Tracer models with three training seeds. Results are Pass^1 / Pass^3 (%):

ModelSeed 20260802 (paper)Seed 1234Seed 5678Mean ± std
Tracer-Low-1.7B24.1 / 11.121.3 / 8.923.3 / 10.822.9 ± 1.4 / 10.3 ± 1.2
Tracer-High-1.7B31.0 / 15.231.3 / 12.732.4 / 15.531.6 ± 0.7 / 14.5 ± 1.5

Every seed outperforms the Qwen3-1.7B base (10.7 / 2.0) by more than 10 points on Pass^1, so the gains are far larger than the seed variance.


Q1. Effect of emotion

We evaluated the same test tasks with a neutral user, using the emotion-free variant of each task. Results are Pass^1 / Pass^3 (%):

AgentEmotional userNeutral userΔ (emotional − neutral)
Qwen3.8-27B50.4 / 31.461.9 / 48.2−11.5 / −16.8
Qwen3-235B-A22B-Thinking-250741.8 / 21.952.6 / 35.4−10.8 / −13.5
Qwen3-235B-A22B-Instruct-250741.7 / 20.549.4 / 31.1−7.7 / −10.6
Muse-Glimmer-30B42.9 / 30.349.5 / 37.8−6.6 / −7.5
MiniMax-M2.746.5 / 26.456.8 / 39.6−10.3 / −13.2
Qwen3-8B29.4 / 13.138.1 / 19.4−8.7 / −6.3
Tracer-High-8B49.1 / 29.958.6 / 41.3−9.5 / −11.4
Tracer-Low-8B37.4 / 20.345.0 / 28.5−7.6 / −8.2

On the same tasks, every model scores 6.6–11.5 points lower in Pass^1 and 6.3–16.8 lower in Pass^3 with the emotional user. Since the task is unchanged, this gap measures robustness to emotional users, beyond general tool-use competence.

Regarding a different emotion per task: given limited compute during the rebuttal period, this experiment is currently running, and we will share the results as soon as it completes.


Q2. Success-only ablation


Tracer trains only on reward +1 trajectories, which complete the task without ###UNSATISFIED###. 526 training rollouts end with ###UNSATISFIED### (Table 5), 351 of them in Tracer-Low’s four domains. For the ablation, We regenerated the data with the same pipeline and removing the user’s emotion instruction and the ###UNSATISFIED### termination, so trajectories are selected by task success alone:

ModelPass^1 with / without signalPass^3 with / without signal
Tracer-High-1.7B31.0 / 29.415.2 / 12.4
Tracer-High-4B45.2 / 42.623.9 / 21.0
Tracer-Low-1.7B24.1 / 20.011.1 / 6.5
Tracer-Low-4B33.8 / 29.318.0 / 14.0

Training with the signal outperforms training without it in all 8 comparisons, improving Pass^1 by 1.6–4.5 points and Pass^3 by 2.8–4.6. These results show that the emotion-conditioned interaction signal improves agent performance.

AuthorsOpenReview ↗

Follow up Q1. Different Emotions per Task

Following our previous response, we completed the additional experiment requested by the reviewer. We evaluated the same 580 benchmark tasks under three conditions: (1) without emotion, (2) with a randomly assigned emotion from the five emotion families, and (3) with the task-relevant emotion selected by our proposed method. The underlying tasks, environments, and success criteria remain unchanged.

Results are reported as Pass^1 / Pass^3 (%):

AgentW/o emotionW/ random emotionW/ relevant emotion
Qwen3.8-27B61.9 / 48.253.3 / 35.250.4 / 31.4
MiniMax-M2.756.8 / 39.649.5 / 30.346.5 / 26.4
Muse-Glimmer-30B49.5 / 37.844.5 / 31.842.9 / 30.3
Qwen3-A22B-235B-Instruct-250749.4 / 31.141.0 / 21.741.7 / 20.5
Qwen3-A22B-235B-Thinking-250752.6 / 35.444.1 / 25.741.8 / 21.9
Tracer-High-8B58.6 / 41.350.5 / 30.649.1 / 29.9
Tracer-Low-8B45.0 / 28.537.8 / 20.337.4 / 20.3
Qwen3-8B38.1 / 19.429.6 / 13.329.4 / 13.1

Both random and task-relevant emotions consistently reduce performance compared to the emotion-free setting. Task-relevant emotions generally produce larger performance drops, particularly in Pass^3. These results demonstrate that emotional conditioning affects agent reliability even when the underlying tasks remain unchanged.

Dear Reviewer rv3w,

As the deadline is nearing, we wanted to gently follow up on our recent submission. Your feedback is highly valuable to us, and we would appreciate any updates or further guidance you might have regarding our revisions and responses.

Thank you for your time and consideration.

Reviewer_XFyMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces τ\tau-Emotion-Bench, a benchmark for evaluating tool-calling agents under emotionally diverse user interactions. It systematically varies user emotions, urgency, politeness, and trust while keeping the underlying task fixed. The authors also propose Tracer, a scalable data-generation and training framework that uses emotion-conditioned user simulators to reward efficient interactions and penalize excessive dependence on users. The benchmark contains 580 executable tasks across 10 domains, and experiments show that even frontier models remain fragile under emotional variation. Training with Tracer-generated trajectories substantially improves smaller models and generalizes to unseen domains and external benchmarks.

Critical strengths
  • Identifies an important and understudied dimension of agent reliability by evaluating performance under emotionally varied user interactions rather than assuming cooperative users.
  • Introduces a large-scale benchmark with 580 executable tasks across 10 domains, substantially increasing the diversity and realism of interaction-based agent evaluation.
  • Demonstrates strong empirical findings that emotionally conditioned interactions significantly impact agent performance and consistency, even for frontier models.
  • Shows consistent gains from Tracer training across multiple model scales and evidence of transfer to unseen domains and external evaluation benchmarks.
Critical weaknesses
  • The benchmark relies heavily on an LLM-based user simulator and interaction termination mechanism, making many conclusions dependent on the fidelity and biases of the simulator rather than real human behavior. Most analyses, including failure categorization and simulator validation, use LLM judges, introducing potential evaluator bias and circularity in the conclusions.
  • The central assumption that emotional variation meaningfully captures real-world user behavior is only weakly validated, with limited evidence that the synthetic emotions accurately reflect actual users interacting with agents.
  • The interaction-aware reward primarily penalizes user dissatisfaction and excessive questioning, but it remains unclear whether this reward aligns with human preferences across different domains and situations.
  • The benchmark measures emotional robustness through synthetic emotional prompts rather than genuine emotional responses emerging from real interactions, potentially limiting ecological validity.
  • Tracer training benefits may stem from additional high-quality supervised data rather than the proposed emotion-conditioned interaction signal itself, and the paper does not fully disentangle these effects.
  • While transfer results are promising, comparisons against competing methods may not be entirely controlled because training datasets, generation pipelines, and supervision signals differ substantially.
Questions for authors
  • How well do the emotion-conditioned user simulators reflect real human behavior, and have the authors validated simulator responses against human interaction data?
  • How sensitive are the benchmark rankings and Tracer improvements to the choice of user simulator model and prompting strategy?
  • Can the authors provide human evaluation demonstrating that the dissatisfaction signal and conversation termination behavior correlate with actual user satisfaction?
  • How much of the Tracer improvement comes from interaction-aware supervision versus simply training on additional successful trajectories?
  • Would the conclusions remain consistent if emotional conditioning were replaced with real user profiles or interaction logs instead of synthetic emotional instructions?
  • Since emotional neglect accounts for a relatively small fraction of observed failures, does the benchmark primarily measure emotional robustness or general tool-use competency under increased interaction complexity?
  • What are the main failure cases where emotionally aware behavior actually changes task outcomes compared to neutral user interactions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer XFyM (1/2)

We thank the reviewer for the careful assessment. To address concerns about LLM judgments, we ran four human studies (3 annotators each), and we added five new experiments for the remaining points.


W2, W4, Q1, Q5: Do simulated users behave like real users?


Annotators independently rated user messages from 80 randomly sampled trajectories (20/30/30 with reward 1/0/−1). They also rated 66 randomly sampled emotion instructions, 3 for each of the 22 sub-emotions.

CheckMean across annotators (range)
Human-likeness of simulator messages (1–5)4.29 (3.98–4.79)
Instruction expresses the target emotion96% (95–98)
Instruction realizes the specified politeness, urgency and trust levels82% (79–86)

The simulator produces human-like messages, and its instructions express the specified emotion. We will list human-validated simulation, rather than real interaction logs, explicitly as a limitation.


W1: Bias and circularity in LLM-judged analyses


Annotators independently checked the judge’s failure categories on 60 randomly sampled failed trajectories.

Judge decisionHuman agreement, mean (range)
Failure category, overall76% (73–77)
Incomplete88%
Missing / extra action79%
Integrity75%
Policy & interaction71%
Wrong tool call67%

The judge’s failure labels largely match human judgment. For hallucinations, the counts in Section 5 are not LLM decisions. Every flagged user turn is manually verified, and only manually confirmed cases are counted.


W3, Q3: Does early termination reflect real dissatisfaction?


We sampled 60 trajectories that ended with reward −1. Annotators read each one in the user's position. 76% (72–81) said they would also have ended the conversation because of the agent's actions.


Q2: Sensitivity to the simulator model and prompt


We re-evaluated a subset of the agents from the paper under a different user-simulator model (GLM-5.2) and a different simulator prompt (ReAct-style). The full set is running and will be reported in the appendix. Results are Pass^1 / Pass^3 (%):

AgentQwen3-235B (paper)GLM-5.2Qwen3-235B + ReAct prompt
Qwen3.8-27B50.4 / 31.470.2 / 60.654.7 / 35.8
Tracer-High-8B49.1 / 29.963.5 / 48.550.0 / 31.2
MiniMax-M2.746.5 / 26.463.2 / 49.248.2 / 29.9
Muse-Glimmer-30B42.9 / 30.356.2 / 48.246.3 / 33.8
Qwen3-235B-A22B-Thinking-250741.8 / 21.958.5 / 41.740.1 / 28.3
Qwen3-235B-A22B-Instruct-250741.7 / 20.557.3 / 40.342.1 / 21.3
Tracer-Low-8B37.4 / 20.348.0 / 31.040.4 / 23.5
Qwen3-8B29.4 / 13.144.4 / 24.833.1 / 15.5
Spearman’s rank correlation ρ vs. paperPass^1Pass^3
GLM-5.2 user0.930.90
ReAct-prompted user0.930.98

Agent rankings remain consistent across user simulators and prompts, with ρ ≥ 0.90 in all settings.


W5, Q4: Is the gain from the interaction signal or just more data?


We regenerated the data with the same pipeline and removing the user’s emotion instruction and the ###UNSATISFIED### termination:

ModelPass^1 with / without signalPass^3 with / without signal
Tracer-Low-1.7B24.1 / 20.011.1 / 6.5
Tracer-Low-4B33.8 / 29.318.0 / 14.0
Tracer-High-1.7B31.0 / 29.415.2 / 12.4
Tracer-High-4B45.2 / 42.623.9 / 21.0

Training with the signal outperforms training without it in all 8 comparisons, improving Pass^1 by 1.6–4.5 points and Pass^3 by 2.8–4.6. These results show that the emotion-conditioned interaction signal improves agent performance.

AuthorsOpenReview ↗

Response to Reviewer XFyM (2/2)

W6: Controlled baseline comparison


We regenerated training data for Simia and EnvScaler baselines from their official repository, using the same teacher model as our pipeline (Qwen3.8-27B), and fine-tuned the same Qwen3 base models with the same recipe (Appendix C). Each baseline receives 2k trajectories, more than Tracer-Low's 1,498. Results are Pass^1 / Pass^3 (%) on all ten domains:

ModelTracer-LowSimiaEnvScaler
Qwen3-1.7B24.1 / 11.15.9 / 0.34.5 / 0.7
Qwen3-4B33.8 / 18.05.6 / 1.09.5 / 2.3
Qwen3-8B37.4 / 20.313.6 / 3.528.7 / 11.1

With more training data than Tracer-Low and the same teacher and base models, the baselines remain far below Tracer at every scale.


Q6: Emotional robustness or general tool-use competence?


We reran the test tasks with a neutral user, using the emotion-free variant of each task. Results are Pass^1 / Pass^3 (%):

AgentEmotional userNeutral userΔ (emotional − neutral)
Qwen3.8-27B50.4 / 31.461.9 / 48.2−11.5 / −16.8
Qwen3-235B-A22B-Thinking-250741.8 / 21.952.6 / 35.4−10.8 / −13.5
Qwen3-235B-A22B-Instruct-250741.7 / 20.549.4 / 31.1−7.7 / −10.6
Muse-Glimmer-30B42.9 / 30.349.5 / 37.8−6.6 / −7.5
MiniMax-M2.746.5 / 26.456.8 / 39.6−10.3 / −13.2
Qwen3-8B29.4 / 13.138.1 / 19.4−8.7 / −6.3
Tracer-High-8B49.1 / 29.958.6 / 41.3−9.5 / −11.4
Tracer-Low-8B37.4 / 20.345.0 / 28.5−7.6 / −8.2

On the same tasks, every model scores 6.6–11.5 points lower in Pass^1 and 6.3–16.8 lower in Pass^3 with the emotional user. Since the task is unchanged, this gap measures robustness to emotional users, beyond general tool-use competence.

The small share of “emotional neglect” in Appendix D.1 does not contradict this result. The error analysis labels what went wrong, not what triggered it: a refund conceded under pressure would be labeled an extra action, and a handoff under distrust a wrong transfer. Emotional pressure therefore mostly surfaces as ordinary execution errors, as the failure cases in Q7 illustrate.


Q7: When does emotion change the task outcome?


We examined the 160 flipped failures of Tracer-High-8B and Tracer-Low-8B, where the agent fails with the emotional user but solves the same task with a neutral one. In 123 of them (77%), a human annotator attributed the failure to the user's emotion:

Failure modeShare
Premature handoff to a human under distrust or frustration28.5%
Failure to de-escalate; user leaves before completion24.4%
Following the user off-task20.3%
Wrong write when the user rushes or disputes15.4%
Conceding unneeded refunds, bookings or changes11.4%

Emotion changes the outcome in two ways:

  • Losing the user (52.9%): the agent fails to keep the emotional user engaged, and the conversation ends early through a handoff or the user leaving.
  • Yielding to pressure (47.1%): the agent follows the emotional user's demands instead of the task and policy.

Dear Reviewer XFyM,

As the deadline is nearing, we wanted to gently follow up on our recent submission. Your feedback is highly valuable to us, and we would appreciate any updates or further guidance you might have regarding our revisions and responses.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comments to All Reviewers

We thank the reviewers (XFyM, rv3w, AnkE) for their valuable feedback.

We are encouraged that the reviewers recognized the importance of evaluating emotional variation in LLM agent interactions (Reviewer XFyM); the diversity and practical value of our benchmark (Reviewers XFyM and AnkE); the finding that emotion-conditioned interactions substantially affect agent performance and consistency, even for frontier models (Reviewer XFyM); the scalability and quality controls of our data-generation pipeline (Reviewer rv3w); and the strong empirical improvements and generalization of Tracer-trained agents (Reviewers XFyM, rv3w, and AnkE).

We have addressed all reviewer comments in the individual responses and will incorporate the remaining changes into the revision. The revised version primarily includes the following significant updates, with new changes highlighted in blue:

  1. In Section 4.2 and Appendix D.1 (Figure 9), we added a paired evaluation of the same tasks with and without emotion conditioning, quantifying the additional difficulty introduced by emotional users (Reviewers XFyM, rv3w, and AnkE).
  2. In Section 4.2 and Appendices D.2 and D.3 (Tables 9 and 10), we added evaluations with a different user-simulator family (GLM-5.2) and an alternative (ReAct-style) simulator prompt, showing consistent agent rankings. (Reviewers XFyM and rv3w).
  3. In Section 4.2 and Appendix D.4 (Table 11), we added controlled comparisons with Simia and EnvScaler, using the same teacher model, data budget and training configuration (Reviewers XFyM, rv3w, and AnkE).
  4. In Section 5 and Appendix D.5 (Table 12), we added a success-only training ablation that removes emotion conditioning and the dissatisfaction signal, isolating the contribution of interaction-aware supervision (Reviewers XFyM, rv3w, and AnkE).
  5. In Section 4.1, we report the evaluation seeds used for the three trials (Reviewer rv3w).
  6. In Appendix C, we clarified the training-data sources: Tracer models are trained only on Tracer-generated data, and the AReaL baselines only on the public AReaL data (Reviewer AnkE).
  7. In Limitations and Future Work, we acknowledged the reliance on simulated users rather than real human interactions as a limitation and identified evaluation with real users as future work (Reviewer XFyM).

We also addressed the following concerns in our individual responses:

  1. Human validation studies of simulated-user realism, the dissatisfaction signal, and the LLM-based judgments (Reviewers XFyM and rv3w).
  2. Additional analyses: a random-emotion control, emotion-induced failures, and training-seed variance (Reviewers XFyM and rv3w).
  3. Methodological clarifications on train/test separation, baseline evaluation, and training-data sizes (Reviewers rv3w and AnkE).

If our rebuttal addresses the concerns, we earnestly and kindly ask the reviewers to consider raising the rating and supporting us for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.