- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TIPER, a framework that evolves user personas to expose robustness failures in interactive LLM agents while preserving task integrity. It combines reflection-guided persona mutation, trajectory-level integrity checks, and task-wise Pareto selection. Experiments across four benchmarks and seven agent models show consistent reductions in task success compared to default user simulators, along with analyses connecting challenging user behaviors to specific agent failure modes.
Critical strengths
- I find the distinction between agent failure and task invalidation particularly important. Simply making users more adversarial can trivially lower task success without revealing meaningful weaknesses. TIPER explicitly addresses this, making its robustness evaluation more convincing.
- The empirical evidence is compelling. The consistent failure exposure across different agents and domains, together with comparisons against predefined behaviors and objective-matched persona optimization, supports the practical value of adaptive persona discovery.
Critical weaknesses
- I'm not fully convinced that the task-wise Pareto mechanism is necessary. The final objective is still to select a single persona with the lowest average score, and Table 4 does not isolate the contribution of this Pareto selection. It also remains unclear whether the gains come primarily from reflection-based search and integrity filtering rather than maintaining a diverse Pareto frontier.
- The relationship between task integrity and evaluation reliability deserves more attention. The search-time gate measures integrity specifically among failed trajectories, while the formal objective concerns overall integrity. Since the final scores also condition on valid trajectories, I would like to understand whether this filtering changes which successes and failures contribute to the reported performance gap.
Questions for authors
- How would TIPER perform if task-wise Pareto selection were replaced with simple top-K selection by average rank-set performance? Would the discovered failure modes become less diverse?
- Have you examined cross-agent transfer of discovered personas? For example, would a persona optimized against GPT-5.4 expose similar failures in other agents? This would help distinguish genuinely agent-specific weaknesses from universally difficult user behaviors.
- Could you report overall trajectory-integrity rates, including successful rollouts, and compare the results on a common set of integrity-valid trajectories?
3 comments
[Q2] Cross-agent transfer experiments.
To further study how well the searched personas transfer and generalize, we froze the personas searched against GPT-5.4 on Telecom and Delivery and evaluated them on two other agents, with all other settings fixed.
[Table R3]. Cross-agent transfer. Reduction in avg@3 points. DS-V4.1-Flash replaces DS-V4-Flash (no longer available), so its numbers are not from Table 2 of the paper.
| Benchmark | Agent | Frozen GPT-5.4 persona | Persona searched against this agent |
|---|---|---|---|
| Telecom | DS-V4.1-Flash | 29.0 | 17.9 |
| Telecom | MiniMax-M3 | 31.5 | 37.0 |
| Delivery | DS-V4.1-Flash | 24.2 | 23.3 |
| Delivery | MiniMax-M3 | 27.5 | 27.5 |
The personas largely transfer. The frozen personas reduce both agents by 24.2–31.5 points, and on MiniMax-M3 they are comparable to an agent-specific search (31.5 vs. 37.0; 27.5 vs. 27.5). This indicates that the personas expose weaknesses shared across agents, while some weaknesses remain agent-specific. Another Telecom persona transfers fully to MiniMax-M3 (20.4 vs. 19.8 on GPT-5.4) but barely to DS-V4.1-Flash (1.9), so per-agent search remains useful. We will add these transfer results to the paper.
[W2 & Q3] Integrity rates and evaluation reliability.
Thanks for raising these. Below we answer three questions: (1) why the search-time gate checks only failures, (2) whether filtering changes which rollouts count, and (3) how the results look on a common set of valid tasks.
The search-time gate is an efficient approximation, and all reported results are based on auditing every rollout. During search, we check integrity only on failures to save audit cost, since only a failed rollout can turn a simulator error into an apparent agent failure. Invalid successes are rare (1 of 371 TIPER successes on τ³-bench), so the gate also keeps the overall invalid rate close to its 5% threshold. The final evaluation then audits every rollout, including successes.
Filtering removes only a few trajectories, equally in both conditions. We audited all 1,290 held-out τ³-bench trajectories behind the paper's Table 1, using the paper's two-judge setup:
[Table R4]. Overall integrity on the held-out τ³-bench trajectories.
| Condition | Overall integrity | Successes removed | Failures removed |
|---|---|---|---|
| Default | 98.9% | 2 of 446 | 5 of 199 |
| TIPER | 98.9% | 1 of 371 | 6 of 274 |
The reductions barely change, whether we remove invalid trajectories or keep only common-valid tasks. Common-valid tasks are those with three valid trials in both conditions, as in Appendix B.4. On LiC and ColBench, the reductions change by at most 1.1 points.
[Table R5]. Change in the reduction (avg@3 points) relative to using all trajectories.
| Domain | Remove invalid trajectories | Common-valid tasks |
|---|---|---|
| Telecom | 0.0 | 0.0 |
| Airline | 0.0 | +0.1 |
| Retail | −1.6 | −1.2 |
| Banking | +0.7 | +2.8 |
We will clarify the search-time gate in Section 3.2 and report the overall integrity rates and the common-valid results in the paper.
Thank you again for the constructive feedback. We hope these results address your questions, and we are happy to discuss further.
Thank you for your response. Most of my concerns are addressed and I would like to maintain my original positive score.
Thank you for the thoughtful review. To address your concerns, we ran the following experiments: (1) replacing Pareto selection with top-K selection and measuring the diversity of what each discovers, (2) evaluating the personas searched against GPT-5.4 on other agents, and (3) auditing all held-out trajectories, including successes.
[W1 & Q1] Task-wise Pareto selection vs. top-K.
We compared Pareto selection with top-K selection on VitaBench Delivery, together with an extra ablation of the reflection module, keeping all other settings fixed, including the search budget:
[Table R1]. Selection and reflection ablations on VitaBench Delivery. Reduction in avg@3 points.
Pareto selection is at least as good as top-3 and better than top-1. Top-3 ties with Pareto selection on GPT-5.4 and is 9.2 points lower on MiniMax-M3; top-1 is 9.2 points lower on both agents. By design, top-K keeps the personas with the best average score, so it can drop a persona that makes the agent fail tasks no other persona does. Pareto selection keeps such a persona, and the search can build on it. This is consistent with GEPA’s comparison of Pareto selection with top-K selection (Agrawal et al., 2026).
Pareto selection also keeps the discovered failures more diverse. We measured two quantities in each search: task coverage, the number of rank-set tasks on which at least one persona in the final active set makes the agent fail, and behavior categories, the number of user-behavior categories covered by all candidates that pass the gate. The categories are the failure-causing behavior types of Section 5.2, and the Pareto and top-3 searches use identical settings.
[Table R2]. Diversity under Pareto and top-3 selection. Task coverage counts rank-set tasks (30 in each setting).
Pareto selection covers more tasks and more behavior categories in every setting, and on Delivery the gap follows Table R1. With GPT-5.4, where top-3 ties with Pareto selection, the coverage is also similar (21 vs. 20), although top-3 explores fewer behavior categories (4 vs. 7). On MiniMax-M3, where top-3 is 9.2 points lower, it covers 3 fewer tasks and half as many behavior categories (2 vs. 4). This is consistent with the mechanism above: top-K drops the personas that make the agent fail tasks no other persona does.
Reflection contributes the most, and integrity filtering is also needed. Without reflection, the reduction falls by about 16 points for both agents (25.0 → 8.3; 27.5 → 11.7), more than the 9.2 points from replacing Pareto selection. Without the integrity gate, the reduction falls from 25.3 to 10.5 on Telecom (Table 4(a) of the paper). As discussed in Section 5.1, we hypothesize that the gate also acts as a regularizer: it keeps proposed candidates from overfitting the search tasks through task-invalidating shortcuts that do not transfer to held-out tasks.
We will add these ablations to Table 4 of the paper, together with the diversity analysis.
References