- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces τ-IF, a multi-variant benchmark for evaluating instruction following in executable, stateful agent environments. Built upon τ-bench Retail and Airline, τ-IF combines task completion with trajectory-level instruction compliance, providing 2,000 verified training trajectories, 5,541 single-step training snapshots, and evaluation sets covering 164 base tasks with five instruction variants each. The benchmark introduces Task-and-Instruction Success (TIS) as its primary metric and evaluates 23 systems, revealing substantial gaps between task completion, instruction compliance, and their conjunction. The authors further compare full-trajectory SFT, fixed-context GRPO/PPO, and online environment RL across three model backbones. Results show that SFT consistently outperforms the implemented RL recipes, while online interaction provides richer reward signals without statistically detectable improvements in TIS.
Critical strengths
-
Meaningful evaluation problem. τ-IF addresses an important limitation of existing instruction-following benchmarks by jointly evaluating executable task outcomes and trajectory-level compliance. The distinction between Task, ISR, CSR, and TIS is well motivated, particularly because successful environment modifications can coexist with instruction violations.
-
Useful benchmark resources and multi-variant design. The release combines verified training trajectories, executable environments, structured constraints, and frozen evaluation variants. Evaluating five instruction configurations per base task while preserving its underlying goal and terminal criterion enables controlled analysis of instruction robustness.
-
Informative empirical findings. The evaluation of 23 systems exposes substantial joint-success limitations, while the training study provides useful evidence that full-trajectory SFT is a strong baseline. The observation that richer online RL rewards do not necessarily improve TIS highlights an important distinction between reward informativeness and objective alignment.
Critical weaknesses
-
Limited instruction-scoring reliability. Instruction compliance relies on a single LLM judge without human calibration or independent judge agreement (Section 3). Since ISR and TIS require all active constraints to pass, judge errors may significantly affect benchmark rankings and conclusions.
-
Confounded training-method comparisons. SFT and RL differ in trajectory coverage, action types, and parameter-update strategies (full fine-tuning vs. LoRA). Consequently, the observed SFT advantage cannot be attributed solely to trajectory supervision or learning paradigm differences.
-
Limited statistical rigor and reward alignment. Experiments use a single training run per configuration, while bootstrap intervals capture task variability rather than training-seed uncertainty. Moreover, Online RL optimizes an additive Task/CSR reward rather than TIS, potentially explaining its limited improvement.
-
Restricted generalization. Training relies exclusively on synthetic Retail tasks, and evaluation reuses 164 public τ-bench tasks. The five variants modify instruction overlays rather than underlying tasks, leaving generalization to unseen environments and workflows insufficiently established.
Questions for authors
-
Judge reliability: Can the authors report human–judge or independent-judge agreement on a representative sample, particularly for cases where Task Success and ISR disagree? How sensitive are TIS rankings and method comparisons to instruction-verdict errors?
-
Fairness of training comparisons: Can the authors disentangle the effects of trajectory coverage, action types, and parameter-update strategies? In particular, would an SFT baseline restricted to the same clean tool-call snapshots, or an RL configuration initialized from the SFT checkpoint, change the central conclusions?
-
Reward alignment: Given that Online RL optimizes an additive Task/CSR reward rather than TIS, can the authors evaluate a conjunction-aligned reward or provide evidence separating reward misalignment from limitations of online optimization?
-
Statistical reliability and generalization: How stable are the main SFT-versus-RL and Online-versus-Fixed differences across independent training seeds? Additionally, how do the authors assess possible public τ-bench exposure effects on the 23-system ranking?