$\tau$-IF: A Multi-Variant Benchmark for Agentic Instruction Following in Executable Environments

LLM agents must complete user goals while obeying standing and task-specific instructions. Their actions can alter state and activate different instructions. Existing agent benchmarks rarely pair executable training data with pre-specified instruction variants for testing this joint capability. To this end, we introduce $\tau$-IF, a benchmark built on executable $\tau$-bench environments. $\tau$-IF releases 2,000 verified training programs with corresponding trajectories and provides two evaluation sets: Test evaluates 164 base tasks using one instruction variant each, whereas Test-Variants evaluates the same tasks under all five variants. Each instance combines a user goal, multi-turn interaction, executable tools, mutable environment state, and machine-scored instruction constraints. Across 23 eligible systems on Test, the best observed Task-and-Instruction Success is 40.24%, although Task and Instruction Success Rate scores reach 73.17% and 75.61%, exposing a joint-success shortfall. We then use $\tau$-IF for an optimizer-paired study across three backbones. Full-trajectory SFT exceeds Fixed-Context GRPO and PPO by 5.08 and 4.80 points, while Online variants add $+0.61$ and $+0.16$ points over Fixed. $\tau$-IF connects open training data, broad capability evaluation, and controlled experimentation in one reproducible platform, establishing verified trajectory supervision as a strong baseline while motivating better rewards and credit assignment.