acceptodds
Withdrawn from ICLR 2027

-IF: A Multi-Variant Benchmark for Agentic Instruction Following in Executable Environments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM agents must complete user goals while obeying standing and task-specific instructions. Their actions can alter state and activate different instructions. Existing agent benchmarks rarely pair executable training data with pre-specified instruction variants for testing this joint capability. To this end, we introduce -IF, a benchmark built on executable -bench environments. -IF releases 2,000 verified training programs with corresponding trajectories and provides two evaluation sets: Test evaluates 164 base tasks using one instruction variant each, whereas Test-Variants evaluates the same tasks under all five variants. Each instance combines a user goal, multi-turn interaction, executable tools, mutable environment state, and machine-scored instruction constraints. Across 23 eligible systems on Test, the best observed Task-and-Instruction Success is 40.24%, although Task and Instruction Success Rate scores reach 73.17% and 75.61%, exposing a joint-success shortfall. We then use -IF for an optimizer-paired study across three backbones. Full-trajectory SFT exceeds Fixed-Context GRPO and PPO by 5.08 and 4.80 points, while Online variants add and points over Fixed. -IF connects open training data, broad capability evaluation, and controlled experimentation in one reproducible platform, establishing verified trajectory supervision as a strong baseline while motivating better rewards and credit assignment.

Primary area:
datasets and benchmarks
Keywords:
agentic instruction following, executable benchmarks, tool-using language agents, multi-variant evaluation, online reinforcement learning, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ck2rOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces τ-IF, a multi-variant benchmark for evaluating instruction following in executable, stateful agent environments. Built upon τ-bench Retail and Airline, τ-IF combines task completion with trajectory-level instruction compliance, providing 2,000 verified training trajectories, 5,541 single-step training snapshots, and evaluation sets covering 164 base tasks with five instruction variants each. The benchmark introduces Task-and-Instruction Success (TIS) as its primary metric and evaluates 23 systems, revealing substantial gaps between task completion, instruction compliance, and their conjunction. The authors further compare full-trajectory SFT, fixed-context GRPO/PPO, and online environment RL across three model backbones. Results show that SFT consistently outperforms the implemented RL recipes, while online interaction provides richer reward signals without statistically detectable improvements in TIS.

Critical strengths
  1. Meaningful evaluation problem. τ-IF addresses an important limitation of existing instruction-following benchmarks by jointly evaluating executable task outcomes and trajectory-level compliance. The distinction between Task, ISR, CSR, and TIS is well motivated, particularly because successful environment modifications can coexist with instruction violations.

  2. Useful benchmark resources and multi-variant design. The release combines verified training trajectories, executable environments, structured constraints, and frozen evaluation variants. Evaluating five instruction configurations per base task while preserving its underlying goal and terminal criterion enables controlled analysis of instruction robustness.

  3. Informative empirical findings. The evaluation of 23 systems exposes substantial joint-success limitations, while the training study provides useful evidence that full-trajectory SFT is a strong baseline. The observation that richer online RL rewards do not necessarily improve TIS highlights an important distinction between reward informativeness and objective alignment.

Critical weaknesses
  1. Limited instruction-scoring reliability. Instruction compliance relies on a single LLM judge without human calibration or independent judge agreement (Section 3). Since ISR and TIS require all active constraints to pass, judge errors may significantly affect benchmark rankings and conclusions.

  2. Confounded training-method comparisons. SFT and RL differ in trajectory coverage, action types, and parameter-update strategies (full fine-tuning vs. LoRA). Consequently, the observed SFT advantage cannot be attributed solely to trajectory supervision or learning paradigm differences.

  3. Limited statistical rigor and reward alignment. Experiments use a single training run per configuration, while bootstrap intervals capture task variability rather than training-seed uncertainty. Moreover, Online RL optimizes an additive Task/CSR reward rather than TIS, potentially explaining its limited improvement.

  4. Restricted generalization. Training relies exclusively on synthetic Retail tasks, and evaluation reuses 164 public τ-bench tasks. The five variants modify instruction overlays rather than underlying tasks, leaving generalization to unseen environments and workflows insufficiently established.

Questions for authors
  1. Judge reliability: Can the authors report human–judge or independent-judge agreement on a representative sample, particularly for cases where Task Success and ISR disagree? How sensitive are TIS rankings and method comparisons to instruction-verdict errors?

  2. Fairness of training comparisons: Can the authors disentangle the effects of trajectory coverage, action types, and parameter-update strategies? In particular, would an SFT baseline restricted to the same clean tool-call snapshots, or an RL configuration initialized from the SFT checkpoint, change the central conclusions?

  3. Reward alignment: Given that Online RL optimizes an additive Task/CSR reward rather than TIS, can the authors evaluate a conjunction-aligned reward or provide evidence separating reward misalignment from limitations of online optimization?

  4. Statistical reliability and generalization: How stable are the main SFT-versus-RL and Online-versus-Fixed differences across independent training seeds? Additionally, how do the authors assess possible public τ-bench exposure effects on the 23-system ranking?

Reviewer_5VszOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces τ-IF, a benchmark that adds explicit, machine-scored instruction constraints to τ-bench Retail and Airline tasks. The constraints cover tool parameters, argument relations, tool order, call budgets, and cross-turn state. An agent counts as successful (TIS) only if it completes the executable task and also passes every active instruction check (ISR). Task success is checked programmatically, while instruction checks are judged by a frozen gpt-oss-120b judge. The release includes 2,000 synthesized and verified Retail training trajectories (Kimi K3 teacher), a Dev set, and 5,541 single-step snapshots. It also includes two evaluation sets over the 164 public τ-bench tasks: Test, with one instruction variant per task, and Test-Variants, with five. Across 23 systems on Test, the best TIS is only 40.24%, even though the best Task and ISR scores reach 73.17% and 75.61% (from different systems). Very few tasks pass all five variants. In a training study on three backbones, full-trajectory SFT beats fixed-context GRPO/PPO by about 5 TIS points, and online RL adds no detectable gain over fixed-context RL. The authors conclude that task completion and instruction compliance are distinct capabilities that current models rarely achieve jointly, and that verified-trajectory SFT is a strong baseline.

Critical strengths
  1. More demanding joint objectives for agents. Requiring task success and full instruction compliance in the same episode is a more realistic target than reporting either component alone. The best Task and ISR scores come from different systems, and joint success tops out at about 40%, although how meaningful this gap is depends on weakness #1.

  2. Variant design that tests robustness. Five instruction variants per task, with the goal and environment held fixed, separate instruction brittleness from task difficulty. The finding that at most about 2.6% of backbone–task pairs pass all five variants is a striking result.

  3. Open training resources and careful provenance. The authors plan to release verified trajectories, single-step snapshots, Dev/Test splits, evaluation code, frozen prompts, and a row-level protocol ledger under permissive licenses. They also report their own exceptions transparently. Both make the benchmark reusable for training as well as evaluation.

Critical weaknesses
  1. The LLM judge is not validated. All instruction verdicts come from one gpt-oss-120b judge served at temperature 0.7, with no human or second-judge check. Because ISR fails a run if any of about 37 checks fails, even a small per-check error rate strongly lowers ISR and TIS. To improve: audit LLM verdicts against human labels and report the judge's accuracy and stability across repeated runs.

  2. The relation between instruction checks and end-state checks is not defined or measured. In τ-bench, policy violations usually also break the end state, so the two signals are tightly coupled. In τ-IF, Table 2 implies TIS ≈ Task × ISR for every system, meaning the added checks are nearly independent of task success. The paper never explains whether this is by design, which makes the headline "joint-success gap" hard to interpret. To improve: report the Task × ISR contingency table and state which constraints can or cannot be detected from the end state.

  3. The comparisons are not controlled. Leaderboard rows use six different evaluation protocols, and the top-ranked system is a protocol exception. The training study uses single seeds, full-parameter SFT versus LoRA RL, and a Fixed-GRPO run with almost no learning signal. To improve: rerun all systems under one protocol, and match settings and seeds across training recipes.

Questions for authors
  1. What is the judge's per-check accuracy against human labels? How many trajectories that humans judge fully compliant fail ISR?
  2. Does the SFT-over-RL gap remain with full-parameter RL, a non-collapsing fixed-context reward, or a TIS-based online reward?
  3. Are the five variants comparable in difficulty? Is low all-five consistency driven by a few hard or conflicting variants?
Reviewer_78CEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

τ-IF extends the Retail and Airline environments of τ-bench to evaluate whether agents complete tasks while following trajectory-level instructions. It provides 2,000 verified training trajectories, a development set, and 164 evaluation tasks with five instruction variants each. Its primary metric, Task-and-Instruction Success (TIS), requires both executable task success and compliance with all active constraints. Experiments evaluate 23 systems and compare SFT, fixed-context RL, and online RL across three backbones.

Critical strengths
  1. Jointly measuring task completion and instruction compliance captures failures that either component alone would miss. Explicit constraint activation makes the evaluation sensitive to the agent’s actual interaction history.

  2. Verified trajectories, executable environments, and five instruction variants per evaluation task support studying both adaptation and robustness within a shared framework.

  3. The supplement provides task-level ledgers, analysis code, optimization diagnostics, and recovery audits. Development-only checkpoint selection and task-cluster bootstrap intervals appropriately account for dependence among instruction variants.

Critical weaknesses
  1. The central instruction metric lacks calibration. Evaluation relies on a single LLM judge without human or independent-judge validation. The serving configuration is stochastic, and TIS requires agreement across dozens of active checks, potentially amplifying individual judgment errors. Consequently, the reliability of absolute scores and model comparisons remains uncertain.

  2. Training comparisons have substantial confounds. SFT uses full-parameter updates, whereas RL uses LoRA, with additional differences in action support, rewards, and training budgets. Moreover, the supplementary audit reports approximately 98% zero-variance reward groups in Qwen fixed-context GRPO and problematic PPO diagnostics. These issues limit the scientific conclusions obtainable from SFT’s advantage and the weak online gains.

  3. Evaluation protocols are not sufficiently comparable. The audit identifies 4,729 whole-task reopens, and earliest-scoreable sensitivity changes six of 21 selected checkpoints. Although qualitative training conclusions survive this analysis, heterogeneous recovery policies and model-specific exceptions still weaken interpretation of the leaderboard as a common-protocol capability comparison.

Questions for authors
  1. What are human-agreement and repeat-judging rates for task-level ISR/TIS? Do model rankings remain stable under an independent evaluation judge?

  2. Would leaderboard ordering and paired model differences persist under a uniform same-step retry policy, alongside strict first-attempt results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.