acceptodds
Under review as a conference paper at ICLR 2027

ProcIF: Evaluating Procedural Instruction Following in Large Language Models

Abstract

Large language models increasingly execute tasks that specify not only what outcome to produce, but also how evidence should be collected, actions should be ordered, failures should be handled, and results should be validated. A correct final answer, however, does not establish that the prescribed process was followed; conversely, procedural compliance does not guarantee a correct outcome. We introduce ProcIF, a cross-domain, two-track procedural instruction following benchmark. It contains 342 instances and 4,157 executable process constraints. Track A adds task-specific procedural constraints to writing, scientific reasoning, workflow synthesis, and tool-use tasks, whereas Track B converts source harness trajectories into visible, observation-conditioned Skills or standard operating procedures. We report strict Procedure Adherence Score (PAS), source-aware Task Success Rate (TSR), and their instance-level conjunction. Across the evaluated models, the best strict PAS and Joint scores reach only 41.23% and 35.96%, respectively. These results show that models often satisfy local requirements while failing the complete procedural contract, making procedural instruction following a challenging and substantially unresolved capability.

Primary area:
datasets and benchmarks
Keywords:
procedural instruction following, evaluation, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qoCcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProcIF evaluates whether LLMs both solve tasks and follow explicit procedures. It contains 342 cross-domain instances with 4,157 executable constraints across task-specific procedures and observation-conditioned SOPs. The benchmark separately reports strict Procedure Adherence, Task Success, and their conjunction. Experiments show that models often achieve correct outcomes while violating procedural requirements.

Critical strengths

1.Clear separation of process and outcome. PAS and TSR expose failures that final-answer evaluation alone would miss. 2.Auditable evaluation design. Most procedural requirements are compiled into deterministic predicates over actions, artifacts, evidence, and runner states, making failures interpretable and reproducible.

Critical weaknesses

1.Track A only approximates actual process execution. Many tasks ask the model to emit a complete self-reported trace in one response. This evaluates structured description of a procedure, but does not fully establish that intermediate operations were genuinely executed. 2.Insufficient robustness analysis. Semantic TSR for 174 instances relies on one model judge without reported human agreement. Moreover, each model-task pair has only one rollout, and strict PAS is confounded by the number of constraints, which differs substantially across tasks and tracks.

Questions for authors

1.For single-turn Track A tasks, which events are externally verified rather than accepted from model-reported artifacts? 2.Was the semantic TSR judge validated against human annotations, and what agreement was obtained? 3.Do model rankings remain stable across repeated rollouts, alternative judges, and controls for constraint count?

Reviewer_fhYbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ProcIF, a benchmark for evaluating whether LLMs can successfully complete tasks while following prescribed procedures. Unlike conventional instruction-following benchmarks that primarily evaluate final outputs, ProcIF separately measures procedural adherence and task success.

The benchmark contains 342 tasks with 4157 procedural constraints. It consists of two tracks: Track A adds explicit steps that models must follow when completing existing tasks, while Track B evaluates whether models can follow given Skills or workflows, including reacting to tool observations and handling failures.

Experiments across 13 language models show substantial gaps between procedural adherence and task success.

Critical strengths

The paper addresses a meaningful limitation of existing agent evaluations: successful task completion does not necessarily imply compliance with the requested procedure. Separately evaluating procedural adherence and task success provides a useful framework for identifying failures that would otherwise be overlooked.

The use of executable predicates over observable trajectories is a valuable contribution. In particular, evaluating procedural properties rather than requiring an exact reference trajectory allows multiple valid executions to receive credit.

Critical weaknesses
  1. A significant portion of the benchmark (149 instances) uses a single-turn setting in which models generate the entire structured process trace alongside the final answer. In these cases, the verifier can check whether the generated artifacts and references satisfy the specified contract, but it cannot establish that the model actually performed the intermediate steps as a genuine sequential process.

  2. The benchmark relies heavily on the correctness and appropriateness of its constructed procedural requirements and corresponding executable predicates. However, it is difficult to assess whether the constructed contracts accurately capture the intended procedures without incorrectly penalizing valid alternative executions.

Questions for authors
  1. How do the authors ensure that the newly constructed procedural requirements are meaningful and representative of real-world workflows? Is there any human evaluation or quantitative analysis supporting the quality of these procedures?
  2. How do the authors validate the correctness of the executable procedural checks and the LLM-based task-success evaluator?
Reviewer_9DbYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ProcIF, a two-track benchmark for evaluating whether language models can complete tasks while following explicit procedural requirements. It contains 342 instances and 4,157 executable process constraints. Track A adds task-specific procedures to writing, scientific reasoning, workflow synthesis, and tool-use tasks, while Track B converts source trajectories into observation-conditioned Skills/SOPs. The paper separately reports Procedure Adherence Score (PAS), Task Success Rate (TSR), and their conjunction (Joint). Across evaluated models, the best strict PAS is 41.23% and the best Joint score is 35.96%, suggesting that models often achieve the requested outcome while violating parts of the prescribed process.

Critical strengths
  1. Clear separation of procedural adherence and task success. PAS, TSR, Joint, and the four outcome quadrants provide a useful diagnostic framework for distinguishing correct outcomes from compliance with explicit procedural requirements.

  2. An auditable, cross-domain evaluation design. The two-track benchmark translates visible requirements into deterministic checks over artifacts, dependencies, and bounded interactions, while allowing alternative compliant trajectories rather than requiring imitation of a hidden reference path.

Critical weaknesses
  1. Benchmark validity needs stronger empirical support. Although human screening is described, the paper does not report annotator agreement or revision/rejection statistics sufficient to assess procedural quality. Analyses separating single-turn trace generation from interactive execution, and controlling for contract complexity, would also strengthen the interpretation of procedural failures.

  2. Evaluation reliability and reproducibility remain insufficiently established. Results use one rollout per model-task pair without uncertainty estimates, and 174 instances involve semantic TSR judgments without reported human calibration. The deferred release of execution and evaluation code further limits independent verification of the benchmark and its results.

Questions for authors
  1. Section 3.2.4 describes human screening for task quality, process fit, and evaluation fit. Could you report annotator numbers, inter-annotator agreement, revision/rejection rates, and representative revisions demonstrating that the procedures are relevant, unambiguous, and jointly satisfiable?

  2. Can you report decoding settings and uncertainty estimates, and assess repeated-rollout stability for representative models? Please also separate single-turn trace generation from interactive execution and examine whether cross-track differences persist after accounting for active-predicate counts and task difficulty.

  3. For the 174 instances with semantic TSR components, how well does GPT-5.6 Luna agree with independent human judgments? Beyond the documented grounding safeguards, can you assess judge-family bias and prompt sensitivity, and clarify when the complete runners and verifiers will be available for inspection?

  4. Table 3 reports 13 models, whereas Figures 3 and 4 display 10 and Section 5.1 aggregates 3,420 model-task pairs. Could you explain the choice of analysis subset and explicitly distinguish subset statistics from claims about all evaluated models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.