- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ProcIF evaluates whether LLMs both solve tasks and follow explicit procedures. It contains 342 cross-domain instances with 4,157 executable constraints across task-specific procedures and observation-conditioned SOPs. The benchmark separately reports strict Procedure Adherence, Task Success, and their conjunction. Experiments show that models often achieve correct outcomes while violating procedural requirements.
Critical strengths
1.Clear separation of process and outcome. PAS and TSR expose failures that final-answer evaluation alone would miss. 2.Auditable evaluation design. Most procedural requirements are compiled into deterministic predicates over actions, artifacts, evidence, and runner states, making failures interpretable and reproducible.
Critical weaknesses
1.Track A only approximates actual process execution. Many tasks ask the model to emit a complete self-reported trace in one response. This evaluates structured description of a procedure, but does not fully establish that intermediate operations were genuinely executed. 2.Insufficient robustness analysis. Semantic TSR for 174 instances relies on one model judge without reported human agreement. Moreover, each model-task pair has only one rollout, and strict PAS is confounded by the number of constraints, which differs substantially across tasks and tracks.
Questions for authors
1.For single-turn Track A tasks, which events are externally verified rather than accepted from model-reported artifacts? 2.Was the semantic TSR judge validated against human annotations, and what agreement was obtained? 3.Do model rankings remain stable across repeated rollouts, alternative judges, and controls for constraint count?