- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes IFHierBench, a benchmark of 600 prompts for hierarchical instruction following in which each constraint is scoped to a nested region of a structured output such as a JSON block inside a markdown section, paired with deterministic hierarchical checkers. The structure level constraints are derived from an analysis of 1,232 real LLM application prompts on GitHub, and the paper evaluates seven models, reporting that prompt level accuracy collapses once constraints move inside a scope and that even the strongest model barely exceeds fifty percent.
Critical strengths
1、The motivating observation is fair: existing instruction following benchmarks treat constraints as a flat list over the whole response, while real prompts increasingly scope constraints to subregions of an artifact. 2、The construction is tidy, every prompt ships with a deterministic checker, and the opening example of a flat checker certifying a response whose content sits in the wrong sections nicely illustrates why scoping matters. 3、The paper is also clear about its pipeline and releases prompts and checkers together.
Critical weaknesses
The central scientific claim of the benchmark is never established. The headline result, that accuracy collapses with constraint depth, is confounded from the start, because four of the five hardest constraint types are exact counting checks such as exact character length and exact word count, and exact counting is notoriously hard even in flat prompts. Without a depth matched control that removes counting constraints, the benchmark cannot say whether nesting or counting is doing the damage, which means its own motivating question stays unanswered by its own data.
The cross model comparison is also not a comparison: proprietary API models run at provider default settings while open weight models run four bit quantization on a single consumer GPU, and no intervals or significance testing appear anywhere, so the ranking tables are decorative. The reasoning trace analysis rests on the authors stripping Claude style enumeration from traces and labeling depth first versus breadth first orders themselves, with no inter rater check. The opening story about developer workflows inverting is asserted rather than evidenced, and the level of typos suggests the text saw few careful passes. A benchmark whose key variable is confounded and whose model comparison is uncontrolled does not give the community a reliable yardstick, and I recommend clear rejection.
Questions for authors
Can you report accuracy stratified by depth after removing all exact counting constraint types, to show the depth effect on its own? Were decoding and quantization conditions matched across proprietary and open weight models, and how does the ranking move if they are?