- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MERIDIAN, a benchmark for long-context instruction following (IF) over real-world documents. It contains 3,131 prompts across five domains (SEC financial filings, U.S. Congressional records, PubMed Central articles, AWS documentation, and English Wikipedia), with contexts from under 8k to 128k+ tokens built by padding a target document with same-domain distractors. Each prompt combines a document task (summarization, classification, closed QA, list extraction) with 2–10 typed constraints drawn from a taxonomy of generic, domain-static and domain-templated constraints. These are split between the system and user prompts and checked by code, LLM, or hybrid verifiers. User prompts are paraphrased by an LLM and validated by a single human annotator, and LLM rubrics are revised until judge-to-judge Krippendorff’s α ≥ 0.667.
Scoring separates a rubric-based Task score (an LLM judge, Sonnet-4.5) from a strict all-or-nothing IF score, and combines them as 0.6·Task + 0.4·IF. The authors evaluate roughly 10 open- and closed-weight models (different subsets per experiment) along three research questions. RQ1: Task and IF scores are only moderately correlated (r ≈ 0.50), and the Combined score drops on average from 45.4 to 38.3 between 8k and 128k. RQ2: moving the user request (first/middle/dispersed vs. last) or delivering it over multiple turns lowers scores by about 1–12 points, mainly through the Task score. RQ3: when one user instruction is moved inside a non-target document, models still obey it 32–49% of the time, and system-prompt IF exceeds user-prompt IF. A post-hoc case study shows that a discourse-coherence score correlates with Task score at r = 0.65 across 11 models.
Critical strengths
S1 – Realistic, multi-axis long-context IF setting with task/IF decoupling. Most IF benchmarks use short, synthetic contexts, while most long-context benchmarks assume a cooperative prompt with no output constraints. MERIDIAN combines real professional documents in five domains, typed constraints including domain-specific ones (e.g., dollar precision, legal citation style, hazard ratios with event counts), and explicit control over context length, constraint count, placement and turn structure. Reporting a judged Task score next to a strict IF score is a useful design choice: Figure 3 and Table 1 show that models keep moderate task quality (Task ≈ 0.4–0.6) while almost never satisfying every constraint (IF ≤ 0.085 for hard prompts). That finding is a concrete, actionable diagnostic that a single aggregate score would hide. This is the main reason the benchmark could be valuable to the community.
S2 – Controlled, paired perturbation design. Placement variants keep the system message, document order and request bytes fixed and are compared within prompt against the native LAST layout (Table 10, n = 873 per model). The multi-turn study likewise pairs identical user-authored content across single- and multi-turn delivery. This attribution-preserving design is more rigorous than many leaderboard-style benchmarks and gives credible evidence that layout alone can shift scores by 2–8 points for some models (e.g., Sonnet-5 MIDDLE: −8.0 Task, p < 0.001).
Critical weaknesses
W1 – Key claims rest on uncontrolled or unvalidated measurements. (a) Injection: the injected items are benign user constraints, and there is no control in which the constraint is absent, so the 32–49% injection rates and the domain gaps (Table 11) may mostly reflect base rates of satisfying the constraint anyway. (b) Hierarchy: system and user constraints appear to be different sets (§3.2, Fig. 1) and no system–user conflicts are reported, so “system IF > user IF” may be a constraint-type effect rather than evidence of instruction hierarchy. (c) Context length: domain and complexity are unevenly spread across length bins (Table 6), and the paper does not say whether the same prompts are padded to each length. (d) Evaluation: one Sonnet-4.5 judge, validated only by judge-to-judge agreement, with no human labels; it also scores same-family models, and each model is sampled once. These issues undermine all three research-question conclusions.
W2 – Limited novelty and inconsistent reporting. The findings largely confirm prior work on long-context IF [1], instruction hierarchy [2, 3], positional sensitivity [4] and prompt injection [5, 6], so the value lies mostly in the benchmark. The reporting is also hard to audit: model subsets change between experiments (9, 10, 4 and 3 models); the placement sample sizes disagree (1,156 vs. 873); Table 2 applies an unpaired test to paired data; IF-legit (Table 5) and IF (Table 1) use different metrics without saying so; and decoding settings and the 0.6/0.4 Combined weighting are neither specified nor justified.
References
[1] X. Wu, M. Wang, Y. Liu, X. Shi, H. Yan, X. Lu, J. Zhu, and W. Zhang. LIFBench: Evaluating the instruction following performance and stability of large language models in long-context scenarios. In Proceedings of ACL, 2025. [2] Z. Zhang, S. Li, Z. Zhang, X. Liu, H. Jiang, X. Tang, Y. Gao, Z. Li, H. Wang, Z. Tan, et al. IHEval: Evaluating language models on following the instruction hierarchy. In Proceedings of NAACL, 2025. [3] Y. Qin, T. Zhang, Y. Shen, W. Luo, H. Sun, Y. Zhang, Y. Qiao, W. Chen, Z. Zhou, W. Zhang, et al. SysBench: Can LLMs follow system message? In Proceedings of ICLR, 2025. [4] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. [5] Z. Li, B. Peng, P. He, and X. Yan. Evaluating the instruction-following robustness of large language models to prompt injection. In Proceedings of EMNLP, 2024. [6] J. Yi, Y. Xie, B. Zhu, E. Kiciman, G. Sun, X. Xie, and F. Wu. Benchmarking and defending against indirect prompt injection attacks on large language models. In Proceedings of KDD, 2025.
Questions for authors
Q1 (RQ1, context length). Is the context-length analysis paired, i.e., is the same (target document, task, constraint set) instantiated at 8k, 16k, …, 128k with only the distractors changing? If not, can you report a paired or stratified analysis (by domain and complexity) showing that the length effect is not a composition effect?
Q2 (RQ3, injection). What is the base rate at which each injected constraint is satisfied when it appears nowhere in the prompt? Please report injection rates net of this base rate, broken down by constraint type.
Q3 (RQ3, instruction hierarchy). In Figure 5 and Tables 7–8, are the same constraints placed in the system prompt and in the user prompt, or are the two sets different (e.g., format vs. content constraints, as §3.2 and Figure 1 suggest)? If they differ, how do you separate a hierarchy effect from a constraint-type effect? Were the system–user conflict prompts described in Appendix B evaluated?
2 comments
Thanks for the response, I will maintain my socre.
Response to Reviewer 1MUN
We appreciate the reviewer's thorough feedback. We hope our responses address all their concerns. We look forward to interacting further on their review.
Weaknesses #1a & 1b, Questions #2 & 3: injection and hierarchy controls
We agree and ran both controls on Opus 5.5 and GPT-6.1 Sol. Without the injected instruction, answers satisfy it 25-27% of the time, so net injection compliance is +5.3 and +8.9 points (p < 0.001). For hierarchy, we restricted the comparison to the 41 instruction types that appear in both channels. System instructions are still followed more often (83% vs. 71% and 85% vs. 69%, p < 0.001), so the effect is not a constraint-type effect. We will fix the Figure 5 caption (lines 426-427).
Weakness #1c & Question #1: context length
The length analysis is not paired (lines 190-195). In a new analysis with one intercept per domain × complexity cell, the newer models show no length decline. We will report this analysis for all models. We will also add a paired subset padded to all five lengths (lines 130-131).
Weakness #1d: judge validation
Judges rejected deliberately violating responses with 0.89-0.98 specificity. We will add these results, a second-family judge and human labels for the camera-ready version of our paper.
Weakness #2: novelty and reporting
We also answered novelty for Reviewer dXEL. Prior findings come from benchmarks that saturate quickly (Ott et al., 2022; Pyatkin et al., 2025; Yen et al., 2025). MERIDIAN tests these effects on real documents with adversarial stumping selection, where only 8.7% of legal candidates passed. The effects survive, and the newer models still reach only 0.27 strict IF. Our new control also shows that the multi-turn loss comes from the model's own intermediate replies, not from turn distance (p = 0.35).
We have already moved Table 2 to paired tests and added all-or-nothing IF to the injection tables. We will address the remaining reporting comments in the revision.
Thank you very much for your review