acceptodds
Under review as a conference paper at ICLR 2027

MERIDIAN: A Long Context, Robust Instruction Following Benchmark

Abstract

We introduce Meridian, a benchmark for long-context (LC) instruction following (IF) over real-world documents. IF is the model capability that allows users to condition model output. Our goals are to enable diagnostics of model failures for this setting by exploring robustness to instruction positions in the prompt and resistance to instruction injection. We design prompts that state document-understanding tasks together with a set of typed constraints incremental to the task. We target one general knowledge domain and four domains relevant to knowledge work (finance, legal, medical, cloud service documentation) both through documents and constraints specific to the domains. We validate instruction prompts via human review. We report Task score, and IF score for the instruction prompts, and a weighted combined score. We test IF across changes in context length, instruction count, and instruction placement; we further explore single- and multi-turn settings. We evaluate current models and show that 1) Combined scores on average degrade (from to ) when context length increases from 8k to 128k tokens, and are sensitive to prompt position; 2) IF score has headroom (max of for the most complex prompts), and is only moderately correlated with writing quality. We plan to release the benchmark and code upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
instruction following, long context, model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1MUNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MERIDIAN, a benchmark for long-context instruction following (IF) over real-world documents. It contains 3,131 prompts across five domains (SEC financial filings, U.S. Congressional records, PubMed Central articles, AWS documentation, and English Wikipedia), with contexts from under 8k to 128k+ tokens built by padding a target document with same-domain distractors. Each prompt combines a document task (summarization, classification, closed QA, list extraction) with 2–10 typed constraints drawn from a taxonomy of generic, domain-static and domain-templated constraints. These are split between the system and user prompts and checked by code, LLM, or hybrid verifiers. User prompts are paraphrased by an LLM and validated by a single human annotator, and LLM rubrics are revised until judge-to-judge Krippendorff’s α ≥ 0.667.

Scoring separates a rubric-based Task score (an LLM judge, Sonnet-4.5) from a strict all-or-nothing IF score, and combines them as 0.6·Task + 0.4·IF. The authors evaluate roughly 10 open- and closed-weight models (different subsets per experiment) along three research questions. RQ1: Task and IF scores are only moderately correlated (r ≈ 0.50), and the Combined score drops on average from 45.4 to 38.3 between 8k and 128k. RQ2: moving the user request (first/middle/dispersed vs. last) or delivering it over multiple turns lowers scores by about 1–12 points, mainly through the Task score. RQ3: when one user instruction is moved inside a non-target document, models still obey it 32–49% of the time, and system-prompt IF exceeds user-prompt IF. A post-hoc case study shows that a discourse-coherence score correlates with Task score at r = 0.65 across 11 models.

Critical strengths

S1 – Realistic, multi-axis long-context IF setting with task/IF decoupling. Most IF benchmarks use short, synthetic contexts, while most long-context benchmarks assume a cooperative prompt with no output constraints. MERIDIAN combines real professional documents in five domains, typed constraints including domain-specific ones (e.g., dollar precision, legal citation style, hazard ratios with event counts), and explicit control over context length, constraint count, placement and turn structure. Reporting a judged Task score next to a strict IF score is a useful design choice: Figure 3 and Table 1 show that models keep moderate task quality (Task ≈ 0.4–0.6) while almost never satisfying every constraint (IF ≤ 0.085 for hard prompts). That finding is a concrete, actionable diagnostic that a single aggregate score would hide. This is the main reason the benchmark could be valuable to the community.

S2 – Controlled, paired perturbation design. Placement variants keep the system message, document order and request bytes fixed and are compared within prompt against the native LAST layout (Table 10, n = 873 per model). The multi-turn study likewise pairs identical user-authored content across single- and multi-turn delivery. This attribution-preserving design is more rigorous than many leaderboard-style benchmarks and gives credible evidence that layout alone can shift scores by 2–8 points for some models (e.g., Sonnet-5 MIDDLE: −8.0 Task, p < 0.001).

Critical weaknesses

W1 – Key claims rest on uncontrolled or unvalidated measurements. (a) Injection: the injected items are benign user constraints, and there is no control in which the constraint is absent, so the 32–49% injection rates and the domain gaps (Table 11) may mostly reflect base rates of satisfying the constraint anyway. (b) Hierarchy: system and user constraints appear to be different sets (§3.2, Fig. 1) and no system–user conflicts are reported, so “system IF > user IF” may be a constraint-type effect rather than evidence of instruction hierarchy. (c) Context length: domain and complexity are unevenly spread across length bins (Table 6), and the paper does not say whether the same prompts are padded to each length. (d) Evaluation: one Sonnet-4.5 judge, validated only by judge-to-judge agreement, with no human labels; it also scores same-family models, and each model is sampled once. These issues undermine all three research-question conclusions.

W2 – Limited novelty and inconsistent reporting. The findings largely confirm prior work on long-context IF [1], instruction hierarchy [2, 3], positional sensitivity [4] and prompt injection [5, 6], so the value lies mostly in the benchmark. The reporting is also hard to audit: model subsets change between experiments (9, 10, 4 and 3 models); the placement sample sizes disagree (1,156 vs. 873); Table 2 applies an unpaired test to paired data; IF-legit (Table 5) and IF (Table 1) use different metrics without saying so; and decoding settings and the 0.6/0.4 Combined weighting are neither specified nor justified.

References

[1] X. Wu, M. Wang, Y. Liu, X. Shi, H. Yan, X. Lu, J. Zhu, and W. Zhang. LIFBench: Evaluating the instruction following performance and stability of large language models in long-context scenarios. In Proceedings of ACL, 2025. [2] Z. Zhang, S. Li, Z. Zhang, X. Liu, H. Jiang, X. Tang, Y. Gao, Z. Li, H. Wang, Z. Tan, et al. IHEval: Evaluating language models on following the instruction hierarchy. In Proceedings of NAACL, 2025. [3] Y. Qin, T. Zhang, Y. Shen, W. Luo, H. Sun, Y. Zhang, Y. Qiao, W. Chen, Z. Zhou, W. Zhang, et al. SysBench: Can LLMs follow system message? In Proceedings of ICLR, 2025. [4] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. [5] Z. Li, B. Peng, P. He, and X. Yan. Evaluating the instruction-following robustness of large language models to prompt injection. In Proceedings of EMNLP, 2024. [6] J. Yi, Y. Xie, B. Zhu, E. Kiciman, G. Sun, X. Xie, and F. Wu. Benchmarking and defending against indirect prompt injection attacks on large language models. In Proceedings of KDD, 2025.

Questions for authors

Q1 (RQ1, context length). Is the context-length analysis paired, i.e., is the same (target document, task, constraint set) instantiated at 8k, 16k, …, 128k with only the distractors changing? If not, can you report a paired or stratified analysis (by domain and complexity) showing that the length effect is not a composition effect?

Q2 (RQ3, injection). What is the base rate at which each injected constraint is satisfied when it appears nowhere in the prompt? Please report injection rates net of this base rate, broken down by constraint type.

Q3 (RQ3, instruction hierarchy). In Figure 5 and Tables 7–8, are the same constraints placed in the system prompt and in the user prompt, or are the two sets different (e.g., format vs. content constraints, as §3.2 and Figure 1 suggest)? If they differ, how do you separate a hierarchy effect from a constraint-type effect? Were the system–user conflict prompts described in Appendix B evaluated?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1MUN

We appreciate the reviewer's thorough feedback. We hope our responses address all their concerns. We look forward to interacting further on their review.

Weaknesses #1a & 1b, Questions #2 & 3: injection and hierarchy controls

We agree and ran both controls on Opus 5.5 and GPT-6.1 Sol. Without the injected instruction, answers satisfy it 25-27% of the time, so net injection compliance is +5.3 and +8.9 points (p < 0.001). For hierarchy, we restricted the comparison to the 41 instruction types that appear in both channels. System instructions are still followed more often (83% vs. 71% and 85% vs. 69%, p < 0.001), so the effect is not a constraint-type effect. We will fix the Figure 5 caption (lines 426-427).

Weakness #1c & Question #1: context length

The length analysis is not paired (lines 190-195). In a new analysis with one intercept per domain × complexity cell, the newer models show no length decline. We will report this analysis for all models. We will also add a paired subset padded to all five lengths (lines 130-131).

Weakness #1d: judge validation

Judges rejected deliberately violating responses with 0.89-0.98 specificity. We will add these results, a second-family judge and human labels for the camera-ready version of our paper.

Weakness #2: novelty and reporting

We also answered novelty for Reviewer dXEL. Prior findings come from benchmarks that saturate quickly (Ott et al., 2022; Pyatkin et al., 2025; Yen et al., 2025). MERIDIAN tests these effects on real documents with adversarial stumping selection, where only 8.7% of legal candidates passed. The effects survive, and the newer models still reach only 0.27 strict IF. Our new control also shows that the multi-turn loss comes from the model's own intermediate replies, not from turn distance (p = 0.35).

We have already moved Table 2 to paired tests and added all-or-nothing IF to the injection tables. We will address the remaining reporting comments in the revision.

Thank you very much for your review

Thanks for the response, I will maintain my socre.

Reviewer_YxVPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MERIDIAN is a benchmark for evaluating instruction following over long, real-world documents across five domains: finance, legal, medicine, cloud documentation, and general knowledge. It contains 3,131 prompts spanning context lengths up to 128k tokens and four task types: summarization, classification, closed question answering, and list extraction.

Each prompt combines a document-understanding task with generic and domain-specific constraints, evaluated using deterministic validators and LLM-based rubrics. The benchmark separately reports task performance, all-or-nothing instruction compliance, and a weighted combined score. It further evaluates instruction placement, multi-turn delivery, and instructions embedded in non-target documents.

The authors evaluate several model families and report declining performance with longer contexts, low complete-compliance rates for complex prompts, sensitivity to instruction placement, and substantial satisfaction of instructions embedded in documents. Human review is used to validate prompt feasibility and paraphrase faithfulness, while rubric refinement uses inter- and intra-judge agreement.

Critical strengths
  1. A useful benchmark spanning several previously separated evaluation dimensions.

MERIDIAN combines real-world documents, professional-domain constraints, long contexts, and robustness transformations within one framework. The coverage is practically relevant and could support more informative model diagnostics than either short-context instruction-following tests or unconstrained long-context tasks alone.

  1. Explicit separation of task success from constraint compliance.

The paper avoids treating a semantically adequate answer as necessarily instruction-compliant. Its combination of fixed per-constraint verifiers, human-reviewed prompts, and separate task/IF reporting provides a strong foundation for analyzing distinct failure modes.

Critical weaknesses
  1. Several robustness experiments do not isolate the capability they claim to measure.

The multi-turn experiment changes system-role instructions into user-role instructions and adds intermediate assistant messages, so its performance differences cannot be attributed solely to turn structure. Similarly, the injection experiment measures whether an embedded instruction is satisfied, without establishing whether that satisfaction exceeds a no-injection baseline.

  1. The reliability and diagnostic interpretation of strict IF scoring are insufficiently established.

With an all-or-nothing metric, complete compliance can become very low simply because several moderately reliable constraints are composed. Likewise, small per-constraint evaluator errors can accumulate into substantial response-level errors. The supplied text reports rubric agreement and refinement, but not sufficient human-grounded evaluation of model-output judgments, broken down by domain, constraint type, and context length. Without complementary per-constraint results and validator audits, the low HARD scores may conflate model failures, compositional difficulty, and evaluator errors.

Questions for authors
  1. Can you isolate turn structure from instruction authority?

Please compare single-turn and multi-turn variants that retain the same system prompt in the system role. How much of the reported degradation remains? Please also specify whether intermediate assistant replies are retained in subsequent context and whether they increase effective context length or introduce premature answers.

  1. What is the counterfactual baseline for injection compliance?

For each injected constraint, what fraction of responses satisfy it when the same document does not contain that instruction?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer YxVP

We appreciate the reviewer's constructive feedback. We hope our responses address all their concerns. We look forward to interacting further on their review.

Weakness #1 & Questions #1–2: multi-turn and injection controls: For multi-turn, the system-to-user change applies to both arms. The paired single-turn control also sends the identical bytes in the user role (Appendix F, lines 1039-1043), and we will make this explicit in §3.3. The remaining difference is the model's own intermediate replies. These are retained in context, capped at 512 tokens, and do cause premature answering: the final answer repeats the previous assistant turn in 55.9–59.6% of conversations. The cost falls on task quality (ΔTask −0.074, 95% CI [−0.087, −0.060]), not strict IF (−0.010, [−0.022, +0.003]). We will also add the system-retained comparison the reviewer requests.

For injection, we agree. We thank the reviewer for this point. We added an absent arm that sends the same prompt, byte for byte, without the injected instruction. Answers satisfy that instruction 26.6% and 25.3% of the time even when it appears nowhere. Net of this base rate, injection compliance is +5.3 and +8.9 points (p < 0.001), and it concentrates in generic format constraints (+8.3 and +10.7). We will report net rates by constraint type and revise our claims to match.

Weakness #2: reliability of strict IF scoring: Low all-or-nothing scores are not only a product of composing constraints. In medicine, independent failures would predict all-or-nothing scores of 0.03–0.10 from per-constraint pass rates, but we observe 0.11–0.20. Verifier errors also do not drive the low scores: removing the four strictest verifiers moves the score only from 0.163 to 0.170. Beyond judge agreement, judges rejected responses that deliberately violate a constraint with 0.89–0.98 specificity. We will add per-constraint results, these audits, and human labels on outputs stratified by domain, constraint type and context length.

Thank you for your response. After considering the response, I will maintain my original score.

Reviewer_dXELOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MERIDIAN evaluates long-context instruction following on 3,131 prompts built from financial, legal, biomedical, cloud-service, and general-domain documents. Each instance specifies a document-understanding task and additional constraints, with other documents used to extend the context. Tasks include summarization, classification, closed question answering, and list extraction. The benchmark separately measures task quality and strict constraint satisfaction using LLM judges and code verifiers. It evaluates changes in context length, instruction placement, single- versus multi-turn delivery, and the treatment of instructions embedded in documents. The paper reports results for existing language models and does not introduce a model-training or inference-improvement algorithm.

Critical strengths

The benchmark uses documents from several practically relevant domains and includes human review of prompt feasibility and paraphrase fidelity. This provides a useful collection of document-grounded instruction-following tasks.

The reported task and constraint scores make it possible to distinguish a substantively adequate answer from one that follows every explicit requirement. The placement and multi-turn experiments provide concrete examples of this distinction.

Critical weaknesses

Limited novelty. The main design components already have clear precedents. LIFBench (ACL 2025), Section 3.3 varies context length by changing the number of documents or list elements. Lost in the Middle (TACL 2024), Section 2.1 places an answer-bearing document among distractor documents and changes context length by varying the number of distractors. HELMET (ICLR 2025), Appendix B.1 also fills contexts with negative passages. Instruction placement is directly studied in BAMBOO (LREC-COLING 2024), Section 4.3 and Table 5, which compares instructions before, after, and at both ends of the context. Long-context instruction following and multi-turn instruction adherence are also established evaluation settings, as illustrated by LIFBench and VerIFY (EACL Findings 2026). Given these precedents for the central components, I find the benchmark's methodological novelty insufficient.

The model evaluation is not sufficiently up to date. The reported panel includes Kimi-K2.5, Qwen3-80B, and Opus-4.8, but does not cover more recent representatives such as Kimi K3, Qwen3.8-Max, and Claude Opus 5.5. These were released on July 16, August 3, and September 22, 2026, respectively. This limits how well the reported rankings and failure patterns characterize the current model landscape and weakens the empirical contribution of the benchmark.

Questions for authors

Which specific designs in MERIDIAN are novel relative to prior benchmarks such as LIFBench, Lost in the Middle, HELMET, BAMBOO, and VerIFY? If the claimed novelty lies in the overall combination, please substantiate what this combination contributes beyond assembling existing components. A concrete comparison with the closest prior work should identify the new evaluation capabilities or findings enabled by your design.

Can you extend the evaluation to more recent models, such as Kimi K3, Claude Opus 5.5, and Qwen3.8, and check whether the reported failure patterns persist? Please clarify the cutoff date used to select the evaluated models.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer dXEL

We appreciate the reviewer's feedback. We hope our responses address all their concerns. We look forward to interacting further on their review.

Weakness #1 & Question #1: novelty relative to prior benchmarks

We thank the reviewer for the pointers. We will cite BAMBOO (Dong et al., 2024) and VerIFY (Robinette et al., 2026) and add a comparison table. Each prior work covers one component:

  • LIFBench (Wu et al., 2025) scores programmatic rubrics on synthetic lists and essays. It has no domain constraints and no task-quality score.
  • Lost in the Middle (Liu et al., 2024) and HELMET (Yen et al., 2025) move the evidence and measure task accuracy. They impose no output constraints.
  • BAMBOO moves the task question within 4k-16k contexts. It has no constraints.
  • VerIFY adds filler turns and checks style instructions. It has no documents and no task scoring.

MERIDIAN combines domain-specific constraints over real professional documents (lines 204-211), separate Task and strict IF scores for the same response (lines 246-257), and byte-matched perturbations (lines 221-243). It also uses adversarial "stumping" selection, which we under-described and will add to §3.2. A prompt is kept only if the best model in a pool follows fewer than 70% of its constraints, and only 8.7% of legal candidates passed. This matters because static benchmarks saturate quickly (Ott et al., 2022; Pyatkin et al., 2025), and most models now score perfectly on needle-in-a-haystack (Yen et al., 2025).

The combination also produces findings that no single component could. Following VerIFY's design, padding from 0 to 16 turns does not significantly reduce task score (p = 0.35, lines 1048-1052). A new paired control replaces the model's intermediate replies with a fixed "Noted." Turn boundaries alone then have no significant effect (−0.9 and +0.6 Combined). The model's own intermediate replies drive the loss (−3.8 and −2.0, p < 0.001).

Weakness #2 & Question #2: recent models

We re-ran every experiment with Claude Opus 5.5 and GPT-6.1 Sol, using our updated scorers. Neither model saturates the benchmark. Combined is 60.9 and 59.8, and strict IF is only 27.2 and 26.8 despite Task scores of 83.3 and 81.8.

Most failure patterns persist:

  • On hard prompts, IF collapses to 11.2 while Task stays at 83.6.
  • System instructions are followed more often than user instructions, even on identical instruction types (+11.9 and +15.5 points).
  • Multi-turn delivery still costs 4.7 and 1.4 Combined points.
  • In the general domain, middle placement costs 2.4 and 2.7 points.

The context-length decline does not persist. Holding domain and complexity fixed, Combined rises by 3.2 points from 8k to 128k, while the paper's models lost 1.7-10.7. We will report this as a finding: newer models close the length gap, but compositional instruction following remains unsolved. Our main experiments ran in August 2026, and we will state this cutoff in §4.1 (line 267).

Thank you for the detailed responses and additional experiments. They help clarify the contribution and address several of my concerns. While some concerns remain, I appreciate the additional evidence and will raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.