- The effect of context length is not sufficiently isolated from increasing context complexity.
This is my primary concern. Across the four length tiers, the benchmark progressively activates additional interference sources and increases the RAG recall cap, from 0–20 in LA to approximately 150 in LD. Consequently, context length, the number of competing information sources, and the volume of retrieved material change together.
The observed degradation therefore cannot be attributed specifically to context length. It may instead reflect some combination of longer inputs, greater evidence density, additional opportunities for source conflict, and exposure to more fabricated or misleading information. Although the paper explicitly frames its design as measuring compound complexity rather than raw token count alone, several central claims emphasize length as the dominant explanatory factor.
The paper would be substantially stronger with controlled comparisons that vary token length while holding the underlying evidence and source configuration fixed, as well as comparisons that vary source count or retrieval volume at approximately fixed token length. Matched or minimally modified versions of the same underlying cases would also help control for differences in item difficulty. Without such experiments, the findings support degradation across increasingly complex context tiers, but a stronger causal interpretation of the independent effect of context length is not fully established.
- The ecological validity of the synthetic adversarial contexts remains uncertain.
The benchmark is motivated by real failure cases from third-party medical applications, which is a useful starting point. However, the interference materials, including fabricated profiles, reports, historical dialogue, memory snippets, and retrieved content, are largely generated synthetically by language models.
This construction enables controlled evaluation, but it also raises questions about whether the benchmark reflects the distribution and difficulty of source-attribution failures in actual medical interactions. The paper does not sufficiently establish how frequently these specific failure modes occur in real-world deployments or whether clinicians consider the constructed scenarios representative of realistic workflows.
The authors acknowledge the need for blinded clinical validation. I consider this more than a minor limitation because the paper makes clinically motivated claims about the significance of the observed failures. Independent clinician review of scenario realism, source ambiguity, and the plausibility of the injected errors would substantially strengthen the benchmark. Where feasible, validation on naturally occurring, appropriately de-identified cases would provide additional evidence of external validity.
- The evaluation protocol introduces potential scoring and model-family biases.
The use of five task formats is a strength, but the scoring procedures have different limitations. Exact-match evaluation may penalize semantically equivalent responses, while LLM-as-judge evaluation introduces dependence on the judge's interpretation of the rubric.
In particular, Claude is used for content generation and as the judge for the generative tasks, and Claude Opus 5 is also among the evaluated models. The authors acknowledge this potential homology bias, but the reported human–machine agreement analysis does not fully establish that the resulting scores are unbiased across model families.
The reported agreement is approximately 78–81% for the three-level faithfulness dimensions, while the much higher agreement on the binary advice and safety dimensions is partly attributable to their simpler scoring criteria. These results are useful, but stronger validation would include independent judges from different model families, blinded human assessment on a larger and more representative sample, and sensitivity analyses showing whether model rankings and the principal conclusions remain stable across judges and scoring schemes.
The paper should also clarify how exact-match scoring handles semantically correct alternatives and how disagreements between automated and human judgments affect aggregate results.
- The statistical analysis requires clearer treatment of item dependence and uncertainty.
The paper provides extensive statistical analyses, including bootstrap confidence intervals, ANOVA, and logistic regression. The repeated observation of a degradation trend across tasks and models is encouraging. Nevertheless, the statistical interpretation could be more rigorous.
The reported analyses involve multiple models evaluated on a shared set of items, and the benchmark's construction may introduce dependencies among cases. The paper should clearly explain how item-level dependence and repeated model observations are handled when estimating uncertainty. In particular, resampling model–item observations independently may not adequately reflect uncertainty associated with the sampled items or the evaluated model set. A cluster-aware analysis that preserves the relevant dependencies would provide a stronger basis for the reported confidence intervals.
There is also an apparent reporting inconsistency in Appendix A.9: the FCC degradation estimate is reported as −10.40 percentage points, whereas its listed confidence interval is [−15.7, −11.2], which does not contain the point estimate. This should be corrected or clarified.
Finally, the paper should distinguish more carefully between statistical significance and practical explanatory importance. A small p-value establishes evidence against a specified null hypothesis; it does not, by itself, establish that context length is the principal causal mechanism behind the observed failures.
- The interpretation of the proposed failure mechanisms is stronger than the direct evidence.
The discussion attributes the observed failures to attention dilution, instruction-following degradation, and competition between sources. The qualitative examples are useful illustrations of these possibilities, but they do not directly establish the underlying mechanisms.
For example, failures in very long contexts are consistent with attention dilution, but they are also consistent with the increased number of misleading sources and the greater complexity of the decision. Similarly, a model accepting a fabricated premise does not necessarily demonstrate general instruction-following degradation without a controlled comparison that isolates the relevant instructions and context components.
The authors appropriately describe these explanations as hypotheses in parts of the discussion. I recommend maintaining this distinction throughout the paper and avoiding causal language unless supported by targeted experiments. Controlled source-removal, source-tagging, and matched-length interventions could help distinguish these mechanisms.