- Limited generalizability of the evaluated retrieval system
The paper investigates position-dependent retrieval accuracy in long documents, but the evaluation relies on a single bag-of-words cosine-similarity retriever with a manually specified positional prior. While this is a valid retrieval baseline, it is unclear whether the observed recovery profiles generalize to other retrieval systems. In particular, bag-of-words similarity does not inherently model sequence order, and the manually specified prior is the only explicitly documented position-dependent scoring component. This makes it difficult to determine whether the observed results reflect general properties of position bias or characteristics of this particular retrieval setup.
Furthermore, the experiments use a single literary corpus and context lengths of at most 360 sentences. Although the source document is book-length, the evaluated contexts are substantially shorter than the full document. The paper does not establish how representative these settings are of broader long-document retrieval scenarios.
- Perfect retrieval accuracy creates a ceiling effect that makes the main conclusions uninformative
Tables 7–9 report mean recovery = 1.0 and minimum recovery = 1.0 for all three context lengths. Thus, the evaluated retriever achieves perfect accuracy regardless of needle position.
Under these circumstances, the absence of an interior dip is not strong evidence against position bias. It may simply indicate that the retrieval task is too easy for the chosen retriever. In particular, the results cannot distinguish between a system genuinely robust to positional variation and an evaluation that lacks sufficient difficulty to expose such variation.
This problem also affects the interpretation of the headline metrics. Table 6 reports a dip growth factor of 0.0 even though both the shortest-context and longest-context dip depths are zero. Since the metric is described as a ratio, this case requires an explicit mathematical convention. Additionally, fitting a quadratic to constant recovery values does not provide meaningful evidence about the location or curvature of a positional minimum.
Author Response to Reviewer UGxk: Compiled Bibliography, Prose, and AI Statement
We thank Reviewer UGxk for the candid critique and take full responsibility for the build and formatting errors in our initial PDF upload. We have remediated all three critical weaknesses and the Ethics concerns in the revised manuscript:
Complete 24-Entry Bibliography Compiled (Zero Plagiarism):
pdflatexexecuted withoutbibtex, plus a package conflict between\usepackage{cite}andnatbib) caused citations to fail to render and omitted the References section. In the revised PDF, all 24 references compile cleanly and contextualize our work within the long-context position-bias literature (including Liu et al., TACL 2024, "Lost in the Middle"; Kamradt, 2023; Press et al., ICLR 2022; Su et al., 2024; Kazemnejad et al., NeurIPS 2023; Xiao et al., ICLR 2024; Hsieh et al., COLM 2024).Mandatory AI Assistance Disclosure Added & Prose Completely Rewritten:
haic) alongside an LLM text-condensing script that introduced robotic boilerplate phrasing and omitted the\section*{AI Assistance Disclosure}section. We also mistakenly selected "No, not at all" on the submission form under the misimpression that the question asked whether AI generated our experimental data (REAL_DATA), rather than text editing.Resolving the Apparent Contradiction Between the Title and the Baseline
1.0Results:1.000(a lexical ceiling measurement trap), even though the continuous score margin Δs(d) followed a U-shape (a2=+0.240,R2=0.998).