- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the relationship between retrieving and using information in long contexts. Using controlled experiments on a synthetic arithmetic task, it compares copying a value with computing from that value while varying dependency distance at fixed context length. It finds a performance gap between these tasks and examines how that gap changes with factors like context structure, model size, and generation budget.
Critical strengths
The fixed-length dependency-distance experiments and paired retrieval/computation comparisons provide useful controlled experiments. They establish a systematic behavioral difference on the same underlying programs, giving the paper empirical value beyond an aggregate accuracy comparison.
Critical weaknesses
-
The additional insight from the task construction is insufficiently established. GSM-Infinite(https://arxiv.org/abs/2502.05252v1) builds on the controlled synthetic reasoning constructions in Physics of Language Models(https://physics.allen-zhu.com/part-2-grade-school-math/) and extends them to long contexts, including symbolic assignments and accuracy comparisons at fixed operation counts. The present paper’s fixed-length distance manipulation and matched retrieval/use queries provide useful additional controls. However, it remains unclear which substantive conclusion these controls establish beyond prior findings. A larger performance drop, or its appearance in a simpler task, needs a clearer explanation of what new understanding it provides.
-
The evaluation supports a behavioral contrast more clearly than the stronger access/use interpretation. Successful copying in a separate query does not establish successful retrieval during a failed computation. Although intermediate values are supplied, computation still requires locating and binding the relevant operands; query dependent retrieval, interference, and composition therefore remain plausible contributors. The response protocol adds uncertainty: the main evaluation uses eight-token raw completions, while Table 5 shows substantial recovery under longer question-based responses. Those comparisons also change prompting and answer extraction, leaving their contributions to the main contrast incompletely separated. The paper acknowledges several of these limitations, but wording about using information the model has “already located” exceeds what the comparisons can establish. This limits both the interpretation of the gap and the explanatory contribution of the study.
Questions for authors
Which specific conclusion depends on the fixed-total-length distance manipulation and matched retrieval/use controls, and goes beyond the findings of Physics of Language Models and GSM-Infinite? How do these controls distinguish the proposed phenomenon from previously observed context-related degradation?
Is “access–use dissociation” intended as an operational difference between query types, or as evidence that information was successfully retrieved during computation but subsequently not used? For the latter interpretation, what evidence distinguishes failed use from query dependent retrieval or operand binding difficulties?
Could you provide a small matched near/far comparison of retrieval, compute, and combine using a supported chat template and budgets sufficient for answers to complete, with prompting and answer extraction held fixed within each budget comparison? Reporting accuracy alongside missing-answer and formatting-error rates would help determine how much of the gap persists beyond the restricted completion tokens.
Discussion Period
Dear Reviewers,
The authors have posted their response. Please take a moment to review it and engage in discussion with both the authors and fellow reviewers, especially on any remaining concerns or points of disagreement. Brief updates to your assessment would be very helpful.
Thank you.