- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper explores how positional information is stored in models, motivated by human working memory's hierarchical structure. They show across 5 LLMs that different structural cues in a prefix and between items can shape how accurately the model retrieves different positional information. Their core results depend on 4-6 conditions that vary the prefix and the sequence, exploring how this impacts absolute/group/slot positional accuracy.
Critical strengths
- There does seem to be a boost in group positional accuracy at the cost of absolute position when moving from L0 to L2, but these are only two prompts (“You will see an ordered sequence of chunks.”, “A larger gap marks a boundary in the sequence.”) and it is unclear what underlying difference between these prompts is causing this behavioral change.
Critical weaknesses
- This paper is incredibly difficult to parse. The connection to human working memory seems mostly unnecessary, and not nearly enough time has been spent motivating the use of the main task studied throughout this work and why this is a useful task for better understanding the internals of LMs.
- Methodologically, running the same task with four different prefixes is insufficient for studying behavior more broadly. There are many confounders, and at minimum you would want multiple prompts per category. Beyond this, the connection between these prompts and the absolute/group/slot retrieval are loose at best and poorly explained.
Questions for authors
- Could the authors please explain the LM exclusion results in Figure 3 and an analysis on the errors of models such as Kimi for L0/L1 where they are not excluded?
- Why did the authors make the following choice: "the L1 vertical range is approximately twice that of L0, similar visual amplitudes understate the stronger slot dependence under L1." if it makes the figure harder to read? Why would you not just keep the same y-scale for both?
- Why is exact positional information necessary in the first place? Very few tasks in practice require absolute position lookup, so why is studying this problem useful in the first place?
4 comments
Full prompts and sensitivity across wording families
Strengths
There does seem to be a boost in group positional accuracy at the cost of absolute position when moving from L0 to L2, but these are only two prompts (“You will see an ordered sequence of chunks.”, “A larger gap marks a boundary in the sequence.”) and it is unclear what underlying difference between these prompts is causing this behavioral change.
W2
Methodologically, running the same task with four different prefixes is insufficient for studying behavior more broadly. There are many confounders, and at minimum you would want multiple prompts per category. Beyond this, the connection between these prompts and the absolute/group/slot retrieval are loose at best and poorly explained.
Our claim is that prompt structure can affect sequential memory in LLMs. We did not claim that all prompts produce the same effect. The changes observed across the four original prefixes provide evidence that prompt structure can change sequential-memory behavior. To examine wording sensitivity, we added three paraphrase families alongside the original P0, testing Gemma-4-31B-IT in EN/ZH on 64 matched base sequences per language. The added methods, results, and response-behavior analyses are reported together in Appendix G.5 of the revised paper (Tables A33–A35).
The families preserve the sequence, targets, worked examples, coordinate questions, and answer-format instructions. For a direct comparison, the EN L0 encoding and the sentence added in L2 are:
| Family | EN L0 encoding | Sentence inserted after the first sentence in L2 |
|---|---|---|
| P0 | You will see an ordered sequence of chunks. Remember the chunks and the order in which they appear. | A larger gap marks a boundary in the sequence. |
| P1 | You will be shown an ordered sequence of chunks. Keep in mind the chunks and the order of their appearance. | A wider gap indicates a boundary in the sequence. |
| P2 | An ordered sequence of chunks will be shown. Retain the chunks and the order in which they come. | A wider gap shows a boundary in the sequence. |
| P3 | The following is an ordered sequence of chunks. Remember the chunks and their order of appearance. | A bigger gap signals a boundary in the sequence. |
We follow the coordinate-accuracy, Local-U, and error-pattern analyses in Secs. 4–5 and Appendices F.2 and F.5. The table shows L2−L0 accuracy changes in percentage points (pp):
| Language | Family | Absolute | Group | Slot |
|---|---|---|---|---|
| EN | P0 | −27.60 | +27.60 | +9.38 |
| EN | P1 | −28.87 | −26.56 | −5.21 |
| EN | P2 | −25.74 | +20.83 | +13.54 |
| EN | P3 | −33.78 | +13.54 | +6.25 |
| ZH | P0 | −26.33 | +17.19 | +3.65 |
| ZH | P1 | −27.95 | +30.21 | +3.65 |
| ZH | P2 | −28.18 | +21.88 | +5.21 |
| ZH | P3 | −26.62 | +29.17 | +2.60 |
Seven of the eight language×family combinations reproduce P0's direction: absolute accuracy decreases and group/slot accuracy increases. The absolute cost repeats in all eight, including L3−L1. These repeated effects support the conclusion across the tested wordings. Raw Local-U increases under L1−L0 and L2−L0 in both languages in all four families, and boundary conditions retain a stronger period-three component in absolute errors.
EN-P1 differs chiefly in its unmarked baseline. L0 group accuracy is 98.96%, versus 48.96%, 50.52%, and 58.85% in P0/P2/P3; L2 stays within 71.35–76.56%. P1 L0 slot accuracy also reaches 100%. Thus P1 already supports highly accurate group reporting without an explicit boundary, changing the subsequent contrast. EN L1−L0 group gains are +49.48, +0.52, +48.44, and +39.58 pp across P0–P3; L3−L2 group gains remain positive in every EN family. ZH retains its original negative L1−L0 group contrast and positive boundary-related group contrasts.
Visible answer behavior helps explain the P1 baseline. Complete ordered group lists occur in 35/192, 163/192, 38/192, and 59/192 EN L0 group responses across P0–P3. All 96 newly correct P1 group responses accompany a bare-integer-to-ordered-list change.
Two paired examples show what this change looks like:
| EN L0 probe/target | P0 answer | P1 response excerpt and final answer |
|---|---|---|
| Group: BR in FABRIC (gold 5) | 4 | Lists all nine words, including 5. FABRIC (FA BR IC); ends 5. |
| Slot: IX in MATRIX (gold 3) | 4 | Identifies MATRIX and lists 1. MA, 2. TR, 3. IX; ends 3. |
This association shows that wording can change visible organization and reporting behavior under the same answer-format instruction. Taken together, the common effects and the EN-P1 exception support prompt-sensitive sequential-memory reporting, with the relative-coordinate gains depending on what the unmarked wording already supports.
Model coverage, Kimi errors, and figure presentation
Q1
Could the authors please explain the LM exclusion results in Figure 3 and an analysis on the errors of models such as Kimi for L0/L1 where they are not excluded?
All five models passed the competence screen in both languages and entered the core fixed-three L0/L1 comparison. Figure 3's narrower coverage concerns the boundary and variable-length follow-ups, which used GPT-5.6 Sol, Kimi-K3, and Gemma-4-31B-IT.
Sol and Kimi had the strongest preliminary positional readouts, and Gemma provided a locally served open-weight model with reliable short-load readout. In preliminary boundary checks, Luna produced out-of-range reports and predominantly overshot on its remaining errors. GLM predominantly undershot. These diagnostics motivated the follow-up selection. The follow-up rationale and calibration results are reported in Appendix D.2.
The original paper already examines errors of models. Sec. 4 and Appendix F.1 show that Kimi-ZH's L1 group decrease is concentrated at later groups and consists of reports of earlier group indices, while slot accuracy remains high. Sec. 5, Figure 6, and Appendix F.5 compare adjacent-position and period-three absolute errors, including Kimi-ZH's grouping-related L0 pattern and the shuffled controls. We added an EN/ZH comparison, a four-panel error-distribution figure, and literal response examples in Appendix F.1 (Figure A1) to make these patterns easier to inspect.
Kimi's L0/L1 behavior differs across languages and coordinates:
| Kimi | Absolute L0 → L1 | Group L0 → L1 | Slot L0 → L1 | Mean group error L0 → L1 | Group L3 |
|---|---|---|---|---|---|
| EN | 94.20 → 86.68% | 23.05 → 27.21% | 61.33 → 75.78% | −1.95 → −1.43 | 98.18% |
| ZH | 97.57 → 83.28% | 83.20 → 41.54% | 94.01 → 97.27% | −0.15 → −0.82 | 99.35% |
All 449 incorrect ZH L1 group reports give an earlier group index, and late-group accuracy falls from 69.80% to 5.49%. EN already underestimates group position in L0, and the premise reduces that underestimation. Visible boundaries in L3 raise group accuracy above 98% in both languages. Thus the premise alone has language-dependent effects on group identification, while explicit boundaries make the groups easier to identify. The added figure shows this contrast directly: EN group reports shift toward the correct index, whereas ZH reports shift away from it toward earlier groups. Its absolute-error panels show mainly one-item slips in EN and prominent three-item slips in ZH.
Examples below are literal saved answers from common error categories; , , and denote the true absolute, group, and slot positions:
| Language/condition | Probe | Target in source word | True | Raw answer |
|---|---|---|---|---|
| EN L1 | Group | VI in SOVIET | (11,4,2) | 3 |
| EN L1 | Absolute | RB in CARBON | (17,6,2) | 18 |
| ZH L1 | Group | 顺 in 顺风车 | (28,10,1) | 9 |
| ZH L0 | Absolute | 魔 in 魔术师 | (40,14,1) | 37 |
In the added count summary on the matched 64-sequence subset of original absolute responses, one-item slips account for 65/69 EN L0 errors and 135/151 EN L1 errors; ZH has 22/38 three-item slips in L0 and 126/241 in L1. The original shuffled comparison removes three-item slips in ZH S0 (0/169), while regular boundaries in S2 restore a positive position-normalized period-three coefficient, 0.155 (Appendix F.5). These patterns connect the ZH error bands to available grouping.
Q2
Why did the authors make the following choice: "the L1 vertical range is approximately twice that of L0, similar visual amplitudes understate the stronger slot dependence under L1." if it makes the figure harder to read? Why would you not just keep the same y-scale for both?
We agree that a common y-axis makes the comparison clearer. Figure 4 now uses identical limits of −15 to 10 pp and ticks every 5 pp for L0 and L1. We revise only the caption’s final sentence: “The generally larger vertical variation under L1 reflects its stronger slot dependence than L0.” The plotted estimates and statistical comparisons are unchanged.
Seeking your feedback on our rebuttal
We sincerely thank you for your detailed feedback and critical questions. We have carefully considered your concerns and provided additional experiments and clarifications.
Our main updates include:
- Task motivation: Clarified the scientific rationale for studying positional retrieval and its connection to human sequential-memory research.
- Prompt robustness (Appendix G.5): Added three paraphrase families, with the absolute-position cost reproduced across all eight language–wording combinations.
- Model coverage and error analysis: Clarified model selection, expanded the Kimi error analysis, and improved Figure 4's presentation.
We would greatly appreciate your feedback on whether our responses have addressed your main concerns. We remain very willing to discuss any unresolved issues or provide further clarification.
Thank you again for your time and consideration.
Task motivation, exact position, and human working memory
W1
Q3
We would like to clarify the scientific rationale of this controlled paradigm. Our study addresses sequential memory in LLMs, an important question about how models retrieve order and position under different structural cues. The Introduction and Related Work already motivate this question through LLM memory research and established human sequential-memory experiments (Secs. 1–2). Thank you for raising the concerns about task utility and the human comparison. We will explain these connections more directly below.
Controlled experiments make it possible to isolate effects that are difficult to distinguish in everyday tasks. A task's scientific value comes from the question it can resolve through precise manipulation and measurement, rather than how frequently that exact task occurs in daily use. Hu and Frank [1] show that auxiliary task demands can obscure language-model capabilities. A controlled sequential-memory task likewise lets us examine structural effects with fewer additional demands from interpreting content and generating an answer.
The underlying question matters for understanding LLM memory and retrieval. Liu et al. [2] show that access to information depends on where it appears in context, while LongMemEval studies the ability to retain and retrieve information across extended interactions [3]. Our question is how structural cues affect sequential memory: whether they change which aspects of the same order a model can report and how its errors depend on that structure. This complements content-retrieval evaluations by examining the organization of order and position themselves (Secs. 1–2).
The experimental design is grounded in human sequential-memory research. Fan et al. [4] study syllables organized into familiar words and distinguish continuous position, word position, and within-word position. We adapt their ZH materials and coordinate distinctions, along with their comparison of adjacent-position and same-slot errors. This supplies an established paradigm and testable behavioral predictions about how grouping influences different reports of order. The human findings motivate the design and predictions.
In our adaptation, the sequence remains available, the target is given, and each request asks for one positional integer (Sec. 3). This focuses the response on position while retaining the segmentation and counting computations involved in the task. We compare cue conditions on matched sequences and targets, asking separately for absolute, group, and within-group slot position. These probes are complementary: absolute position provides a whole-sequence reference, allowing us to distinguish a general improvement from a change that favors one coordinate over another. Group and slot queries presuppose a grouped sequence, whereas the absolute query asks only for continuous position. Absolute reports therefore also let us examine spontaneous sensitivity to the latent hierarchy under L0, without specifying a group-relative coordinate in the question. Absolute responses also provide signed error distances. Varying group length then distinguishes predictions based on a fixed numerical displacement from those based on a target's position within a group (Sec. 6). The design thus connects a sequential-memory question to controlled interventions and measurements that can discriminate between behavioral accounts.
[1] Hu, Jennifer, and Michael C. Frank. "Auxiliary task demands mask the capabilities of smaller language models." Conference on Language Modeling (COLM).
[2] Liu, Nelson F., et al. "Lost in the middle: How language models use long contexts." Transactions of the association for computational linguistics 12 (2024): 157-173.
[3] Wu, Di, et al. "Longmemeval: Benchmarking chat assistants on long-term interactive memory." International Conference on Learning Representations (ICLR).
[4] Fan, Ying, et al. "Two-dimensional neural geometry underpins hierarchical organization of sequence in human working memory." Nature Human Behaviour 9.2 (2025): 360-375.