acceptodds
Under review as a conference paper at ICLR 2027

Prompt Structure Reorganizes In-Context Sequential Memory in LLMs

Abstract

Large language models (LLMs) are sensitive to where information appears in their context. Research on human sequential memory shows that position is not a single quantity: beyond its absolute rank in the whole sequence, an item in a grouped sequence carries a global rank, the position of its group, and a local rank, its position within the group, which are represented along separable neural dimensions. Existing evaluations of LLMs ask what appears at a position, not where an item is, so whether LLMs hold several positional descriptions of an item, and whether the prompt determines which one is accessible, remains unknown. Here we show, across five LLMs probed with controlled English and Chinese sequences of familiar words, that structural cues reorganize positional retrieval. Stating that the sequence consists of familiar words mainly improves reports of a target's group and within-group position. Making group boundaries visible improves these reports further, but often lowers exact absolute-position accuracy and redirects absolute-position errors toward the displayed grouping. These boundary effects persist after the familiar-word structure is destroyed and when group lengths vary, where exact absolute reports can collapse while reported positions remain strongly rank-correlated with true positions. Retrieval thus depends on the match between the organization a cue emphasizes and the coordinate a query demands, paralleling hierarchical working memory in humans, although in LLMs this organization is largely cue-dependent. Prompts therefore act as organizational interventions on in-context sequential memory, and memory systems should be evaluated across the positional readouts they support and for interference between them.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sequential memory, large language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TbWWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores how positional information is stored in models, motivated by human working memory's hierarchical structure. They show across 5 LLMs that different structural cues in a prefix and between items can shape how accurately the model retrieves different positional information. Their core results depend on 4-6 conditions that vary the prefix and the sequence, exploring how this impacts absolute/group/slot positional accuracy.

Critical strengths
  • There does seem to be a boost in group positional accuracy at the cost of absolute position when moving from L0 to L2, but these are only two prompts (“You will see an ordered sequence of chunks.”, “A larger gap marks a boundary in the sequence.”) and it is unclear what underlying difference between these prompts is causing this behavioral change.
Critical weaknesses
  • This paper is incredibly difficult to parse. The connection to human working memory seems mostly unnecessary, and not nearly enough time has been spent motivating the use of the main task studied throughout this work and why this is a useful task for better understanding the internals of LMs.
  • Methodologically, running the same task with four different prefixes is insufficient for studying behavior more broadly. There are many confounders, and at minimum you would want multiple prompts per category. Beyond this, the connection between these prompts and the absolute/group/slot retrieval are loose at best and poorly explained.
Questions for authors
  • Could the authors please explain the LM exclusion results in Figure 3 and an analysis on the errors of models such as Kimi for L0/L1 where they are not excluded?
  • Why did the authors make the following choice: "the L1 vertical range is approximately twice that of L0, similar visual amplitudes understate the stronger slot dependence under L1." if it makes the figure harder to read? Why would you not just keep the same y-scale for both?
  • Why is exact positional information necessary in the first place? Very few tasks in practice require absolute position lookup, so why is studying this problem useful in the first place?
4 comments
AuthorsOpenReview ↗

Task motivation, exact position, and human working memory

W1

This paper is incredibly difficult to parse. The connection to human working memory seems mostly unnecessary, and not nearly enough time has been spent motivating the use of the main task studied throughout this work and why this is a useful task for better understanding the internals of LMs.

Q3

Why is exact positional information necessary in the first place? Very few tasks in practice require absolute position lookup, so why is studying this problem useful in the first place?

We would like to clarify the scientific rationale of this controlled paradigm. Our study addresses sequential memory in LLMs, an important question about how models retrieve order and position under different structural cues. The Introduction and Related Work already motivate this question through LLM memory research and established human sequential-memory experiments (Secs. 1–2). Thank you for raising the concerns about task utility and the human comparison. We will explain these connections more directly below.

Controlled experiments make it possible to isolate effects that are difficult to distinguish in everyday tasks. A task's scientific value comes from the question it can resolve through precise manipulation and measurement, rather than how frequently that exact task occurs in daily use. Hu and Frank [1] show that auxiliary task demands can obscure language-model capabilities. A controlled sequential-memory task likewise lets us examine structural effects with fewer additional demands from interpreting content and generating an answer.

The underlying question matters for understanding LLM memory and retrieval. Liu et al. [2] show that access to information depends on where it appears in context, while LongMemEval studies the ability to retain and retrieve information across extended interactions [3]. Our question is how structural cues affect sequential memory: whether they change which aspects of the same order a model can report and how its errors depend on that structure. This complements content-retrieval evaluations by examining the organization of order and position themselves (Secs. 1–2).

The experimental design is grounded in human sequential-memory research. Fan et al. [4] study syllables organized into familiar words and distinguish continuous position, word position, and within-word position. We adapt their ZH materials and coordinate distinctions, along with their comparison of adjacent-position and same-slot errors. This supplies an established paradigm and testable behavioral predictions about how grouping influences different reports of order. The human findings motivate the design and predictions.

In our adaptation, the sequence remains available, the target is given, and each request asks for one positional integer (Sec. 3). This focuses the response on position while retaining the segmentation and counting computations involved in the task. We compare cue conditions on matched sequences and targets, asking separately for absolute, group, and within-group slot position. These probes are complementary: absolute position provides a whole-sequence reference, allowing us to distinguish a general improvement from a change that favors one coordinate over another. Group and slot queries presuppose a grouped sequence, whereas the absolute query asks only for continuous position. Absolute reports therefore also let us examine spontaneous sensitivity to the latent hierarchy under L0, without specifying a group-relative coordinate in the question. Absolute responses also provide signed error distances. Varying group length then distinguishes predictions based on a fixed numerical displacement from those based on a target's position within a group (Sec. 6). The design thus connects a sequential-memory question to controlled interventions and measurements that can discriminate between behavioral accounts.

[1] Hu, Jennifer, and Michael C. Frank. "Auxiliary task demands mask the capabilities of smaller language models." Conference on Language Modeling (COLM).

[2] Liu, Nelson F., et al. "Lost in the middle: How language models use long contexts." Transactions of the association for computational linguistics 12 (2024): 157-173.

[3] Wu, Di, et al. "Longmemeval: Benchmarking chat assistants on long-term interactive memory." International Conference on Learning Representations (ICLR).

[4] Fan, Ying, et al. "Two-dimensional neural geometry underpins hierarchical organization of sequence in human working memory." Nature Human Behaviour 9.2 (2025): 360-375.

AuthorsOpenReview ↗

Full prompts and sensitivity across wording families

Strengths

There does seem to be a boost in group positional accuracy at the cost of absolute position when moving from L0 to L2, but these are only two prompts (“You will see an ordered sequence of chunks.”, “A larger gap marks a boundary in the sequence.”) and it is unclear what underlying difference between these prompts is causing this behavioral change.

W2

Methodologically, running the same task with four different prefixes is insufficient for studying behavior more broadly. There are many confounders, and at minimum you would want multiple prompts per category. Beyond this, the connection between these prompts and the absolute/group/slot retrieval are loose at best and poorly explained.

Our claim is that prompt structure can affect sequential memory in LLMs. We did not claim that all prompts produce the same effect. The changes observed across the four original prefixes provide evidence that prompt structure can change sequential-memory behavior. To examine wording sensitivity, we added three paraphrase families alongside the original P0, testing Gemma-4-31B-IT in EN/ZH on 64 matched base sequences per language. The added methods, results, and response-behavior analyses are reported together in Appendix G.5 of the revised paper (Tables A33–A35).

The families preserve the sequence, targets, worked examples, coordinate questions, and answer-format instructions. For a direct comparison, the EN L0 encoding and the sentence added in L2 are:

FamilyEN L0 encodingSentence inserted after the first sentence in L2
P0You will see an ordered sequence of chunks. Remember the chunks and the order in which they appear.A larger gap marks a boundary in the sequence.
P1You will be shown an ordered sequence of chunks. Keep in mind the chunks and the order of their appearance.A wider gap indicates a boundary in the sequence.
P2An ordered sequence of chunks will be shown. Retain the chunks and the order in which they come.A wider gap shows a boundary in the sequence.
P3The following is an ordered sequence of chunks. Remember the chunks and their order of appearance.A bigger gap signals a boundary in the sequence.

We follow the coordinate-accuracy, Local-U, and error-pattern analyses in Secs. 4–5 and Appendices F.2 and F.5. The table shows L2−L0 accuracy changes in percentage points (pp):

LanguageFamilyAbsoluteGroupSlot
ENP0−27.60+27.60+9.38
ENP1−28.87−26.56−5.21
ENP2−25.74+20.83+13.54
ENP3−33.78+13.54+6.25
ZHP0−26.33+17.19+3.65
ZHP1−27.95+30.21+3.65
ZHP2−28.18+21.88+5.21
ZHP3−26.62+29.17+2.60

Seven of the eight language×family combinations reproduce P0's direction: absolute accuracy decreases and group/slot accuracy increases. The absolute cost repeats in all eight, including L3−L1. These repeated effects support the conclusion across the tested wordings. Raw Local-U increases under L1−L0 and L2−L0 in both languages in all four families, and boundary conditions retain a stronger period-three component in absolute errors.

EN-P1 differs chiefly in its unmarked baseline. L0 group accuracy is 98.96%, versus 48.96%, 50.52%, and 58.85% in P0/P2/P3; L2 stays within 71.35–76.56%. P1 L0 slot accuracy also reaches 100%. Thus P1 already supports highly accurate group reporting without an explicit boundary, changing the subsequent contrast. EN L1−L0 group gains are +49.48, +0.52, +48.44, and +39.58 pp across P0–P3; L3−L2 group gains remain positive in every EN family. ZH retains its original negative L1−L0 group contrast and positive boundary-related group contrasts.

Visible answer behavior helps explain the P1 baseline. Complete ordered group lists occur in 35/192, 163/192, 38/192, and 59/192 EN L0 group responses across P0–P3. All 96 newly correct P1 group responses accompany a bare-integer-to-ordered-list change.

Two paired examples show what this change looks like:

EN L0 probe/targetP0 answerP1 response excerpt and final answer
Group: BR in FABRIC (gold 5)4Lists all nine words, including 5. FABRIC (FA BR IC); ends 5.
Slot: IX in MATRIX (gold 3)4Identifies MATRIX and lists 1. MA, 2. TR, 3. IX; ends 3.

This association shows that wording can change visible organization and reporting behavior under the same answer-format instruction. Taken together, the common effects and the EN-P1 exception support prompt-sensitive sequential-memory reporting, with the relative-coordinate gains depending on what the unmarked wording already supports.

AuthorsOpenReview ↗

Model coverage, Kimi errors, and figure presentation

Q1

Could the authors please explain the LM exclusion results in Figure 3 and an analysis on the errors of models such as Kimi for L0/L1 where they are not excluded?

All five models passed the competence screen in both languages and entered the core fixed-three L0/L1 comparison. Figure 3's narrower coverage concerns the boundary and variable-length follow-ups, which used GPT-5.6 Sol, Kimi-K3, and Gemma-4-31B-IT.

Sol and Kimi had the strongest preliminary positional readouts, and Gemma provided a locally served open-weight model with reliable short-load readout. In preliminary boundary checks, Luna produced out-of-range reports and predominantly overshot on its remaining errors. GLM predominantly undershot. These diagnostics motivated the follow-up selection. The follow-up rationale and calibration results are reported in Appendix D.2.

The original paper already examines errors of models. Sec. 4 and Appendix F.1 show that Kimi-ZH's L1 group decrease is concentrated at later groups and consists of reports of earlier group indices, while slot accuracy remains high. Sec. 5, Figure 6, and Appendix F.5 compare adjacent-position and period-three absolute errors, including Kimi-ZH's grouping-related L0 pattern and the shuffled controls. We added an EN/ZH comparison, a four-panel error-distribution figure, and literal response examples in Appendix F.1 (Figure A1) to make these patterns easier to inspect.

Kimi's L0/L1 behavior differs across languages and coordinates:

KimiAbsolute L0 → L1Group L0 → L1Slot L0 → L1Mean group error L0 → L1Group L3
EN94.20 → 86.68%23.05 → 27.21%61.33 → 75.78%−1.95 → −1.4398.18%
ZH97.57 → 83.28%83.20 → 41.54%94.01 → 97.27%−0.15 → −0.8299.35%

All 449 incorrect ZH L1 group reports give an earlier group index, and late-group accuracy falls from 69.80% to 5.49%. EN already underestimates group position in L0, and the premise reduces that underestimation. Visible boundaries in L3 raise group accuracy above 98% in both languages. Thus the premise alone has language-dependent effects on group identification, while explicit boundaries make the groups easier to identify. The added figure shows this contrast directly: EN group reports shift toward the correct index, whereas ZH reports shift away from it toward earlier groups. Its absolute-error panels show mainly one-item slips in EN and prominent three-item slips in ZH.

Examples below are literal saved answers from common error categories; ii, gg, and ss denote the true absolute, group, and slot positions:

Language/conditionProbeTarget in source wordTrue (i,g,s)(i,g,s)Raw answer
EN L1GroupVI in SOVIET(11,4,2)3
EN L1AbsoluteRB in CARBON(17,6,2)18
ZH L1Group顺 in 顺风车(28,10,1)9
ZH L0Absolute魔 in 魔术师(40,14,1)37

In the added count summary on the matched 64-sequence subset of original absolute responses, one-item slips account for 65/69 EN L0 errors and 135/151 EN L1 errors; ZH has 22/38 three-item slips in L0 and 126/241 in L1. The original shuffled comparison removes three-item slips in ZH S0 (0/169), while regular boundaries in S2 restore a positive position-normalized period-three coefficient, 0.155 (Appendix F.5). These patterns connect the ZH error bands to available grouping.

Q2

Why did the authors make the following choice: "the L1 vertical range is approximately twice that of L0, similar visual amplitudes understate the stronger slot dependence under L1." if it makes the figure harder to read? Why would you not just keep the same y-scale for both?

We agree that a common y-axis makes the comparison clearer. Figure 4 now uses identical limits of −15 to 10 pp and ticks every 5 pp for L0 and L1. We revise only the caption’s final sentence: “The generally larger vertical variation under L1 reflects its stronger slot dependence than L0.” The plotted estimates and statistical comparisons are unchanged.

AuthorsOpenReview ↗

Seeking your feedback on our rebuttal

We sincerely thank you for your detailed feedback and critical questions. We have carefully considered your concerns and provided additional experiments and clarifications.

Our main updates include:

  • Task motivation: Clarified the scientific rationale for studying positional retrieval and its connection to human sequential-memory research.
  • Prompt robustness (Appendix G.5): Added three paraphrase families, with the absolute-position cost reproduced across all eight language–wording combinations.
  • Model coverage and error analysis: Clarified model selection, expanded the Kimi error analysis, and improved Figure 4's presentation.

We would greatly appreciate your feedback on whether our responses have addressed your main concerns. We remain very willing to discuss any unresolved issues or provide further clarification.

Thank you again for your time and consideration.

Reviewer_mFRBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This submission examines whether, when an LLM answers whether a model can report where a known item occurred, the positional description that is easier to report changes depending on the organization emphasized by a cue. The experiments evaluate absolute-position accuracy (counts items from the left across the whole sequence), group-position accuracy (counts familiar-word groups from the left), and within-group slot accuracy (counts items from the left within the target’s group) for the same target, using controlled English and Chinese sequences of familiar words split into displayed items. The two manipulated cues are (L) a lexical cue (a familiar-word premise stated before the sequence) and (S) a boundary cue (visibly marked group boundaries, rendered as larger spacing). The main results are as follows: (i) familiar-word prompts improve group/slot reports, (ii) visible boundaries strengthen that gain but hurt exact absolute-position accuracy, (iii) the boundary effect persists after word structure is destroyed, and (iv) with variable group lengths, exact absolute reports can collapse while rank-correlation remains high.

Critical strengths
  1. Whereas existing evaluations of LLMs ask what appears at a position, this study is novel in asking where a particular item is under multiple coordinates and in trying to examine interference between those coordinates. I am not familiar with this topic, so I am not fully convinced why this new question matters or how it matters, but I can see that this work adds a new question that may inform prompt design and the evaluation of memory systems.
  2. The experimental design is careful overall, and the differences across conditions appear attributable to the differences in cues. Therefore, the results also look sound to me, although I have not been able to check them thoroughly.
  3. The appendix provides detailed experimental settings, and the supplementary materials include the frozen materials and code, which shows an effort to ensure reproducibility.
Critical weaknesses
  1. I do not see any particular critical weaknesses in this submission, but the scope of the claims appears to shift somewhat within the paper, so I would recommend revising the wording carefully to make it consistent throughout. My understanding is that this study is limited to behavioral-level observation and evaluation of LLMs and does not extend to mechanisms such as internal representations of information in memory. This is stated explicitly, for example, in Sec 7, "Our evidence is behavioral, so it does not identify the computation behind the readout", and in Appendix A, "Our evidence characterizes retrieval behavior. Identifying the computations underlying structure-relative influence, including persistent internal variables, transient readout strategies, or attention allocation, requires activation-level or causal analyses". However, there are also statements such as the Fig 1 caption, "Structural cues change how positional information is organized in sequential memory. ... This trade-off implies that prompt structure changes how LLMs organize information in sequential memory". Here, "organization" may be read as referring not only to a behavioral-level phenomenon but also to organization at the level of internal representations, so I think some care is needed. One possible source of confusion is that the terms "organization/reorganization" may be used across different levels, including the grouped structure of the input sequence, the coordinate system used to describe position, the observed readout behavior, and the internal memory representation. In particular, it seems unclear what "reorganizes memory" in the abstract and Fig 1 is intended to refer to, and this may be a source of confusion.
Questions for authors
  1. What is the motivation for inserting spaces both within words and between words in L2? It seems to me that a setting with no spaces within words and spaces only between words would be more natural in ordinary language.
3 comments
AuthorsOpenReview ↗

Behavioral scope, terminology, and presentation

W1

I do not see any particular critical weaknesses in this submission, but the scope of the claims appears to shift somewhat within the paper, so I would recommend revising the wording carefully to make it consistent throughout. My understanding is that this study is limited to behavioral-level observation and evaluation of LLMs and does not extend to mechanisms such as internal representations of information in memory...In particular, it seems unclear what "reorganizes memory" in the abstract and Fig 1 is intended to refer to, and this may be a source of confusion.

Your distinction between behavioral findings and internal representations matches the scope of our evidence. We use organization and reorganization to describe patterns of positional reports and their changes under prompt interventions. We have made this meaning explicit while retaining the paper's conclusions.

In Sec. 1, we retain the original conclusion and add an operational definition: “Here, organization refers to the behavioral pattern of positional-report accuracy and errors, and reorganization to changes in that pattern under different prompt conditions.” This distinguishes the observed behavior from an explanation of its internal implementation:

LevelMeaning in the paper
Input groupingGroup structure in the materials and displayed cues
Position coordinateAbsolute, group, or within-group position
Behavioral organizationAccuracy and error patterns of positional reports
Internal representationA possible explanation requiring further mechanistic evidence

At the two locations you identify, we made focused wording changes:

  1. Abstract. We replace “in LLMs this organization is largely cue-dependent” with “in LLMs this behavior is largely cue-dependent.” The surrounding argument is retained.
  2. Figure 1 caption. The title now reads “Structural cues change the behavior of LLMs in sequential memory.” The final sentence of panel (d) now reads “This trade-off shows that prompt structure changes the behavior of LLMs in sequential memory.” The rest of the caption is retained.

Beyond the two passages you identify, we added the behavioral definition of organization and reorganization in Sec. 1 (Introduction) of the revised paper, retaining the original conclusion. We also changed Figure 1's organization labels to positional readout: “Prompt-conditioned memory organization” becomes “Prompt-conditioned positional readout,” “Memory organization” becomes “Positional readout,” and “Prompt-conditioned organization” becomes “Prompt-conditioned readout.” These label changes make the behavioral meaning clear when the figure is read independently. We retain organization and reorganization elsewhere under this definition.

AuthorsOpenReview ↗

Itemization and natural-word format

Q1

What is the motivation for inserting spaces both within words and between words in L2? It seems to me that a setting with no spaces within words and spaces only between words would be more natural in ordinary language.

The experimental unit is a displayed chunk, so the display makes each chunk individually identifiable, including within a word. Spatial separation provides a textual counterpart to the successive presentation of discrete units in the human sequential-memory paradigm. It lets us manipulate group boundaries while keeping the tested units visible. Without within-word separation, the model must additionally recover those units from the string, introducing a segmentation demand alongside positional retrieval.

Your proposed natural-word display is therefore useful for testing how the result extends beyond this itemized assay. We added a Gemma-4-31B-IT experiment on the same 64 base sequences per language (Appendix G.4, Table A32). N0 concatenates whole words, and N2 adds three spaces only between words; neither separates items within words.

The table shows absolute-probe accuracy without/with boundaries and boundary-minus-unmarked changes for all three probes. Accuracies are percentages; changes are percentage points (pp).

FormatBasesLanguageAbsolute, unmarked → boundaryΔ absoluteΔ groupΔ slot
Original itemized 3/664EN92.19 → 64.58−27.60+27.60+9.38
Natural words64EN39.81 → 71.88+32.07−3.65−13.02
Original itemized 3/664ZH88.54 → 62.21−26.33+17.19+3.65
Natural words64ZH41.55 → 58.16+16.61+50.52−3.12

Natural N0 lowers absolute accuracy from 92.19% to 39.81% in EN and from 88.54% to 41.55% in ZH. Recovering an item's global position now involves identifying units within the continuous string. Adding word boundaries in N2 raises accuracy by 32.07 and 16.61 pp, respectively, consistent with boundaries helping this segmentation. The original itemized comparison measures the effect of grouping with item boundaries already supplied while the natural comparison additionally involves recovering those units. Display and explanatory wording change together in this added experiment.

Intact word spelling can also help local reporting. EN N0 slot accuracy is 95.31%, compared with 84.90% in original L0. All 29 paired slot responses that change from incorrect to correct also change from a bare integer to a numbered explanation. For example, for IX in MATRIX, L0 answers 4, whereas N0 identifies MATRIX, lists 1. MA; 2. TR; 3. IX, and answers 3. Such responses support a route through word recognition and within-word decomposition. These coordinate-specific changes show how presentation affects the procedures available for positional reporting.

These results explain the role of explicit item separation in our design: it keeps the tested units identifiable while we manipulate grouping. Extending to natural strings additionally engages segmentation and word recognition, yielding a different pattern of costs and benefits.

The original boundary-surface control (Appendix G.1, Figures A2–A3) addresses a related, narrower question. It varies only the boundary rendering, keeping ordinary item gaps and the lexical premise fixed. Across Sol and Gemma in EN/ZH, all four surfaces reduce adjacent-position error influence and increase the period-three error component relative to L1. Together, these controls show repeated error geometry across boundary surfaces with itemization fixed, while changing item visibility in natural displays can alter accuracy contrasts.

AuthorsOpenReview ↗

Seeking your feedback on our rebuttal

We sincerely thank you for your constructive feedback and for recognizing the novelty and careful experimental design of our work.

We have made the following updates in response to your suggestions:

  • Behavioral scope and terminology: Clarified the meaning of "organization" and "reorganization" in Sec. 1, and revised the Abstract and Figure 1 to distinguish behavioral findings from claims about internal representations.
  • Natural-word display (Appendix G.4): Added the suggested experiment without within-word spacing, further clarifying the role of itemization in our experimental design.

We would greatly appreciate your feedback on whether these revisions have satisfactorily addressed your concerns. Your recognition of our work is incredibly encouraging to us. We would be grateful for your continued support during the discussion phase.

Thank you again for your time and consideration.

Reviewer_KGxZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how prompt structure affects positional retrieval in LLMs. The authors distinguish absolute position, group position, and within-group position, and test whether lexical cues and visible group boundaries affect these three types of judgments differently. Experiments across five LLMs in English and Chinese show that structural cues generally improve group and within-group position accuracy, but often reduce absolute-position accuracy. The authors further examine error patterns using shuffled sequences, variable group lengths, and alternative boundary markers. They find that visible boundaries systematically change absolute-position errors, even when the original word structure is removed.

Critical strengths

The paper asks an interesting question that has received relatively little attention. Most work on LLM memory examines whether information at a particular position can be retrieved. Here, the authors instead ask whether models can report different positional coordinates of the same item. The connection to hierarchical working memory in humans provides a useful motivation.

The experiments are also fairly thorough. I particularly appreciate that the authors analyze error patterns instead of relying only on accuracy, and that they include structure-destroyed and variable-length controls. The finding that adding explicit boundaries can substantially reduce absolute-position accuracy while improving group-position accuracy is interesting and worth investigating.

Critical weaknesses

First, I am not convinced that the results reflect a reorganization of sequential memory. The sequence remains in the prompt when the model answers. With explicit boundaries, the model may simply adopt a different counting strategy, for example, counting groups first and then estimating the absolute position. This could explain why group and slot accuracy improve while absolute-position accuracy decreases. The authors test a fixed-length conversion rule in the variable-length experiments, which is useful. However, ruling out this particular rule does not rule out other counting or numerical readout biases. This distinction matters because the paper's main claim concerns memory organization, while the experiments directly measure positional reports.

Second, the variable-length results do not support the hierarchical interpretation. In the fixed-length experiments, the authors interpret errors of ±3 positions as consistent with confusing adjacent groups while preserving within-group position. However, since every group contains exactly three items, these errors could also reflect a simple periodic displacement. The variable-length experiment is important because it can separate these explanations. However, Figure 7b shows no reliable same-slot preference in any of the six model-language pairs. The subsequent regression results indicate that group and slot positions influence absolute-position errors, but they do not establish that models preserve within-group positions when confusing groups. I think this negative result deserves more attention in the discussion. The evidence for a specifically hierarchical error pattern is not clear.

Questions for authors

Could the authors test whether the drop in absolute-position accuracy persists when models are instructed to count the displayed items explicitly? This would help clarify whether the effect reflects positional retrieval or a counting strategy induced by the grouping.

How do the authors interpret the lack of a reliable same-slot preference in Figure 7b? Given that this analysis was designed to distinguish hierarchical transpositions from fixed numerical displacements, I think its implications for the main interpretation should be discussed more directly.

The sequence remains available throughout the trial. Is there a way to separate the effect of structural information presented before the sequence from the effect of using that information when answering the question? It would be useful to know whether introducing the same structural cue only at retrieval produces a similar pattern.

7 comments
AuthorsOpenReview ↗

Available context and the memory claim

We thank the reviewer for recognizing the interest of the positional-coordinate question and the value of our error analyses and controls. Below we address counting strategies, the interpretation of variable-length errors, and cue placement.

W1

First, I am not convinced that the results reflect a reorganization of sequential memory. The sequence remains in the prompt when the model answers.

The task comes from human sequential-memory research. Grouped probed-recall paradigms distinguish the group containing a target from its position within that group [1]. More directly, Fan et al. [2] study global and local positional representations in human working memory using syllable sequences organized into words. We adopt their ZH word banks and the distinction between continuous, group, and within-group positions. Thus, the sequence task and its positional coordinates have a direct basis in an established human memory paradigm.

The important difference between testing human and AI is how the sequence remains accessible. Human participants can retain information after the stimulus disappears. In the standard Transformer inference setting used here, independent requests have no persistent sequence-specific memory outside the supplied context. Information about the sequence must therefore remain accessible in that context for the model to answer. Removing it without providing another memory channel would remove the task information, rather than reproduce a human retention interval. Keeping the sequence in context is the architectural adaptation that makes this comparison possible.

This is also consistent with existing LLM evaluations. LongBench tests retrieval and question answering over supplied long contexts [3]. LongMemEval evaluates conversational memory through indexing, retrieval, and reading of interaction histories [4]. Although their tasks differ from ours, both assess recovery of information made accessible to the model at answer time. The availability of context is therefore compatible with an operational evaluation of LLM memory.

We regard this as a reasonable transfer of the sequential-memory paradigm to LLMs, with the context-access difference made explicit by the term “in-context sequential memory.” We agree that our evidence is behavioral, as stated in Sec. 7 and Appendix A. The paradigm provides a controlled test of how prompts affect positional retrieval: we compare cue conditions for matched sequences and targets, and use structure-destroyed and variable-length controls to test alternative explanations. These comparisons support the paper's conclusion that structural prompts change positional-report accuracy and error patterns in the tested settings.

[1] McNicol, Don, and Andrew Heathcote. "Representation of order information: An analysis of grouping effects in short-term memory." Journal of Experimental Psychology: General 115.1 (1986): 76.

[2] Fan, Ying, et al. "Two-dimensional neural geometry underpins hierarchical organization of sequence in human working memory." Nature Human Behaviour 9.2 (2025): 360-375.

[3] Bai, Yushi, et al. "Longbench: A bilingual, multitask benchmark for long context understanding." Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers). 2024.

[4] Wu, Di, et al. "Longmemeval: Benchmarking chat assistants on long-term interactive memory." In Proceedings of International Conference on Learning Representations (ICLR), 2025.

AuthorsOpenReview ↗

Counting strategies and explicit-counting intervention

W1

With explicit boundaries, the model may simply adopt a different counting strategy, for example, counting groups first and then estimating the absolute position...The authors test a fixed-length conversion rule in the variable-length experiments, which is useful. However, ruling out this particular rule does not rule out other counting or numerical readout biases. This distinction matters because the paper's main claim concerns memory organization, while the experiments directly measure positional reports.

Q1

Could the authors test whether the drop in absolute-position accuracy persists when models are instructed to count the displayed items explicitly? This would help clarify whether the effect reflects positional retrieval or a counting strategy induced by the grouping.

Counting is itself one way to perform sequential-memory retrieval in this in-context setting. Itemwise counting treats the sequence as a continuous chain, while groupwise counting uses group membership and within-group position to reconstruct an absolute index. A prompt-induced change between these strategies is therefore compatible with a change in the behavioral organization of sequential memory. The strategy account helps explain how the observed reorganization could arise.

To further test this strategy account, we added an explicit-counting experiment (Appendix G.2, Table A29). The added EN instructions for the absolute probe were:

Itemwise

First number the displayed chunks of the test sequence one at a time from the far left, continuing across every gap without restarting the count. Then, on its own final line, write exactly: Final answer: <integer>

Groupwise

First decide which complete word the target chunk belongs to. Then count how many displayed chunks each earlier complete word actually contains, using each word's own actual number of chunks rather than assuming that every word has the same number, and add those counts together. Then add the target chunk's position inside its own word. Then, on its own final line, write exactly: Final answer: <integer>

We tested all three procedures under both L1 and L3 on 48 variable-length base sequences per language for Gemma-4-31B-IT and a prespecified 16-sequence subset for GPT-5.6 Sol. Direct reuses the original paper’s responses for the same sequences and targets. Each column gives that procedure’s L3−L1 accuracy change in percentage points (pp).

ModelLanguageDirectItemwiseGroupwise
GemmaEN−33.330.00+1.85
GemmaZH−19.540.00−8.70
Sol (16 bases)EN−26.390.00−2.78
Sol (16 bases)ZH−70.830.00−1.82

Itemwise accuracy reaches 100% in both L1 and L3 for each model and language, so all itemwise differences are zero. ZH Gemma groupwise counting instead retains an 8.70 pp absolute cost while L3 group and slot accuracy reach 100%; the joint group-minus-absolute boundary contrast is +13.56 pp. This procedure also produces structured errors: all 53 equidistant identifying errors in L3 choose the same-slot alternative.

Taken together, specifying the procedure strongly changes the absolute boundary effect. In ZH Gemma, direct reports and explicit groupwise counting share a cross-coordinate dissociation. Their L3−L1 absolute accuracy falls by 19.54 and 8.70 pp, respectively, while group accuracy rises by 23.61 and 4.86 pp. Slot accuracy increases in direct reporting and is already at ceiling in both groupwise conditions. The shared pattern concerns coordinate accuracy.

Group-based counting therefore provides a concrete candidate explanation for the original effect. The explicit intervention shows what an instructed procedure can generate. Whether the original direct reports recruit that computation is a further mechanistic question, testable by tracing group/slot computations and causally perturbing their contribution to absolute-position errors.

AuthorsOpenReview ↗

Variable-length evidence and the negative same-slot result

W2

Second, the variable-length results do not support the hierarchical interpretation...However, Figure 7b shows no reliable same-slot preference in any of the six model-language pairs. The subsequent regression results indicate that group and slot positions influence absolute-position errors, but they do not establish that models preserve within-group positions when confusing groups.

Q2

How do the authors interpret the lack of a reliable same-slot preference in Figure 7b? Given that this analysis was designed to distinguish hierarchical transpositions from fixed numerical displacements, I think its implications for the main interpretation should be discussed more directly.

We admit that Figure 7b does not establish a general slot-preserving transposition preference in the variable-length materials. The original Figures 7a and 7c examine the broader error pattern: whether reports retain global ordinal association and whether group/slot coordinates influence numerical errors. To examine that influence further, we added boundary-sensitive error comparisons (Appendix F.6, Table A22).

With groups of 2, 3, and 4 items, errors form broader, position-dependent deviations rather than one repeated numerical offset. Across all six pairs, L3 reports remain strongly rank-correlated with true positions (Spearman's ρ=0.980\rho = 0.980–0.9910.991), despite exact accuracy of 16.67–69.44% (Figure 7a). For example, Sol-ZH has only 16.67% exact accuracy but ρ=0.982\rho = 0.982. This indicates strong global ordinal association alongside substantial numerical error.

Figure 7c asks whether those errors depend on structural coordinates. It tests group influence at fixed absolute position and slot, and slot influence at fixed absolute position and group. The paired L3−L1 slope changes are reported in Appendix F.6, Table A21:

Model–languageGroup-slope change [95% CI]Slot-slope change [95% CI]
Sol-EN+0.071 [−0.580, +0.731]+0.136 [+0.006, +0.278]
Sol-ZH+2.041 [+1.069, +3.031]+0.798 [+0.477, +1.090]
Kimi-EN+1.082 [+0.572, +1.617]+0.447 [+0.250, +0.641]
Kimi-ZH+1.170 [+0.522, +1.847]+0.605 [+0.288, +0.945]
Gemma-EN+0.571 [+0.243, +0.912]−0.063 [−0.208, +0.096]
Gemma-ZH+0.761 [+0.199, +1.314]+0.247 [−0.049, +0.574]

Group-slope changes are reliably positive in five of six pairs, and slot-slope changes in four. For Sol-ZH, +2.041 means that, at the same absolute position and slot, the boundary-related signed-error shift grows by about two positions per additional group. Thus, grouping changes the mapping from sequence position to reported number: later structural coordinates push that number upward in the affected pairs. The group effect is robust to response-range treatment and layout variation (Appendix F.8).

The added comparisons hold the correct position and candidate error number fixed and ask whether the boundary-related change in that error's probability differs when it crosses a real group boundary versus stays within a group. Four pairs show a nonzero contrast: Sol-EN +3.12 pp, Kimi-EN −4.61 pp, Gemma-EN −6.69 pp, and Gemma-ZH −2.02 pp. These exploratory contrasts show boundary-sensitive error selection, with model-dependent directions.

The original fixed-three conversion offers a partial account: error alignment strengthens with boundaries in all six pairs, but the rule explains only 1.8–8.7% of L3 signed-error variance (Sec. 6; Appendix F.7). Taken together, the variable-length evidence supports a systematic influence of structural coordinates on absolute-position reporting while retaining strong global ordinal association. The common pattern is a structure-dependent numerical distortion, rather than the specific slot-preserving transposition proposed for the fixed-length errors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall response

We thank the reviewers for their careful reading and for suggesting experiments that sharpen the paper's interpretation. Building on our point-by-point replies, we summarize what the additional evidence tells us about the central question.

Our study addresses a basic question about sequential memory in LLMs. How does prompt structure change which positional relations can be retrieved and how errors are organized? We extend content-retrieval evaluations by probing three positional descriptions of the same known item: absolute, group, and within-group position. Matched sequences and targets separate changes in overall difficulty from changes in access to particular coordinates. This controlled paradigm draws on human sequential-memory research and isolates structural influences that are difficult to separate in everyday tasks.

Our contribution is a controlled demonstration that prompts reshape positional access and error patterns. Across controlled EN/ZH experiments, lexical cues mainly facilitate group/slot reporting, while visible boundaries can strengthen relative-coordinate access, reduce exact absolute accuracy, and redirect errors toward grouping. Structure-destroyed and variable-length controls test whether these patterns depend on familiar words or fixed numerical displacements. These findings provide a basis for studying how LLMs retrieve sequential information. We have made the behavioral meaning of organization explicit in Sec. 1 and aligned the Abstract and Figure 1 with it.

Can a counting strategy explain the effect? Counting is itself a strategy for sequential-memory retrieval in this in-context paradigm. Itemwise counting treats the sequence as a continuous chain, while groupwise counting uses group membership and within-group position to reconstruct an absolute index. How prompts alter the use of these procedures is part of the sequential-memory question. We added explicit-counting interventions in Gemma and GPT-5.6 Sol (Appendix G.2, Table A29). With explicit enumeration, itemwise counting reaches 100% absolute accuracy both with and without visible boundaries. In Gemma-ZH, groupwise counting instead retains an 8.70 pp absolute cost, while group and slot accuracy reach 100% with boundaries. Group-based counting thus provides a concrete candidate explanation for the original coordinate-specific effect. The intervention establishes what an instructed procedure can generate and makes its contribution to the original direct reports a testable mechanistic question.

Does giving the same cue only at retrieval reproduce the result? We moved the familiar-word premise from before the sequence to immediately before the question, with the sequence still available (Appendix G.3, Tables A30–A31). Across Sol, Kimi, and Gemma in EN/ZH, late-cue absolute accuracy stays within 1.10 pp of the L0 baseline. Some group/slot benefits remain, especially in Gemma-EN, but the late cue does not generally reproduce the early cue's combined accuracy and within-group profile pattern. This supports early guidance of how the subsequent sequence is organized for positional reporting, beyond supplying lexical information at answer time.

How much depends on wording? We added three paraphrase families alongside the original wording for Gemma on 64 matched sequences per language (Appendix G.5, Tables A33–A35). The absolute boundary cost recurs in all eight language-by-family combinations and seven also reproduce the original direction of group and slot gains. The EN-P1 exception is informative. Its unmarked group accuracy already reaches 98.96%, compared with 48.96% under the original wording. Complete ordered group lists accompany 163/192 responses, versus 35/192 originally, under the same answer-format instruction. Wording changes both baseline performance and visible reporting behavior.

What do the variable-length errors establish? Reports retain strong global ordinal association even when exact accuracy falls. Group-coordinate influence strengthens reliably in five of six model-language pairs, and slot influence in four (Sec. 6; Appendix F.6, Table A21). The added comparisons also show boundary-sensitive selection of matched numerical errors in four pairs, with model-dependent directions (Table A22). These findings support structure-dependent numerical distortions.

The revised paper also reports the suggested natural-word display control (Appendix G.4, Table A32), which additionally engages segmentation, and expands the Kimi error analysis (Appendix F.1, Figure A1). Together, the original controls and new interventions make the behavioral contribution more precise and identify concrete procedures for further mechanistic study. We would especially welcome the reviewers' assessment of how the counting and cue-placement results bear on the strategy explanation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.